Repository navigation
Update coasts package to version 4.20.0: updated documentation to re… #528
data-pipeline.yaml
on: push
Build R container
2m 55s
Enrich taxa with FishBase data
2m 25s
Predict fishing tracks and aggregate to H3 grid
15m 46s
Export spatial outputs as JSON/GeoJSON for web
3m 40s
Annotations
6 warnings and 120 notices
|
WARN
89 of 946 taxa names matched no FishBase/SeaLifeBase species and were dropped: Abalistes stellaris, Scaridae, Brachyura, Carangoides armatus, Actinopterygii, Solea bleekeri, Parupeneus macronema, Charybdis feriatus, Chromis multilineata, Sorsogona prionota, Liza alata, Selachimorpha (Pleurotremata), Leiognathus equulus, Octopus cyaneus, NA, Haemulidae (=Pomadasyidae), Ulua mentalis, Carangoides fulvoguttatus, Pristotis jerdoni, Sepiidae, Sepiolidae, Synagrops microlepis, Valamugil cunnesius, Pomadasys multimaculatum, Carangoides ferdau, Valamugil spp, Sepia prashadi, Sepia latimanus, Stegastes apicalis, Carangoides coeruleopinnatus, Caesionidae, Algae, Epinephelus chabaudi, Secutor insidiator, Labridae (ex Scaridae), Synodontis zambesiensis, Histioteuthis dofleini, Etrumeus teres, Rhamphochromis brevis, Stegastes dorsopunicans, Carangoides gymnostethus, Herklotsichthys quadrimaculat., Liza vaigiensis, Sepia pharaonis, Parupeneus heptacanthus, Leiognathus leuciscus, Chromis intercrusma, Auxis thazard, A. rochei, Synagrops adeni, Carangoides oblongus, Stolephorus commersonii, Chelon macrolepis, Thryssa baelama, Premnas biaculeatus, Upeneus arge, Alectis indica, Pomacentrus agassizi, Thunnini, Octopus aegina, Loligo duvauceli, Carangoides orthogrammus, Plectorhinchus orientalis, Carangoides equula, Carangoides malabaricus, Parapenaeopsis sculptilis, Chromis margaritifer, Thryssa vitrirostris, Carangoides chrysophrys, Liza dumerili, Paracaesio xanthura, Loliginidae, Ommastrephidae, Carangoides dinema, Stegastes fasciolatus, Reptantia, Strongylura leiura, Zebrasoma veliferum, Ambassis productus, Himantura granulatus, Crenimugil crenilabis, Valamugil seheli, Secutor ruconius, E. fuscoguttatus x E. lanceolatus, Abudefduf luridus, Ambassis commersoni, Sepia trygonina, Chromis dimidiata, Cociella crocodila, Carangoides plagiotaenia, Leiognathus dussumieri, Pomadasys opercularis
|
|
WARN
Failed to create 2dsphere index: Index build failed: 7fb076c7-7985-4934-9301-cfe9ff6758b2: Collection portal.wio_gaul2 ( 81c75ec6-9281-466c-9abe-660d48fd3cae ) :: caused by :: collection scan stopped. totalRecords: 18; durationMillis: 161ms; phase: collection scan; collectionScanPosition: RecordId(18); readSource: kMajorityCommitted :: caused by :: Can't extract geo keys: *** _id: ObjectId('6ac27a81b3a34f7553098f14'), iso3_code: "TZA", gaul1_name: "Kaskazini Pemba", gaul2_name: "Micheweni", geometry: *** type: "MultiPolygon", coordina
|
|
WARN
Failed to create 2dsphere index: Index build failed: aa7bc972-925c-4538-9425-58ff71ef1575: Collection portal.wio_gaul1 ( 2ecfa951-6ae6-4f78-ac90-0f457e2caa4b ) :: caused by :: collection scan stopped. totalRecords: 5; durationMillis: 158ms; phase: collection scan; collectionScanPosition: RecordId(5); readSource: kMajorityCommitted :: caused by :: Can't extract geo keys: *** _id: ObjectId('6ac27a7db3a34f7553098ee7'), iso3_code: "TZA", gaul1_name: "Tanga", geometry: *** type: "MultiPolygon", coordinates: [ [ [ [ 39.0352, -5.3714 ], [ 39
|
|
WARN
77 trip(s) revised by PDS are too long to predict; their files stay as they are and will be queued again next run
|
|
WARN
Skipping 198 trip(s) spanning > 5 days (IDs: 10579226, 10667906, 10797328, 11764837, 11781714, 11781722, 11781728, 11781947, 11785296, 11798762, 11798812, 11800140, 11801183, 11801206, 11819908, 11824638, 12047334, 13538978, 13889043, 13897835, 14020005, 14026248, 14133813, 14159300, 14248868, 14248869, 14251332, 14256294, 14300374, 14410128, 14410129, 14410132, 14410141, 14410142, 14410143, 14410145, 14410146, 14410147, 14410148, 14410154, 14410157, 14410160, 14410169, 14410170, 14419581, 14426872, 14435867, 14437813, 14452384, 14454042, 14455928, 14478602, 14478852, 14479232, 14484759, 14484948, 14486032, 14490643, 14499230, 14499292, 14505638, 14517256, 14517421, 14530361, 14541455, 14576754, 14578065, 14578438, 14578791, 14596027, 14640775, 14641243, 14657319, 14671211, 14672093, 14692374, 14754754, 14773203, 14847024, 14880620, 14907966, 14917078, 14927555, 14951120, 14951492, 14951780, 14952607, 14954865, 14965855, 14969765, 14980949, 14981319, 15005613, 15005747, 15006983, 15019233, 15038361, 15040373, 15040491, 15040651, 15040664, 15048463, 15050619, 15051707, 15055352, 15056600, 15059190, 15069591, 15074704, 15083360, 15086066, 15110498, 15111803, 15116206, 15118268, 15118393, 15131805, 15143641, 15156437, 15157630, 15211281, 15216883, 15216902, 15220930, 15225522, 15247602, 15256975, 15257586, 15262108, 15265117, 15271764, 15282083, 15287034, 15287035, 15287036, 15287037, 15287041, 15297669, 15311711, 15318293, 15319384, 15320346, 15334836, 15342909, 15345308, 15351019, 15351064, 15351150, 15352515, 15359797, 15360417, 15362089, 15382632, 15408671, 15421878, 15445740, 15448144, 2335178, 2359445, 2361328, 2365619, 2373334, 2404095, 2412993, 2437365, 2445535, 2446863, 2713356, 2721594, 2809967, 2814249, 2815140, 3088569, 3251255, 3295739, 3453864, 4474333, 4781534, 4781603, 4782486, 4782831, 4782896, 4814203, 4997708, 5152619, 5168468, 5851840, 7355604, 7520544, 8636700, 8655459, 8657128, 8663566, 8915080, 9342968, 9556078, 9697060, 9776875)
|
|
WARN
4 trips failed to fetch or were empty
|
|
Build R container
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Using bucket: peskas-api-prod
|
|
INFO
Retrieving wf-surveys-merged__20261002171551_e507498__.parquet
|
|
INFO
Using bucket: zanzibar-prod
|
|
INFO
Retrieving wf-adnap-surveys-merged__20261003040556_7027eef__.parquet
|
|
INFO
Using bucket: mozambique-prod
|
|
INFO
Retrieving kenya-surveys-matched-trips__20261004152157_9f57c51__.parquet
|
|
INFO
Using bucket: kenya-prod
|
|
INFO
Downloading merged trip data from regional buckets...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Match GPS and survey data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Ingest static assets
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Uploading taxa-fishbase-enriched__20261004154253_81ee894__.parquet to cloud storage
|
|
INFO
Uploading enriched taxa data (5356 rows, 3931 species, FishBase / SeaLifeBase latest) to cloud storage
|
|
INFO
Joining and deduplicating biological data
|
|
INFO
Fetching biological data from FishBase / SeaLifeBase
|
|
INFO
Filtering to FAO area(s) 51, 57
|
|
INFO
Expanding 949 taxa against FishBase / SeaLifeBase latest
|
|
INFO
Downloading taxa metadata from cloud storage
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Enrich taxa with FishBase data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Uploading pds-trips__20261004154336_81ee894__.parquet to cloud storage
|
|
INFO
Kept 124980 of 139129 trips from devices currently under: WorldFish - Mozambique, Totalenergies Mozambique TEPMA1 (formerly Syberintel), Kenya, Kenya AABS, WorldFish - Kenya, WorldFish - Tanzania AP, WorldFish - Zanzibar, MAF / WorldFish
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Ingest GPS trips
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Processing complete. Successfully processed 2 tracks.
|
|
INFO
Processing 2 new tracks in parallel...
|
|
INFO
Setting up parallel processing with 3 workers...
|
|
INFO
Checking existing tracks in cloud storage...
|
|
INFO
Reading trip IDs...
|
|
INFO
Downloading pds-trips__20261004154336_81ee894__.parquet
|
|
INFO
Getting trips file from cloud storage...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Ingest GPS tracks
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Uploading grid summaries...
|
|
SUCCESS
Track preprocessing complete
|
|
INFO
Uploading preprocessed tracks...
|
|
INFO
Processing 9383 tracks in parallel...
|
|
INFO
Setting up parallel processing with 3 workers...
|
|
INFO
Listing raw tracks...
|
|
INFO
Checking existing preprocessed tracks...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Preprocess GPS tracks data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Pushing performance_metrics MongoDB...
|
|
INFO
Pushing fisher statistics to MongoDB...
|
|
INFO
Merging datasets datasets...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Export fishers stats
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Uploading metrics_gaul1 data to MongoDB
|
|
INFO
Pushing regional time series metrics to MongoDB...
|
|
INFO
Index may need to be created manually via MongoDB shell
|
|
INFO
Uploading wio_gaul2 data to MongoDB
|
|
INFO
Index may need to be created manually via MongoDB shell
|
|
INFO
Uploading wio_gaul1 data to MongoDB
|
|
INFO
Pushing combined geospatial data to MongoDB...
|
|
INFO
Loading geospatial data from cloud storage...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Export WIO data
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Downloading 116 predicted track files with 3 workers...
|
|
INFO
116 new files to aggregate (skipping 124739 already done)
|
|
INFO
120 aggregated file(s) deleted or replaced upstream: withdrawing 93 trip(s) from the effort store
|
|
INFO
Loaded per-trip effort store: 639092 rows, 103101 trips
|
|
INFO
Listing predicted track files...
|
|
INFO
Retrieving pds-trips__20261004154336_81ee894__.parquet
|
|
INFO
Building trip -> gear/country lookup...
|
|
INFO
Model version: 0.1.0
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
INFO
Skipping 124662 trips at v0.1.0, 318 to process (191 of them refreshed after a PDS revision)
|
|
INFO
Re-predicting 191 trips revised by PDS since they were fetched
|
|
INFO
Deleting 6 file(s): their trip is no longer listed by PDS
|
|
INFO
Total trips to consider: 124980
|
|
INFO
Fetching trips from PDS API (from 2018-01-01)...
|
|
INFO
Loading device registry...
|
|
INFO
Model version: 0.1.0
|
|
INFO
Using Python: /opt/ssfaitk/bin/python3
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Predict fishing tracks and aggregate to H3 grid
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Trip effort: 2424 rows | 488 trips | 1452 H3 cells
|
|
INFO
Downloaded 100,977 fishing points across 488 trips
|
|
INFO
Downloading 552 matched track files (out of 124855 total predicted)
|
|
INFO
Downloading matched predicted tracks for CPUE model ...
|
|
INFO
Matched trips: 969 catch records | 552 unique PDS trips | 3 countries | 2025-03-03 -> 2026-09-26
|
|
INFO
Retrieving trips-matched__20261004154022_81ee894__.parquet
|
|
INFO
Downloading trips-matched (all countries) ...
|
|
INFO
=== CPUE pipeline | H3 res = 9 | method = weighted ===
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Run spatial CPUE model
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
|
INFO
Uploaded: timor_leste_effort_gear_h3res9__20261004163357_81ee894__.zip
|
|
INFO
Uploaded: mozambique_effort_gear_h3res9__20261004163351_81ee894__.zip
|
|
INFO
Uploaded: tanzania_effort_gear_h3res9__20261004163348_81ee894__.zip
|
|
INFO
Uploaded: zanzibar_effort_gear_h3res9__20261004163346_81ee894__.zip
|
|
INFO
Uploaded: kenya_effort_gear_h3res9__20261004163342_81ee894__.zip
|
|
INFO
Exporting gear shapefiles for: Kenya, Zanzibar, Tanzania, Mozambique, Timor-Leste
|
|
INFO
Retrieving predicted-pds-h3_grid_r9__20261004162432_81ee894__.parquet
|
|
INFO
Loading effort grid (predicted-pds-h3_grid_r9) ...
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
INFO
Uploaded: pds-h3-effort-gear-r9__20261004163204_81ee894__.json
|
|
INFO
Uploaded: pds-h3-effort-r9__20261004163200_81ee894__.json
|
|
INFO
Study period inferred: 2018-02-07 to 2026-10-03 (3161 days)
|
|
INFO
Collapsing effort grid over gear/country ...
|
|
INFO
Effort grid: 111402 rows | 60165 cells
|
|
INFO
Retrieving predicted-pds-h3_grid_r9__20261004162432_81ee894__.parquet
|
|
INFO
Loading effort grid (predicted-pds-h3_grid_r9) ...
|
|
INFO
=== Web spatial export | H3 res = 9 ===
|
|
INFO
Using configutation: production
|
|
INFO
Loading configuration file...
|
|
Export spatial outputs as JSON/GeoJSON for web
"The ubuntu-latest label will migrate to Ubuntu 26 beginning October 19, 2026. For more information, see https://github.com/actions/runner-images/issues/14748"
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
WorldFishCenter~peskas.coasts~ZM97TB.dockerbuild
|
84.6 KB |
sha256:e40dab25e86c7e2fda2199efc9f4301f8dada3b84fbaa43132712af5d6ed8602
|
|