Skip to content

feat: use original Copernicus provider to list queryables of wekeo_ecmwf #1897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 8 commits into
base: develop
Choose a base branch
from
Draft

feat: use original Copernicus provider to list queryables of wekeo_ecmwf #1897

wants to merge 8 commits into from

Conversation

@dalpasso
Copy link
Collaborator

@dalpasso dalpasso commented Oct 28, 2025

The provider wekeo_ecmwf doesn't provide the constrains and form URLs. This causes a different list of queryables compared to the original Copernicus provider.

Description

This PR enhances the WekeoECMWFSearch plugin with dynamic discover queryables. A list of configurations is given in the new property dynamic_discover_queryables. The configuration of the first collection selector that matches kwargs[field].startswith(prefix) is used. If no one is found, it falls back to standard behaviors. Follows and example of configuration.

      - collection_selector: # cop_ads
          - field: dataset
            prefix: EO:ECMWF:DAT:CAMS
        discover_queryables:
          fetch_url: null
          product_type_fetch_url: null
          constraints_url: https://ads.atmosphere.copernicus.eu/api/catalogue/v1/collections/{dataset#wekeo_to_cop_collection(EO:ECMWF:DAT:)}/constraints.json
          form_url: https://ads.atmosphere.copernicus.eu/api/catalogue/v1/collections/{dataset#wekeo_to_cop_collection(EO:ECMWF:DAT:)}/form.json
      - collection_selector: # cop_cds
          - field: dataset
            prefix: EO:ECMWF:DAT:SATELLITE
          - field: dataset
            prefix: EO:ECMWF:DAT:SEASONAL
          - field: dataset
            prefix: EO:ECMWF:DAT:INSITU
          - field: dataset
            prefix: EO:ECMWF:DAT:DERIVED
          - field: dataset
            prefix: EO:ECMWF:DAT:REANALYSIS
          - field: dataset
            prefix: EO:ECMWF:DAT:SIS
        discover_queryables:
          fetch_url: null
          collection_fetch_url: null
          constraints_url: https://cds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset#wekeo_to_cop_collection(EO:ECMWF:DAT:)}/constraints.json
          form_url: https://cds.climate.copernicus.eu/api/catalogue/v1/collections/{dataset#wekeo_to_cop_collection(EO:ECMWF:DAT:)}/form.json

A new converter wekeo_to_cop_collection has been added to convert the name of a collection from the WEkEO format to the Copernicus format (e.g. EO:ECMWF:DAT:SATELLITE_CARBON_DIOXIDE -> satellite-carbon-dioxide).

A bug was fixed in the metadata formatter to allow the usage of colons : as argument of the converter function (e.g. {dataset#wekeo_to_cop_collection(EO:ECMWF:DAT:)}).

@dalpasso dalpasso self-assigned this Oct 28, 2025
@dalpasso dalpasso marked this pull request as draft October 28, 2025 08:33
@github-actions
Copy link
Contributor

github-actions bot commented Oct 28, 2025 •
edited
Loading

Test Results

    4 files  ±0      4 suites  ±0   3m 45s ⏱️ +23s
  610 tests +1    609 ✅ +1  1 💤 ±0  0 ❌ ±0 
2 440 runs  +4  2 434 ✅ +4  6 💤 ±0  0 ❌ ±0 

Results for commit d1a9f2a. ± Comparison against base commit ea8aebe.

♻️ This comment has been updated with latest results.

@eodag-bot
Copy link
Collaborator

eodag-bot commented Oct 28, 2025 •
edited
Loading

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         243      10  95.88%   102-113, 619
config.py                                      451      32  92.90%   81-83, 92, 100, 104-106, 177, 188, 205-206, 728-730, 854-857, 901-902, 911-912, 946, 1026, 1047, 1055, 1085-1090, 1092
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    800      48  94.00%   338, 605, 636, 678-681, 719, 763, 797, 842-847, 871, 957, 1296, 1338, 1372-1383, 1487-1490, 1527-1547, 1616-1621, 1633-1636, 1954, 2192, 2255-2256, 2285-2286
api/search_result.py                           126      12  90.48%   87, 96, 103, 117, 288, 293-294, 328, 342, 365-366, 372
api/product/__init__.py                         18       2  88.89%   60, 62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        211      18  91.47%   74-76, 245-246, 264-265, 371, 398, 499, 519, 543-546, 555-558, 605
api/product/metadata_mapping.py                780      52  93.33%   124-126, 218-223, 244, 302-303, 410, 462-463, 500, 521-524, 547, 559-560, 601, 624, 654-659, 718-723, 735, 743, 975, 1150, 1159-1163, 1180-1185, 1318, 1341, 1350, 1372, 1377, 1429, 1501, 1522, 1548, 1562, 1587, 1633, 1702, 1774
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     23       0  100.00%
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      18  89.60%   105-110, 145-150, 154, 192, 214, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      10  89.80%   166-168, 215-216, 242-244, 300-301
plugins/apis/usgs.py                           181      26  85.64%   155, 261, 295, 330-332, 337, 365-366, 371, 401-408, 419-424, 446-452, 483
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                66      27  59.09%   28-30, 66-69, 76-79, 85, 89, 93, 104-120, 135-161
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        381      71  81.36%   267, 301, 339-342, 362-366, 461-463, 467, 498-499, 505-509, 540, 605-613, 677-770, 781-786, 824, 850, 895-897, 949, 1129
plugins/download/base.py                       260      35  86.54%   129, 158, 305-306, 364-365, 407, 411-422, 436, 504-508, 538-539, 564-572, 574-578, 628, 649, 671, 679
plugins/download/http.py                       545      77  85.87%   238, 280-283, 345-348, 351, 358-363, 394-396, 413, 428, 486, 521, 535, 549, 557-561, 577-582, 593, 611, 647-650, 671, 681, 688, 708, 843, 875, 905-914, 950, 975-976, 995-1000, 1009, 1024-1026, 1030, 1033, 1048-1049, 1125-1126, 1130, 1182, 1228-1229, 1235, 1245, 1271, 1306, 1326, 1363-1365
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         187      18  90.37%   107, 111, 135-141, 198-201, 294, 315, 448, 486, 508-511, 520
plugins/search/build_search_result.py          498      79  84.14%   245-246, 282, 285, 323, 326, 357-359, 570-581, 713, 715, 775-787, 808, 820, 846, 891, 941-956, 1004, 1029, 1032, 1036, 1076-1096, 1137, 1164-1165, 1174-1183, 1247, 1262, 1268, 1287-1296, 1413-1414, 1455, 1458, 1467-1469, 1526, 1572
plugins/search/cop_marine.py                   252      47  81.35%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 232, 238, 242, 246, 257, 268-269, 277, 309, 313, 334, 338, 342, 346, 350-354, 360-363, 366-380, 397, 446-450, 455, 467
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/qssearch.py                     726      77  89.39%   412-413, 530-531, 554-555, 567-571, 628-644, 761-764, 899, 906, 918, 975, 996, 999-1000, 1009-1010, 1019-1020, 1029-1030, 1057, 1109, 1192, 1215, 1288-1289, 1295, 1385, 1480-1484, 1550, 1553, 1559-1560, 1581, 1609-1621, 1628, 1660-1662, 1672-1678, 1708, 1731, 1736-1737, 1752, 1758, 1768, 1852, 1856, 1867, 1891, 1904, 1912-1922, 1959-1963
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            79      14  82.28%   100-126, 170, 183, 227
types/__init__.py                              161      45  72.05%   67, 71, 80-84, 95-107, 135-137, 144-149, 194, 213, 216, 220, 254, 264-280, 285, 287, 309, 314, 322, 332
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             84       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              576      63  89.06%   71, 160-182, 195-196, 205-231, 234, 249, 329-333, 408-412, 492, 532-533, 562, 937-940, 948-949, 956-958, 991, 1010-1011, 1040, 1058-1059, 1199, 1363, 1452, 1467, 1476, 1488, 1623-1629, 1663
utils/cache.py                                  22       0  100.00%
utils/dates.py                                  57       4  92.98%   63-65, 138
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       7  81.58%   51, 53, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      13  94.58%   200-203, 245, 258, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                         8998    1244  86.17%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
api/product/metadata_mapping.py             +7       0  +0.06%
plugins/search/build_search_result.py      +14     +15  -2.64%
utils/s3.py                                  0      +1  -0.42%
TOTAL                                      +21     +16  -0.15%

Results for commit: d1a9f2a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@eodag-bot
Copy link
Collaborator

eodag-bot commented Oct 28, 2025 •
edited
Loading

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         243      10  95.88%   102-113, 619
config.py                                      451      32  92.90%   81-83, 92, 100, 104-106, 177, 188, 205-206, 728-730, 854-857, 901-902, 911-912, 946, 1026, 1047, 1055, 1085-1090, 1092
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    800      48  94.00%   338, 605, 636, 678-681, 719, 763, 797, 842-847, 871, 957, 1296, 1338, 1372-1383, 1487-1490, 1527-1547, 1616-1621, 1633-1636, 1954, 2192, 2255-2256, 2285-2286
api/search_result.py                           126      12  90.48%   87, 96, 103, 117, 288, 293-294, 328, 342, 365-366, 372
api/product/__init__.py                         18       2  88.89%   60, 62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        211      18  91.47%   74-76, 245-246, 264-265, 371, 398, 499, 519, 543-546, 555-558, 605
api/product/metadata_mapping.py                780      52  93.33%   124-126, 218-223, 244, 302-303, 410, 462-463, 500, 521-524, 547, 559-560, 601, 624, 654-659, 718-723, 735, 743, 975, 1150, 1159-1163, 1180-1185, 1318, 1341, 1350, 1372, 1377, 1429, 1501, 1522, 1548, 1562, 1587, 1633, 1702, 1774
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     23       0  100.00%
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      18  89.60%   105-110, 145-150, 154, 192, 214, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      10  89.80%   166-168, 215-216, 242-244, 300-301
plugins/apis/usgs.py                           181      26  85.64%   155, 261, 295, 330-332, 337, 365-366, 371, 401-408, 419-424, 446-452, 483
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                66      27  59.09%   28-30, 66-69, 76-79, 85, 89, 93, 104-120, 135-161
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        381      71  81.36%   267, 301, 339-342, 362-366, 461-463, 467, 498-499, 505-509, 540, 605-613, 677-770, 781-786, 824, 850, 895-897, 949, 1129
plugins/download/base.py                       260      37  85.77%   129, 158, 225-227, 305-306, 364-365, 407, 411-422, 436, 504-508, 538-539, 564-572, 574-578, 628, 649, 671, 679
plugins/download/http.py                       545      77  85.87%   238, 280-283, 345-348, 351, 358-363, 394-396, 413, 428, 486, 521, 535, 549, 557-561, 577-582, 593, 611, 647-650, 671, 681, 688, 708, 843, 875, 905-914, 950, 975-976, 995-1000, 1009, 1024-1026, 1030, 1033, 1048-1049, 1125-1126, 1130, 1182, 1228-1229, 1235, 1245, 1271, 1306, 1326, 1363-1365
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         187      18  90.37%   107, 111, 135-141, 198-201, 294, 315, 448, 486, 508-511, 520
plugins/search/build_search_result.py          498      79  84.14%   245-246, 282, 285, 323, 326, 357-359, 570-581, 713, 715, 775-787, 808, 820, 846, 891, 941-956, 1004, 1029, 1032, 1036, 1076-1096, 1137, 1164-1165, 1174-1183, 1247, 1262, 1268, 1287-1296, 1413-1414, 1455, 1458, 1467-1469, 1526, 1572
plugins/search/cop_marine.py                   252      47  81.35%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 232, 238, 242, 246, 257, 268-269, 277, 309, 313, 334, 338, 342, 346, 350-354, 360-363, 366-380, 397, 446-450, 455, 467
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/qssearch.py                     726      77  89.39%   412-413, 530-531, 554-555, 567-571, 628-644, 761-764, 899, 906, 918, 975, 996, 999-1000, 1009-1010, 1019-1020, 1029-1030, 1057, 1109, 1192, 1215, 1288-1289, 1295, 1385, 1480-1484, 1550, 1553, 1559-1560, 1581, 1609-1621, 1628, 1660-1662, 1672-1678, 1708, 1731, 1736-1737, 1752, 1758, 1768, 1852, 1856, 1867, 1891, 1904, 1912-1922, 1959-1963
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            79      14  82.28%   100-126, 170, 183, 227
types/__init__.py                              161      45  72.05%   67, 71, 80-84, 95-107, 135-137, 144-149, 194, 213, 216, 220, 254, 264-280, 285, 287, 309, 314, 322, 332
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             84       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              576      63  89.06%   71, 160-182, 195-196, 205-231, 234, 249, 329-333, 408-412, 492, 532-533, 562, 937-940, 948-949, 956-958, 991, 1010-1011, 1040, 1058-1059, 1199, 1363, 1452, 1467, 1476, 1488, 1623-1629, 1663
utils/cache.py                                  22       0  100.00%
utils/dates.py                                  57       4  92.98%   63-65, 138
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       7  81.58%   51, 53, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                         8998    1245  86.16%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
api/product/metadata_mapping.py             +7       0  +0.06%
plugins/search/build_search_result.py      +14     +15  -2.64%
TOTAL                                      +21     +15  -0.14%

Results for commit: d1a9f2a

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@dalpasso dalpasso

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dalpasso @eodag-bot