Skip to content

Commit de34cf5

Browse files
sbrunatosbrunato
committed
feat(core)!: STAC formatted properties (#1730)
1 parent 5aaa691 commit de34cf5

File tree

150 files changed

+55901
-30277
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

150 files changed

+55901
-30277
lines changed

‎.github/workflows/fetch.yml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Fetch product types
1+
name: Fetch collections
22

33
on:
44
workflow_dispatch:
@@ -8,8 +8,8 @@ on:
88
- cron: "0 6 * * *"
99

1010
jobs:
11-
fetch-product-types:
12-
name: Fetch providers for new product types
11+
fetch-collections:
12+
name: Fetch providers for new collections
1313
runs-on: ubuntu-latest
1414
steps:
1515
- name: Checkout the repo
@@ -28,22 +28,22 @@ jobs:
2828
uv venv
2929
source .venv/bin/activate
3030
uv pip install ".[all-providers]"
31-
- name: Fetch and update external product types reference
31+
- name: Fetch and update external collections reference
3232
env:
3333
EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__USERNAME: ${{ secrets.EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__USERNAME }}
3434
EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__PASSWORD: ${{ secrets.EODAG__WEKEO_MAIN__AUTH__CREDENTIALS__PASSWORD }}
3535
run: |
3636
source .venv/bin/activate
37-
export JSON_OUTPUT_FILE="eodag/resources/ext_product_types.json"
38-
export JSON_REF_FILE=$(python -c "import eodag; print(eodag.config.EXT_PRODUCT_TYPES_CONF_URI)")
37+
export JSON_OUTPUT_FILE="eodag/resources/ext_collections.json"
38+
export JSON_REF_FILE=$(python -c "import eodag; print(eodag.config.EXT_COLLECTIONS_CONF_URI)")
3939
eodag -vvv discover --storage ${JSON_OUTPUT_FILE}
4040
# add empty line at end of file
4141
sed -i -e '$a\' ${JSON_OUTPUT_FILE}
4242
git config user.name "github-actions[bot]"
4343
git config user.email "'github-actions[bot]@users.noreply.github.com"
4444
git add "${JSON_OUTPUT_FILE}"
45-
git commit -m "chore: update external product types reference" || exit 0
46-
echo "Update external product types reference from daily fetch. See [Python API User Guide / Product types discovery](https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/2_providers_products_available.html#Product-types-discovery)" >> $GITHUB_STEP_SUMMARY
45+
git commit -m "chore: update external collections reference" || exit 0
46+
echo "Update external collections reference from daily fetch. See [Python API User Guide / Collections discovery](https://eodag.readthedocs.io/en/latest/notebooks/api_user_guide/1_providers_products_available.ipynb#Collections-discovery)" >> $GITHUB_STEP_SUMMARY
4747
echo '### Changed files' >> $GITHUB_STEP_SUMMARY
4848
echo 'May be truncated.' >> $GITHUB_STEP_SUMMARY
4949
COMMIT_SHA=$(git rev-parse HEAD)
@@ -61,9 +61,9 @@ jobs:
6161
- name: Create Pull Request
6262
uses: peter-evans/create-pull-request@v6
6363
with:
64-
branch: external-product-types-ref-update
64+
branch: external-collections-ref-update
6565
delete-branch: true
66-
title: 'chore: update external product types reference'
66+
title: 'chore: update external collections reference'
6767
body: ${{ env.UPDATE_SUMMARY }}
6868
labels: |
6969
automated pr

‎.gitignore‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -104,9 +104,9 @@ docs/notebooks/tutos/*workspace*
104104
examples/eodag_workspace
105105
tests/resources/user_conf.yml
106106
search_results.geojson
107-
ext_product_types.json
108-
!eodag/resources/ext_product_types.json
109-
!tests/resources/ext_product_types.json
107+
ext_collections.json
108+
!eodag/resources/ext_collections.json
109+
!tests/resources/ext_collections.json
110110

111111
# IDE
112112
# Pycharm

‎BREAKING_CHANGES.rst‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,261 @@ Migration guide
33

44
Full changelog available in `Release history <changelog.html>`_.
55

6+
v4.0.0b1
7+
++++++++
8+
9+
STAC-Formatted API and Results
10+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11+
12+
Migration from `OGC OpenSearch Extension for Earth Observation <https://docs.ogc.org/is/13-026r9/13-026r9.html>`_
13+
to `SpatioTemporal Asset Catalog (STAC) <https://github.com/radiantearth/stac-spec>`_ for data representation.
14+
15+
This impacts both the API and CLI.
16+
17+
Renamed properties
18+
^^^^^^^^^^^^^^^^^^
19+
20+
The list starts with common STAC properties, then lists STAC extensions properties, and finally EODAG-specific
21+
properties.
22+
23+
.. list-table::
24+
:header-rows: 1
25+
26+
* - v3.x.x property
27+
- v4.x.x property
28+
* - ``productType``
29+
- ``collection``
30+
* - ``startTimeFromAscendingNode``
31+
- ``start_datetime``
32+
* - ``completionTimeFromAscendingNode``
33+
- ``end_datetime``
34+
* - ``abstract``
35+
- ``description``
36+
* - ``accessConstraint``
37+
- ``license``
38+
* - ``keyword``
39+
- ``keywords``
40+
* - ``creationDate``
41+
- ``created``
42+
* - ``modificationDate``
43+
- ``updated``
44+
* - ``organisationName``
45+
- ``providers: [{"roles": ["producer"]}]``
46+
* - ``platform``
47+
- ``constellation``
48+
* - ``platformSerialIdentifier``
49+
- ``platform``
50+
* - ``instrument``
51+
- ``instruments[]``
52+
* - ``resolution``
53+
- ``gsd``
54+
* - ``cloudCover``
55+
- ``eo:cloud_cover``
56+
* - ``snowCover``
57+
- ``eo:snow_cover``
58+
* - ``tileIdentifier``
59+
- ``grid:code``
60+
* - ``gridSquare``
61+
- ``mgrs:grid_square``
62+
* - ``latitudeBand``
63+
- ``mgrs:latitude_band``
64+
* - ``utmZone``
65+
- ``mgrs:utm_zone``
66+
* - ``storageStatus``
67+
- ``order:status``
68+
* - ``processingLevel``
69+
- ``processing:level``
70+
* - ``acquisitionType``
71+
- ``product:acquisition_type``
72+
* - ``productType``
73+
- ``product:type`` or ``collection``
74+
* - ``timeliness``
75+
- ``product:timeliness``
76+
* - ``dopplerFrequency``
77+
- ``sar:frequency_band``
78+
* - ``polarisationChannels`` or ``polarizationChannels``
79+
- ``sar:polarizations[]``
80+
* - ``sensorMode``
81+
- ``sar:instrument_mode``
82+
* - ``swathIdentifier``
83+
- ``sar:instrument_mode`` or ``sar:beam_ids[] (geodes)``
84+
* - ``cycleNumber``
85+
- ``sat:orbit_cycle``
86+
* - ``orbitDirection``
87+
- ``sat:orbit_state``
88+
* - ``orbitNumber``
89+
- ``sat:absolute_orbit``
90+
* - ``relativeOrbitNumber``
91+
- ``sat:relative_orbit``
92+
* - ``doi``
93+
- ``sci:doi``
94+
* - ``illuminationAzimuthAngle``
95+
- ``view:sun_azimuth``
96+
* - ``illuminationElevationAngle``
97+
- ``view:sun_elevation``
98+
* - ``illuminationZenithAngle``
99+
- ``view:incidence_angle``
100+
* - ``productVersion``
101+
- ``version``
102+
* - ``defaultGeometry``
103+
- ``eodag:default_geometry``
104+
* - ``downloadLink``
105+
- ``eodag:download_link``
106+
* - ``mtdDownloadLink``
107+
- ``eodag:mtd_download_link``
108+
* - ``sensorType``
109+
- ``eodag:sensor_type``
110+
* - ``quicklook``
111+
- ``eodag:quicklook``
112+
* - ``thumbnail``
113+
- ``eodag:thumbnail``
114+
* - ``combinedOrderId``
115+
- ``eodag:combined_order_id``
116+
* - ``downloadId``
117+
- ``eodag:download_id``
118+
* - ``message``
119+
- ``eodag:order_message``
120+
* - ``orderId``
121+
- ``eodag:order_id``
122+
* - ``orderStatus``
123+
- ``eodag:order_status``
124+
* - ``orderStatusLink``
125+
- ``eodag:status_link``
126+
* - ``percent``
127+
- ``eodag:percent``
128+
* - ``productInfo``
129+
- ``eodag:product_info``
130+
* - ``productPath``
131+
- ``eodag:product_path``
132+
* - ``requestParams``
133+
- ``eodag:request_params``
134+
* - ``searchLink``
135+
- ``eodag:search_link``
136+
* - ``tilePath``
137+
- ``eodag:tile_path``
138+
* - ``providerProductType``
139+
- ``_collection`` or ``provider_collection``
140+
141+
Python API updates
142+
^^^^^^^^^^^^^^^^^^
143+
144+
All product-types related classes and methods have been renamed with collections.
145+
146+
.. list-table::
147+
:header-rows: 1
148+
149+
* - v3.x.x Python API
150+
- v4.x.x Python API
151+
* - ``EODataAccessGateway.available_providers(product_type=None, ...)``
152+
- :meth:`~eodag.api.core.EODataAccessGateway.available_providers` ``(collection=None, ...)``
153+
* - ``EODataAccessGateway.discover_product_types()``
154+
- :meth:`~eodag.api.core.EODataAccessGateway.discover_collections`
155+
* - ``EODataAccessGateway.fetch_product_types_list()``
156+
- :meth:`~eodag.api.core.EODataAccessGateway.fetch_collections_list`
157+
* - ``EODataAccessGateway.guess_product_type()``
158+
- :meth:`~eodag.api.core.EODataAccessGateway.guess_collection`
159+
* - ``EODataAccessGateway.list_product_types()``
160+
- :meth:`~eodag.api.core.EODataAccessGateway.list_collections`
161+
* - ``EODataAccessGateway.update_product_types_list()``
162+
- :meth:`~eodag.api.core.EODataAccessGateway.update_collections_list`
163+
* - ``EOProduct.product_type``
164+
- :attr:`~eodag.api.product._product.EOProduct.collection`
165+
166+
CLI updates
167+
^^^^^^^^^^^
168+
169+
Check latest `CLI User Guide <cli_user_guide.rst>`_ for complete guidelines.
170+
171+
.. list-table::
172+
:header-rows: 1
173+
174+
* - v3.x.x Command and options
175+
- v4.x.x Command and options
176+
* - ``eodag search -p, --productType TEXT``
177+
- ``eodag search -c, --collection TEXT``
178+
* - ``eodag search --query provider=TEXT``
179+
- ``eodag search -p, --provider TEXT``
180+
* - ``eodag search -i, --instrument TEXT``
181+
- ``eodag search --instruments TEXT``
182+
* - ``eodag search -P, --platform TEXT``
183+
- ``eodag search --constellation TEXT``
184+
* - ``eodag search -t, --platformSerialIdentifier TEXT``
185+
- ``eodag search --platform TEXT``
186+
* - ``eodag search -c, --cloudCover INTEGER RANGE``
187+
- ``eodag search --cloud-cover INTEGER RANGE``
188+
* - ``eodag search -L, --processingLevel TEXT``
189+
- ``eodag search --processing-level TEXT``
190+
* - ``eodag search -S, --sensorType TEXT``
191+
- ``eodag search --sensor-type TEXT``
192+
* - ``eodag list -i, --instrument TEXT``
193+
- ``eodag list --instruments TEXT``
194+
* - ``eodag list -P, --platform TEXT``
195+
- ``eodag list --constellation TEXT``
196+
* - ``eodag list -t, --platformSerialIdentifier TEXT``
197+
- ``eodag list --platform TEXT``
198+
* - ``eodag list -L, --processingLevel TEXT``
199+
- ``eodag list --processing-level TEXT``
200+
* - ``eodag list -S, --sensorType TEXT``
201+
- ``eodag list --sensor-type TEXT``
202+
203+
204+
Environment variables renamed
205+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
206+
207+
Check updated list of environment variables in `Getting started / Configure EODAG / environment variables
208+
<getting_started_guide/configuration.html#core-configuration-using-environment-variables>`_.
209+
210+
.. list-table::
211+
:header-rows: 1
212+
213+
* - v3.x.x environment variable
214+
- v4.x.x environment variable
215+
* - ``EODAG_PRODUCT_TYPES_CFG_FILE``
216+
- ``EODAG_COLLECTIONS_CFG_FILE``
217+
* - ``EODAG_EXT_PRODUCT_TYPES_CFG_FILE``
218+
- ``EODAG_EXT_COLLECTIONS_CFG_FILE``
219+
* - ``EODAG_STRICT_PRODUCT_TYPES``
220+
- ``EODAG_STRICT_COLLECTIONS``
221+
222+
Files renamed
223+
^^^^^^^^^^^^^
224+
225+
.. list-table::
226+
:header-rows: 1
227+
228+
* - v3.x.x file path
229+
- v4.x.x file path
230+
* - ``docs/_static/product_types_information.csv``
231+
- ``docs/_static/collections_information.csv``
232+
* - ``docs/_static/eodag_fetch_product_types.png``
233+
- ``docs/_static/eodag_fetch_collections.png``
234+
* - ``docs/add_product_type.rst``
235+
- ``docs/add_collection.rst``
236+
* - ``docs/getting_started_guide/product_types.rst``
237+
- ``docs/getting_started_guide/collections.rst``
238+
* - ``eodag/resources/product_types.yml``
239+
- ``eodag/resources/collections.yml``
240+
* - ``eodag/resources/ext_product_types.json``
241+
- ``eodag/resources/ext_collections.json``
242+
* - ``tests/resources/ext_product_types.json``
243+
- ``tests/resources/ext_collections.json``
244+
* - ``tests/resources/ext_product_types_free_text_search.json``
245+
- ``tests/resources/ext_collections_free_text_search.json``
246+
* - ``tests/resources/file_product_types_modes.yml``
247+
- ``tests/resources/file_collections_modes.yml``
248+
* - ``tests/resources/file_product_types_override.yml``
249+
- ``tests/resources/file_collections_override.yml``
250+
* - ``tests/resources/stac/product_type_queryables.json``
251+
- ``tests/resources/stac/collection_queryables.json``
252+
* - ``utils/product_types_information_to_csv.py``
253+
- ``utils/collections_information_to_csv.py``
254+
255+
External collections reference configuration file is now hosted as
256+
`https://cs-si.github.io/eodag/eodag/resources/ext_collections.json
257+
<https://cs-si.github.io/eodag/eodag/resources/ext_collections.jsons>`_. See `API user guide / Providers and products
258+
/ Collections discovery <notebooks/api_user_guide/1_providers_products_available.ipynb#Collections-discovery>`_ for more
259+
information.
260+
6261
v3.0.0b3
7262
++++++++
8263

‎README.rst‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,9 @@ EODAG (Earth Observation Data Access Gateway) is a command line tool and a plugi
4242
aggregating results and downloading remote sensed images while offering a unified API for data access regardless of the
4343
data provider. The EODAG SDK is structured around three functions:
4444

45-
* List product types: list of supported products and their description
45+
* List collections: list of supported products and their description
4646

47-
* Search products (by product type or uid): searches products according to the search criteria provided
47+
* Search products (by collection or uid): searches products according to the search criteria provided
4848

4949
* Download products: download product “as is"
5050

@@ -108,7 +108,7 @@ Example usage for interacting with the api in your Python code:
108108
dag = EODataAccessGateway()
109109
110110
search_results = dag.search(
111-
productType='S2_MSI_L1C',
111+
collection='S2_MSI_L1C',
112112
geom={'lonmin': 1, 'latmin': 43.5, 'lonmax': 2, 'latmax': 44}, # accepts WKT polygons, shapely.geometry, ...
113113
start='2021-01-01',
114114
end='2021-01-15'
@@ -137,9 +137,9 @@ Start playing with the CLI:
137137

138138
- To search for some products::
139139

140-
eodag search --productType S2_MSI_L1C --box 1 43 2 44 --start 2021-03-01 --end 2021-03-31
140+
eodag search --collection S2_MSI_L1C --box 1 43 2 44 --start 2021-03-01 --end 2021-03-31
141141

142-
The request above searches for ``S2_MSI_L1C`` product types in a given bounding box, in March 2021. It saves the results in a GeoJSON file (``search_results.geojson`` by default).
142+
The request above searches for ``S2_MSI_L1C`` collections in a given bounding box, in March 2021. It saves the results in a GeoJSON file (``search_results.geojson`` by default).
143143

144144
Results are paginated, you may want to get all pages at once with ``--all``, or search products having 20% of maximum coud cover with ``--cloudCover 20``. For more information on available options::
145145

@@ -153,11 +153,11 @@ Start playing with the CLI:
153153

154154
eodag download --quicklooks --search-results search_results.geojson
155155

156-
- To list all available product types and supported providers::
156+
- To list all available collections and supported providers::
157157

158158
eodag list
159159

160-
- To list available product types on a specified supported provider::
160+
- To list available collections on a specified supported provider::
161161

162162
eodag list -p creodias
163163

0 commit comments

Comments
 (0)