Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation for new system GSS #31

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Pressure levels

**Proměnné datasetu ERA5 hourly data on pressure levels (reanalysis-era5-pressure-levels)**
**Variables of the ERA5 hourly data on pressure levels dataset (reanalysis-era5-pressure-levels)**

| Name | Unit | Description |
|------|------|-------------|
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Single levels

**Proměnné datasetu ERA5 hourly data on single levels (reanalysis-era5-single-levels)**
**Variables of the ERA5 hourly data on single levels dataset (reanalysis-era5-single-levels)**

| Name | Unit | Description |
|------|------|-------------|
Expand Down
Binary file added docs/related/collgs/copsi_community.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/copsi_extended_search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/copsi_first_page.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/copsi_geo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/copsi_idps.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/copsi_map.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
38 changes: 21 additions & 17 deletions docs/related/collgs/era5.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,51 +13,55 @@ ERA5 is available on:
- Single levels
- Pressure levels: 1000/975/950/925/900/875/850/825/800/775/750/700/650/600/550/500/450/400/350/300/250/225/200/175/150/125/100/70/50/30/20/10/7/5/3/2/1

## Redistribuce ERA5 dat
## Redistribution of ERA5 Data

V rámci distribuce ERA5 dat jsou uživatelům poskytovány datasety [ERA5 hourly data on single levels](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=overview) a [ERA5 hourly data on pressure levels](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=overview). Data v těchto redistribuovaných datasetech zahrnují celé území České republiky, jež je specifikováno bounding boxem [48.48, 12.07, 51.08, 19.00](http://bboxfinder.com/#48.480000,12.070000,51.080000,19.00000).
As part of the ERA5 data distribution, users are provided with the datasets [ERA5 hourly data on single levels](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=overview) and [ERA5 hourly data on pressure levels](https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=overview). These redistributed datasets cover the entire territory of the Czech Republic, specified by the bounding box [48.48, 12.07, 51.08, 19.00](http://bboxfinder.com/#48.480000,12.070000,51.080000,19.00000).

K dnešnímu dni (7. listopadu 2023) jsou stažena data od října 2023 do srpna 2018, přičemž jsou stále stahovány starší data, až do prvního poskytovaného měsíce ledna 1940.
As of today (November 7, 2023), data from October 2023 to August 2018 have been downloaded, and older data is still being downloaded, up to the first available month of January 1940.

Seznamy proměnných, jež jsou v datasetech zahrnuty, jsou dostupné na stránkách:
The list of variables included in the datasets is available at the following pages:

- [ERA5 reanalysis-single-levels variables](../../../related/collgs/ERA5_reanalysis-era5-single-levels_variables)
- [ERA5 reanalysis-pressure-levels variables](../../../related/collgs/ERA5_reanalysis-era5-pressure-levels_variables)

Veřejnosti jsou data zpřístupněna bez potřeby registrace prostřednictvím STAC katalogu na adresách [https://stac.cesnet.cz/collections/reanalysis-era5-single-levels](https://stac.cesnet.cz/collections/reanalysis-era5-single-levels) a [https://stac.cesnet.cz/collections/reanalysis-era5-pressure-levels](https://stac.cesnet.cz/collections/reanalysis-era5-pressure-levels).
The data is available to the public without the need for registration via the STAC catalog at the following addresses:

Data nejsou připravována pro specifický dotaz uživatele, ale jsou předpřipravená, kdy pro každý dostupný měsíc uvažovaného datasetu jsou k dispozici čtyřy produktové typy:
- [https://stac.cesnet.cz/collections/reanalysis-era5-single-levels](https://stac.cesnet.cz/collections/reanalysis-era5-single-levels)
- [https://stac.cesnet.cz/collections/reanalysis-era5-pressure-levels](https://stac.cesnet.cz/collections/reanalysis-era5-pressure-levels)

The data is not prepared for specific user queries but is pre-processed. For each available month of the considered dataset, four product types are available:

- Reanalysis
- Ensemble mean
- Ensemble members
- Ensemble spread

Dle specifikace dat mohou být dále měněna a upravována a to až tři měsíce zpětně.
According to the data specification, they can be further modified and adjusted up to three months retrospectively.

Pro reflektování této skutečnosti jsou nyní data stahována do konce měsíce předcházejícímu aktuálnímu měsíci. Tedy například v listopadu bude posledním staženým měsícem říjen. Je ale nutné mít na paměti, že tato data se ještě můžou měnit.
To reflect this, data is now downloaded up to the end of the month preceding the current month. For example, in November, the last downloaded month will be October. However, it should be noted that this data may still change.

Dále pak dochází k aktualizaci a finalizaci dat, kdy je po 2. dni aktuálního měsíce stažen aktuálnímu měsící čtyři měsíce předcházející měsíc s finálními daty. Například tedy v listopadu budou stažena finální data pro červenec.
Additionally, the data undergoes an update and finalization process, where the data for the current month is downloaded along with the final data for the previous four months after the 2nd day of the current month. For example, in November, final data for July will be downloaded.

## Postup stažení
## Download Procedure

Data pro každý uvažovaný dataset a produktový typ jsou distribuovány ve formátu `.grib` a obsahují kompletní hodinová data pro všechny dostupné proměnné uvažovaného měsíce a pro dataset `reanalysis-era5-pressure-levels` také pro všechny úrovně tlaku.
Data for each considered dataset and product type are distributed in `.grib` format and contain complete hourly data for all available variables of the considered month. For the `reanalysis-era5-pressure-levels` dataset, data is also available for all pressure levels.

Jednotlivé měsíce jsou organizovány v ploché struktuře:
Individual months are organized in a flat structure:

`/collections/reanalysis-era5-single-levels/items/YYYY-MM-reanalysis-era5-single-levels`

Takovýto API endpoint vrací geojson daného itemu, přičemž pro stažení dat je významný node *assets*.
This API endpoint returns the GeoJSON of the given item, with the *assets* node being significant for downloading the data.

Node *assets* sestává ze čtyř dalších nodů, totiž jednotlivých produktových typů:
The *assets* node consists of four additional nodes, corresponding to the individual product types:

- reanalysis
- ensemble-mean
- ensemble-members
- ensemble-spread

V každém z těchto nodů produktových typů je obsažen klíč `href` jež odkazuje na požadovaný `.grib` soubor.
Each of these product-type nodes contains an `href` key that points to the desired `.grib` file.

## License

## Licence
The data is available under the [License to use Copernicus Products](https://www.copernicus.eu/en/access-data/copyright-and-licences).

Data jsou dostupná pod licencí [License to use Copernicus Products](https://www.copernicus.eu/en/access-data/copyright-and-licences).
134 changes: 134 additions & 0 deletions docs/related/collgs/gss_login.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
## User Authentication in GSS

This section describes user authentication in the GSS (Gael Store Service) system, including both interactive login via the graphical COPSI component and obtaining and exchanging tokens for machine use.

Both methods require registration.

Instructions on using this software can be found in the section on [working with Sentinel satellite data](./sentinel.md).

## Graphical Interface

![copsi_first_page.png](copsi_first_page.png)

When accessing the COPSI application at [https://collgs.cesnet.cz](https://collgs.cesnet.cz), a login window will appear.

After confirming by clicking _Sign In_, you will be redirected to select the community you want to log in with.

![copsi_community.png](copsi_community.png)

If you do not have an account in any of the communities, choose the option _E-INFRA CZ_. Then select your affiliated institution or choose a social identity (e.g., ORCID).

![copsi_idps.png](copsi_idps.png)

Complete the login process. If this is your first login, you will be redirected to a registration form.

Fill in the required details in the form. By submitting the completed application, an account is created in the IAM Perun system.

The account remains active for two years, and you will receive an email notification before the account expires to extend its validity.

![perun_registration.png](perun_registration.png)

Based on the selected community during login, you will either be redirected back to the COPSI application, or your application will wait for approval by the organization administrator.

If, after approval, the application redirects you to a page informing you that you are already registered, try accessing the application in an incognito window or clearing your cookies.

## Machine Access

To download products via the OData API, authentication using an access token must be added to the requests.

Tokens can be obtained at [https://keycloak.grid.cesnet.cz/token-portal](https://keycloak.grid.cesnet.cz/token-portal).

![token_portal_intro.png](token_portal_intro.png)

On the portal page, proceed by selecting _Authorise_. If you are generating tokens for the first time, you will need to register.

Otherwise, log in similarly to the process described in [Graphical Interface](#graphical-interface).

![token_portal.png](token_portal.png)

On the _Generate Tokens_ page, the following items are available:

**Access Token** - Used for authentication and authorization when accessing the API. It is valid for 8 hours.

**Refresh Token** - Used to renew the access token without the need to log in again. It is valid for 30 days. Each time an access token is exchanged, a new refresh token is issued, and the previous one becomes invalid. For the next token exchange, the new refresh token must be used. The new refresh token will also have a validity of 30 days.

**Client ID and Client Secret** - Used to authenticate the application during token exchange.

**Token Exchange Command** - This command allows you to obtain a new access token and refresh token with full validity when the access token expires.

**Token Management Link** - Provides access to the application for managing your account and tokens. Here, you can log out from a device and revoke the refresh token.

### How to Work with Tokens

#### 1. Accessing the API with an Access Token

After obtaining an access token, include it in your API requests as an authorization header:

Authorization: Bearer <access_token>

Adding this header to your requests allows the API to verify your identity and grant the necessary permissions.

A GSS OData API call may look as follows:

```shell
curl -X GET "https://collgs.cesnet.cz/odata/v1/Products?\$filter=startswith(Name,'S1')%20and%20contains(Name,'SLC')" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-H "Accept: application/json"
```

#### 2. Refreshing the Access Token Using the Refresh Token

When the access token expires, you can use the refresh token to obtain a new access token without logging in again.

The process involves sending a POST request, which is pre-configured as a _curl_ command on the Token Portal page. If you are using Python version 3, you may need to replace _python_ with _python3_ in the command.

This request will return a new access token with an 8-hour validity, along with a refresh token valid for 30 days.

The response in JSON format will look as follows:

{
"access_token": "eyJh....gywQ",
"expires_in": 300,
"refresh_expires_in": 28800,
"refresh_token": "eyJh....xjVg",
"token_type": "Bearer",
"id_token": "eyJh....zLMA",
"not-before-policy": 0,
"session_state": "80....ff",
"scope": "openid email profile"
}

You can use the new access token for further requests in GSS, and replace the old refresh token with the new one.

#### 3. Secure Token Storage

**Access Token**:
Store in memory (e.g., a variable or cache) only while it is active. Since it has a shorter validity period, there is no need for long-term storage.

**Refresh Token**:
It is recommended to store in a secure location, such as a database or encrypted storage.

Since the refresh token has a longer validity period, it should be stored securely with limited access.

#### 4. Token Revocation

If necessary, tokens can be revoked using the account management application linked at the bottom of the Token Portal.

Revocation (logging out) is located under **Account Security > Device Activity**.

Use the _Sign out_ option to log out a specific device, or _Sign out all devices_ for a complete logout.

![token_revoke.png](token_revoke.png)

### Token Exchange

If you are already working with tokens from EGI Check-in, it is possible to exchange them for tokens granting access to the GSS system without further login. In this case, send a valid access token issued by EGI Check-in via POST method to the endpoint [https://keycloak.grid.cesnet.cz/realms/collgs/protocol/openid-connect/token](https://keycloak.grid.cesnet.cz/realms/collgs/protocol/openid-connect/token) and attach the following to the request body:

{
"client_id": "token-exchange",
"grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
"subject_token": {EGI access token},
"subject_issuer": "https://aai.egi.eu/auth/realms/egi"
}

In the response, you will receive an access token that can be used for GSS API calls.
Binary file added docs/related/collgs/gss_overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/related/collgs/gss_polygons.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
48 changes: 27 additions & 21 deletions docs/related/collgs/index.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,46 @@
# O službě
# About the Service

Toto je přehled všech stránek v kategorii *Copernicus Collaborative Ground Segment – CollGS*.
This is an overview of all pages in the category *Copernicus Collaborative Ground Segment – CollGS*.

CESNET provozuje český národní uzel iniciativy Evropské kosmické agentury (ESA) [Spolupracující pozemní segment Sentinel](http://collgs.czechspaceportal.cz).
CESNET operates the Czech national node of the European Space Agency's (ESA) initiative [Sentinel Collaborative Ground Segment](http://collgs.czechspaceportal.cz).

Družice Sentinel jsou páteří a základním zdrojem dat evropského programu pozorování Země [Copernicus](http://copernicus.gov.cz).
Sentinel satellites are the backbone and primary data source of the European Earth observation program [Copernicus](http://copernicus.gov.cz).

Národní uzel slouží k uložení a distribuci dat Sentinel pokrývajících území České republiky. Přístup k datům je pro všechny uživatele zdarma.
The national hub serves for storing and distributing Sentinel data covering the territory of the Czech Republic.

V závislosti na povaze zpracování a následného využití dat může CESNET nabídnout také doplňkové služby jako výpočetní či úložné kapacity.
In addition to Sentinel data, we also provide access to datasets from Landsat and ERA5, each detailed on separate pages in this documentation.

Access to the data is free for all users.

Depending on the nature of data processing and subsequent usage, CESNET can also offer supplementary services such as computing or storage capacities.

![pic](sentinel-strip.jpg)

## Registrace a správa účtu
## Registration and Account Management

For accessing Sentinel data, you first need to register.

Účet se registruje vyplněním formuláře na [Webu národního uzlu](https://dhr1.cesnet.cz/#/self-registration).
The registration form will appear upon your first login to the GSS (Gael Store Service) system.

Po vyplnění formuláře Vám systém zašle e-mailovou zprávu, na jejímž základě svou registraci potvrdíte.
After completing the form, the system will send you an email, through which you can confirm your registration.

Po úspěšné registraci můžete okamžitě přistupovat k datům národního uzlu. Možné způsoby přístupu k datům popisuje stránka [Přístup k datům v národním uzlu CollGS](../../related/collgs/pristup-k-datum).
Once successfully registered, you can access the data of the national hub.

Velmi kvalitní dokumentace nejen k práci s uživatelským rozhraním ale také k datové struktuře produktů je k dispozici na [stránkách Czech Space Portal](https://collgs.czechspaceportal.cz/uzivatelska-prirucka/manual-ke-collgs/).
Possible methods of accessing the data are described on the page about [accessing Sentinel satellite data](../../related/collgs/sentinel).

O veškerou manipulaci se svým účtem, kterou nemůžete udělat sami, požádejte na adrese <[email protected]>.
For any account-related actions that you cannot perform yourself, please contact <[email protected]>.

## Naše spolupráce
## Our Collaboration

Přístup k datům z národního uzlu CollGS využívají uživatelské skupiny z mnoha oborů. Těsnější spolupráci s námi navázaly např. tyto projekty:
User groups from various fields utilize access to data from the national CollGS hub. For example, the following projects have established closer collaboration with us:

- [EGI Open Data Platform &ndash; EGI DataHub](https://datahub.egi.eu)
- [Projekt C-SCALE](https://c-scale.eu)
- [Sentinel 1 (SAR) data pre-processing v multi-cloudovém prostředí](https://github.com/SimonNtz/SAR_app)
- [Mapové produkty Ústavu pro hospodářskou úpravu lesů](http://geoportal.uhul.cz/mapy/MapyDpz.html)
- [EGI Open Data Platform &ndash; EGI DataHub](https://datahub.egi.eu)
- [C-SCALE Project](https://c-scale.eu)
- [Sentinel 1 (SAR) Data Pre-Processing in a Multi-Cloud Environment](https://github.com/SimonNtz/SAR_app)
- [Mapping Products of the Institute for Forest Management](http://geoportal.uhul.cz/mapy/MapyDpz.html)

## Kontakt
## Contact

Chcete-li přímo kontaktovat tým poskytující služby pozemního segmentu, použijte adresu <[email protected]>.
If you wish to contact the team providing ground segment services directly, use the address <[email protected]>.

Na uvedenou adresu můžete směrovat veškeré dotazy, hlášení problémů i své návrhy a připomínky.
You can send all inquiries, problem reports, as well as your suggestions and feedback to this address.
Loading