Project organisation¶
This page describes the RS-Python components from a functional point of view (what you can do with them), then explains where they are implemented in the RS-PYTHON GitHub organisation.
Components for the users¶
Reference System web services¶
The web services are implemented by rs-server and rs-dpr-service. They are all protected by authentication and fine-grained authorisation (see User rights and API keys) and documented in a single Swagger page.
- External data sources: STAC API services to search the Copernicus ground segment in read-only mode: CADIP acquisition stations (raw data sessions), AUXIP instances (auxiliary data) and PRIP production services (Level-0, Level-1 and Level-2 products).
- Catalog: a STAC API service to search, read and update the RS-Python catalog, stored in a PostgreSQL database. Each user owns their collections. The catalog products are also exposed with the OData PRIP interface of the ESA Production Interface Delivery Points.
- Staging: an OGC API - Processes service that copies products from the external data sources into the RS-Python object storage and catalog.
- Processing: an OGC API - Processes service that runs the DPR processors on a Dask cluster.
- OSAM (Object Storage Access Manager): get your S3 credentials to download the products from the object storage.
STAC browsers¶
The STAC Browser web interfaces allow to browse the products of the stations and of the catalog without writing any code.
DPR processors¶
The DPR processors generate the Sentinel products. They are delivered by the EOPF project as Python packages, embedded into Docker images by RS-Python, and run on the Reference System Dask clusters.
Prefect¶
Prefect runs and monitors the processing flows:
- on-demand flows, started by a user from the Prefect dashboard or from Python code,
- systematic flows, started automatically when new data is available (see Systematic processing),
- custom flows, written by the users themselves (see Create a custom flow).
The flows call the Reference System services with the rs-client-libraries.
JupyterHub¶
JupyterHub provides a virtual environment in which the RS-Python libraries are already installed. Use its notebooks to call the services, test your code and customise your Prefect flows.
Dask¶
Dask runs the DPR processors and the staging tasks in parallel. The Dask dashboard shows the progress of a processor run at a lower level than Prefect.
API Key Manager¶
The API Key Manager creates the API keys that hold your authentication and authorisation information, used to call the services and run the Prefect flows.
Components for the administrators¶
In addition to the components above, the administrators use:
- the Keycloak administration console, to manage the user accounts and their rights,
- the object storage provider console,
- the OSAM service, to synchronise the user accounts and rights between Keycloak and the S3 object storage,
- Grafana and the OpenTelemetry instrumentation, to monitor the logs, metrics, traces and resources of the platform.
Source code organisation¶
The source code is hosted in the RS-PYTHON GitHub organisation. Each repository
has a README.md file that describes its content.
Documentation repositories¶
| Repository | Description |
|---|---|
| rs-documentation | System-level documentation; aggregates the documentation of all the other repositories into this website. |
| rs-demo | Jupyter notebooks showcasing the features of the system, and the local mode to run the whole system on your computer. |
| RS-PYTHON.github.io | Sources of the project landing page. |
Python repositories¶
| Repository | Description |
|---|---|
| rs-server | The rs-server web services: STAC access to the external data sources and to the catalog, staging, OSAM, frontend. |
| rs-dpr-service | The web service that retrieves the task tables and triggers the DPR processors in a Dask cluster. |
| rs-client-libraries | The rs_client library (which simplifies calling the services), the rs_workflows Prefect flows and the rs_common utilities. |
| rs-testmeans | Mockups of the external data sources (CADIP, AUXIP, PRIP, LTA) and of a DPR processor, used for the tests and the demos. |
| opentelemetry-instrumentation-eopf | OpenTelemetry instrumentation for EOPF. |
STAC extensions¶
| Repository | Description |
|---|---|
| cadip-stac-extension | Describes Copernicus CADIP sessions from the acquisition ground stations as STAC items. |
| ownership-stac-extension | Expresses the ownership of STAC collections and items. |
| download-stac-api-extension | STAC API extension that adds a dedicated endpoint to download an asset. |
Infrastructure repositories¶
| Repository | Description |
|---|---|
| rs-infra-core | Terraform and Ansible playbooks to create the Kubernetes cluster and deploy the core infrastructure components. |
| rs-infra-monitoring | The monitoring components (Grafana, Prometheus, Loki, Tempo, Alloy) deployed on top of the core infrastructure. |
| rs-workflow-env | The virtual environment components (JupyterHub, Prefect, Dask Gateway, API Key Manager) and the Docker images of the Dask clusters and of JupyterHub. |
| rs-server-deployment | The deployment of the rs-server services, rs-dpr-service, STAC browsers, databases and mockups on top of the core infrastructure. |
| rs-helm | The Helm charts of the rs-server services, rs-dpr-service, STAC browsers and mockups. |
| rs-workflow-deployment | The deployment of the Prefect flows and automations delivered by default. |
The security components (e.g. Wazuh and NeuVector) are deployed from a separate, non-public repository.
Tool repositories¶
| Repository | Description |
|---|---|
| actions | Reusable GitHub Actions workflows and composite actions for all the RS-PYTHON repositories. |
The dependencies between the repositories and their CI/CD are described in Repositories and CI/CD.