Skip to content

rs-server developer manual

This page describes how to develop on rs-server. The general guidelines of the project (coding environment, code style, git workflow, CI/CD) are described in the Developers and contributors section of the documentation.

Repository structure

rs-server
├── docs/                  # This documentation: Markdown pages (doc/), PlantUML sources (uml/) and images (images/)
├── resources/             # Development scripts (configuration files synchronisation, dependency updates...)
├── services/
│   ├── adgs/              # rs-server-adgs (AUXIP instances)
│   ├── cadip/             # rs-server-cadip (CADIP stations)
│   ├── catalog/           # rs-server-catalog
│   ├── common/            # rs_server_common, shared by all the services
│   ├── frontend/          # rs-server-frontend (aggregated Swagger page)
│   ├── osam/              # rs-server-osam
│   ├── prip/              # rs-server-prip (PRIP instances)
│   └── staging/           # rs-server-staging
├── tests/                 # Tests of the station services and of the common package
├── pyproject.toml         # Poetry project that includes all the services
└── CHANGELOG.md

Each service folder contains:

  • rs_server_<service>/: the Python package of the service,
  • config/: its configuration files, e.g. the STAC mapping and the virtual collections of the stations,
  • tests/: its tests (for some of them),
  • pyproject.toml: its own Poetry project, used to build its wheel package,
  • .github/Dockerfile: its Docker image, built by the CI (see Repositories and CI/CD).

The rs_server_common package contains the code shared by the services:

Module Content
authentication Authentication with API keys (checked by the API Key Manager) and OAuth2 (Keycloak), authorisation checks, authentication to the external stations.
stac_api_common, stac_cql2, stac_sortables Common implementation of the STAC API for the station services: collections from the configuration, conversion of the STAC search parameters and CQL2 filters into OData requests, pagination.
data_retrieval Access to the stations with EODAG.
s3_storage_handler Access to the S3 object storage, bucket configuration.
utils Logging, OpenTelemetry instrumentation, utilities.

Set up and test

poetry install --with dev
poetry run opentelemetry-bootstrap -a install
poetry run pre-commit install

# Run the tests (the catalog tests start a pgstac database with Docker)
poetry run pytest -m unit

The tests of the catalog use pytest-docker to run a pgstac database: Docker must be installed.

To run the services with their dependencies (databases, station mockups, Dask, Prefect, Jupyter...), and to test your local source code without rebuilding the Docker images, use the local mode.

Configuration files

The configuration files of the station services (*_ws_config.yaml, *_search_config.yaml, rs-server.yaml...) are generated from templates by the resources/sync_config_files.py script, which also copies them to the rs-demo (local mode), rs-helm and rs-server-deployment repositories. Don't modify the generated files directly: modify the templates and run the script again.

Document the endpoints

The endpoints are documented in their OpenAPI description, displayed in the Swagger page: describe each endpoint, its parameters and its responses in the docstring and the FastAPI decorators of its Python function. After a modification, the aggregated OpenAPI specification of the frontend is regenerated by the CI.

Add a service

  1. Create the services/<service> folder, with the same structure as the other services.
  2. Add its package to the root pyproject.toml and to the CI (publish-binaries.yml workflow).
  3. Add it to the frontend services.yml and to the build_aggregated_openapi.sh script.
  4. Add its Helm chart to rs-helm, its deployment to rs-server-deployment, and its container to the local mode.
  5. Add it to the Python API documentation of rs-documentation (.github/scripts/build_pydoc.sh).

Add a station

  1. Add the station URL and credentials in the station secrets (rs-server.yaml file in local mode, rs-server-station-secrets in cluster mode).
  2. Add the station to the configuration of its service and its virtual collections to the *_search_config.yaml file.
  3. Create the rs_<service>_<station>_read and rs_processes_staging_download_<station> Keycloak roles.