Initialise the processing environment¶
After the deployment of the platform, and for each new user group (each Prefect instance), the administrators initialise the shared processing environment. The rs-demo repository provides Jupyter notebooks for these tasks: clone it in your JupyterHub session and run them from there.
1. Prefect block env-vars¶
The env-vars Prefect secret block contains the environment
variables shared by all the users of the Prefect instance. The flows read it at startup (see
Authentication). Initialise it with the
notebooks/init-prefect/env-vars.ipynb
notebook, that copies the values of the JupyterHub environment and asks for the others:
| Variable | Description |
|---|---|
RSPY_WEBSITE, RSPY_UAC_CHECK_URL, RSPY_PREFECT_URL |
URLs of the RS-Python services, of the API Key Manager and of Prefect (copied from JupyterHub). |
DASK_GATEWAY_ADDRESS, DASK_GATEWAY_PUBLIC |
Internal and public addresses of the Dask Gateway (copied from JupyterHub). |
JUPYTERHUB_API_TOKEN |
Token used by the flows and the services to connect to the Dask clusters (Dask Gateway uses the JupyterHub authentication). |
TEMPO_ENDPOINT |
OpenTelemetry endpoint for the traces (copied from JupyterHub). |
PREFECT_BUCKET_NAME, PREFECT_BUCKET_FOLDER |
Temporary bucket and folder used by the flows to share files, e.g. the payload files of the processors. |
POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_PI_DB |
Connection to the performance indicators database (from the rs_performance_indicator inventory variables). |
RSPY_HOST_OSAM |
Internal URL of the OSAM service. |
Each user then creates their own env-vars-<login> block, see
Set up your environment. The processing flows delivered by default run
on behalf of a technical owner (e.g. copernicus): create its account and its block the same way.
2. Prefect variables¶
The Prefect variables contain the default settings of the flows:
| Variable | Content | Documentation |
|---|---|---|
processing-storage-configuration |
Where the processors read and write their inputs, outputs and auxiliary files (S3 buckets, shared disks...). Initialise it with the notebooks/init-prefect/processing-storage-configuration.ipynb notebook, from the template delivered with rs-client-libraries. |
Storage configuration |
<processing_id>-default-setting |
Default parameters of the processing flows of a processing (s1l0, s1ard, s3l0, s3l1olci, s3l2olci): processor, pipeline, Dask cluster, input and target collections... They are checked against the ProcessorDefaultSettings.schema.json JSON schema. |
Customise the DPR processing |
The flows don't create the catalog collections where they publish their products: create them, on behalf of the
technical owner of the processing flows, before running the flows (with POST /catalog/collections, see
Endpoints). For each processing, these are the collections of the
generated_product_to_collection_identifier of its <processing_id>-default-setting variable: the collection_name,
or the product type in lowercase if it is not given (e.g. s03sral0c for the S03SRAL0C product type), as the
collection names are in lowercase by convention and the STAC collection identifiers are case sensitive. The
processing flows check that they exist at their very beginning, and fail with the list of the missing collections
otherwise. The Level-0 processings also need the s0<mission>-cadip-session collections where the sessions are
staged.
3. Dask clusters¶
Start the Dask clusters needed by the flows, see Add and delete Dask clusters.
The staging Dask cluster is kept running automatically by the dask-cluster-staging application of
rs-server-deployment.
4. Flows and automations¶
The flows and automations delivered by default are deployed by rs-workflow-deployment. Check that they are visible in the Deployments and Automations pages of the Prefect dashboard.
5. Users¶
Create the user accounts and assign their permissions, then run the OSAM synchronisation.