Skip to content

rs-workflow-env

The virtual environment of RS-Python, the Copernicus Reference System Python: the components that support the processing orchestration, deployed on top of the core infrastructure, and the Docker images of JupyterHub and of the Dask clusters.

Overview

The apps folder contains the applications deployed with the Ansible playbooks of rs-infra-core (see Deploy RS-Python):

Application Description
jupyterhub JupyterHub: one JupyterLab session per user, with the RS-Python libraries, see JupyterHub.
prefect3-server, 01-prefect3-db The Prefect server and its database, protected by Keycloak.
prefect3-worker-integrated, prefect3-worker-sandbox, prefect3-worker-monitoring The Prefect Kubernetes workers of the three work pools.
prefect3-grafanadatasource The Grafana data source of the Prefect database.
dask-gateway Dask Gateway, that creates the Dask clusters, with the JupyterHub authentication, see Add and delete Dask clusters.
apikeymanager, 01-apikeymanager-db The API Key Manager and its database, see User rights and API keys.
01-listenerset The Gateway API listeners of the public host names (processing.<domain>, apikeymanager.<domain>, dash-dask.processing.<domain>).
gitlab-runner-cpm A GitLab runner used to run the EOPF CPM tests on the cluster.

The docker folder contains the Docker images built by the CI of this repository:

Folder Images
docker/base Base images: Python, Jupyter, Dask (dask-gateway) for the cluster and local modes.
docker/eopf Dask images of the DPR processors, see Dask images.
docker/jupyter The JupyterHub image, with the rs-client-libraries and one Jupyter kernel per Dask version.
docker/jupyter-cpm A JupyterLab image with the EOPF CPM development tools.

The Python and Dask versions of the Dask clusters are listed in docker/scripts/dask-cluster-versions.yml.

Getting started

Deploy the virtual environment from the bastion, after the core infrastructure:

cd ~/rs-infra-core
ansible-playbook apps.yaml -i inventory/mycluster/hosts.yaml -e '{"package_paths": ["~/rs-workflow-env/apps/"]}'

The parameters are the jupyterhub, prefect3server, prefect3worker, apikeymanager and dask variables of the inventory. After the deployment:

  1. generate a JupyterHub API token and configure it for the staging service and the Dask clusters (see Dask Gateway),
  2. set the concurrency limits of the work pools if needed (see Prefect work pools),
  3. initialise the processing environment.

Several user groups

Each user group (e.g. each organisation) can have its own virtual environment: its own Prefect server and workers, JupyterHub and Dask Gateway, in its own namespace, with its own CPU and RAM resources. Duplicate the applications with different names and namespaces, as described in Isolation between user groups.

Documentation

Repository Relationship
rs-infra-core The core infrastructure and the Ansible playbooks used to deploy this repository.
rs-workflow-deployment The Prefect flows deployed in this environment.
rs-client-libraries The library installed in the JupyterHub image.
rs-dpr-service The service that runs the processors on the Dask clusters.
rs-documentation Online documentation of the whole project.