rs-workflow-deployment¶
Deployment of the Prefect flows of RS-Python and of the automations of the systematic processing.
Overview¶
The flows are developed in rs-client-libraries (rs_workflows
package). To be run on the cluster, a flow must be registered in the Prefect server as a
deployment, attached to a
work pool. This repository is a package of applications deployed
by the apps.yaml playbook of rs-infra-core, after
rs-workflow-env (Prefect server and workers) and
rs-server-deployment (services).
Each application is a Kubernetes Job that runs in the Prefect image, with a ConfigMap:
- the deployment applications (
01-*and02-*) clone rs-client-libraries at a given version (RS_CLIENT_LIBRARIES_REF) and runprefect deploywith a prefect.yaml file, that lists the deployments, their entry points and their work pool. When a flow runs, the worker clones the same version of rs-client-libraries (git_clonepull step). - the automation application (
03-systematic-processing-automations) resolves the identifiers of the target deployments from their names and creates all the Prefect automations of the systematic processing, in a single job. Each automation is a JSON file of itsautomationsdirectory, where the identifier of a deployment is written${deployment_id:<flow name>/<deployment name>}. The genericscripts/create-automations.shscript resolves these placeholders and creates an automation from each JSON file. The JSON files and the script are mounted in the job from ConfigMaps generated by itskustomization.yaml, and they are not templated by Ansible. No automation is created if a target deployment is missing. To add an automation, add its JSON file to theautomationsdirectory and to theconfigMapGeneratorofkustomization.yaml.
| Application | Work pool | Deployments or automation |
|---|---|---|
01-operations-flows |
monitoring-k8s-pool |
Synchronize OSAM Accounts, Update OSAM Account, Collect OBS Logs, Consolidate OBS Logs, Clean old prefect workers |
01-search-flows |
processing-integrated-k8s-pool |
Search Auxip, Search Cadip, Search Catalog, Search Prip |
01-staging-flow |
processing-integrated-k8s-pool |
Stage Generic |
02-benchmarking-flows |
processing-integrated-k8s-pool |
Benchmark Processor |
02-conversion-flows |
processing-integrated-k8s-pool |
Convert ADF, Convert ADF Group, Convert ADF Scheduled, Convert SAFE |
02-on-demand-processing-flows |
processing-integrated-k8s-pool |
Process Generic, Process L0, Process S1 L0, Process S1 ARD Reference, Process S1 ARD, Process S3 L0, Process S3 L1 OLCI, Process S3 L1 OLCI Systematic, Process S3 L2 OLCI, Process S3 L2 OLCI Systematic |
02-on-demand-staging-flows |
processing-integrated-k8s-pool |
Stage Aux CQL2, Stage Aux, Stage Cadip, Stage Prip |
02-olci-quicklook-flows |
processing-integrated-k8s-pool |
S3L1 OLCI quicklook, S3L2 OLCI quicklook, see its README |
03-systematic-processing-automations |
CADIP staging completed -> L0 (runs Process L0, that runs the Level-0 flow of the mission of the session), S03OLCL0_ generated -> S3 L1 OLCI, S03OLCEFR generated -> S3 L2 OLCI, S3 L1 OLCI quicklook on quicklook-inputs-ready, S3 L2 OLCI quicklook on quicklook-inputs-ready |
The flows and their parameters are described in Prefect flows, and the automations in Systematic processing.
Getting started¶
Deploy the applications from the bastion:
cd ~
git clone https://github.com/RS-PYTHON/rs-workflow-deployment.git
cd ~/rs-infra-core
ansible-playbook apps.yaml -i inventory/mycluster/hosts.yaml -e '{"package_paths": ["~/rs-workflow-deployment/apps/"]}'
The deployments must exist before the automations that target them: the NN- prefixes give the deployment order.
The automation job creates new automations each time it is run: delete the existing automations in the Prefect web
interface before running it again, otherwise the flows are triggered twice.
To deploy a new version of the flows, change the version of rs-client-libraries in the configmap.yaml (branch of
the git_clone step) and job.yaml (RS_CLIENT_LIBRARIES_REF) files of the applications, then run the playbook
again. To add your own flows, see
Write a custom flow.
Documentation¶
Related repositories¶
- rs-client-libraries: source code of the flows.
- rs-workflow-env: Prefect server, workers and work pools.
- rs-infra-core: Kubernetes infrastructure and deployment playbooks.