Common flows¶
This page describes the flows that are not specific to a mission: search, staging, conversion, operation and benchmarking. The processing flows are described in Level-0 flows, the mission pages and Customise the DPR processing. The source code of the flows is in rs_workflows.
The env parameter of the flows is described in Prefect flows.
Search flows¶
These flows search products and return a STAC ItemCollection. They are mainly used as sub-flows, or to test a
filter before running a staging or a processing.
| Flow | Parameters | Description |
|---|---|---|
search-cadip |
cadip_collection_identifier, session_identifier, error_if_empty |
Search a session in a CADIP collection (e.g. s1_ins for the Sentinel-1 sessions of a station). |
search-aux |
aux_cql2, source, error_if_empty |
Search auxiliary files with a CQL2 filter, in a source: auxip (default), catalog, prip, cadip, cdse or earthdatahub. |
search-prip |
prip_cql2, prip_collection, error_if_empty |
Search products in the PRIP stations with a CQL2 filter. |
search-catalog |
catalog_cql2, collections, error_if_empty |
Search items in the catalog with a CQL2 filter. |
search-earthdatahub |
edh_cql2, error_if_empty |
Search products in the EarthDataHub of Destination Earth. |
The CQL2 filters are given in JSON, e.g.:
{"op": "and", "args": [
{"op": "=", "args": [{"property": "product:type"}, "AUX_PP1"]},
{"op": "t_intersects", "args": [{"property": "datetime"}, {"interval": ["2026-08-01T00:00:00Z", "2026-08-02T00:00:00Z"]}]}
]}
Staging flows¶
These flows copy products from the external data sources into the catalog, with the staging service.
| Flow | Parameters | Description |
|---|---|---|
stage-generic |
stac_input, catalog_collection_identifier, asset_names |
Stage any STAC items: a Feature or FeatureCollection (e.g. the result of a search flow), or a URL that returns a FeatureCollection. |
stage-cadip-with-options |
cadip_collection_identifier, session_identifier, catalog_collection_identifier, staging_retries, staging_retry_delay |
Search a CADIP session and stage it, with retries. |
stage-cadip-latest |
cadip_collection, owner_identifier |
Stage the latest session of a CADIP station (searched over the last 10 hours). |
stage-cadip-selection |
cadip_collection, owner_identifier |
Search the sessions of the last 10 hours of a CADIP station, ask the user to select one, and stage it. |
stage-aux |
start_datetime, end_datetime, product_type, catalog_collection_identifier |
Stage the auxiliary files of a type that cover a time interval (ValCover policy). |
stage-aux-cql2 |
cql2_filter, catalog_collection_identifier, timeout_seconds, source, selected_assets |
Search auxiliary files with a CQL2 filter in a source (see search-aux) and stage the ones that are not already in the catalog. Used by the processing flows to stage the ADFs. |
stage-prip |
start_datetime, end_datetime, product_type, prip_collection, catalog_collection_identifier |
Stage the PRIP products of a type in a time interval. |
The items already in the catalog are not staged again.
Conversion flows¶
These flows convert legacy products and auxiliary files into the Zarr format used by the EOPF processors.
| Flow | Parameters | Description |
|---|---|---|
convert-adf |
adf_input: ADF type, time interval, satellite or CQL2 filter, collections mapping |
Stage the legacy auxiliary files needed to build an ADF of the given type, convert them to Zarr, store the result in the object storage and publish it in the catalog. |
convert-adf-group |
period_start_datetime, period_end_datetime, configuration, owner |
Convert a set of ADF types over a period: the past part of the period is converted immediately, the future part is scheduled with convert-adf-scheduled. A sample configuration is given in convert-aux-sample.json. |
convert-adf-scheduled |
adf_type, cql2_filter_without_date, period, auxiliary_product_to_collection_identifier |
Convert the ADFs of the last period, started by a schedule. |
import-adf-from-obs |
configuration, owner, obs_id, rehearsal_mode |
Import ADF files already in Zarr format from an object storage bucket into the catalog. |
convert-safe |
conversion_input |
Stage a legacy SAFE product, convert it to its EOPF Zarr counterpart with the conv_safe_zarr process of rs-dpr-service, and publish it. If stac_input is already an item of the RS-Python catalog (its href, or a feature collection whose item has a self link to it, with the hostname of the public or cluster-internal URL of the catalog), the SAFE product is read from the catalog instead of being staged again, and its item is kept after the conversion. |
Sentinel-1 calibration ADF¶
Use adf_type: S01__ADF_CALIB to request legacy AUX_CAL products from ADGS
(source: auxip) and produce ADF_CALIB Zarr products for Sentinel-1 L0 processing.
For example, pass these fields in AdfProcessIn, together with the usual env
and search time interval (or a custom cql2_filter):
{
"adf_type": "S01__ADF_CALIB",
"auxiliary_product_to_collection_identifier": [
{"product_type": "AUX_CAL", "collection_name": "S1_AUX", "source": "auxip"},
{"product_type": "ADF_CALIB", "collection_name": "S1_ADF"}
]
}
Staging, archive extraction, upload and catalog publication use the existing flow.
Conversion itself runs locally, without the upstream S3-to-S3 CLI: the adapter
in rs_workflows/adf_conversion/S01__ADF_CALIB.py calls save_aux_cal_xml_to_zarr
from the vendored aux_file_conversion.py. Runtime dependencies are dpath,
NumPy, xarray (with DataTree) and Zarr, installed with this library.
The helper is copied unchanged from
S1 L1 ADF converter,
revision e5dae104384562dcd662e7cf749d45b286ed9e87 (Apache-2.0, Copyright 2026 B-Open).
It preserves the upstream calibration groups, complex elevation antenna patterns,
angular coordinates and calibration constants. Output names follow the upstream
convention, e.g. S01A_ADF_CALIB_V20190228T092500_G20240327T102320.zarr.
The flow supplies staged STAC items to the adapter through the temporary
WORK/aux_cal_items.json file. The adapter matches each extracted SAFE to its
staged item ID or asset filename, preserves the ADGS start_datetime and
end_datetime, and adds root metadata for catalog publication. Missing or
ambiguous metadata and unsupported legacy calibration data fail explicitly;
no validity end date is invented. For a standalone local run, supply that same
metadata file, set ADF_OUTPUT, and invoke the adapter with the input directory
and --working_dir.
Operation flows¶
These flows are run by the operators, from the monitoring-k8s-pool work pool. They run on behalf of technical
users (operator-osam, operator-quota), whose Prefect blocks must be initialised.
| Flow | Parameters | Description |
|---|---|---|
synchronize-osam-accounts |
Synchronise the object storage accounts with the Keycloak accounts, see OSAM. | |
update-osam-account |
user_name |
Apply the object storage rights of a user. Publishes the obs-rights artifact. |
initialize-obs-db |
Create the tables of the object storage quota monitoring database. | |
collect-obs-logs |
platform, max_files, threshold_minute |
Collect the object storage access logs, see Object storage quota monitoring. |
consolidate-obs-logs |
retention_hours |
Consolidate the old access logs per user, bucket and hour. |
initialize-pi-db |
Create the tables of the performance indicators database. | |
cleanup-offline-workers |
work_pools_csv, max_age_days |
Delete the Prefect workers that are offline for more than max_age_days. Scheduled every day. |
Benchmarking¶
The benchmark-processor flow (processor_name, processor_version, scenario_name) reads its settings from the
benchmarking-<processor_name>-settings Prefect variable and publishes the benchmarking-result artifact. Its
implementation is a mockup.