Skip to content

rs-server-deployment

Deployment of the services of RS-Python on a Kubernetes cluster: rs-server services, rs-dpr-service, STAC browsers, databases and station mockups.

Overview

This repository is a package of applications deployed on the cluster created by rs-infra-core, with its apps.yaml playbook, after rs-workflow-env. Each folder of apps is an application, made of a Kustomize file that references a Helm chart of rs-helm and Kubernetes manifests, templated with the variables of the inventory. The NN- prefixes give the deployment order.

Applications Description
01-cloudnative-pgstac The catalog database: a PostgreSQL cluster managed by CloudNativePG, with the pgSTAC schema, its scheduled backups in the object storage, its metrics and its Grafana data source.
01-pi-db, 01-pygeoapi-db, 01-s3quotamonitoring-db The databases of the performance indicators, of the jobs of the staging service (pygeoapi) and of the object storage quota monitoring, with their Grafana data sources.
01-dask-cluster-staging A launcher that keeps a Dask cluster running in Dask Gateway, for the staging service.
01-rs-catalog-staging-configmap The bucket configuration of the catalog and of the staging: the bucket and the expiration delay of the products, per owner, collection and product type, see Object storage buckets and data lifecycle.
01-rs-server-station-secrets The URLs and credentials of the external stations (CADIP, AUXIP, PRIP, LTA...), see External data sources.
02-rs-server-osam The Object Storage Access Manager.
rs-server-adgs, rs-server-cadip, rs-server-prip The services that give a STAC access to the external stations.
rs-server-catalog The catalog service.
rs-server-staging The staging service.
rs-server-frontend The frontend: home page and aggregated OpenAPI documentation of the services.
rs-dpr-service-py-<python>-dask-<dask> The instances of rs-dpr-service, one per Python and Dask version used by the processors, see rs-dpr-service deployment.
stac-browser-auxip, stac-browser-cadip, stac-browser-catalog, stac-browser-prip The STAC Browser web interfaces of the stations and of the catalog, see Browse products.
mockup-* The station mockups of rs-testmeans, for the test platforms only.

All the services are deployed in the processing namespace, and the databases in the database namespace.

Catalog database initialization

At each deployment of 01-cloudnative-pgstac, the init-pgstac-db job (job-init-pgstac.yaml) initializes or updates the catalog database. All its SQL can be run again on an existing database:

  1. pypgstac migrate creates the pgSTAC schema, or migrates it to the version of the job image.
  2. The catalog user gets the pgstac_read, pgstac_ingest and pgstac_admin roles, and the pgstac, public search path.
  3. The script of configmap-extensions.yaml customizes pgSTAC for rs-server-catalog: - it enables the context setting of pgSTAC (number of matching items in the search results); - it registers in pgstac.stac_extensions the STAC extensions of the inventory variable rs_server_catalog.extensions, with their JSON schemas, downloaded by an init container; - it registers in pgstac.queryables the queryables of the inventory variable rs_server_catalog.queryables, and sets the definition of the expires queryable (a UTC date-time); - it sets up the search by externalIds (see Search by external identifiers): the externalIds queryable, the external_ids_contains CQL2 operator in pgstac.cql2_ops, whose SQL template is a jsonb containment on properties.externalIds, and the GIN indexes on this expression that serve it, items_external_ids_idx on pgstac.items (so on all its partitions) and queryable_index_template_external_ids_idx on pgstac.queryable_index_template (so that the pgSTAC maintenance never drops it). It also removes the external_ids_tokens function used before pgSTAC 0.10.

The local mode of rs-demo sets up the search by externalIds with the same SQL, and the tests of rs-server-catalog with SQL_CREATE_EXTERNAL_IDS_OPERATOR and SQL_CREATE_EXTERNAL_IDS_INDEXES of external_ids.py: keep them aligned.

Getting started

Deploy the applications from the bastion, after the infrastructure installation and the deployment of rs-workflow-env:

cd ~
git clone https://github.com/RS-PYTHON/rs-server-deployment.git
cd ~/rs-infra-core
ansible-playbook apps.yaml -i inventory/mycluster/hosts.yaml -e '{"package_paths": ["~/rs-server-deployment/apps/"]}'

# Deploy (or update) a single application, e.g. the catalog service
ansible-playbook apps.yaml -i inventory/mycluster/hosts.yaml -e '{"package_paths": ["~/rs-server-deployment/apps/"]}' -e app=rs-server-catalog

On a production platform, don't deploy the mockup-* applications: remove them from your copy of the package. The parameters of the applications (rspy, rs_server_station_secrets, rs_station_mockup... sections: image versions, buckets, stations, resources) are in the host_vars/setup/apps.yml file of the inventory.

Documentation