Skip to content

Running Modes

In this page, we will see how to run the Jupyter notebooks on cluster, local and hybrid mode.

Run on cluster mode

On cluster mode, we run the Jupyter notebooks from our JupyterHub session deployed on the cluster. They connect to the services deployed on the RS-Server website (=cluster). Authentication is required for this mode.

Prerequisites

Run the demos on cluster mode

  • Open a JupyterHub session.

  • Open a terminal, check that rs-client-libraries is installed by running:

    pip show rs-client-libraries # should show the name, version, ...

  • Optional: save your API key into your ~/.env file so it is loaded automatically by your notebooks:

    1. Go to the menu View -> Show Hidden Files
    2. In the file browser in the left panel, check that you are in your / home folder.
    3. Double-click the .env file to open it using the integrated text editor.
    4. Add this line to the file, save it with Ctrl-S and close the editor:
    # Replace by your value
export RSPY_APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

  • On the left, in the file explorer, go to the demos or tutorial folder and double-click a notebook to open it:

Jupyter Notebook

Install a new rs-client-libraries version

Option 1: update the JupyterHub image (affects everyone)

  1. Vérify in the CI/CD that the last rs-client-libraries modifications were merged into the develop branch: https://github.com/RS-PYTHON/rs-client-libraries/actions/workflows/publish-binaries.yml

  2. Ask the rs-infrastructure administrator to run a new CI/CD workflow to publish this rs-client-libraries version into a new JupyterHub image.

Option 2: from a wheel package (affects only you)

  1. In the CI/CD, click on the last rs-client-libraries branch workflow that you want to use: https://github.com/RS-PYTHON/rs-client-libraries/actions/workflows/publish-binaries.yml, go to the Artifacts section and download the .whl package file.

  2. Or alternatively, build the wheel yourself from your local rs-client-libraries project by running: poetry build --format wheel

  3. Upload the .whl package file to your JupyterHub session, open a Terminal and run:

    # Uninstall the old version. Note: this fails if we do it for the first time because 
# we try to uninstall the root installation of the library, but this this OK.
pip uninstall -y rs-client-libraries 2>/dev/null

# You may have conflicts between dependencies installed for the root user
# and the current user. You can uninstall all current user dependencies with:
# for dep in $(pip freeze | cut -d "@" -f1); do pip uninstall -y $dep 2>/dev/null; done

# The old rs-client-libraries version is still installed for the root user.
# This is the version you (=current user) use by default.
pip show rs-client-libraries | grep Location # should display: /opt/conda/lib/python3.11/site-packages

# Install the new version for the current user.
pip install rs_client_libraries-<version>-py3-none-any.whl

Run on local mode

On local mode, docker-compose and Docker images are used to run services and libraries locally (not on a cluster). There is no authentication for this mode.

Prerequisites

Run the demos on local mode

To pull the latest Docker images, run:

# Login into the project ghcr.io (GitHub Container Registry)
# Username: your GitHub login
# Password: your personal access token (PAT) created above
docker login https://ghcr.io/v2/rs-python

# From the local-mode directory, pull the images
cd ./local-mode
docker compose pull

Then to run the demos:

# Still from the local-mode directory, if you're not there yet
cd ./local-mode

# Run all services.
# Note: in case of port conflicts, you can kill all your running docker containers with:
# docker rm -f $(docker ps -aq)
docker compose down -v; docker compose up # -d for detached

# Note: we always need to call 'down' before 'up' or we'll have errors
# when the stac database will initialize a second time.

Near the end of the logs you will see some Jupyter information e.g: 

jupyter | To access the server, open this file in a browser:
jupyter |     ...
jupyter | Or copy and paste one of these URLs:
jupyter |     ...
jupyter |     http://127.0.0.1:8888/lab?token=612cb124335d9ab80a5a6414631a7df186b2401234050001

Open (ctrl-click) the http://127.0.0.1:8888/lab?token=... link to open the Jupyter web client (=Jupyter Notebook) in your browser.

Note: the token is auto-generated by Jupyter and changes everytime you relaunch the containers. So after relaunching, your old Jupyter web session won't be available anymore.

To show the Jupyter logs from another terminal, run:

docker compose logs jupyter

On the left, in the file explorer, go to the demos or tutorial folder and double-click a notebook to open it:

Jupyter Notebook

# When you're done, shutdown all services and volumes (-v)
# with Ctrl-C (if not in detached mode i.e. -d) then:
docker compose down -v

# You can use this to remove all docker volumes 
# (use with care if you have other docker containers)
docker volume prune

How does it work

The docker-compose.yml file uses Docker images to run all the necessary container services for the demos :

These containers are run locally (not on a cluster). The Jupyter notebooks accessed from http://127.0.0.1:8888 are run from the containerized Jupyter server, not from your local environment. This Jupyter environment contains all the Python modules required to call the rs-server HTTP endpoints.

How to run your local rs-server code in this environment

It can be helpful to use your last rs-server code version to debug it or to test modifications without pushing them and rebuilding the Docker image. Follow these steps:

  1. Go to the local-mode directory and run: 

    cp 'docker-compose.yml' 'docker-compose-debug.yml'

  2. If your local rs-server github repository is under /my/local/rs-server, modify the docker-compose-debug.yml file to mount your local rs-server services: 

    # e.g.
rs-server-adgs:
  # ...
  volumes:
    - /my/local/rs-server/services/common/rs_server_common:/usr/local/lib/python3.11/site-packages/rs_server_common
    - /my/local/rs-server/services/adgs/rs_server_adgs:/usr/local/lib/python3.11/site-packages/rs_server_adgs
    - /my/local/rs-server/services/adgs/config:/usr/local/lib/python3.11/site-packages/config
    # - and any other useful files ...

  3. Run the demo with: 

    # Still from the local-mode directory, if you're not there yet
cd ./local-mode

# Run all services
docker compose down -v; docker compose -f docker-compose-debug.yml up # -d for detached

Run on hybrid mode

On hybrid mode, we run the Jupyter notebooks locally, but they connect to the services deployed on the RS-Server website (=cluster). Authentication is required for this mode.

Prerequisites

  • You have access to the RS-Server website: https://rspy.ops.rs-python.eu/docs
  • You have generated an API key from the RS-Server website.
  • You have saved the S3 bucket configuration in you local file: ~/.s3cfg
  • Python is installed on your system.

You also need the rs-client-libraries project:

  • If you have its source code, install it with:

    cd /path/to/rs-client-libraries
pip install poetry
poetry install --with dev,demo
poetry run opentelemetry-bootstrap -a install

  • Or if you only have its .whl package, install it with: 

    pip install rs_client_libraries-*.whl
# then install jupyter lab
pip install jupyterlab

Run the demos on hybrid mode

From your terminal in the rs-demo, run:

export RSPY_APIKEY=your_api_key # see the prerequisites

# NOTE: at CS France premises, use this to deactivate the proxy which causes random errors
unset no_proxy ftp_proxy https_proxy http_proxy

# To use your local rs-client-libraries source code
cd /path/to/rs-client-libraries
# git checkout develop && git pull # maybe take the latest default branch
poetry run /path/to/rs-demo/hybrid-mode/start-jupyterlab.sh

# Or if you have installed it from rs_client_libraries-*.whl, 
# just run
/path/to/rs-demo/hybrid-mode/start-jupyterlab.sh

The Jupyter web client (=Jupyter Notebook) opens in a new tab of your browser.

WARNING: the cluster is shut down from 18h30 to 8h00 each night and on the weekends.

How to check your Python interpreter used in notebooks

In a notebook cell, run: 

import sys
print(sys.executable)

If you use the rs-client-libraries poetry environment, it should show something like: 

${HOME}/.cache/pypoetry/virtualenvs/rs-client-libraries-xxxxxxxx-py3.11/bin/python