Skip to content

rs_dpr_service/s1_slc_s1ard_format.md

<< Back to index

Convert the Sentinel-1 TOPSAR SLC products read by CPM 3.0.0 into the format expected by the S1 ARD processor.

The S1 ARD processor (s1-ard 1.2.0) runs with CPM 3.0.0rc4 and validates its SLC inputs against the Zarr layout written by CPM 2.x: one sub-product per burst AND per polarization, flat other_metadata, sar:polarization, a processing_history indexed by product level, and line/pixel coordinates.

Since CPM 3.0.0, the SAFE conversion joins both polarizations of each burst along a polarization dimension (CPM issue #1042: https://gitlab.eopf.copernicus.eu/cpm/eopf-cpm/-/issues/1042), splits other_metadata by polarization and drops several annotation attributes. Its processing history parser is also broken (only one "Unknown" entry). So S1 ARD rejects the SLC products converted by CPM 3.0.0.

CPM 3.0.0 also mixes up the polarizations of the data (e.g. the 'VV' label contains the VH pixels and noise LUTs): the real polarization of the measurements is found from their pixels, and the noise LUTs are read from the XML files.

This module rebuilds the S1 ARD layout from the DataTree read by CPM 3.0.0 (before it is written, lazily, without copying the measurement data) and from the XML and TIFF files of the SAFE product. The result is NOT compatible with the CPM 3.0.0 SLC model: this is written in its STAC metadata.

This module runs in the EOPF Dask worker (see safe_to_zarr.py), where eopf, lxml and xarray are installed.

SafeAnnotations

XML annotations of a Sentinel-1 TOPSAR SLC SAFE product, read once for all the bursts.

Source code in docs/rs-dpr-service/rs_dpr_service/s1_slc_s1ard_format.py
class SafeAnnotations:
    """XML annotations of a Sentinel-1 TOPSAR SLC SAFE product, read once for all the bursts."""

    def __init__(self, safe_path: str):
        # pylint: disable=import-outside-toplevel
        from eopf.store.product_specific.S01 import parse_annotations

        self.safe_path = safe_path
        self.history = parse_processing_history(os.path.join(safe_path, "manifest.safe"))
        # Annotation metadata, before CPM 3.0.0 deletes its "deprecated" attributes
        self.metadata = {}
        for path, metadata in parse_annotations(safe_path).items():
            if match := _XML_FILE_PATTERN.search(os.path.basename(path)):
                self.metadata[(match[1].upper(), match[2].upper())] = metadata
        self.annotations = _xml_files(safe_path, "annotation/s1*.xml")
        self.calibrations = _xml_files(safe_path, "annotation/calibration/calibration-*.xml")
        self.noises = _xml_files(safe_path, "annotation/calibration/noise-*.xml")
        self.rfis = _xml_files(safe_path, "annotation/rfi/rfi-*.xml")
        self.measurements = _xml_files(safe_path, "measurement/s1*.tiff")

build_other_metadata(annotation, cpm_metadata, burst_id, image_number=None)

Rebuild the CPM 2.x other_metadata of one burst and polarization.

Parameters:

Name Type Description Default
annotation dict[str, Any]

metadata of the annotation XML file, from eopf parse_annotations()

required
cpm_metadata dict[str, Any]

other_metadata[] written by CPM 3.0.0

required
burst_id str

burst identifier

required
image_number str | None

image number of the annotation XML file. CPM 3.0.0 writes the same image number for all the polarizations, so it is read from the XML file.

None
Source code in docs/rs-dpr-service/rs_dpr_service/s1_slc_s1ard_format.py
def build_other_metadata(
    annotation: dict[str, Any],
    cpm_metadata: dict[str, Any],
    burst_id: str,
    image_number: str | None = None,
) -> dict[str, Any]:
    """
    Rebuild the CPM 2.x other_metadata of one burst and polarization.

    Parameters:
        annotation: metadata of the annotation XML file, from eopf parse_annotations()
        cpm_metadata: other_metadata[<polarization>] written by CPM 3.0.0
        burst_id: burst identifier
        image_number: image number of the annotation XML file. CPM 3.0.0 writes the same image number for all the
            polarizations, so it is read from the XML file.
    """
    annotation = _rename_keys(copy.deepcopy(annotation), {"t0": "t_0"})
    product_information = annotation["general_annotation"]["product_information"]
    processing = annotation["image_annotation"]["processing_information"]
    swath_params = {key: value for key, value in processing["swath_proc_params_list"][0].items() if key != "swath"}
    input_dimensions = {key: value for key, value in processing["input_dimensions_list"][0].items() if key != "swath"}
    metadata: dict[str, Any] = {
        key: cpm_metadata[key] for key in ("eopf_category", "title", "image_number") if key in cpm_metadata
    }
    if image_number is not None:
        metadata["image_number"] = image_number
    metadata.update(
        {
            "timeliness_category": product_information["timeliness_category"],
            "projection": product_information["projection"],
            "range_sampling_rate": product_information["range_sampling_rate"],
            "processing_information": {**swath_params, "input_dimensions": input_dimensions},
            "quality_information": annotation["quality_information"],
            "general_annotation": annotation["general_annotation"],
            "image_annotation": annotation["image_annotation"],
            "swath_timing": annotation["burst_timing"],
            "swath_merging": annotation["swath_merging"],
            "burst_id": burst_id,
        },
    )
    return metadata

is_s1_topsar_slc_safe(safe_uri)

Return True if the SAFE product is a Sentinel-1 IW or EW SLC product.

Source code in docs/rs-dpr-service/rs_dpr_service/s1_slc_s1ard_format.py
def is_s1_topsar_slc_safe(safe_uri: str) -> bool:
    """Return True if the SAFE product is a Sentinel-1 IW or EW SLC product."""
    return bool(S1_TOPSAR_SLC_SAFE_PATTERN.match(os.path.basename(str(safe_uri).rstrip("/"))))

parse_processing_history(manifest_path)

Read the processing history of a Sentinel-1 SAFE manifest, indexed by product level.

This implements what the CPM 3.0.0 ToProcessingHistoryS01 formatter is meant to do: the CPM version tests XML attributes with "name" in node (which looks for child elements in lxml), so it never descends into the processing tree and returns a single "Unknown" entry dated 1970. S1 ARD reads processing_history["Level-1 Intermediate SLC Product"][0].

Source code in docs/rs-dpr-service/rs_dpr_service/s1_slc_s1ard_format.py
def parse_processing_history(manifest_path: str) -> dict[str, list[dict[str, Any]]]:
    """
    Read the processing history of a Sentinel-1 SAFE manifest, indexed by product level.

    This implements what the CPM 3.0.0 ToProcessingHistoryS01 formatter is meant to do: the CPM version
    tests XML attributes with `"name" in node` (which looks for child elements in lxml), so it never
    descends into the processing tree and returns a single "Unknown" entry dated 1970.
    S1 ARD reads processing_history["Level-1 Intermediate SLC Product"][0].
    """
    root = etree.parse(manifest_path).getroot()  # nosec B320 (local SAFE manifest of the conversion input)
    nodes = root.xpath(
        "metadataSection/metadataObject[@ID='processing']/metadataWrap/xmlData/safe:processing",
        namespaces={"safe": _SAFE_NAMESPACE},
    )
    history: dict[str, list[dict[str, Any]]] = {}

    def walk(node: Any, is_product: bool):
        if _local_name(node) == "processing":
            entry = _history_entry(node)
            history.setdefault(_history_level(node, entry, is_product), []).append(entry)
            is_product = False
        for child in node:
            # The processing history of the auxiliary data files is not of interest
            if isinstance(child.tag, str) and _AUX_MARKER not in child.attrib.get("name", _AUX_MARKER):
                walk(child, is_product)

    for node in nodes:
        walk(node, True)

    # Like CPM 2.x: entries sorted by time, and levels sorted by their first entry. EOPF convert() then adds the
    # SAFE product name and the CPM entry to the latest entry.
    for entries in history.values():
        entries.sort(key=lambda entry: entry["time"])
    return dict(sorted(history.items(), key=lambda item: item[1][0]["time"]))

to_s1ard_format(product, safe_path)

Convert a Sentinel-1 TOPSAR SLC product read by CPM 3.0.0 into the format expected by S1 ARD 1.2.0.

Parameters:

Name Type Description Default
product DataTree

product read from the SAFE by eopf.store.open_datatree (CPM 3.0.0)

required
safe_path str

local path of the SAFE product, to read its XML files

required

Returns:

Type Description
DataTree

A new DataTree, with one burst per polarization. The measurement data stays lazy.

Source code in docs/rs-dpr-service/rs_dpr_service/s1_slc_s1ard_format.py
def to_s1ard_format(product: xr.DataTree, safe_path: str) -> xr.DataTree:
    """
    Convert a Sentinel-1 TOPSAR SLC product read by CPM 3.0.0 into the format expected by S1 ARD 1.2.0.

    Parameters:
        product: product read from the SAFE by eopf.store.open_datatree (CPM 3.0.0)
        safe_path: local path of the SAFE product, to read its XML files

    Returns:
        A new DataTree, with one burst per polarization. The measurement data stays lazy.
    """
    annotations = SafeAnnotations(safe_path)
    result = xr.DataTree(dataset=product.to_dataset(inherit=False), name=product.name)
    result.attrs = copy.deepcopy(product.attrs)
    result.attrs.setdefault("stac_discovery", {}).setdefault("properties", {}).update(S1ARD_FORMAT_PROPERTIES)
    for name, burst in product.children.items():
        for burst_name, node in _split_burst(name, burst, annotations).items():
            result[burst_name] = node
    return result