Output metadata#

Note

This page is a first draft.

The output dataset describes its HEALPix grid with the conventions selected in the metadata settings (all of them by default), and keeps the STAC metadata of the input datasets.

Selecting the conventions#

settings = hc.ConvertSettings.from_dict(
    {
        "metadata": {
            "zarr_conventions": [
                "dggs",
                "multiscales",
                {
                    "name": "cf",
                    "version": "1.13",
                    "configuration": {"healpix_grid_mapping": True},
                },
            ]
        },
        "group_settings": {...},
    }
)

Each item of zarr_conventions is a convention name or a mapping with a name and, optionally, a version and a configuration. The conventions left out are not written.

DGGS Zarr convention#

Each HEALPix group lists the DGGS Zarr convention in its zarr_conventions attribute, and describes its grid in a dggs attribute:

{
    "name": "healpix",
    "refinement_level": 7,
    "indexing_scheme": "nested",
    "ellipsoid": {
        "name": "wgs84",
        "semi_major_axis": 6378137.0,
        "inverse_flattening": 298.257223563,
    },
    "spatial_dimension": "cells",
    "coordinate": "cell_ids",
    "compression": "none",
}

Multiscales Zarr convention#

A multiscale group (see multiscale_settings in Conversion settings) lists the Multiscales Zarr convention and the DGGS Zarr convention in its zarr_conventions attribute. Its multiscales attribute lists its groups, one per refinement level (layout), with their HEALPix grid if add_healpix_positioning is set (it is not by default: the attributes of a level group describe its own grid). As the convention asks, the asset of a layout item is a path relative to the group holding the metadata, e.g. "7" for a converted level and "pyramids/4" for a level derived in a child group (see HEALPix pyramids).

The layout item of a derived level also names the converted level it is aggregated from (derived_from, a path like asset), the transformation from that level (transform) and how its cells are aggregated (resampling_method, also given once for the whole group since every derived level of a pyramid uses the same method):

{
  "layout": [
    { "asset": "7" },
    {
      "asset": "6",
      "derived_from": "7",
      "transform": { "scale": [2.0] },
      "resampling_method": "mean"
    },
    {
      "asset": "4",
      "derived_from": "7",
      "transform": { "scale": [8.0] },
      "resampling_method": "mean"
    }
  ],
  "resampling_method": "mean"
}

The convention gives scale per axis, and a HEALPix group has a single one, the cell dimension: the factor is the length of a cell relative to a cell of the source level, [2.0] one level coarser and [8.0] three levels coarser. Since HEALPix tessellates a 2D surface, a cell aggregates the square of that, 4 and 64 cells of the source level.

A derived level is aggregated from a converted level and never from another derived level, so its scale grows with the distance to that level.

CF conventions#

With the CF conventions (version 1.13 or newer, the first one defining the HEALPix grid mapping):

the root group has a Conventions attribute (e.g., "CF-1.13");
if healpix_grid_mapping is set (the default), each HEALPix group has a crs grid mapping variable (grid_mapping_name = "healpix"), referenced by the grid_mapping attribute of the data variables, and the cell ids coordinate has standard_name = "healpix_index".

Variable attributes#

The attributes of the input variables (units, long_name, etc.) are copied to the resampled variables, except the ones matching the exclude_attrs patterns of the group’s metadata settings, and the ones set by its override_attrs patterns (see Conversion settings).

STAC metadata#

The root group of the output dataset has a stac_discovery attribute: a STAC item merged from the stac_discovery attributes of the input datasets, with

the geometry and bounding box of the output dataset, and an id derived from the output path;
derived_from links to the input datasets;
provenance properties following the STAC processing extension: the date of the conversion (processing:datetime), the versions of healpix-convert and healpix-resample (processing:software), the conversion settings (processing:expression, whose format is the URL of the settings JSON schema of the recorded healpix-convert version) and the inputs (processing:lineage).

Set metadata.stac.include_conversion_settings to False to leave processing:expression out (not recommended).

Merging the input items#

With several input datasets, the properties, extensions, links and assets of the output item are merged from theirs, with the strategy chosen per kind by merge_properties, merge_extensions, merge_links and merge_assets:

Strategy	Kept	Available for
`"pick_first"`	what the first input dataset has (the default)	all four
`"join_inner"`	what every input dataset has: their intersection	all four
`"join_outer"`	what any input dataset has: their union	extensions, links

Properties and assets are merged by name, and "join_inner" keeps an entry only when every input has it with the same value — a datetime that differs across the inputs does not describe the output either. They have no "join_outer": a union of mappings has to invent a rule for the keys whose value differs across the inputs, and any such rule describes the output less well than leaving the entry out.

Extensions and links are merged whole — two of them are the same one when they are equal, which is all a link has to go on — so "join_outer" is meaningful there, and lists them in the order they first appear.

With a single input dataset the strategies are all equivalent. The other fields of the item are unaffected: the geometry, the bounding box and the id describe the output, and stac_version must match across the inputs or the conversion fails.

Excluding and setting STAC properties, extensions, links and assets#

Once merged, metadata.stac drops and sets what the output item propagates from the input items:

"metadata": {
    "stac": {
        "exclude_properties": ["internal:.*"],
        "override_properties": {"license": "CC-BY-4.0"},
        "exclude_extensions": ["https://stac-extensions.github.io/projection/.*"],
        "override_extensions": ["https://example.org/ext/v1.0.0/schema.json"],
        "exclude_links": ["self|parent"],
        "override_links": [
            {"rel": "canonical", "href": "https://example.org/out.zarr"}
        ],
        "override_assets": {"data": {"href": "s3://public/out.zarr"}},
    }
}

The four exclude_* settings list exact names or regular expressions matched against the whole name. What that name is depends on the kind:

Setting	Matched against	Override
`exclude_properties`	the property name	`override_properties`: name → value
`exclude_extensions`	the extension schema URL	`override_extensions`: list of URLs
`exclude_links`	the link’s `rel`	`override_links`: list of link objects
`exclude_assets`	the asset key	`override_assets`: key → asset object

Links have no identifier, so rel is what they are matched on; a link without one matches the empty string. An override wins over an exclusion, and the override_* of a list (extensions, links) appends what is not there yet rather than replacing.

exclude_assets is the one setting that is not empty by default: it is [".*"], so the input’s assets are not propagated. They point at the input files, which the output does not hold, and an output item advertising files that are not its own is worse than one advertising none. Set it to [] to keep them, or describe the output’s own files with override_assets.

These settings apply to the propagated metadata only. The STAC metadata written by healpix-convert itself keeps precedence: the processing:* properties and the processing extension above are written after the settings are applied, so the output stays traceable whatever they say. The one exception is include_conversion_settings, which decides whether the settings themselves are recorded.

Note

The output item is not validated against the extensions it lists: keeping the properties and the extensions consistent with each other is left to the data provider.

Consolidated metadata#

Consolidating the metadata of a Zarr group copies the metadata of all its descendants into its own metadata document, so that a reader opens the whole hierarchy below it with a single read instead of one read per node (a lot faster on an object store).

finalize_healpix_dataset(), the last stage of the conversion, writes it, and no other stage does: such a copy is stale as soon as a group or an array is written next to it. Converting the data does not make it stale, though, since it only writes array chunks.

The metadata.consolidate settings select the groups it is written for:

"metadata": {
    "consolidate": {
        "root_group": True,
        "healpix_groups": True,
        "multiscale_groups": True,
        "derived_level_groups": True,
        "groups": ["provenance"],
    }
}

Setting	Consolidates the metadata of
`root_group`	the root group, i.e. the whole output dataset
`healpix_groups`	each group converted to HEALPix
`multiscale_groups`	each multiscale group, i.e. all the levels it holds
`derived_level_groups`	each group holding a derived level of a pyramid
`groups`	the given paths, in the output dataset

The four flags are set by default, so every group a reader is expected to open on its own is consolidated: the consolidated metadata of the root group can describe much more than what such a reader needs. groups is empty by default, and its paths are those of the output dataset, not of the input dataset(s), the root group being "/". A path that names no group of the output dataset is reported with a warning when the staging cache is created (its structure is known there, before anything is converted) and skipped when the metadata is consolidated.

Note

Consolidated metadata is not part of the Zarr v3 specification (it is an extension that Zarr-Python writes and reads). A reader that ignores it opens the output dataset all the same, group by group.