S3 object stores

Note

This page is a first draft.

The input and output paths can be URLs of an S3 object store (s3://bucket/path/dataset.zarr): Zarr accesses them through fsspec and s3fs, which healpix-convert depends on.

Output dataset

prepare_healpix_dataset(), convert_group_to_healpix() and create_healpix_dataset() take an output_storage_options argument, passed to zarr.open_group() to open the output dataset. For example, for an S3-compatible object store:

output_path = "s3://my-bucket/output.zarr"
output_storage_options = {
    "endpoint_url": "https://s3.example.com",
    "key": "...",
    "secret": "...",
}

hc.prepare_healpix_dataset(
    cache, settings, output_path, output_storage_options=output_storage_options
)
hc.convert_group_to_healpix(
    "measurements/land_cover",
    cache=cache,
    settings=settings,
    output_path=output_path,
    output_storage_options=output_storage_options,
)

Pass the same output_storage_options to every call writing to the same output dataset.

Input datasets

The input datasets are opened with xarray.open_datatree(), without storage options: S3 inputs are read with the default configuration of s3fs, e.g., credentials from the environment or from the AWS configuration files.

When the staging cache is passed to other jobs as JSON (see the Quickstart), each job opens the input datasets again from their paths, so the inputs must be readable from every job.

The paths of the input datasets are also recorded in the STAC metadata of the output dataset (see Output metadata).