# S3 object stores

:::{note}
This page is a first draft.
:::

The input and output paths can be URLs of an S3 object store
(`s3://bucket/path/dataset.zarr`): Zarr accesses them through
[fsspec](https://filesystem-spec.readthedocs.io) and
[s3fs](https://s3fs.readthedocs.io), which `healpix-convert` depends on.

## Output dataset

{py:func}`~healpix_convert.prepare_healpix_dataset`,
{py:func}`~healpix_convert.convert_group_to_healpix` and
{py:func}`~healpix_convert.create_healpix_dataset` take an
`output_storage_options` argument, passed to {py:func}`zarr.open_group` to open
the output dataset. For example, for an S3-compatible object store:

```python
output_path = "s3://my-bucket/output.zarr"
output_storage_options = {
    "endpoint_url": "https://s3.example.com",
    "key": "...",
    "secret": "...",
}

hc.prepare_healpix_dataset(
    cache, settings, output_path, output_storage_options=output_storage_options
)
hc.convert_group_to_healpix(
    "measurements/land_cover",
    cache=cache,
    settings=settings,
    output_path=output_path,
    output_storage_options=output_storage_options,
)
```

Pass the same `output_storage_options` to every call writing to the same
output dataset.

## Input datasets

The input datasets are opened with {py:func}`xarray.open_datatree`, without
storage options: S3 inputs are read with the default configuration of s3fs,
e.g., credentials from the environment or from the AWS configuration files.

When the staging cache is passed to other jobs as JSON (see the
{doc}`../tutorials/quickstart`), each job opens the input datasets again from
their paths, so the inputs must be readable from every job.

The paths of the input datasets are also recorded in the STAC metadata of the
output dataset (see {doc}`metadata`).