Parallel conversion with Dask

Note

This page is a first draft.

When a Dask distributed client is active, convert_group_to_healpix() and create_healpix_dataset() convert the chunks of a group in parallel on the cluster: each chunk is one task. Without a client, the chunks are converted one after the other.

from distributed import Client

import healpix_convert as hc

client = Client()  # a local cluster, or the address of an existing scheduler

hc.convert_group_to_healpix(
    "measurements/land_cover",
    cache=cache,
    settings=settings,
    output_path=output_path,
)

Only groups with a chunk method other than "no_chunk" are converted in parallel (see Conversion settings): with "no_chunk", the whole group is resampled at once in the calling process.

Options

convert_group_to_healpix() has two options for the Dask conversion:

• resources: the worker resources needed by each chunk task, e.g., {"GPU": 1}, passed to distributed.Client.map().

• load_input_data: if True, the input data of the group is persisted in the distributed memory of the cluster before the conversion (or loaded in memory without a client). This can speed up the conversion, but needs enough memory for the whole input group.

Combine with parallel jobs

chunk_indices limits a call to some chunks of the group: each job of a workflow can convert its own share of the chunks, and each job can use a Dask cluster to convert them in parallel. See the Quickstart.