healpix_resample.psf (PSF / multi-point HEALPix regridding)#
healpix_resample.psf provides GPU-friendly sparse regridding from unstructured geographic samples
(longitude/latitude) to a subset of HEALPix pixels at a target resolution (nside = 2**level).
In contrast to a pure nearest-neighbor operator, this class builds a local, multi-point Gaussian kernel around each sample (a “PSF”-like footprint) and can solve an inverse problem to estimate a HEALPix field that best explains the observed samples.
The implementation is designed for large N and batched values (B, N) on CUDA using PyTorch sparse
operators.
What the class does#
Given:
sample coordinates
(lon, lat)of shape(N,)sample values
valof shape(N,)or(B, N)a HEALPix
level(thusnside = 2**level)a neighbourhood size
Npt(number of nearby HEALPix cells per sample)
The class:
Selects nearby HEALPix cells for each sample using local neighbourhood search (avoids building an
N × Npixdistance matrix).Computes Gaussian weights as a function of distance (meters) with scale
sigma_m.Builds two sparse operators:
Mof shape(N, K): maps a HEALPix field on K kept pixels to sample points (forward model is viaMTbelow)MTof shape(K, N): maps sample values back to the HEALPix subset (adjoint-like accumulation)
Provides:
resample(val): estimate the HEALPix field (hval) from samples by solving a damped least-squares problem with Conjugate Gradient (CG)invert(hval): project a HEALPix field back to sample locations
Note on geodesy: distances are computed in meters and the class supports the Earth ellipsoid WGS84 and the HEALPix authalic definition through its geometry helper.
Mathematical view (high level)#
Let:
ybe the sample values(B, N)hbe the unknown HEALPix field(B, K)on the kept pixelsMbe(N, K)andMTbe(K, N)
A reference field is computed by weighted back-projection:
x_ref = y @ M(shape(B, K))
Then the solver estimates an update delta by minimizing a damped normal equation:
minimize
|| (x_ref + delta) @ MT - y ||^2 + lam * ||delta||^2
This is solved with CG using matrix-vector products only:
A(v) = (v @ MT) @ M + lam * v
Finally:
h = x_ref + delta
M and MT are not Euclidean transposes of each other (each is normalized against a
different axis of the raw weight matrix – per-HEALPix-cell for M, per-source-sample for
MT). They are, however, an exact adjoint pair with respect to a pair of weighted inner
products induced by those same normalizers (Dx on the HEALPix side, Dy on the source-sample
side). CG’s own dot products are evaluated in the Dx-weighted metric (self.cell_weight)
rather than the Euclidean one, which is what makes CG a theoretically justified solver here
rather than an empirical choice – see the accompanying paper for the full derivation.
Conservative rebinning: weighting by pixel area#
By default, M’s per-cell normalization treats every source sample as contributing equally,
regardless of how much physical area it actually represents. On a grid where pixel area varies
appreciably (e.g. a reduced Gaussian grid, where longitude spacing shrinks towards the poles),
this introduces a small but systematic bias: the local average is really an average over
samples, not over area.
Passing area bakes the per-sample pixel area directly into M’s construction (raw weights
g_ik are multiplied by the sample’s area a_i before normalizing), making the reconstruction
an area-weighted local average instead – a conservative rebinning. MT is unaffected: the
area factor cancels out of its own per-sample normalization algebraically, which matches the
fact that the HEALPix side needs no such weight (HEALPix cells are equal-area / iso-surface by
construction).
op = PSFResampler(lon_deg=lon, lat_deg=lat, level=level, area=area) # explicit weights
op = PSFResampler(lon_deg=lon, lat_deg=lat, level=level) # area="auto" (default)
area=None(default) orarea="auto": the per-sample area is estimated automatically from the grid geometry, assuming samples share latitude “rings” (regular lat/lon grids, reduced Gaussian grids such as ECMWF’s N-grids). If no such structure is detected (e.g. a grid regular in a different projection such as UTM, or scattered points), silently falls back to a uniform weight of1.0per sample – the same as before this option existed.area=<array>: use an explicit per-sample area/weight instead (own units; only ratios matter).
Local area-weighting alone does not guarantee exact global conservation – it removes the
local bias, but small residual imbalances can remain (e.g. near the edge of the retained-cell
set). For an exact guarantee, combine it with resample(..., conservative=True):
res = op.resample(val, conservative=True)
which applies a minimum-distortion correction (a Lagrange-multiplier solve reusing the same
weighted CG machinery) so that mean(hval) exactly matches the true area-weighted mean of
val – equivalent to conserving the area-integrated total, since HEALPix cells are
equal-area. See the accompanying paper (Section “Optional Flux-Conservation Constraint”) for
the full derivation, and docs/tutorials for a worked example on real ERA5 data.
Bounded fit: clip#
Solving for a field sharper than the samples’ PSF is a deconvolution: across a sharp,
high-contrast edge the solution over- and undershoots, which can give negative values for a
non-negative field (reflectance, concentration, precipitation) and values above the input
maximum. lam damps it but does not remove it.
clip=(lo, hi) adds hard bounds to the fit:
res = op.resample(val, lam=0.1, clip=(0.0, None)) # non-negative field
res = op.resample(val, lam=0.1, clip=(None, 1.0)) # upper bound only
res = op.resample(val, lam=0.1, clip=("input", "input")) # within the input range, per batch row
Each side is None (not constrained), a number, or "input" (the min, or max, of the row’s
valid samples). The default clip=None is the unconstrained fit.
The fit then solves the same weighted least-squares problem under lo <= h <= hi:
minimize || h @ MT - y ||^2_Dy + lam * || h - x_ref ||^2_Dx subject to lo <= h <= hi
with an active-set loop around the same weighted CG:
a cell at a bound whose gradient points out of the box is held at that bound and removed from the free unknowns;
CG solves for the free cells, with the held cells still in the forward model
h @ MT;cells that step out of the box are clamped and held at the next pass;
a held cell whose gradient points back into the box is released (Karush-Kuhn-Tucker conditions); the loop stops when nothing changes, or after
max_outerpasses (default 10).
Because a held cell still enters the forward model, the samples it can no longer explain leave a residual that its free neighbours take up. A cell held at 0 pulls its neighbours down slightly, instead of being zeroed after the fact; the result is the minimum of the constrained problem, which clipping the unconstrained result is not.
Lower and upper bounds are not equivalent. A lower bound such as 0 is usually physical. An upper bound is a stronger choice: a bright, sharp feature smoothed by the input PSF is legitimately reconstructed above the input values. Capping it at the input maximum spreads the energy the capped cells cannot carry over their neighbours, which widens the feature. Bounding one side also reduces the ringing on the other side, since the oscillation that follows an overshoot is no longer driven.
With conservative=True, the conservation correction is computed on the free cells only;
free cells it would take out of the box are held at their bound and the correction is computed
again. The output then satisfies both the bounds and the conservation constraint whenever the
target mean is reachable inside the box.
ResampleResults.n_clipped_low and n_clipped_high give the number of cells held at each bound
(per batch row); they are None without clip.
Constructor#
PSFResampler(
lon_deg, lat_deg,
level,
out_cell_ids=None,
Npt=9,
sigma_m=None,
threshold=0.1,
area=None,
fill_missing_out_cells=False,
nest=True,
radius=6371000.0,
ellipsoid="WGS84",
dtype=torch.float32,
device="cpu",
# forwarded to KNeighborsResampler through **kwargs (see "ring_* parameters" below):
# ring_weight, ring_search_init, ring_search_max, num_threads
verbose=False,
)
Key parameters#
lon_deg, lat_deg: sample coordinates in degrees, shape(N,),float32orfloat64(the geometry is computed infloat64).level: HEALPix level (nside = 2**level).out_cell_ids: restrict the output to a caller-specified subset oflevel-resolution cells (e.g. fromsubset_for_parent_cellwhen processing one coarse region at a time). Seefill_missing_out_cellsbelow for what happens when a requested cell has too little real kernel support. If none of the requested cells has support (e.g. a chunk at the edge of the input whose nearby samples all lie just outside it), the constructor does not raise: it builds an operator withK = 0, andresample()returnscell_idsof size 0 andcell_dataof shape(B, 0), with or withoutfill_missing_out_cells. Withoutout_cell_ids, an empty result still raisesRuntimeError("No HEALPix cell passed the threshold ...").Npt: number of neighbouring HEALPix cells used per sample.sigma_m: Gaussian length scale in meters.If
None, a default scale based on the HEALPix pixel area is used:sigma = sqrt(4*pi / (12*4**level)) * R.
threshold: global pruning threshold on accumulated raw (area-independent) weights; pixels with too little kernel support are discarded. This reducesKand keeps the operator compact.area: per-sample pixel area/weight of the source grid, for conservative rebinning (see above).None/"auto"(default) estimates it automatically when possible, else falls back to uniform.fill_missing_out_cells(defaultFalse): only relevant without_cell_ids. Some requested output cells can end up with too little real kernel support – an empty or near-emptyMcolumn, typically becauseout_cell_idsforce-included a cell the KNN search wouldn’t have retained on its own (e.g. a river-mouth cell in an ocean model, the original motivating case for this fallback). Filling such a cell requires an expensive, unvectorized per-cell fallback search.False(default): skip that fallback entirely. Weakly-supported requested cells come back asnaninresample()’s output rather than an approximate value – correct and fast, and the recommended default especially when combined withsubset_for_parent_cell(which can force-include many such cells at once).True: restore the original approximate-fallback-fill behaviour (a non-NaN but approximate value for these cells). Opt into this only if a value is specifically needed instead of a gap, and the construction-time cost is acceptable.
nest: HEALPix indexing scheme (nested ifTrue).device,dtype: PyTorch device and dtype for all matrices and computations.ring_*parameters: control the local neighbourhood expansion strategy in the geometry helper.ring_weightis the neighbourhood used to decide which cells are kept; each sample then looks for itsNptnearest kept cells in rings growing fromring_search_inittoring_search_max(whenNone,ring_search_maxisring_search_init + 2, and a value belowring_search_initis raised to it). A sample that does not findNptcells withinring_search_maxrings keeps the ones it found; this happens when the samples are much sparser than the cells. If no sample reaches any kept cell, the constructor raises aRuntimeErrorinstead of returning an empty operator.verbose: prints CG progress and area-estimation diagnostics.
Stored attributes (after initialization)#
N: number of samples.K: number of kept HEALPix pixels.cell_ids:(K,)HEALPix pixel ids retained after thresholding.hi:(N, Npt)indices intocell_idsfor each sample (the chosen neighbours).d_m:(N, Npt)distances in meters for each neighbour link.area:(N,)per-sample pixel area/weight used to buildM(see above).cell_weight:(K,)theDxweight (raw, area-weighted per-cell accumulated weight) CG uses for its inner product.M: sparse CSR(N, K)operator.MT: sparse CSR(K, N)operator.
Methods#
resample(val, lam=0.0, max_iter=100, tol=1e-8, x0=None, return_info=False, conservative=False, clip=None, max_outer=10)#
Estimate a HEALPix field from samples.
Input:
val:(N,)or(B, N)lam: damping / Tikhonov regularizationx0: optional initial guess fordelta(shape(B, K))conservative: apply the exact minimum-distortion flux-conservation correction (see above)clip:Noneor(lo, hi), hard bounds imposed inside the fit (see “Bounded fit” above)max_outer: maximum number of passes of the bounded fit
Output: a
ResampleResultswithcell_data(hval, shape(K,)or(B, K)),cell_ids, CG diagnostics (cg_residual_norms,cg_niters) and, withclip, the number of cells held at each bound (n_clipped_low,n_clipped_high)
invert(hval)#
Project HEALPix field(s) back to sample locations.
Input:
hval(K,)or(B, K)Output: reconstructed samples
(N,)or(B, N)
get_cell_ids()#
Return the kept HEALPix pixel ids as a NumPy array (K,).
Typical workflow#
import torch
from healpix_resample import PSFResampler
op = PSFResampler(
lon_deg=lon, lat_deg=lat,
level=level, Npt=9,
device="cuda", dtype=torch.float32,
area=area, # optional; omit for "auto" (see above)
)
# Estimate HEALPix field on the kept pixels, exactly conservative
res = op.resample(val, lam=1e-3, max_iter=200, tol=1e-7, conservative=True)
hval = res.cell_data
# Reconstruct values at the original sample points
val_hat = op.invert(hval)
# Access the HEALPix pixel ids corresponding to hval
cell_ids = op.get_cell_ids()
Notes and practical tips#
Choose
Nptaccording to the desired smoothness / footprint:small
Npt→ more local, less smoothlarger
Npt→ smoother but more compute
Tune
sigma_m:smaller
sigma_m→ sharper PSF, more local influencelarger
sigma_m→ smoother field but can blur features
Use
lamto stabilize inversion when sampling is sparse/irregular:lam = 0is pure least squareslam > 0damps high-frequency or poorly constrained modes
Use
clip, not a largerlam, to keep the output physical at sharp edges (see “Bounded fit” above):lamreduces the over- and undershoots but also smooths the whole field;cliponly acts on the cells that leave the boxfor a non-negative field,
clip=(0.0, None)is the natural setting; add an upper bound only when values above it are known to be impossiblecheck
res.n_clipped_low/res.n_clipped_high: a large count means the fit is strongly constrained, and may call for a largersigma_morlameach pass of the bounded fit runs one CG solve, so
clipcosts up tomax_outertimes the unconstrained fit
The operator only returns a subset of HEALPix pixels (
cell_ids), not the full sky map. This is intentional for memory/performance on regional problems.