Ingest#

Translates raw NEXRAD Level-II polar format into a Cartesian grid dataset in the system’s canonical internal representation.

Read and regrid NEXRAD Level-II files to Cartesian xarray.Dataset.

This module handles loading raw NEXRAD archive files and transforming them into regular Cartesian grids using Py-ART. The output is an xarray.Dataset containing gridded reflectivity and other radar fields, enabling downstream analysis (segmentation, projection, cell detection).

Key capabilities: - Reads NEXRAD Level-II (.gz) files - Regrids to configurable Cartesian grid (default: 41x201x201 m) - Saves intermediate NetCDF files for data lineage - Handles errors gracefully (logs and returns None)

Author: Bhupendra Raut

class adapt.modules.ingest.module.RadarDataLoader(config)#

Bases: object

Load and regrid NEXRAD radar files to Cartesian grid xarray.Dataset.

This class handles the two-stage process of transforming raw NEXRAD data:

  1. Read: Loads NEXRAD Level-II archive files using Py-ART

  2. Regrid: Transforms from polar to Cartesian grid using distance-weighted interpolation (configurable algorithm, default: Cressman weighting)

The output is an xarray.Dataset with dimensions (z, y, x) and data variables including reflectivity and velocity. Radar metadata (lat/lon/alt) is preserved in dataset attributes.

Configuration#

Expects config dict with two sub-dicts:

  • readerdict

    • file_format : str, file type (default: “nexrad_archive”)

  • regridderdict

    • grid_shape : tuple, (nz, ny, nx) vertical/horizontal grid points (default: (41, 201, 201))

    • grid_limits : tuple, ((z_min, z_max), (y_min, y_max), (x_min, x_max)) in meters from radar center (default: ((0, 20000), (-100000, 100000), (-100000, 100000)))

    • roi_func : str, influence radius function (default: “dist_beam” - distance from radar ray). Other options: “constant_distance”, “dist”

    • min_radius : float, minimum radius for influence (meters, default: 1750)

    • weighting_function : str, interpolation weighting (default: “cressman”) Other options: “linear”, “nearest”, “barnes”

Notes

  • Not thread-safe: create separate loader instances for concurrent use

  • Regridding takes 5-15 seconds per file (CPU-bound)

  • NetCDF output is always compressed (zlib, level 9)

  • All methods return None on failure (logged, not raised)

  • Radar object is a temporary Py-ART object; only the xarray.Dataset is kept

Examples

>>> config = {
...     "reader": {"file_format": "nexrad_archive"},
...     "regridder": {"grid_shape": (41, 201, 201), "min_radius": 1750.0}
... }
>>> loader = RadarDataLoader(config)
>>> ds = loader.load_and_regrid("20250305_KLOT.gz")
>>> print(ds.data_vars)  # reflectivity, velocity, etc.
load_and_regrid(filepath, grid_kwargs=None)#

Read and regrid a NEXRAD file in one call (convenience method).

Combines read() and regrid() operations for simpler usage when both steps are needed in sequence. This is the primary entry point for most use cases.

Parameters:

  • filepath (Path | str) – Path to NEXRAD Level-II (.gz) file.

  • grid_kwargs (dict | None) – Regridding parameter overrides. Passed to regrid().

Returns:

Cartesian grid xarray.Dataset if successful, None if: - File does not exist or cannot be read - Regridding fails

Return type:

Dataset | None

Notes

  • Preferred method over separate read() + regrid() calls

Examples

>>> loader = RadarDataLoader(config)
>>> ds = loader.load_and_regrid("20250305_KLOT.gz")
>>> if ds is not None:
...     print(f"Grid shape: {ds.reflectivity.shape}")
...     cells = segment_cells(ds)  # Downstream processing
read(filepath)#

Read a NEXRAD archive file into a Py-ART Radar object.

Loads the raw NEXRAD Level-II file (.gz format) and parses it into Py-ART’s Radar object, which contains polar coordinate reflectivity, velocity, and other fields indexed by ray and gate.

Parameters:

filepath (Path | str) – Path to the NEXRAD Level-II file. Must exist on disk.

Returns:

Py-ART Radar object if successful, None if: - File does not exist - File format is unsupported - Read operation fails

Return type:

object

Notes

  • File existence is validated before attempting read

  • Errors are logged; exception stack trace is included

  • Radar object is large (~100-200 MB) and should be passed immediately to regrid() for processing and cleanup

  • Read time: 1-3 seconds per file (mostly decompression)

Examples

>>> loader = RadarDataLoader(config)
>>> radar = loader.read("20250305_KLOT.gz")
>>> if radar is not None:
...     print(f"Loaded {radar.nrays} rays x {radar.ngates} gates")
regrid(radar, grid_kwargs=None)#

Transform a Py-ART Radar object from polar to Cartesian grid.

Performs distance-weighted interpolation to convert irregular polar coordinates (typical of radar) to a regular Cartesian grid suitable for machine learning and image processing tasks. Radar metadata (latitude, longitude, altitude) is preserved in dataset attributes.

Parameters:

  • radar (Any) – Input Py-ART Radar object (from read() method).

  • grid_kwargs (dict | None) –

    Override regridding parameters. Merged with config defaults. Supported keys:

    • grid_shapetuple of int

      (nz, ny, nx) grid dimensions

    • grid_limitstuple of tuple

      ((z_min, z_max), (y_min, y_max), (x_min, x_max)) in meters

    • roi_funcstr

      Radius of influence function

    • min_radiusfloat

      Minimum radius in meters

    • weighting_functionstr

      Interpolation algorithm (“cressman”, “barnes”, “linear”)

Returns:

Cartesian grid xarray.Dataset with dimensions (z, y, x) and variables (reflectivity, velocity, etc.) if successful.

Return type:

Dataset | None

Notes

  • Regridding is CPU-intensive: 5-15 seconds per file

  • Grid defaults from config are merged with grid_kwargs overrides

  • Radar location attributes (latitude, longitude, altitude) are added to dataset.attrs for metadata tracking

Examples

>>> loader = RadarDataLoader(config)
>>> radar = loader.read("20250305_KLOT.gz")
>>> ds = loader.regrid(radar, grid_kwargs={"grid_shape": (50, 250, 250)})
>>> print(ds.reflectivity.shape)  # (50, 250, 250)