Working with Vertically Pointing Radar Data Using PyART, Xarray, and hvPlot

This notebook will walk through how to utilize existing packages in the Python ecosystem (PyART, Xarray, and hvPlot) to visualize data from a vertically pointing Ka-band radar.

By the end of this notebook, you will learn how to create the following visualization:

Data Overview

Before starting up this notebook, make sure to download your data from the ARM data portal, using this link!

We are using data from a snow storm on 14 March 2022.

Ka-band ARM zenith radar (KAZR) Instrument

An image of the KAZR radar collecting the data is shown below:

From the official ARM KAZR Documentation, “The Ka-band ARM zenith radar (KAZR) remotely probes the extent and composition of clouds at millimeter wavelengths. The KAZR is a zenith-pointing Doppler radar that operates at a frequency of approximately 35 GHz. The main purpose of this radar is to determine the first three Doppler moments (reflectivity, vertical velocity, and spectral width) at a range resolution of approximately 30 meters from near-ground to nearly 20 km in altitude.”

Imports

Before running through this notebook, you will need to install :

• matplotlib

• pyart

• hvplot

• holoviews

import matplotlib.pyplot as plt
import glob
import pyart
import pandas as pd
import hvplot.xarray
from bokeh.models.formatters import DatetimeTickFormatter
import xarray as xr
import holoviews as hv
import warnings

warnings.filterwarnings("ignore")
hv.extension("bokeh")

Plot Using PyART

Let’s first try plotting using the Python ARM Radar Toolkit (PyART)!

Load in the data using pyart

We can use the pyart.io.read() function for this dataset!

radar = pyart.io.read("guckazrcfrgeM1.a1.20220314.000003.nc")

This returns a radar object, which is the primary data model in PyART

radar

<pyart.core.radar.Radar at 0x7f7dbd517400>

If we wanted to take a look at what fields are available, we can print out radar.fields, formatting that as a list

list(radar.fields)

['linear_depolarization_ratio',
 'mean_doppler_velocity',
 'mean_doppler_velocity_crosspolar_v',
 'reflectivity',
 'reflectivity_crosspolar_v',
 'signal_to_noise_ratio_copolar_h',
 'signal_to_noise_ratio_crosspolar_v',
 'spectral_width',
 'spectral_width_crosspolar_v']

Let’s take a look at reflectivity and mean_doppler_velocity

Plot the Data in PyART

We will use the RadarDisplay as our mapping class, and the plot_vpt, which is made to work with verticall pointing radars (which is what KAZR is!)

display = pyart.graph.RadarDisplay(radar)
display.plot_vpt("reflectivity", time_axis_flag=True, edges=False)

We can also take a look at the mean_doppler_velocity, which is the radial velocity (wind speed) away from the radar (from the ground in this case)

display = pyart.graph.RadarDisplay(radar)
display.plot_vpt(
    "mean_doppler_velocity", time_axis_flag=True, edges=False, cmap="pyart_balance"
)

Clean Up the Plots a Bit

We can zoom in a bit on the lower part of the atmosphere here (below 7 km) since there is a substantial amount of noise above that, and combine our plots into a single figure using plt.subplot()

fig = plt.figure(figsize=(20, 8))

display = pyart.graph.RadarDisplay(radar)

# Create an axis for reflectivity
ax1 = plt.subplot(211)

# Plot reflectivity
display.plot_vpt(
    "reflectivity", fig=fig, ax=ax1, vmin=-20, vmax=40, time_axis_flag=True, edges=False
)

# Limit the y-axis to 0 to 7 km
plt.ylim(0, 7)

# Label the y-axis
plt.ylabel("Time (UTC)")

# Create an axis for velocity
ax2 = plt.subplot(212)

# Plot doppler velocity
display.plot_vpt(
    "mean_doppler_velocity",
    fig=fig,
    ax=ax2,
    vmin=-7,
    vmax=7,
    time_axis_flag=True,
    edges=False,
    cmap="pyart_balance",
)

# Limit the y-axis to 0 to 7 km
plt.ylim(0, 7)

# Label the y-axis
plt.ylabel("Time (UTC)")

# Add some space in between the plots
plt.subplots_adjust(hspace=0.5)

Plot Up Our Analysis Using Xarray + hvPlot

Let’s try another method of reading in and visualization… focusing on using xarray and hvPlot, which are key components of the Pangeo stack

Read the Data Using Xarray

Xarray has a similar .open() function, titled open_dataset

ds = xr.open_dataset("guckazrcfrgeM1.a1.20220314.000003.nc")
ds

<xarray.Dataset>
Dimensions:                             (time: 1736, range: 600, frequency: 1,
                                         sweep: 1, r_calib: 1)
Coordinates:
  * time                                (time) datetime64[ns] 2022-03-14T00:0...
  * frequency                           (frequency) float32 3.489e+10
  * range                               (range) float32 100.7 ... 1.806e+04
    azimuth                             (time) float32 0.0 0.0 0.0 ... 0.0 0.0
    elevation                           (time) float32 90.0 90.0 ... 90.0 90.0
Dimensions without coordinates: sweep, r_calib
Data variables: (12/38)
    base_time                           datetime64[ns] 2022-03-14T00:00:03
    time_offset                         (time) datetime64[ns] 2022-03-14T00:0...
    linear_depolarization_ratio         (time, range) float32 ...
    mean_doppler_velocity               (time, range) float32 ...
    mean_doppler_velocity_crosspolar_v  (time, range) float32 ...
    reflectivity                        (time, range) float32 ...
    ...                                  ...
    longitude                           float64 -107.0
    altitude                            float64 2.886e+03
    altitude_agl                        float64 3.0
    lat                                 float64 38.96
    lon                                 float64 -107.0
    alt                                 float64 2.886e+03
Attributes: (12/31)
    command_line:             kazrcfr_ingest -s guc -f M1
    Conventions:              ARM-1.3 CF/Radial-1.4 instrument_parameters rad...
    process_version:          ingest-kazrcfr-1.3-0.el7
    dod_version:              kazrcfrge-a1-1.1
    input_source:             /data/collection/guc/guckazrM1.00/KAZR_MOMENTS_...
    site_id:                  guc
    ...                       ...
    scan_mode:                vertically-pointing
    scan_name:                vpt
    software_version:         1.7.6 (Wed Mar 23 17:10:35 UTC 2016 leachman
    title:                    ARM KAZR Moments
    doi:                      10.5439/1498936
    history:                  created by user dsmgr on machine procnode2 at 2...

xarray.Dataset

• Dimensions:
  • time: 1736
  • range: 600
  • frequency: 1
  • sweep: 1
  • r_calib: 1
• Coordinates: (5)
  • time
    (time)
    datetime64[ns]
    2022-03-14T00:00:00.453818 ... 2...

    long_name :

    Time in seconds since volume start

    standard_name :

    time

    array(['2022-03-14T00:00:00.453818000', '2022-03-14T00:00:02.527426000',
           '2022-03-14T00:00:04.601027000', ..., '2022-03-14T00:59:54.002026000',
           '2022-03-14T00:59:56.075634000', '2022-03-14T00:59:58.149219000'],
          dtype='datetime64[ns]')

  • frequency
    (frequency)
    float32
    3.489e+10

    long_name :

    Transmit center frequency

    units :

    Hz

    standard_name :

    radiation_frequency

    meta_group :

    instrument_parameters

    array([3.489e+10], dtype=float32)

  • range
    (range)
    float32
    100.7 130.7 ... 1.803e+04 1.806e+04

    long_name :

    Range to measurement volume

    units :

    m

    meters_between_gates :

    29.979246

    meters_to_center_of_first_gate :

    100.679245

    spacing_is_constant :

    True

    standard_name :

    projection_range_coordinate

    axis :

    radial_range_coordinate

    array([  100.679245,   130.6585  ,   160.63774 , ..., 17998.29    ,
           18028.268   , 18058.248   ], dtype=float32)

  • azimuth
    (time)
    float32
    ...

    long_name :

    Azimuth angle from true north

    units :

    degree

    standard_name :

    sensor_to_target_azimuth_angle

    axis :

    radial_azimuth_coordinate

    array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)

  • elevation
    (time)
    float32
    ...

    long_name :

    Elevation angle from horizontal plane

    units :

    degree

    standard_name :

    sensor_to_target_elevation_angle

    axis :

    radial_elevation_coordinate

    array([90., 90., 90., ..., 90., 90., 90.], dtype=float32)

• Data variables: (38)
  • base_time
    ()
    datetime64[ns]
    ...

    string :

    2022-03-14 00:00:03 0:00

    long_name :

    Base time in Epoch

    ancillary_variables :

    time_offset

    array('2022-03-14T00:00:03.000000000', dtype='datetime64[ns]')

  • time_offset
    (time)
    datetime64[ns]
    ...

    long_name :

    Time offset from base_time

    ancillary_variables :

    base_time

    array(['2022-03-14T00:00:00.453818000', '2022-03-14T00:00:02.527426000',
           '2022-03-14T00:00:04.601027000', ..., '2022-03-14T00:59:54.002026000',
           '2022-03-14T00:59:56.075634000', '2022-03-14T00:59:58.149219000'],
          dtype='datetime64[ns]')

  • linear_depolarization_ratio
    (time, range)
    float32
    ...

    long_name :

    Linear depolarization ratio, channel unspecified

    units :

    dB

    standard_name :

    radar_linear_depolarization_ratio

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • mean_doppler_velocity
    (time, range)
    float32
    ...

    long_name :

    Radial mean Doppler velocity, positive for motion away from the instrument

    units :

    m/s

    standard_name :

    radial_velocity_of_scatterers_away_from_instrument

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • mean_doppler_velocity_crosspolar_v
    (time, range)
    float32
    ...

    long_name :

    Doppler velocity, crosspolar for vertical channel

    units :

    m/s

    standard_name :

    radial_velocity_of_scatterers_away_from_instrument

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • reflectivity
    (time, range)
    float32
    ...

    long_name :

    Equivalent reflectivity factor

    units :

    dBZ

    standard_name :

    equivalent_reflectivity_factor

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • reflectivity_crosspolar_v
    (time, range)
    float32
    ...

    long_name :

    Equivalent reflectivity factor, crosspolar for vertical channel

    units :

    dBZ

    standard_name :

    equivalent_reflectivity_factor

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • signal_to_noise_ratio_copolar_h
    (time, range)
    float32
    ...

    long_name :

    Signal-to-noise ratio, horizontal channel

    units :

    dB

    standard_name :

    radar_signal_to_noise_ratio_copolar_h

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • signal_to_noise_ratio_crosspolar_v
    (time, range)
    float32
    ...

    long_name :

    Signal-to-noise ratio, Cross-polar for vertical channel

    units :

    dB

    standard_name :

    radar_signal_to_noise_ratio_crosspolar_v

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • spectral_width
    (time, range)
    float32
    ...

    long_name :

    Spectral width

    units :

    m/s

    standard_name :

    radar_doppler_spectrum_width

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • spectral_width_crosspolar_v
    (time, range)
    float32
    ...

    long_name :

    Spectral Width, Crosspolar for Vertical Channel

    units :

    m/s

    standard_name :

    radar_doppler_spectrum_width

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [1041600 values with dtype=float32]

  • fixed_angle
    (sweep)
    float32
    ...

    long_name :

    Ray target fixed angle

    units :

    degree

    array([90.], dtype=float32)

  • n_samples
    (time)
    float64
    ...

    long_name :

    Number of Samples used to compute moments

    units :

    1

    meta_group :

    instrument_parameters

    standard_name :

    number_of_samples_used_to_compute_moments

    array([15., 15., 15., ..., 15., 15., 15.])

  • noise_figure
    (time)
    float32
    ...

    long_name :

    Receiver noise figure estimated from noise source using y-factor method

    units :

    dB

    array([-67.8043, -67.8043, -67.8043, ..., -67.8043, -67.8043, -67.8043],
          dtype=float32)

  • nyquist_velocity
    (time)
    float32
    ...

    long_name :

    Unambiguous doppler velocity

    units :

    m/s

    meta_group :

    instrument_parameters

    array([7.968252, 7.968252, 7.968252, ..., 7.968252, 7.968252, 7.968252],
          dtype=float32)

  • prt
    (time)
    float32
    ...

    long_name :

    Pulse repetition time

    units :

    s

    array([0.00027, 0.00027, 0.00027, ..., 0.00027, 0.00027, 0.00027],
          dtype=float32)

  • pulse_width
    (time)
    float32
    ...

    long_name :

    Transmitter pulse width

    units :

    s

    meta_group :

    instrument_parameters

    array([3.e-07, 3.e-07, 3.e-07, ..., 3.e-07, 3.e-07, 3.e-07], dtype=float32)

  • r_calib_radar_constant_copol
    (r_calib)
    float32
    ...

    long_name :

    Calibrated radar constant copolar

    units :

    dB

    meta_group :

    radar_calibration

    standard_name :

    calibrated_radar_constant_h_channel

    array([-8.996191], dtype=float32)

  • r_calib_radar_constant_crosspol
    (r_calib)
    float32
    ...

    long_name :

    Calibrated radar constant crosspolar

    units :

    dB

    meta_group :

    radar_calibration

    standard_name :

    calibrated_radar_constant_h_channel

    array([-13.405288], dtype=float32)

  • r_calib_two_way_radome_loss_h
    (r_calib)
    float32
    ...

    long_name :

    Radar calibration two way radome loss horizontal channel

    units :

    dB

    standard_name :

    radar_calibration_two_way_radome_loss_h_channel

    array([2.], dtype=float32)

  • radar_beam_width_h
    ()
    float32
    ...

    long_name :

    Half power radar beam width horizontal channel

    units :

    degree

    array(0.19)

  • radar_beam_width_v
    ()
    float32
    ...

    long_name :

    Half power radar beam width vertical channel

    units :

    degree

    array(0.19)

  • radar_measured_sky_noise_h
    (time)
    float32
    ...

    long_name :

    Measured sky noise, horizontal channel

    units :

    dBm

    array([-69.03703 , -69.035995, -69.0373  , ..., -69.03814 , -69.05106 ,
           -69.05984 ], dtype=float32)

  • radar_measured_sky_noise_v
    (time)
    float32
    ...

    long_name :

    Measured sky noise, vertical channel

    units :

    dBm

    array([nan, nan, nan, ..., nan, nan, nan], dtype=float32)

  • radar_measured_transmit_power
    (time)
    float32
    ...

    long_name :

    Radar measured transmit peak power

    units :

    dBm

    meta_group :

    instrument_parameters

    standard_name :

    radar_transmit_power

    array([52.143, 52.143, 52.143, ..., 52.143, 52.143, 52.143], dtype=float32)

  • receiver_gain_copol
    (time)
    float32
    ...

    long_name :

    Receiver gain copol

    units :

    1

    array([37.396, 37.396, 37.396, ..., 37.396, 37.396, 37.396], dtype=float32)

  • sweep_end_ray_index
    (sweep)
    float64
    ...

    long_name :

    Index of last ray in sweep

    units :

    1

    array([1735.])

  • sweep_mode
    (sweep)
    |S22
    ...

    long_name :

    Scan mode for sweep

    units :

    1

    array([b'vertical pointing '], dtype='|S22')

  • sweep_number
    (sweep)
    float64
    ...

    long_name :

    Sweep index number 0 based

    units :

    1

    array([0.])

  • sweep_start_ray_index
    (sweep)
    float64
    ...

    long_name :

    Index of first ray in sweep

    units :

    1

    array([0.])

  • unambiguous_range
    (time)
    float32
    ...

    long_name :

    Unambiguous Range

    units :

    m

    meta_group :

    instrument_parameters

    standard_name :

    unambiguous_range

    array([40479.477, 40479.477, 40479.477, ..., 40479.477, 40479.477, 40479.477],
          dtype=float32)

  • latitude
    ()
    float32
    ...

    long_name :

    Latitude

    units :

    degree_N

    standard_name :

    latitude

    valid_min :

    -90.0

    valid_max :

    90.0

    array(38.956158)

  • longitude
    ()
    float32
    ...

    long_name :

    Longitude

    units :

    degree_E

    standard_name :

    longitude

    valid_min :

    -180.0

    valid_max :

    180.0

    array(-106.987854)

  • altitude
    ()
    float32
    ...

    long_name :

    Altitude

    units :

    m

    standard_name :

    altitude

    array(2886.)

  • altitude_agl
    ()
    float32
    ...

    long_name :

    Altitude above ground level

    units :

    m

    standard_name :

    height

    array(3.)

  • lat
    ()
    float32
    ...

    long_name :

    North latitude

    units :

    degree_N

    standard_name :

    latitude

    valid_min :

    -90.0

    valid_max :

    90.0

    array(38.956158)

  • lon
    ()
    float32
    ...

    long_name :

    East longitude

    units :

    degree_E

    standard_name :

    longitude

    valid_min :

    -180.0

    valid_max :

    180.0

    array(-106.987854)

  • alt
    ()
    float32
    ...

    long_name :

    Altitude above mean sea level

    units :

    m

    standard_name :

    altitude

    array(2886.)

• Attributes: (31)

  command_line :

  kazrcfr_ingest -s guc -f M1

  Conventions :

  ARM-1.3 CF/Radial-1.4 instrument_parameters radar_parameters radar_calibration

  process_version :

  ingest-kazrcfr-1.3-0.el7

  dod_version :

  kazrcfrge-a1-1.1

  input_source :

  /data/collection/guc/guckazrM1.00/KAZR_MOMENTS_20220314-000005.dat

  site_id :

  guc

  platform_id :

  kazrcfrge

  facility_id :

  M1

  data_level :

  a1

  location_description :

  Surface Atmosphere Integrated Field Laboratory (SAIL), Mt Crested Butte, Colorado

  datastream :

  guckazrcfrgeM1.a1

  antenna_altitude :

  3 m

  antenna_diameter :

  2 m

  comment :

  digital_rx_dec :

  24

  dualpol :

  0

  fft_len :

  512

  filter_length :

  67

  institution :

  U.S. Department of Energy Atmospheric Radiation Measurement (ARM) Climate Research Facility

  n_gates :

  1024

  num_spectral_averages :

  15

  pulse_compression_ratio :

  13.3

  range_gate_spacing_m :

  29.979246

  range_offset_ch1 :

  -1.4 m

  range_offset_ch2 :

  70.7 m

  scan_mode :

  vertically-pointing

  scan_name :

  vpt

  software_version :

  1.7.6 (Wed Mar 23 17:10:35 UTC 2016 leachman

  title :

  ARM KAZR Moments

  doi :

  10.5439/1498936

  history :

  created by user dsmgr on machine procnode2 at 2022-03-14 01:49:03, using ingest-kazrcfr-1.3-0.el7

Subset for Time, Reflectivity and Velocity

Xarray has some helpful subsetting tools - including isel() which is index-based selection. In this case, since we don’t want to visualize the entire time period, we select the first 600 time steps

ds = ds.isel(time=range(600))

We are interested in looking at reflectivity and velocity here, and we can drop a couple of extra coordinates we don’t need (azimuth and elevation) since we won’t use them

ds = ds[["reflectivity", "mean_doppler_velocity"]].drop(["elevation", "azimuth"])
ds

<xarray.Dataset>
Dimensions:                (time: 600, range: 600)
Coordinates:
  * time                   (time) datetime64[ns] 2022-03-14T00:00:00.453818 ....
  * range                  (range) float32 100.7 130.7 ... 1.803e+04 1.806e+04
Data variables:
    reflectivity           (time, range) float32 ...
    mean_doppler_velocity  (time, range) float32 ...
Attributes: (12/31)
    command_line:             kazrcfr_ingest -s guc -f M1
    Conventions:              ARM-1.3 CF/Radial-1.4 instrument_parameters rad...
    process_version:          ingest-kazrcfr-1.3-0.el7
    dod_version:              kazrcfrge-a1-1.1
    input_source:             /data/collection/guc/guckazrM1.00/KAZR_MOMENTS_...
    site_id:                  guc
    ...                       ...
    scan_mode:                vertically-pointing
    scan_name:                vpt
    software_version:         1.7.6 (Wed Mar 23 17:10:35 UTC 2016 leachman
    title:                    ARM KAZR Moments
    doi:                      10.5439/1498936
    history:                  created by user dsmgr on machine procnode2 at 2...

xarray.Dataset

• Dimensions:
  • time: 600
  • range: 600
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2022-03-14T00:00:00.453818 ... 2...

    long_name :

    Time in seconds since volume start

    standard_name :

    time

    array(['2022-03-14T00:00:00.453818000', '2022-03-14T00:00:02.527426000',
           '2022-03-14T00:00:04.601027000', ..., '2022-03-14T00:20:38.395940000',
           '2022-03-14T00:20:40.469427000', '2022-03-14T00:20:42.542946000'],
          dtype='datetime64[ns]')

  • range
    (range)
    float32
    100.7 130.7 ... 1.803e+04 1.806e+04

    long_name :

    Range to measurement volume

    units :

    m

    meters_between_gates :

    29.979246

    meters_to_center_of_first_gate :

    100.679245

    spacing_is_constant :

    True

    standard_name :

    projection_range_coordinate

    axis :

    radial_range_coordinate

    array([  100.679245,   130.6585  ,   160.63774 , ..., 17998.29    ,
           18028.268   , 18058.248   ], dtype=float32)

• Data variables: (2)
  • reflectivity
    (time, range)
    float32
    ...

    long_name :

    Equivalent reflectivity factor

    units :

    dBZ

    standard_name :

    equivalent_reflectivity_factor

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [360000 values with dtype=float32]

  • mean_doppler_velocity
    (time, range)
    float32
    ...

    long_name :

    Radial mean Doppler velocity, positive for motion away from the instrument

    units :

    m/s

    standard_name :

    radial_velocity_of_scatterers_away_from_instrument

    unpacking_details :

    Can be unpacked as follows: multiply packed values by scale_factor field attribute, then add the add_offset field attribute.

    [360000 values with dtype=float32]

• Attributes: (31)

  command_line :

  kazrcfr_ingest -s guc -f M1

  Conventions :

  ARM-1.3 CF/Radial-1.4 instrument_parameters radar_parameters radar_calibration

  process_version :

  ingest-kazrcfr-1.3-0.el7

  dod_version :

  kazrcfrge-a1-1.1

  input_source :

  /data/collection/guc/guckazrM1.00/KAZR_MOMENTS_20220314-000005.dat

  site_id :

  guc

  platform_id :

  kazrcfrge

  facility_id :

  M1

  data_level :

  a1

  location_description :

  Surface Atmosphere Integrated Field Laboratory (SAIL), Mt Crested Butte, Colorado

  datastream :

  guckazrcfrgeM1.a1

  antenna_altitude :

  3 m

  antenna_diameter :

  2 m

  comment :

  digital_rx_dec :

  24

  dualpol :

  0

  fft_len :

  512

  filter_length :

  67

  institution :

  U.S. Department of Energy Atmospheric Radiation Measurement (ARM) Climate Research Facility

  n_gates :

  1024

  num_spectral_averages :

  15

  pulse_compression_ratio :

  13.3

  range_gate_spacing_m :

  29.979246

  range_offset_ch1 :

  -1.4 m

  range_offset_ch2 :

  70.7 m

  scan_mode :

  vertically-pointing

  scan_name :

  vpt

  software_version :

  1.7.6 (Wed Mar 23 17:10:35 UTC 2016 leachman

  title :

  ARM KAZR Moments

  doi :

  10.5439/1498936

  history :

  created by user dsmgr on machine procnode2 at 2022-03-14 01:49:03, using ingest-kazrcfr-1.3-0.el7

Plot a Quick Plot using Xarray

We can use the .plot() with our dataset if we want to get a quick-look at the data

ds.reflectivity.plot(x="time", cmap="pyart_HomeyerRainbow", vmin=-20, vmax=40)

<matplotlib.collections.QuadMesh at 0x7f7dad654f10>

This provided a similar view to our plot above - but what if we wanted to plot this interactively? We can use hvPlot which is a high-level interactive plotting library, meant to work with xarray.Datasets along with other popular components of the Python ecosystem.

Extract Date Information for Labels

Before we jump into the plotting, let’s define some helpful formatting tools.

First, we can parse the date from our dataset:

date = pd.to_datetime(ds.time.values[0]).strftime("%d %b %Y")
date

'14 Mar 2022'

For the interactive plot, we need to specify how our x-axis (time) labels are formatted

formatter = DatetimeTickFormatter(minutes="%H:%M UTC")

Create our Holoviews Plots

We create two interactive plots, one for reflectivity and the other doppler velocity.

Included in each plotting call are additional parameters:

• x : which field to use as the x-axis

• cmap : which colormap to use

• clim : color bar limits (equivalent to vmin/vmax)

• hover : whether to plot an interactive data inspection tool

• colorbar : whether to include a colorbar with the plot

• clabel : colorbar label

• title : Plot title (works at the subplot level)

• xformatter : formatter for the x-axis, in this case, our time

• xlabel : label

ref_plot = ds.reflectivity.hvplot.quadmesh(
    x="time",
    cmap="pyart_HomeyerRainbow",
    clim=(-20, 40),
    ylim=(0, 7000),
    hover=True,
    colorbar=True,
    clabel="Reflectivity (dBZ)",
    title=f"{ds.platform_id} \n{ds.location_description} \n{date}",
    xformatter=formatter,
    xlabel="Time (UTC)",
).opts(
    fontsize={
        "title": 10,
        "labels": 10,
        "xticks": 10,
        "yticks": 10,
    }
)

vel_plot = ds.mean_doppler_velocity.hvplot.quadmesh(
    x="time",
    cmap="pyart_balance",
    clim=(-7, 7),
    ylim=(0, 7000),
    clabel="Velocity (m/s)",
    title=f"{ds.platform_id} \n{ds.location_description} \n{date}",
    xformatter=formatter,
    xlabel="Time (UTC)",
).opts(fontsize={"title": 10, "labels": 10, "xticks": 10, "yticks": 10})

View the Final Visualization

Now that we have prepared our plots, let’s take a look at the output!

We can combine our plots by adding them (plot1 + plot2) together, specifying we desire a single column (.cols(1)).

(ref_plot + vel_plot).cols(1)

Try Using Rasterize

We would ideally use rasterize=True here to utilize Datashader to lazily visualize our data, viewing our data a coarse resolution intially, and accessing the higher resolution data when we zoom in on particular regions.

Unfortunately, run into the following bug when trying this on our data…

ref_plot_raster = ds.reflectivity.hvplot.quadmesh(
    x="time",
    cmap="pyart_HomeyerRainbow",
    clim=(-20, 40),
    ylim=(0, 7000),
    rasterize=True,
    hover=True,
    colorbar=True,
    clabel="Reflectivity (dBZ)",
    title=f"{ds.platform_id} \n{ds.location_description} \n{date}",
    xformatter=formatter,
    xlabel="Time (UTC)",
).opts(
    fontsize={
        "title": 10,
        "labels": 10,
        "xticks": 10,
        "yticks": 10,
    }
)

ref_plot_raster

WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 54168-03-13 00:07:33')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=(numpy.datetime64('2022-03-13T23:59:59.417014000'), numpy.datetime64('2022-03-14T00:20:43.579705000')), y_range=(0, 7000))
WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 54168-03-13 00:07:33')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=(numpy.datetime64('2022-03-13T23:59:59.417014000'), numpy.datetime64('2022-03-14T00:20:43.579705000')), y_range=(0, 7000))
WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 54168-03-13 00:07:33')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=(numpy.datetime64('2022-03-13T23:59:59.417014000'), numpy.datetime64('2022-03-14T00:20:43.579705000')), y_range=(0, 7000))
WARNING:param.dynamic_operation: Callable raised "OutOfBoundsDatetime('Out of bounds nanosecond timestamp: 54168-03-13 00:07:33')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=(numpy.datetime64('2022-03-13T23:59:59.417014000'), numpy.datetime64('2022-03-14T00:20:43.579705000')), y_range=(0, 7000))

Traceback (most recent call last):
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/plotting/util.py", line 277, in get_plot_frame
    return map_obj[key]
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1043, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in resolve
    return key, map_obj[key]
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1043, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in resolve
    return key, map_obj[key]
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1043, in dynamic_operation
    key, obj = resolve(key, kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1032, in resolve
    return key, map_obj[key]
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1344, in __getitem__
    val = self._execute_callback(*tuple_key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 1111, in _execute_callback
    retval = self.callback(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/spaces.py", line 708, in __call__
    ret = self.callable(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1044, in dynamic_operation
    return apply(obj, *key, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1036, in apply
    processed = self._process(element, key, kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/util/__init__.py", line 1018, in _process
    return self.p.operation.process_element(element, key, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/operation.py", line 194, in process_element
    return self._apply(element, key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/operation.py", line 141, in _apply
    ret = self._process(element, key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/operation/datashader.py", line 1514, in _process
    element = element.map(op, predicate)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/data/__init__.py", line 207, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/data/__init__.py", line 1223, in map
    return super(Dataset, self).map(*args, **kwargs)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/dimension.py", line 710, in map
    return map_fn(self) if applies else self
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/operation.py", line 214, in __call__
    return self._apply(element)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/core/operation.py", line 141, in _apply
    ret = self._process(element, key)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/holoviews/operation/datashader.py", line 1133, in _process
    data[x.name] = data[x.name].astype('datetime64[us]').astype('int64')
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/common.py", line 1497, in astype
    return apply_ufunc(
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/computation.py", line 1159, in apply_ufunc
    return apply_dataarray_vfunc(
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/computation.py", line 286, in apply_dataarray_vfunc
    result_var = func(*data_vars)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/computation.py", line 750, in apply_variable_ufunc
    data = as_compatible_data(data)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/variable.py", line 234, in as_compatible_data
    data = _possibly_convert_objects(data)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/xarray/core/variable.py", line 178, in _possibly_convert_objects
    return np.asarray(pd.Series(values.ravel())).reshape(values.shape)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/pandas/core/series.py", line 451, in __init__
    data = sanitize_array(data, index, dtype, copy)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/pandas/core/construction.py", line 567, in sanitize_array
    subarr = _try_cast(data, dtype, copy, raise_cast_failure)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/pandas/core/construction.py", line 726, in _try_cast
    return sanitize_to_nanoseconds(arr, copy=copy)
  File "/Users/mgrover/opt/anaconda3/envs/pyart-dev/lib/python3.9/site-packages/pandas/core/dtypes/cast.py", line 1717, in sanitize_to_nanoseconds
    values = conversion.ensure_datetime64ns(values)
  File "pandas/_libs/tslibs/conversion.pyx", line 257, in pandas._libs.tslibs.conversion.ensure_datetime64ns
  File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 54168-03-13 00:07:33

Key Takeaways

As we saw in this post, we have a few different options to plot vertically pointed radar data, including

• PyART (RadarDisplay)

• Xarray (Matplotlib)

• hvPlot (hvplot.Quadmesh)

Each approach has its own pros/cons, but one thing is clear: it will be helpful to continue to push for xarray integration with the pyart.Radar object, so that interactive plots like the one shown at the end of this post will be easier to create and share!

We ran into issues using rasterize with the hvPlot example - which is a bug that will need to examined further, as adding a rasterize capability with these data will enable a more performant, and shareable workflow since the notebooks including the rendered visualizing will be much smaller than the current implementation.

Since we include all of the data points, the total size of the notebook is ~50 MB, which makes visualizing and sharing the rendered plot difficult.

If you are interested in additional plots from this case, check out our SAIL Jupyterbook, documenting Jupyter Notebooks related to radar data analysis from the Surface Atmosphere Integrated Field Laboratory (SAIL) campaign!

New Docs   Fixing Weird Patterns When Plotting NEXRAD Level 3 Data