act.qc.QCFilter.datafilter#

QCFilter.datafilter(variables=None, rm_assessments=None, rm_tests=None, verbose=False, del_qc_var=False, no_NaN=False)[source]#

Method to apply quality control variables to data variables by changing the data values in the dataset using quality control variables. The data is updated with failing data set to NaN. This can be used to update the data variable in the xarray dataset for use with xarray methods to perform analysis on the data since those methods don’t read the quality control variables.

Parameters:

  • variables (None or str or list of str) – Data variable names to process. If set to None will update all data variables.

  • rm_assessments (str or list of str) – Assessment names listed under quality control varible flag_assessments to exclude from returned data. Examples include [‘Bad’, ‘Incorrect’, ‘Indeterminate’, ‘Suspect’]

  • rm_tests (int or list of int) – Test numbers listed under quality control variable to exclude from returned data. This is the test number (or bit position number) not the mask number.

  • verbose (boolean) – Print processing information.

  • del_qc_var (boolean) – Option to delete quality control variable after processing. Since the data values can not be determined after they are set to NaN and xarray method processing would also process the quality control variables, the default is to remove the quality control data variables. Defaults to False.

  • no_NaN (boolean) – Should the returned Xarray Dataset use NaN as the missing value indicator. If Xarray did not convert the _FillValue or missing_value to NaN upon reading the same missing value indicator should be used. If set to true will try to determine the current missing_value or _FillValue set in the file and use that value. If neither are set as a variable attribute it will use the default value (most likely -9999).

Examples

from act.io.arm import read_arm_netcdf
from act.tests import EXAMPLE_MET1

ds = read_arm_netcdf(EXAMPLE_MET1)
ds.clean.cleanup()

var_name = "atmos_pressure"

ds_1 = ds.nanmean()

ds.qcfilter.add_less_test(var_name, 99, test_assessment="Bad")
ds.qcfilter.datafilter(rm_assessments="Bad")
ds_2 = ds.nanmean()

print("All_data =", ds_1[var_name].values)
All_data = 98.86098
print("Bad_Removed =", ds_2[var_name].values)
Bad_Removed = 99.15148