Horel Group

John Horel group - MesoWest, TRAX/eBUS, etc.

This module contains classes and functions for working with the Horel group data in the CHPC UATAQ filesystem.

Module Attributes

HOREL_DIR	Horel group directory
UUTRAX_DIR	UUTRAX directory
UUTRAX_PILOT_DIR	UUTRAX pilot directory
MESOWEST_DIR	MesoWest operational data directory (SODAR archive is read by uataq.sodar)
PILOT_PHASE	Pilot phase time ranges for UUTRAX data
lvl_data_dirs	Data levels for UUTRAX data
column_mapping	Horel to UATAQ column mapping

Classes

HorelCSVFile(path, instrument)	Class for parsing CSV files from the Horel group.
HorelCSVFinalizedFile(path, instrument)	Class for parsing finalized CSV
HorelFile(path, instrument)	Abstract base class for Horel data files.
HorelGroup()	A class representing the Horel group data space in the CHPC UATAQ filesystem.
HorelH5File(path, instrument)	Class for parsing H5 files from the Horel group.

uataq.filesystem.groupspaces.horel.HOREL_DIR: str = '/uufs/chpc.utah.edu/common/home/horel-group'

Horel group directory

uataq.filesystem.groupspaces.horel.UUTRAX_DIR: str = '/uufs/chpc.utah.edu/common/home/horel-group/uutrax'

UUTRAX directory

uataq.filesystem.groupspaces.horel.UUTRAX_PILOT_DIR: str = '/uufs/chpc.utah.edu/common/home/horel-group/uutrax_pilot'

UUTRAX pilot directory

uataq.filesystem.groupspaces.horel.MESOWEST_DIR: str = '/uufs/chpc.utah.edu/common/home/horel-group/oper/mesowest'

MesoWest operational data directory (SODAR archive is read by uataq.sodar)

uataq.filesystem.groupspaces.horel.PILOT_PHASE: dict[str, TimeRange] = {'TRX01': TimeRange(start=2014-11-11 00:00:00, stop=2018-11-19 20:03:59), 'TRX02': TimeRange(start=2016-02-04 00:00:00, stop=2018-11-19 18:53:53)}

Pilot phase time ranges for UUTRAX data

uataq.filesystem.groupspaces.horel.lvl_data_dirs: dict[str, list] = {'final': ['/uufs/chpc.utah.edu/common/home/horel-group/uutrax'], 'qaqc': ['/uufs/chpc.utah.edu/common/home/horel-group/uutrax'], 'raw': ['/uufs/chpc.utah.edu/common/home/horel-group/uutrax_pilot', '/uufs/chpc.utah.edu/common/home/horel-group/uutrax']}

Data levels for UUTRAX data

uataq.filesystem.groupspaces.horel.column_mapping: dict[str, dict[str, str]] = {'2b_205': {'2B_Air_Flow_Rate': 'Flow_Lpm', '2B_Internal_Air_Pressure': 'Internal_P_hPa', '2B_Internal_Air_Temperature': 'Internal_T_C', '2B_Ozone_Concentration': 'O3_ppb', 'FL2B': 'Flow_Lpm', 'OZNE': 'O3_ppb', 'Ozone_Data_Flagged': 'QAQC_Flag', 'PS2B': 'Internal_P_hPa', 'TC2B': 'Internal_T_C'}, '2b_405': {'2B405_Air_Flow_Rate': 'Flow_Lpm', '2B405_Cell_O3_Flow_Rate': 'O3_Flow_mLpm', '2B405_Internal_Air_Pressure': 'Internal_P_hPa', '2B405_Internal_Air_Temperature': 'Internal_T_C', '2B405_NO2_Concentration': 'NO2_ppb', '2B405_NOX_Concentration': 'NOx_ppb', '2B405_NO_Concentration': 'NO_ppb', 'FLNO': 'Flow_Lpm', 'FO3N': 'O3_Flow_mLpm', 'NO1C': 'NO_ppb', 'NO2C': 'NO2_ppb', 'NOXC': 'NOx_ppb', 'PSNO': 'Internal_P_hPa', 'TCNO': 'Internal_T_C'}, 'cr1000': {'Battery_Voltage': 'Battery_Voltage_V', 'Bus_Box_Temperature': 'Logger_T_C', 'Bus_Top_Relative_Humidity': 'Ambient_RH_pct', 'Bus_Top_Temperature': 'Ambient_T_C', 'TICC': 'Logger_T_C', 'TRNR': 'Ambient_RH_pct', 'TRNT': 'Ambient_T_C', 'Train_Box_Temperature': 'Logger_T_C', 'Train_Top_Relative_Humidity': 'Ambient_RH_pct', 'Train_Top_Temperature': 'Ambient_T_C', 'VOLT': 'Battery_Voltage_V'}, 'gps': {'Elevation': 'Altitude_msl', 'GELV': 'Altitude_msl', 'GLAT': 'Latitude_deg', 'GLON': 'Longitude_deg', 'GPS_Data_Flagged': 'QAQC_Flag', 'GPS_Direction': 'Course_deg', 'GPS_RMC_Valid': 'Status', 'GPS_Speed': 'Speed_m_s', 'GTIM': 'Instrument_Time', 'Latitude': 'Latitude_deg', 'Longitude': 'Longitude_deg', 'NSAT': 'N_Satellites', 'RDIR': 'Course_deg', 'RSPD': 'Speed_m_s', 'RSTS': 'Status'}, 'metone_es405': {'ERRR': 'Status', 'ES405_Air_Flow_Rate': 'Flow_Lpm', 'ES405_Error_Code': 'Status', 'ES405_Internal_Air_Pressure': 'Internal_P_hPa', 'ES405_Internal_Air_Temperature': 'Internal_T_C', 'ES405_Internal_Relative_Humidity': 'Internal_RH_pct', 'ES405_PM10_Concentration': 'PM10_ugm3', 'ES405_PM1_Concentration': 'PM1_ugm3', 'ES405_PM2.5_Concentration': 'PM2.5_ugm3', 'ES405_PM4_Concentration': 'PM4_ugm3', 'FLOW': 'Flow_Lpm', 'INRH': 'Internal_RH_pct', 'ITMP': 'Internal_T_F', 'PM01': 'PM1_ugm3', 'PM04': 'PM4_ugm3', 'PM10': 'PM10_ugm3', 'PM2.5_Data_Flagged': 'QAQC_Flag', 'PM25': 'PM2.5_ugm3', 'PRES': 'Internal_P_hPa'}, 'metone_es642': {'ERRR': 'Status', 'ES642_Air_Flow_Rate': 'Flow_Lpm', 'ES642_Error_Code': 'Status', 'ES642_Internal_Air_Pressure': 'Ambient_P_hPa', 'ES642_Internal_Air_Temperature': 'Ambient_T_C', 'ES642_Internal_Relative_Humidity': 'Internal_RH_pct', 'ES642_PM2.5_Concentration': 'PM2.5_ugm3', 'FLOW': 'Flow_Lpm', 'INRH': 'Internal_RH_pct', 'ITMP': 'Ambient_T_F', 'PM2.5_Data_Flagged': 'QAQC_Flag', 'PM25': 'PM2.5_ugm3', 'PRES': 'Ambient_P_hPa'}}

Horel to UATAQ column mapping

class uataq.filesystem.groupspaces.horel.HorelFile(path, instrument)

Abstract base class for Horel data files.

path

The file path.

Type:

str

period

The period of the data file.

Type:

pd.Period

logger

The logger name.

Type:

str

date_slicer

A slice object to extract the date from the file name.

Type:

slice

file_freq

The file frequency.

Type:

str

ext

The file extension.

Type:

str

time_col

The time column name.

Type:

str

instrument

The instrument name.

Type:

str

usecols(col)

Check if a column should be used based on the instrument.

convert_nodata(data, nodata=-9999.0)

Convert NoData values to NaN.

coerce_numeric(data, exclude='Time_UTC')

Coerce columns to numeric.

date_slicer: slice = slice(6, 13, None)

file_freq: str = 'M'

time_col: str

__init__(path, instrument)

Initialize a HorelFile subclass object.

The instrument parameter is used to filter columns based on the instrument name.

Parameters:

• path (str) – The file path.

• instrument (str) – The instrument name - used to filter columns.

usecols(col)

Check if a column should be used based on the instrument.

Parameters:

col (str) – The column name.

Returns:

True if the column should be used, False otherwise.

Return type:

bool

format_time(data, **kwargs)

Format the time column in the data DataFrame.

Parameters:

• data (pd.DataFrame) – The data DataFrame.

• **kwargs (dict) – Additional keyword arguments to pass to pd.to_datetime.

Returns:

The data DataFrame with the time column formatted as Time_UTC.

Return type:

pd.DataFrame

static convert_nodata(data, nodata=-9999.0)

Convert NoData values to NaN.

Parameters:

• data (pd.DataFrame) – The data DataFrame.

• nodata (float) – The NoData value.

Returns:

The data DataFrame with NoData values converted to NaN.

Return type:

pd.DataFrame

static coerce_numeric(data, exclude='Time_UTC')

Coerce columns to numeric.

Parameters:

• data (pd.DataFrame) – The data DataFrame.

• exclude (str | Sequence[str]) – Columns to exclude from coercion.

Returns:

The data DataFrame with columns coerced to numeric.

Return type:

pd.DataFrame

class uataq.filesystem.groupspaces.horel.HorelH5File(path, instrument)

Class for parsing H5 files from the Horel group.

path

The file path.

Type:

str

period

The period of the data file.

Type:

pd.Period

logger

The logger name.

Type:

str

date_slicer

A slice object to extract the date from the file name.

Type:

slice

file_freq

The file frequency.

Type:

str

ext

The file extension.

Type:

str

time_col

The time column name.

Type:

str

instrument

The instrument name.

Type:

str

usecols(col)

Check if a column should be used based on the instrument.

convert_nodata(data, nodata=-9999.0)

Convert NoData values to NaN.

coerce_numeric(data, exclude='Time_UTC')

Coerce columns to numeric.

logger: str = 'campbellsci'

ext: str = 'h5'

time_col: str = 'EPOCHTIME'

parse()

Parse the H5 file and return a DataFrame.

Returns:

A DataFrame containing the parsed data.

Return type:

pd.DataFrame

class uataq.filesystem.groupspaces.horel.HorelCSVFile(path, instrument)

Class for parsing CSV files from the Horel group.

path

The file path.

Type:

str

period

The period of the data file.

Type:

pd.Period

logger

The logger name.

Type:

str

date_slicer

A slice object to extract the date from the file name.

Type:

slice

file_freq

The file frequency.

Type:

str

ext

The file extension.

Type:

str

time_col

The time column name.

Type:

str

instrument

The instrument name.

Type:

str

usecols(col)

Check if a column should be used based on the instrument.

convert_nodata(data, nodata=-9999.0)

Convert NoData values to NaN.

coerce_numeric(data, exclude='Time_UTC')

Coerce columns to numeric.

parse()

Parse the CSV file and return a DataFrame.

logger: str = 'horel-pipeline'

ext: str = 'csv.gz'

time_col: str = 'Timestamp'

parse()

Parse the CSV file and return a DataFrame.

Returns:

A DataFrame containing the parsed data.

Return type:

pd.DataFrame

class uataq.filesystem.groupspaces.horel.HorelCSVFinalizedFile(path, instrument)

Class for parsing finalized CSV

path

The file path.

Type:

str

period

The period of the data file.

Type:

pd.Period

logger

The logger name.

Type:

str

date_slicer

A slice object to extract the date from the file name.

Type:

slice

file_freq

The file frequency.

Type:

str

ext

The file extension.

Type:

str

time_col

The time column name.

Type:

str

final_patterns

A list of patterns to filter columns.

Type:

list[str]

instrument

The instrument name.

Type:

str

usecols(col)

Check if a column should be used based on the instrument.

convert_nodata(data, nodata=-9999.0)

Convert NoData values to NaN.

coerce_numeric(data, exclude='Time_UTC')

Coerce columns to numeric.

parse()

Parse the CSV file, finalize the data, and return a DataFrame.

final_patterns = ['UTC', 'Ambient', 'ppb', 'ugm3', 'deg', 'm_s']

parse()

Parse the CSV file, finalize the data, and return a DataFrame.

Returns:

A DataFrame containing the finalized data.

Return type:

pd.DataFrame

class uataq.filesystem.groupspaces.horel.HorelGroup

A class representing the Horel group data space in the CHPC UATAQ filesystem.

name

The group name.

Type:

str

datafiles

A dictionary mapping datafile keys to DataFile classes.

Type:

dict[str, Type[DataFile]]

get_highest_lvl(SID, instrument)

Get the highest data level for a given site and instrument.

get_files(SID, instrument, lvl, logger)

Get list of file paths for a given site, instrument, and level.

get_datafile_key(instrument, lvl, logger)

Get the datafile key based on the instrument, level, and logger.

get_datafiles(SID, instrument, lvl, logger, time_range, pattern=None)

Returns a list of data files for a given level and time range.

name: str = 'horel'

datafiles: dict[str, type[DataFile]] = {'final': <class 'uataq.filesystem.groupspaces.horel.HorelCSVFinalizedFile'>, 'qaqc': <class 'uataq.filesystem.groupspaces.horel.HorelCSVFile'>, 'raw': <class 'uataq.filesystem.groupspaces.horel.HorelH5File'>}

static get_highest_lvl(SID, instrument)

Return “final” when the site has finalized files, else “raw”.

Return type:

str

get_files(SID, instrument, lvl, logger='campbellsci')

List horel files for the instrument.

Maps UATAQ instrument names onto the horel directory names. See GroupSpace.get_files().

Return type:

list[str]

get_datafile_key(instrument, lvl, logger)

Return the data file key.

campbellsci is the only horel logger, so the key depends on the level alone.

Return type:

str

get_datafiles(SID, instrument, lvl, logger, time_range, pattern=None)

Return a list of data files for a given level and time range.

Extends DataFile.get_datafiles by supplying the instrument name to the DataFile subclass.

Parameters:

• SID (str) – The site ID.

• instrument (str) – The instrument name.

• lvl (str) – The data level.

• logger (str) – The logger name.

• time_range (TimeRange) – The time range.

• pattern (str | None) – The pattern to match file names.

Returns:

A list of data files.

Return type:

list[DataFile]

static standardize_data(instrument, data)

Rename horel columns to UATAQ names and apply per-instrument fixes.

The fixes: GPS flags, Fahrenheit temperatures. See GroupSpace.standardize_data().

Return type:

DataFrame