Contributing#

Contributions are welcome. CONTRIBUTING.md explains how to set up a development environment, run the checks, and propose a change. AGENTS.md is the short guide for contributors and coding agents: the rules the code keeps, the commands, and the gotchas. This page holds the development setup on CHPC and the longer notes it points to: the module layout, how to add an instrument or a group space, how the docs are organized and generated, and what the mobile clocks, the MetOne swap files, data availability and the SODAR archive need.

Development setup#

This sets up a copy of uataq to change and send back, the way its maintainer works. Three tools do the work:

  • uv installs the packages the project needs into a .venv folder inside your copy of the code, at the versions recorded in uv.lock.

  • just runs the project’s named tasks, listed in its justfile (just --list shows them). uv sync installs it, and you run it as uv run just.

  • pre-commit checks your changes for style problems each time you git commit, before the commit is made.

uataq needs uv 0.12 or newer, and CHPC’s uv module is older (0.11.14 in October 2026). Install your own copy with pip. Do this in a shell where the uataq environment is not active: run deactivate first, or pip stops with Can not perform a '--user' install.

module load python
pip install --user uv
uv --version

The last command should print uv 0.12 or newer. Then get the code and set it up as the Getting Started steps of CONTRIBUTING.md say: clone, uv sync (about two minutes on CHPC) and install the pre-commit hooks.

To run the checks that CI runs (lint, types, docstrings and the tests), use a compute node, not the login node:

srun --account=lin-np --partition=lin-np --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=8G --time=00:30:00 uv run just quality-check

It takes about a minute and ends with the number of tests that passed. The tests do not need the archive, so you can also develop on your own computer. CONTRIBUTING.md has the rest of the workflow: branches and pull requests.

To work on the code while you use it for your own analysis, install your clone into the environment from the steps above (an editable install, which runs the code in your folder as you change it):

source ~/venvs/uataq/bin/activate
cd uataq
pip install -e .

If uv sync fails with Required uv version ... does not match the running version, you are running CHPC’s uv module: run module unload uv and open a new shell so your own copy comes first.

Module layout#

src/uataq/
  __init__.py        read_data, get_obs, get_recent_obs, get_network_obs;
                     re-exports laboratory, get_site, Network, DEFAULT_GROUP,
                     TimeRange and the modules below
  config.json        sites and instruments; read at import (importlib.resources)
  _laboratory.py     Laboratory: builds sites and instruments from config.json
  sites.py           Site (stationary) and MobileSite (GPS-located rows)
  instruments.py     Instrument ABC, one subclass per model, and `catalog`
  network.py         Network: many sites, one pollutant (GeoDataFrame);
                     get_availability / plot_availability
  pollutants.py      column grammar {pollutant}[d|channel]_{units}[_cal|_raw]:
                     concentration_columns, parse_column, POLLUTANTS
  gps.py             haversine, bearing, estimate_speed_course,
                     gps_time_from_time_of_day
  sodar.py           Horel/MesoWest SODAR profiles (Sodar, list_stations);
                     standalone, not in config.json, not imported by uataq
  timerange.py       TimeRange and the TimeRangeTypes alias
  errors.py          ReaderError and its subclasses, ParserError, EmptyResultWarning,
                     DroppedPollutantWarning, ...
  filesystem/
    __init__.py      DEFAULT_GROUP, get_group; re-exports core
    core.py          DataFile, GroupSpace, groups, lvls, list_files,
                     filter_datafiles, parse_datafiles, cpu_count
    groupspaces/     one module per research group's archive (lin, horel)
tests/               pytest
docs/                Sphinx; docs/_ext/ holds two local extensions

How a read flows: laboratory (a process-wide singleton built at import) holds the Site objects; get_site(SID) returns one, and the top-level functions delegate to it. A Site asks each Instrument which group to read each part of the range from (plan_reads); the instrument gets its files from that group’s GroupSpace (filesystem.groups[name]), whose DataFile classes parse them; the frames are standardized to uataq’s column names and concatenated in time. In one Site.read_data call, a file that several instruments read is parsed once (horel’s finalized CSVs hold a whole platform): filesystem.core._parse_once holds a dict, by path, until the call returns, and HorelCSVFile.parse keeps its parse there for the other instruments, which copy their own columns. filesystem.lvls ranks the levels (raw < qaqc < calibrated < final).

Adding an instrument or a group space#

A change at a site (an instrument installed, removed or swapped) must be written in config.json, or uataq cannot read the data; Configuration lists its keys.

A new instrument. An Instrument subclass in instruments.py (and a DataFile in the group’s module if the format is new), its entry in config.json with dates and notes, and parser tests on synthetic lines.

A new group space. Add filesystem/groupspaces/<name>.py with a GroupSpace subclass, its DataFile classes and a column_mapping dict (the group’s column names to uataq’s, applied by its standardize_data), modeled on lin.py and horel.py, and register it at the bottom of that module: filesystem.groups["<name>"] = <Name>Group(). Nothing else needs to import it: groupspaces/__init__.py star-imports every non-underscore module in the folder. Its API page appears by itself (“Functions and modules” lists filesystem recursively).

Writing the docs#

Each fact is written on one page, and the other pages link to it (the rules for writing are in AGENTS.md). This table says which page holds which kind of fact. A change that moves a fact, or renames or retitles a page, updates its row here.

Fact

Page

What UATAQ is, a first example, how to cite the data

the front page (docs/index.rst)

Install, Jupyter on Open OnDemand, Slurm jobs, updating and pinning

Installation

A first read, UTC and local time, hourly means, what to write down

Your first data

Why the examples show recorded output: they read the archive and are not run in the build

Your first data

Which sites measure what, when a site has data, the newest data

Find sites and data

Several pollutants and sites, get_network_obs, read_data, large reads

Read data

The data levels, which level a read gives, ID, what final keeps, flags, empty _cal

Data levels and flags

Mobile sites: positions, the two clocks, TRX01’s two LGR names, raw-only months

Mobile sites

Errors and empty results

When there is nothing to read

Sites, instruments, their dates, glossary

Sites and instruments

Columns of each instrument and level, column names

Columns

Arguments: site, instruments, pollutants, format, group, level, time range (every form), processes

Input parameters

Config keys

Configuration

Development setup: clone, uv sync, pre-commit hooks, the checks, branches and pull requests

CONTRIBUTING.md

What CHPC changes in that setup, and the developer notes

Contributing (this page)

Function and class details, what a module is for

the docstrings, on the generated API pages (Functions and modules)

The front page (docs/index.rst) lists four sections, which are the navbar: Getting started, Guides and Reference (each a folder of docs/ with an index.rst that lists its pages, in the left sidebar) and this page. A new page goes in the folder of its section and on that section’s index.rst.

Examples that run in the build. In an .. ipython:: python block, @savefig name.png above a plot keeps the figure, and :okexcept: or :okwarning: allows a block meant to raise or warn.

Examples that need the archive. These pages show the output of a run on CHPC in plain code-block or pycon blocks, because the docs build (and CI) has no archive: your first data, find sites and data (except its first section), read data, data levels and flags, mobile sites, errors, columns and input parameters.

The site and instrument tables are generated. The site-catalog and instrument-catalog directives in docs/reference/sites_and_instruments.rst (docs/_ext/catalog.py) build them from uataq.laboratory and uataq.instruments.catalog on every build.

The API reference is generated. docs/reference/functions_and_modules.rst lists the functions and modules, and each module, class, function and class member gets a page in docs/api/ (gitignored; a :toctree: pointing outside docs/ litters the repo root). A module’s docstring is its page’s text. docs/_templates/autosummary/ and docs/_ext/api_pages.py (from python-template, which keeps class members from being described twice) build the pages.

Clocks on mobile platforms#

The TRAX and bus platforms carry instruments from two groups, each stamped by its own logger clock:

  • lin: the instruments and the GPS are stamped by the platform’s Pi computer, whose clock cannot always be trusted. The Pi’s time is kept as Pi_Time, and lin’s GPS supplies Time_UTC, which is GPS time.

  • horel: Time_UTC is the CR1000 logger’s clock, which runs ahead of GPS time (by how much: Two groups, two clocks). horel’s raw GPS files log the receiver’s own time of day (GTIM, read as Instrument_Time), so read_data() adds GPS_Time_UTC for horel, dated from the logger clock across midnight and NaT on rows without a fix. The index stays on the logger clock, so it still lines up with the logger’s other columns. horel’s qaqc and final files have no GTIM: their ozone and particulate rows stay on logger time, and nothing in them can correct it.

locate() therefore joins each group’s rows to that group’s own GPS on its own clock: lin’s on Pi_Time, horel’s on Time_UTC (there the instrument and GPS values are one record). Joining horel’s rows to lin’s GPS on the Pi clock would place them that many seconds along the track, and drop the rows with no lin GPS record at their second. locate replays each instrument’s plan_reads to split its frame by group, and _gps_group raises ValueError rather than borrow another group’s GPS.

The Pi’s clock has itself been hours to days off GPS time (on TRX01 about 70 minutes behind all of 2016-11-09 and 10, and up to 5 days in 2022-11). A GPS read selects fixes by Time_UTC, GPS time, while lin’s rows were selected by the Pi’s clock, so locate reads lin’s GPS a week past the rows’ range on each side (_PI_CLOCK_MARGIN in sites.py); otherwise the rows near the ends of a range would find no fix and be dropped. A located lin row’s Time_UTC can then fall outside the requested range by as much as the Pi was off.

Each located row carries GPS_Group ("lin" or "horel"): which GPS located it, and so which clock its Time_UTC is on. A TRAX read with no group can mix the two groups’ GPS, which log at different rates and with different columns, so filter or compare by it. A GPS group that cannot be read leaves only its own rows unlocated, with a warning; if none can be read, locate raises ReaderError.

The MetOne swap files#

At five MetOne swaps (ES642 to ES405) horel split the month’s raw files in two: {SID}_{YYYY}_{MM}_esampler.h5 holds the ES405’s rows from the swap on, and {SID}_{YYYY}_{MM}_esampler.h5_prior405upgrade the ES642’s rows from the start of the month to the swap (which months, and that they are raw only: Months that are only raw). TRX01 to TRX03’s 2b and cr1000 folders have the same split for January 2025, so their ozone and GPS rows before the swap are only in a prior405upgrade file too. The two files of a month share no timestamp.

The suffix is uataq.filesystem.groupspaces.horel.PRIOR_405_SUFFIX. HorelGroup.get_datafiles adds the suffixed file to every raw read except the ES405’s, and clipping each instrument to its installed window keeps the two MetOne models’ rows apart. The clipped range drives both the file selection and the row slice, and both are needed: groups reuse a file name across an instrument change (horel calls both MetOne models esampler), so a monthly file that straddles a swap holds the replacement’s rows. At each swap the ES642’s removal_date and the ES405’s installation_date are the same instant, so a sample at that instant belongs to the ES405 alone. Other suffixes in horel’s raw folders (TRX01_2024_02_cr1000.h5_bad, which is empty, and the pilot archive’s cr1000/old/*.h5_old) are not read. The four LI-COR 6262 to 7000 swaps (DBK, HEB, RPK, SUG) are in separate lin folders, so their files cannot mix, but the same date rule holds. BUS01’s swap is a sixth MetOne swap with no split files: it fell in a gap of several months with no file at all, and the instant is the ES405’s first raw row.

For some months after the swaps at BUS01 and BUS02, horel’s qaqc and final files name the ES405’s values with the ES642’s column names (ES642_PM2.5_Concentration). The ES405’s es642_column_dates in config.json says when (Configuration): a file read for the ES405 takes those columns as its own for rows inside that window, and drops them outside it. BUS02’s window starts at the ES405’s installation: its files from March to May 2025 hold the ES405’s flow, temperature, humidity, pressure and error code under the ES642’s names, but their PM2.5 under that name is empty, so qaqc has those diagnostics and no PM2.5 for the period. MetOne_ES405.get_datafiles hands the window to the files.

The pilot archive (uutrax_pilot) and the main one (uutrax) both have a raw file for November 2018, the month TRX01’s and TRX02’s pilot phase ended. The pilot file goes on to the end of the month with a copy of the main file’s row once a minute, so HorelH5File.parse keeps a pilot file’s rows only before the stop in uataq.filesystem.groupspaces.horel.PILOT_PHASE.

Data availability internals#

get_availability() parses every file rather than trusting file listings: horel’s qaqc and final files hold a whole platform, so a file existing says nothing about one pollutant (it can hold -9999 ozone for most of a month while the raw 2B file is full).

  • A time bin counts for a level when any row in it has a non-null value in a column concentration_columns() picks (O3_ppb, CO2d_ppm_cal, CO2d_ppm_raw, BC6_ngm3; not diagnostics such as O3_Meas_mV or O3_ppb_std). _raw columns count (raw=True), as in Network.get_obs: they are the only populated ones for TRX01’s lgr_ugga_manual_cal. Each bin takes the highest level present, so “measured but not in final” shows as qaqc or raw.

  • With no group, every group operating the instrument is checked, ignoring group_dates: availability reports what each archive holds, while get_obs reads one group per portion.

  • One pool runs over the whole network’s files, and each worker reduces its file to bins at once, so memory is one file per worker. A file that cannot be read is logged and skipped, and the count is reported. Rows with a NaT Time_UTC are dropped before standardize_data.

  • plot_availability() (matplotlib, an optional import) colors levels by uataq.network.LEVEL_COLORS, one hue from light (raw) to dark (final). Compute the availability once and pass it in to restyle.

  • Network("all", pollutant) is every configured site that measures it. Ozone at every site over every year is thousands of files: run it in a Slurm job with num_processes="max", from a script file (see A year or more).

The SODAR archive#

uataq.sodar reads the Horel group’s MesoWest SODAR archive, under sodar_data/hdf5archive/ in uataq.filesystem.groupspaces.horel.MESOWEST_DIR:

  • {SID}_full_metadata_log.h5: /metagroup/metadata and /metagroup/variables, the variables indexed by SHORTNAME with a MULT scale factor each.

  • {SID}_{YYYY}_{MM}_sodar.h5: the table /obsdata/observations, with STATION_ID, DATTIM (epoch seconds) and one (n_levels,) integer column per profile field (HEIGHT, WS, WD, MXWS). -9999 becomes NaN, and each variable in the metadata log is divided by its MULT.

Stations seen: USDR1 to USDR5, WSLSA, WSMLI. Stray files (USDR2_2018_12_sodar_p1.h5, all_sodar_metadata.h5, _Processed/) do not match the exact-name pattern and are ignored. Monthly files are concatenated along Time_UTC with an outer join, which pads with NaN if the number of range gates changes; get_winds_at_height() matches the height at every timestamp.

SODAR is standalone, outside config.json, Instrument and GroupSpace, because its stations are MesoWest stations rather than UATAQ sites, its output is two-dimensional (Time_UTC, level), which the pandas DataFile pipeline cannot carry, and its scale factors live in a per-station file outside the data file. Only lvl="raw" exists; any other level raises ValueError.