Data levels and flags#

Note

The output of the examples on this page is from runs on CHPC on 12 October 2026 (why it is recorded).

The archive keeps each instrument’s data at up to four data levels: the same measurements, processed further at each level. This page says what each level holds, which level each function reads, what the flags mean and what the final level keeps. The columns of each level are on Columns.

The four levels#

From least to most processed:

raw

The instrument’s data as it was logged.

qaqc

Every row, with the pipeline’s QAQC flag in QAQC_Flag. Flagged rows are kept.

calibrated

The qaqc rows calibrated against reference tanks, with the calibration fit. Only the lin group’s CO2 and CH4 analyzers have this level.

final

The rows that pass, with the pipeline’s default flags applied: flagged rows are dropped, and for the CO2 and CH4 analyzers so are the rows that measured calibration tanks.

final is the ready-to-share level: it can go to people outside the group as it is. The lower levels are for finer control. Which one to use depends on how clean the data need to be for what you are doing, and on how much control you want over the QAQC. The pipeline describes the levels too, in its README.

For the CO2 and CH4 analyzers (the lgr_ugga, licor_6262 and licor_7000), each level holds:

Level

What it has

final

The concentrations and H2O_ppm, for the outside-air rows that passed QAQC. No ID or flag columns.

calibrated

Every row, with the calibrated and uncalibrated concentrations, the calibration fit, the ID_CO2 (and ID_CH4) columns and QAQC_Flag.

qaqc

Every row, with all of the analyzer’s columns, ID, ID_CO2 (and ID_CH4), and QAQC_Flag. Not calibrated.

raw

The analyzer’s columns as they were logged, with ID but without ID_CO2, ID_CH4 or QAQC_Flag. The lgr_ugga clocks ran on Mountain time until 29 December 2015, when they were switched to UTC: uataq converts the times logged before that day to UTC and leaves out the rows of the day itself, as the pipeline does for qaqc. Both passes through the hour that repeats when daylight saving ends (1 to 2 a.m.) are taken as daylight time, as in the pipeline.

Which level a read gives#

  • get_obs() and get_recent_obs() read final. They take no level: one given raises an error (see When there is nothing to read).

  • get_network_obs() reads the highest level each instrument has. That is final too, except at a horel vehicle with no finalized files at all, which is read at raw.

  • read_data() reads the level you give as lvl. With none, it reads the highest level the instrument has, final when it has one.

Not every instrument has every level. Only the lin group’s CO2 and CH4 analyzers have calibrated, and TRX01’s lgr_ugga_manual_cal does not. The horel group’s instruments have raw, qaqc and final, and some months of them are only at raw (see Mobile sites). Asking for a level an instrument does not have raises an error that lists the levels it has (see When there is nothing to read).

The same day at two levels#

The examples below read the lgr_ugga at WBB on 2 June 2025. final is its highest level, so read_data() reads it with no lvl:

>>> import pandas as pd
>>> import uataq
>>> final = uataq.read_data("WBB", "lgr_ugga", time_range="2025-06-02")["lgr_ugga"]
>>> final.columns.tolist()
['CO2d_ppm_cal', 'CO2d_ppm_raw', 'CH4d_ppm_cal', 'CH4d_ppm_raw', 'H2O_ppm']
>>> len(final)
8390

The qaqc level of the same day has every row the analyzer logged, with the columns that say what each row is:

>>> qaqc = uataq.read_data("WBB", "lgr_ugga", lvl="qaqc", time_range="2025-06-02")["lgr_ugga"]
>>> len(qaqc)
8845
>>> qaqc["QAQC_Flag"].value_counts()
QAQC_Flag
 0    8586
-2     259
Name: count, dtype: int64
>>> cal = uataq.read_data("WBB", "lgr_ugga", lvl="calibrated", time_range="2025-06-02")["lgr_ugga"]

The 8,845 rows are the 8,390 final ones, 259 rows flagged -2 (Flags) and 196 rows that measured calibration tanks (What the analyzer was sampling).

What the analyzer was sampling#

The analyzers do not only sample the outside air. A valve switches them to the reference tanks for a few minutes every few hours, and the rows logged while the lines are flushed after a switch are marked as well. Three columns say which is which.

ID is the label logged with each row. For the lgr_ugga it is text, the description set on the analyzer’s Multi Inlet Unit (the valve box). ID_CO2 and ID_CH4 are numbers that the pipeline makes from it:

>>> qaqc.groupby(["ID", "ID_CO2", "ID_CH4"]).size()
ID                      ID_CO2   ID_CH4
~406.49~1.91             406.49   1.91       70
~478.24~2.62             478.24   2.62       63
~520.00~5.63             520.00   5.63       63
~Atmosphere~Atmosphere  -10.00   -10.00    8390
~flush~flush            -99.00   -99.00     259
dtype: int64

ID_CO2 / ID_CH4

Meaning

-10

The outside air (ID is ~Atmosphere~Atmosphere).

-99

A flush (ID is ~flush~flush).

a positive number

A reference tank. The number is the tank’s known CO2 (or CH4) in ppm, which ID holds as ~CO2~CH4.

So for the lgr_ugga, select the outside air with ID_CO2 == -10. ID == -10 selects no rows, because ID is text.

For the licor_6262 and licor_7000, ID is already a number, with the same codes, and ID_CO2 is equal to it. They have no ID_CH4:

>>> licor = uataq.read_data("SUG", "licor_7000", lvl="qaqc", time_range="2025-06-02")["licor_7000"]
>>> licor["ID"].value_counts()
ID
-10.00     7992
-99.00      459
 398.22      60
 453.73      60
 487.55      60
 524.29       5
Name: count, dtype: int64
>>> (licor["ID"] == licor["ID_CO2"]).all()
np.True_

What the final level keeps#

The flush rows are flagged (-2), but the reference tank rows have flag 0, like the outside air:

>>> pd.crosstab(qaqc["ID_CO2"], qaqc["QAQC_Flag"])
QAQC_Flag   -2     0
ID_CO2
-99.00     259     0
-10.00       0  8390
 406.49      0    70
 478.24      0    63
 520.00      0    63

So QAQC_Flag >= 0 alone keeps the tank rows. The final level is the calibrated rows of the outside air that pass QAQC:

>>> kept = cal[(cal["ID_CO2"] == -10) & (cal["QAQC_Flag"] >= 0)]
>>> len(kept), len(final)
(8390, 8390)
>>> kept.index.equals(final.index)
True

The same holds for the licor_7000 at SUG that day (7,992 rows). The final level follows the calibrated level’s flags, which can differ from the qaqc level’s because the calibration adds flags of its own. For the licor_6262 at SUG in June 2019, the calibration flagged 566 rows -4 (517 of them outside air; the other 49 measured tanks, which the rule drops anyway), and the rule above gives the 238,632 final rows from the calibrated level, but 239,149 from the qaqc level.

Flags#

QAQC_Flag is in the qaqc and calibrated levels of the lin group’s instruments, and in the qaqc level of the horel group’s ozone and particle monitors. Zero and positive flags pass; a negative flag fails, and its number says why. The final level has no flag column: it holds only rows that passed.

What each number means is on the pipeline’s QAQC flag page, grouped by instrument, with the calibration’s own flags under Calibration. It names the columns as they are in the pipeline’s files (see Columns, “In every table”): its Flow_CCmin, for example, is uataq’s Flow_mLpm, or Flow_Lpm for the 2b_205.

The horel group’s ozone (2b_205) and particle (metone_es642, metone_es405) monitors mark a flagged row with 1, and uataq turns that into QAQC_Flag = -1, so their flag is 0 (passes) or -1 (flagged). Their final level is the qaqc rows that pass and have a value: an ozone or PM2.5 concentration. The metone_es642’s Ambient_T_C and Ambient_P_hPa (the sampler’s own temperature and pressure) stay in final, but only on rows that have a PM2.5 value. For the 2b_205 on TRX01 on 1 June 2024:

>>> o3 = uataq.read_data("TRX01", "2b_205", lvl="qaqc", group="horel", time_range="2024-06-01")["2b_205"]
>>> o3.columns.tolist()
['O3_ppb', 'Flow_Lpm', 'Internal_T_C', 'Internal_P_hPa', 'QAQC_Flag']
>>> o3["QAQC_Flag"].value_counts()
QAQC_Flag
 0    39134
-1     4066
Name: count, dtype: int64
>>> final_o3 = uataq.read_data("TRX01", "2b_205", group="horel", time_range="2024-06-01")["2b_205"]
>>> kept = o3[(o3["QAQC_Flag"] >= 0) & o3["O3_ppb"].notna()]
>>> kept.index.equals(final_o3.index)
True

The -1 of a horel row means only that the horel group flagged it. The flag numbers on the pipeline’s page are the lin group’s.

When _cal is empty#

Some rows of the final level have a _raw value and an empty _cal. There are two causes you can see in the data.

The instrument has no calibrated level. The pipeline then fills only the _raw columns. This is the case for TRX01’s lgr_ugga_manual_cal, whose _raw columns hold the value the LGR software calibrated, not an uncalibrated one:

>>> trax = uataq.read_data("TRX01", "lgr_ugga_manual_cal", time_range="2024-09-01")["lgr_ugga_manual_cal"]
>>> trax.notna().sum()
CO2d_ppm_cal       0
CO2d_ppm_raw    7200
CH4d_ppm_cal       0
CH4d_ppm_raw    7200
H2O_ppm         7200
dtype: int64

There was no calibration fit at that time (CO2d_n is 0). These rows are not flagged, so they are in the final level. A row gets a fit only after the analyzer has measured the tanks, and the first tank measurement of a new year comes some hours in. At WBB on 1 January 2025:

>>> jan1 = uataq.read_data("WBB", "lgr_ugga", time_range="2025-01-01")["lgr_ugga"]
>>> empty = jan1["CO2d_ppm_cal"].isna() & jan1["CO2d_ppm_raw"].notna()
>>> int(empty.sum()), jan1.index[empty].max()
(824, Timestamp('2025-01-01 02:14:01'))

The calibrated level shows the fit starting as the analyzer measures the three tanks, one after another. Here is the first row with each value of CO2d_n:

>>> cal = uataq.read_data("WBB", "lgr_ugga", lvl="calibrated", time_range="2025-01-01T02")["lgr_ugga"]
>>> cols = ["CO2d_ppm_raw", "CO2d_ppm_cal", "CO2d_n", "CO2d_b", "CO2d_rsq", "ID_CO2"]
>>> cal.drop_duplicates("CO2d_n")[cols]
                     CO2d_ppm_raw  CO2d_ppm_cal  CO2d_n    CO2d_b  CO2d_rsq  ID_CO2
Time_UTC
2025-01-01 02:00:00      445.0774           NaN       0       NaN       NaN  -10.00
2025-01-01 02:15:48      406.0314    406.425037       1  0.000000       NaN  406.49
2025-01-01 02:18:44      477.9684    478.134889       2 -1.685318       NaN  478.24
2025-01-01 02:21:40      519.6304    519.871474       3 -0.750496  0.999997  520.00

The same happens after the last tank measurement of a year (at WBB from 23:14 on 31 December 2024). Rows can lack a fit at other times too; CO2 and CH4 are empty on the same rows. Find them with the line below rather than assuming where they are.

To find these rows in your data:

empty = df["CO2d_ppm_cal"].isna() & df["CO2d_ppm_raw"].notna()

Their _raw value is not calibrated.