Estimate a background#
This example estimates the methane background under two years of made-up hourly data from a city site, with each of lair’s background methods, and checks each against the background the data were made from. Three rules come out of it:
Each method finds the clean part of the data in its own way.
rolling_baseline()andphase_shift_corrected_baseline()take a low quantile over a window.thoning()fits a smooth curve through every value it is given, spikes included, so give it data that is already clean, such as afternoon means or a clean site’s record.A rolling window must be longer than the longest stretch without clean air. Over a pollution episode of several days, the low quantile of a day is the episode, not the background.
Select the afternoon in local time. The hours of a UTC series are not the hours of the day at the site: put the data in local time first (see the clock page).
Note
This page does not run when the docs build. lair.background needs NOAA’s
CCG filter, which lair downloads the first time it is imported (see
The first import downloads a file), and scipy, from the science extra
(see Optional dependencies). The output below was recorded from one
run of this code on 2026-10-29, with lair 2026.12.11.dev9+g55e932b18.
The methods#
Function |
Needs |
Returns |
Use it for |
|---|---|---|---|
a |
the rolling low quantile, smoothed with a rolling mean over the same window, at the times of the data |
a continuous record from one site, when every window holds some clean air |
|
a |
a low quantile from windows ahead of and behind each time, smoothed, at the times of the data |
one-second data from a moving platform, whose background changes within the hour |
|
a |
NOAA’s smooth curve (a polynomial trend and yearly harmonics, plus the
residuals smoothed) at |
a record that is already background: afternoon means, flasks, a clean site |
|
CarbonTracker’s files and a set of points, such as the ends of STILT back trajectories |
the mean mole fraction at the points, in ppm |
air from upwind, with no measurement from the site (see From upwind) |
The data#
Two years of hourly methane, in ppm, stamped in UTC. The true background rises 0.01 ppm a year and peaks in winter. On top of it, the city adds an enhancement: large at night, under the night-time inversion, and small in the afternoon. For six days in January 2023 a winter inversion holds the city’s methane in all day.
In [1]: import numpy as np
...: import pandas as pd
...:
...: rng = np.random.default_rng(0)
...: times = pd.date_range("2022-01-01", "2023-12-31 23:00", freq="1h") # UTC
...: years = (times - times[0]).days / 365
...: true = pd.Series(
...: 1.95 + 0.01 * years + 0.008 * np.cos(2 * np.pi * (times.day_of_year - 30) / 365),
...: index=times,
...: )
...: hour = (times.hour - 7) % 24 # Mountain Standard Time
...: night = (hour >= 19) | (hour < 9)
...: episode = (times >= "2023-01-20") & (times < "2023-01-26")
...: enhancement = rng.exponential(np.where(night, 0.10, 0.01)) + np.where(episode, 0.15, 0)
...: ch4 = (true + enhancement + rng.normal(0, 0.002, times.size)).rename("ch4")
...: ch4.describe().round(3)
Out[1]:
count 17520.000
mean 2.023
std 0.089
min 1.943
25% 1.968
50% 1.985
75% 2.047
max 2.826
Name: ch4, dtype: float64
A rolling low quantile#
rolling_baseline() needs nothing but the data. It takes
the 1st percentile over a window centered on each time, then smooths it with a
rolling mean over the same window:
In [2]: from lair.background import rolling_baseline
...:
...: day = rolling_baseline(ch4) # window="24h", q=0.01
...: two_weeks = rolling_baseline(ch4, window="14d")
...: check = pd.DataFrame({"true": true, "24h": day, "14d": two_weeks})
...: check.loc["2023-01-18":"2023-01-27"].at_time("19:00").round(3)
Out[2]:
true 24h 14d
2023-01-18 19:00:00 1.968 1.969 1.967
2023-01-19 19:00:00 1.968 1.978 1.967
2023-01-20 19:00:00 1.968 2.104 1.967
2023-01-21 19:00:00 1.968 2.118 1.968
2023-01-22 19:00:00 1.968 2.120 1.968
2023-01-23 19:00:00 1.969 2.120 1.968
2023-01-24 19:00:00 1.969 2.119 1.968
2023-01-25 19:00:00 1.969 1.999 1.968
2023-01-26 19:00:00 1.969 1.967 1.968
2023-01-27 19:00:00 1.969 1.969 1.968
Outside the episode both windows hold clean afternoons, and both follow the true background. During the episode no hour of the day is clean, so the 24-hour window rises with the city’s methane. The 14-day window still holds clean days on either side.
Afternoon means and a smooth curve#
thoning() fits NOAA’s curve (see
The first import downloads a file for where it comes from) through every value it is
given. Given every hour, it follows the mean of the data, enhancements
included. Given afternoon means, it follows the afternoon, when the city’s
methane is mixed through a deep boundary layer.
The data are in UTC, so put them in local time first, as the clock page does, then keep the afternoon hours and average each day:
In [3]: from lair.background import thoning
...: from lair.clock import AFTERNOON, UTC2MST
...:
...: local = UTC2MST(ch4.to_frame(), driver="pandas")["ch4"]
...: afternoon = local[local.index.hour.isin(AFTERNOON)].resample("D").mean()
...: afternoon.head(3).round(3)
Out[3]:
2022-01-01 00:00:00-07:00 1.965
2022-01-02 00:00:00-07:00 1.965
2022-01-03 00:00:00-07:00 1.974
Freq: D, Name: ch4, dtype: float64
Each day is labeled at its local midnight. On the UTC series the same lines
would average 12:00 to 16:59 UTC, which is 5 to 9 in the morning in Utah, under
the night-time inversion. smooth_time evaluates the curve at the hourly
times, to put it beside the others:
In [4]: every_hour = thoning(ch4)
...: afternoons = thoning(afternoon, smooth_time=ch4.index)
...: int(afternoons.isna().sum())
Out[4]: 23
The 23 missing values are the hours before the first afternoon mean and after the last: the curve is NaN outside the time range of its data.
Side by side#
How far above the true background each estimate is, on average over every hour, and over the hours of the episode:
In [5]: methods = pd.DataFrame(
...: {
...: "rolling_baseline, 24h": day,
...: "rolling_baseline, 14d": two_weeks,
...: "thoning, every hour": every_hour,
...: "thoning, afternoon means": afternoons,
...: }
...: )
...: above = methods.sub(true, axis=0)
...: pd.DataFrame({"all hours": above.mean(), "episode": above[episode].mean()}).round(3)
Out[5]:
all hours episode
rolling_baseline, 24h 0.001 0.128
rolling_baseline, 14d -0.001 -0.001
thoning, every hour 0.063 0.084
thoning, afternoon means 0.011 0.035
The curve through every hour sits above the background by about the mean enhancement, night-time plumes included. The curve through the afternoon means sits above it by about the afternoon’s enhancement, which is small at this made-up site and larger at a real site in the middle of a city. Both rise during the episode, when the afternoons are not clean either. The rolling low quantile is close whenever its window holds clean air.
One-second data#
phase_shift_corrected_baseline() is for one-second data
from a moving platform, a car or a train, whose background can change within
the hour. Here, two hours of it: a background that rises 0.05 ppm over 20
minutes, and 30 short plumes.
In [6]: from lair.background import phase_shift_corrected_baseline
...:
...: seconds = pd.date_range("2024-06-01 18:00", periods=7200, freq="1s") # UTC, 1 Hz
...: i = np.arange(seconds.size)
...: true_drive = 1.98 + 0.05 * np.clip((i - 3000) / 1200, 0, 1)
...: plumes = np.zeros(seconds.size)
...: for center in rng.integers(0, seconds.size, 30):
...: width = rng.integers(5, 30) # seconds
...: plumes += rng.exponential(0.3) * np.exp(-0.5 * ((i - center) / width) ** 2)
...: drive = pd.Series(true_drive + plumes + rng.normal(0, 0.003, i.size), index=seconds)
...:
...: fast = pd.DataFrame(
...: {
...: "phase_shift_corrected_baseline": phase_shift_corrected_baseline(drive),
...: "rolling_baseline, 1h": rolling_baseline(drive, window="1h"),
...: }
...: )
...: fast.sub(true_drive, axis=0).resample("10min").mean().round(3)
Out[6]:
phase_shift_corrected_baseline rolling_baseline, 1h
2024-06-01 18:00:00 0.010 -0.006
2024-06-01 18:10:00 -0.001 -0.006
2024-06-01 18:20:00 -0.004 -0.006
2024-06-01 18:30:00 -0.003 -0.006
2024-06-01 18:40:00 0.003 -0.005
2024-06-01 18:50:00 0.004 -0.016
2024-06-01 19:00:00 -0.006 -0.035
2024-06-01 19:10:00 -0.009 -0.040
2024-06-01 19:20:00 -0.007 -0.032
2024-06-01 19:30:00 -0.007 -0.025
2024-06-01 19:40:00 -0.006 -0.019
2024-06-01 19:50:00 -0.006 -0.012
Both use a one-hour window. While the background is flat, both stay within about 0.01 ppm of it. Where it rises, from 18:50 to 19:10, the rolling quantile, centered on each time, falls up to 0.04 ppm below it and is still 0.01 ppm below at the end. The phase shift corrected baseline, which looks ahead and behind separately, stays within about 0.01 ppm.
From upwind#
The methods above estimate the background from the site’s own data. The other
way is to take it from upwind: sample a global model, such as NOAA’s
CarbonTracker, where the air arriving at the site came from. That needs no
clean hour at the site, and it is the way lair’s maintainer took for the Salt
Lake Valley methane inversion of his first paper:
CarbonTracker-CH4 version CT-CH4-2025, sampled with
background() at the ends of 24-hour STILT back
trajectories released from the site in the afternoon, and averaged to one value
per day. How to sample CarbonTracker at a set of points is on the
CarbonTracker page.
Next#
thoning_filter() returns NOAA’s filter itself, which
also gives the trend and the growth rate. For mobile data on a fixed route,
lair.transects.enhancement() subtracts each transit’s own low percentile
instead.