fips.Posterior.data_worth#

Posterior.data_worth(by, W)[source]#

Return what each group of observations adds to the certainty of each aggregate, and its share of DOFS.

For each group (a site, a platform, a month), the posterior standard deviation of each aggregate without the group’s observations and with them (this posterior), without solving again. Removing the observations \(g\) changes the posterior covariance by a rank \(m_g\) update,

\[\hat{S}_{-g} = \hat{S} + K_g P_{gg}^{-1} K_g^T, \qquad P = (H S_0 H^T + S_z/\gamma)^{-1},\]

with \(K_g\) the gain’s columns for the group and \(P_{gg}\) its block of \(P\) (the block inverse of the innovation covariance). It is exact whatever the model-data mismatch, also when it correlates the errors of one group with another’s: it is the posterior of the problem without the group’s rows and columns. For \(W\), \(W K_g\) is a part of \(W K\) (k x m), so nothing n x n is formed.

A group’s share of DOFS is the sum of \((H K)_{ii}\) over its observations (the observation self-sensitivity, Cardinali et al., 2004); the shares of groups that hold every observation once sum to dofs.

What a group is worth by itself is the other reading: the aggregate’s standard deviation with only the group’s observations, on the prior, from a rank \(m_g\) update of \(S_0\),

\[S_{g} = S_0 - S_0 H_g^T F_g^{-1} H_g S_0, \qquad F_g = H_g S_0 H_g^T + R_{gg} / \gamma,\]

with \(H_g\) the group’s rows of \(H\) and \(R_{gg}\) its own block of the model-data mismatch. It is not the difference of the other columns: two groups that see the same part of the state each take much off the prior alone and little on top of one another, and the two readings agree only for a single group. Because it uses \(R_{gg}\) only, it ignores any correlation between the group’s errors and the other groups’, which std_without and std_with keep: a problem whose mismatch correlates sites gives each site, alone, the benefit of independent errors from the rest.

Parameters:

  • by (Any) – The levels of the observations’ labels to group by, as desroziers() and pandas.DataFrame.groupby() take them: "site", "block" (a platform per block), ["site", pd.Grouper(level="obs_time", freq="MS")]. Observations whose block lacks a level are in no group of it (they are never removed).

  • W (Operator) – An aggregation, as cov() takes it: one row per aggregate (a regional annual total, a trend as C @ W). For every state element, the identity on the state’s space.

Returns:

One row per group and aggregate, indexed by the group’s levels and then the aggregation’s rows (a group level with the name of one of those, block, is called obs_block): n_obs (the group’s observations), dofs (its share of DOFS), std_without (the aggregate’s posterior standard deviation without the group), std_with (with it: uncertainty()’s posterior_std) and reduction, \(1 - \sigma_{with} / \sigma_{without}\), the fraction the group takes off on top of the other observations; std_alone (the standard deviation with only the group’s observations, on the prior) and reduction_alone, \(1 - \sigma_{alone} / \sigma_{prior}\), the fraction the group takes off the prior by itself (prior_std in uncertainty()). A variance below zero by more than roundoff, which only a covariance that is not positive semidefinite can give, makes its standard deviation and the reduction NaN, with a logged warning (as in uncertainty()), and so does one in the alone column or in the prior’s.

Return type:

DataFrame

Raises:

  • TypeError – If W is not an Operator, or the method cannot give the blocks of \(P\) and the self-sensitivity (its solution lacks innovation_precision or self_sensitivity).

  • ValueError – If the columns of W do not have exactly the state’s labels, or by puts no observation in a group.

See also

uncertainty

The prior and posterior uncertainty of each aggregate.

candidate_worth

What observations not in the problem would add.

dofs

The degrees of freedom for signal, the sum of the shares.

References

Cardinali, C., Pezzulli, S., & Andersson, E. (2004). Influence-matrix diagnostic of a data assimilation system. Q.J.R. Meteorol. Soc., 130, 2767-2786.

Examples

One flux observed at two sites, three times at A and once at B; the flux’s own row (W the identity):

>>> import numpy as np
>>> import pandas as pd
>>> import fips
>>> state = fips.Space({"flux": pd.Index(["a"], name="cell")})
>>> labels = pd.MultiIndex.from_tuples(
...     [("A", 0), ("A", 1), ("A", 2), ("B", 0)], names=["site", "hour"]
... )
>>> observed = fips.Space({"co2": labels})
>>> problem = fips.Problem(
...     prior=fips.Vector(np.zeros(1), state),
...     prior_cov=fips.Covariance.diagonal(1.0, state),
...     obs=fips.Vector(np.ones(4), observed),
...     obs_cov=fips.Covariance.diagonal(1.0, observed),
...     forward=fips.Operator(np.ones((4, 1)), observed, state),
... )
>>> post = fips.solve(problem)
>>> W = fips.Operator(np.eye(1), state, state)
>>> table = post.data_worth(by="site", W=W).droplevel("block").reset_index()
>>> table.iloc[:, :7].round(3)
  site cell  n_obs  dofs  std_without  std_with  reduction
0    A    a      3   0.6        0.707     0.447      0.368
1    B    a      1   0.2        0.500     0.447      0.106

Alone on the prior, site A (three observations) leaves the flux at a standard deviation of 0.5 and site B (one) at 0.707, taking 0.5 and 0.293 off the prior’s 1; on top of the other site, B adds only 0.106 and A 0.368:

>>> table[["site", "std_alone", "reduction_alone"]].round(3)
  site  std_alone  reduction_alone
0    A      0.500            0.500
1    B      0.707            0.293