Sequential Statistical Methods

for Reliable Machine Learning

Jeremias Sulam

Trustworthy AI Workshop · Simons Institute · October 2026

Why audit deployed models

Sepsis alerts

Epic’s sepsis model at Michigan Medicine: AUC 0.63 against the 0.76–0.83 reported, 67% of cases missed.

COVID-19

Sepsis alerts up 43% across 24 hospitals in four US health systems — with 35% fewer patients in the hospital (electives cancelled).

GPT-4

GPT-4 identifying prime numbers: 84% accurate in March 2023, 51% in June.

Checking a model once, before deployment, is not enough.

[Wong et al, 2021a,b][Chen, Zaharia & Zou, 2023]

Auditing a model

Each prediction gives an audit score Z_t \in [-1,1], e.g. Z_t = f(X_t) - Y_t, i.i.d. from some P.

Is \E_P[Z] = 0? That is, does the model work as intended?

We need a sequential test \phi_t \in \{0,1\} (1: reject, flag the model): \sup_{P:\,\E_P[Z]=0}\ P\big(\exists\, t:\ \phi_t = 1\big)\ \le\ \alpha

[Chugg et al, 2023][Dai et al, 2025]

Testing by betting

Is the coin fair? Heads Z_t = +1, tails Z_t = -1. Start at W_0 = 1, bet \lambda_t:

W_t = W_{t-1}\,(1 + \lambda_t Z_t), \qquad \lambda_t \text{ chosen before seeing } Z_t

t my bet \lambda_t flip W_t
1 0 (I know nothing) H 1.00
2 0.2 (lean heads) H 1.20
3 0.3 (more) T 0.84
4 0.15 (back off) H 0.97
5 0.2 H 1.16

[Shafer, 2021][Waudby-Smith & Ramdas, 2024]

Testing by betting

If the coin is fair, no strategy makes money on average:

\E[W_t \mid Z_1,\dots,Z_{t-1}] = W_{t-1}

So (Ville) \quad P_{H_0}\big(\exists\, t:\ W_t \ge 1/\alpha\big) \le \alpha.

Reject when W_t \ge 1/\alpha. Same for any bounded Z_t with H_0:\ \E[Z]=0.

[Ville, 1939]

Stopping time and power

Stopping time \ \tau = \min\{t:\ W_t \ge 1/\alpha\}. \quad Powerful test \iff small \E[\tau].

How to choose \lambda_t? ONS, universal portfolios, mixtures, aGRAPA, …

With ONS, \ \ln W_t \gtrsim \Delta^2 t/4 for \Delta = \E[Z], so that \ \E[\tau] = \mathcal O\big(\tfrac{1}{\Delta^2}\ln\tfrac{1}{\alpha\Delta^2}\big).

[Cutkosky & Orabona, 2018][Chugg et al, 2023][Waudby-Smith & Ramdas, 2024]

Two questions

Q1: How can we quickly test if a model deployed across many locations or groups is not working as expected?

global hypothesis testing

Q2: How do we quantify uncertainty in predictors over different groups making minimal assumptions and with potential shifts?

online conformal prediction

In both cases we want guarantees that

  • hold at any time, since we check continuously,
  • hold for every group, not just on average,
  • need no tuning, since we can’t anticipate the shift.

Q1: How can we quickly test if a model deployed across many locations or groups is not working as expected?

global hypothesis testing

Q2: How do we quantify uncertainty in predictors over different groups making minimal assumptions and with potential shifts?

online conformal prediction

Groups

A group is a function \ c:\mathcal X \to [0,1]: how much x belongs to it.

imaging modality

  • chest X-ray
  • lung CT
  • mammogram
  • retinal OCT

demographics

  • female, 50+
  • male, under 30
  • on Medicare

soft membership

  • P(\text{smoker} \mid x)
  • P(\text{Medicaid} \mid x)

c(X_t) = \big(\underset{\text{chest X-ray}}{1},\ \underset{\text{lung CT}}{0},\ \underset{\text{female, 50+}}{1},\ \underset{P(\text{smoker}\mid x)}{0.5},\ \dots\big) Groups may overlap.

Auditing every group

At each time t we observe one audit score per group, Z_{1,t},\dots,Z_{k,t}.

H_0:\ \E[Z_i] = 0\ \text{ for all } i \qquad\text{vs.}\qquad H_1:\ \E[Z_i] \ne 0\ \text{ for some } i

Bonferroni

Track the max-wealth over all groups: reject when \ \tfrac{1}{\cR{k}}\max_{i} W_{i,t} \ge \tfrac1\alpha

Stopping time: \ \E[\tau] = \mathcal O\big(\tfrac{1}{\Delta_{\max}^2}\ln\tfrac{\cR{k}}{\alpha\,\Delta_{\max}^2}\big)

  • threshold \cR{k}/\alpha: +\ln \cR{k} in time, however many groups are non-null
  • only the largest wealth counts

[Chugg et al, 2023]

Merging evidence

Product W^{\text{prod}}_t = \prod_{i=1}^k W_{i,t} independent across groups

Average W^{\text{ave}}_t = \frac1k\sum_{i=1}^k W_{i,t} any dependence across groups

Streams are i.i.d. in time.

Both are nonnegative martingales under H_0, so both reject at 1/\alpha.

How fast does each one stop?

[Vovk & Wang, 2021][Ramdas & Wang, 2025]

Sparse and dense alternatives

Expected stopping times

sparse (one group) dense (all groups)
Bonferroni \ln\frac1\alpha + \ln k \ln\frac1\alpha + \ln k
Product \ln\frac1\alpha + {\color{cc0000}k\ln k} {\color{1baf7a}\frac1k}\ln\frac1\alpha
Average \ln\frac1\alpha + \ln k \ln\frac1\alpha
Balanced \ln\frac1\alpha + \ln k {\color{1baf7a}\frac1k}\ln\frac1\alpha

leading terms for large k, small \alpha, fixed signal

W^{\text{bal}}_t = \tfrac12\, W^{\text{ave}}_t + \tfrac12\, W^{\text{prod}}_t \quad\Longrightarrow\quad \E[\tau] \le \min\{T_{\text{prod}},\, T_{\text{bonf}}\} under independence of streams

Note: Bonferroni also provides strong FWER control; merging does not.

[Bharti, Pal, S., 2026]

Zero-shot medical imaging

ConceptCLIP (medical vision–language model), zero-shot, Z = f(X) - Y, k = 10 groups: brain CT, brain MRI, chest X-ray, mammogram, retinal OCT, colon endoscopy, … Eight are biased; after debiasing, only colon endoscopy.

8 of 10 non-null: product and balanced first

1 of 10 non-null: Bonferroni and average first

[Nie et al, 2025][Kim et al, 2019]

Q1: How can we quickly test if a model deployed across many locations or groups is not working as expected?

global hypothesis testing

  • One betting test per group.
  • Merge the wealth: \tfrac12 average +\ \tfrac12 product.
  • Stops nearly as fast as the best rule for either sparse or dense regimes, without knowing which.

Q1: How can we quickly test if a model deployed across many locations or groups is not working as expected?

global hypothesis testing

Q2: How do we quantify uncertainty in predictors over different groups making minimal assumptions and with potential shifts?

online conformal prediction

Online prediction intervals

Patient t arrives at the emergency department. We predict the length of stay \hat Y_t, post \mathcal I_t = [\hat Y_t - \tau_t,\ \hat Y_t + \tau_t], then observe Y_t.

Covered when \ S_t := |Y_t - \hat Y_t| \le \tau_t.

Goal: for any sequence (X_t,Y_t), \qquad \tfrac1T\textstyle\sum_{t\le T} \mathbf 1\{S_t \le \tau_t\}\ \longrightarrow\ 1-\alpha

No distributional assumptions: any shift, even adversarial.

[Gibbs & Candès, 2021]

Tracking the quantile loss online

pinball loss of the radius: \quad \ell_t(\tau) = \max\{(1-\alpha)(S_t-\tau),\ \alpha(\tau - S_t)\}

subgradient at \tau_t: g_t = \mathbf 1\{S_t \le \tau_t\} - (1-\alpha)

ACI subgradient descent: \tau_{t+1} = \tau_t - \eta\, g_t a miss widens by \eta(1-\alpha), a cover shrinks by \eta\alpha

The subgradients add up to the coverage error: \MisCov_T = \Big|\tfrac1T\textstyle\sum_{t} \mathbf 1\{S_t\le\tau_t\} - (1-\alpha)\Big| = \tfrac{1}{T}\big|\textstyle\sum_t g_t\big|

\eta needs to be set in advance and is hard to tune.

[Gibbs & Candès, 2021]

Coverage through betting

Each sample is a coin flip: miss (Y_t\notin\mathcal I_t) or cover (Y_t\in\mathcal I_t). We want misses at rate \alpha.

The coin: \ Z_t = \mathbf 1\{\text{miss}_t\} - \alpha \in \{-\alpha,\ 1-\alpha\}. Then, the wealth: W_t = W_{t-1}\,(1 + \lambda_t Z_t)

If \ \varepsilon_T = \tfrac1T\sum_{t\le T} \mathbf 1\{\text{miss}_t\} - \alpha = 0, no bettor makes money. Otherwise, the bettor makes money.

[Shafer & Vovk, 2001][Vovk, Takemura & Shafer, 2005]

Coverage through betting

A universal portfolio strategy (for choosing \lambda_t; a mixture over all constant bets) nearly matches the best constant bet:

\ln W_T \ \ge\ \max_\lambda \ln W_T(\lambda) - \tfrac12\ln(\pi T)

[Cover, 1991][Cover & Ordentlich, 1996]

Coverage through betting (proof sketch)

Idea: choose the interval size so that the bettor cannot make money.

\tau_t := \lambda_t\, W_{t-1} \qquad\text{(the amount bet on a miss)}

Then \ W_t = W_{t-1} - \tau_t\, g_t, \quad so \quad W_T = 1 - \sum_{t\le T} \tau_t\, g_t.

If coverage is wrong (\varepsilon_T \ne 0), the wealth grows: \ln W_T \ \gtrsim\ \frac{\varepsilon_T^2\, T}{2\alpha(1-\alpha)} - \tfrac12\ln(\pi T)

But the wealth is bounded by the scores: -\tau_t g_t \le (1-\alpha)\,S_t \quad\Longrightarrow\quad W_T \le 1 + (1-\alpha)\sum_{t\le T} S_t

So: \qquad |\varepsilon_T| = \mathcal O\big(\sqrt{\ln T / T}\big) \qquad \text{whenever } S_t \le D\,t^q

[Orabona & Pál, 2016][Liu, Dobriban & Orabona, 2026 (UP-OCP)]

Coverage for every group

For groups c_1,\dots,c_k with T_j = \sum_{t\le T} c_j(X_t), we want, for every j, \frac{1}{T_j}\sum_{t=1}^T \mathbf 1\{S_t \le \tau_t\}\, c_j(X_t)\ \longrightarrow\ 1-\alpha

From one group to many

As before, a goal for every group j, with T_j = \sum_{t\le T} c_j(X_t):

\frac{1}{T_j}\sum_{t=1}^T \mathbf 1\{S_t \le \tau_t\}\, c_j(X_t)\ \longrightarrow\ 1-\alpha

As before, one betting game per group.

Bettor j makes money if group j is mis-covered.

But it can’t, because we adapt the radius: \ \tau_{t,j} = \lambda_{t,j}\, W_{t-1,j}.

Problem: this provides as many radii as groups a sample belongs to.

[Liu, Dobriban & Orabona, 2026]

POGO: one bettor per group

Portfolios for Online Group cOnformal prediction

Radius linear in group membership, as in GCACI Ramalingam et al, 2025:

\tau_t(X_t) = \langle \theta_t,\ c(X_t)\rangle = \theta_{\text{X-ray}}\, c_{\text{X-ray}}(X_t) + \theta_{50+}\, c_{50+}(X_t) + \cdots

  • one universal-portfolio bettor per group, starting with W_{0,j} = 1/k
  • group j only plays when present: \ g_{t,j} = g_t\, c_j(X_t)
  • each bettor sets one coordinate: \ \theta_{t,j} = W_{t-1,j}\,\lambda_{t,j} (controls \sum_j W_{t,j})

[Ramalingam et al, 2025][Bharti, Pal, Teneggi, S., 2026]

POGO: coverage guarantee

Theorem [Bharti, Pal, Teneggi, S.] If S_t \le D\,t^q, then for every group j \MisCov_T(j)\ \lesssim\ \sqrt{\frac{2\alpha(1-\alpha)\,L}{T_j}} + \frac{L}{T_j}, \qquad L = \ln T + \ln k + C \ln T from the bound on the wealth and the universal portfolio, \ln k from starting at 1/k; C depends on D, q, \alpha. Full statement in backup.

For T_j \asymp \rho_j T: \ \MisCov_T(j) = \mathcal O\big(\sqrt{\alpha\ln(kT)/T} + \ln(kT)/T\big).

group miscoverage tuning scores
GCACI \mathcal O\big(\sqrt{(k + 1/\eta)/T}\big) \eta bounded
UP-OCP (marginal only) \mathcal O\big(\sqrt{\alpha\ln T/T} + \ln T/T\big) none S_t \le D\,t^q
POGO \mathcal O\big(\sqrt{\alpha\ln(kT)/T} + \ln(kT)/T\big) none S_t \le D\,t^q

[Ramalingam et al, 2025][Angelopoulos et al, 2025][Liu et al, 2026]

POGO: coverage guarantee

Theorem [Bharti, Pal, Teneggi, S.] If S_t \le D\,t^q, then for every group j \MisCov_T(j)\ \lesssim\ \sqrt{\frac{2\alpha(1-\alpha)\,L}{T_j}} + \frac{L}{T_j}, \qquad L = \ln T + \ln k + C \ln T from the bound on the wealth and the universal portfolio, \ln k from starting at 1/k; C depends on D, q, \alpha. Full statement in backup.

✓ no step size to tune
✓ marginal coverage included (add a group with everyone, c_0 \equiv 1)

Simulation: a shift in a rare group

k=10 groups; at t=10^4 the scores of a rare group (5% of patients) jump

POGO \eta=0.1 \eta=0.01 \eta=0.001
mean radius 0.18 0.24 0.18 0.18
jump per step 0.01 0.09 0.02 0.02

ICU length of stay and stock prices

MIMIC-IV length of stay; groups by race, insurance, sex

Delta (DAL) daily opening price

lowest group coverage vs. average radius, target 1-\alpha from 0.75 to 0.95; up and to the left is better

Take-home messages

  • Auditing and uncertainty quantification after deployment are sequential problems; betting gives anytime-valid answers.
  • Auditing k groups: a balanced merge of the wealth is fast whether one group or many are non-null.
  • Online conformal: set the interval width by betting; coverage holds on every group, for any sequence, with no step size.

Thank you

Beepul Bharti JHU

Ambar Pal Amazon

Jacopo Teneggi JHU

Global Sequential Testing for Multi-Stream Auditing
arXiv:2602.21479

Parameter-Free and Group Conditional Online Conformal Prediction
arXiv:2606.00419

Supported by NSF CAREER award 2239787 and NIH/NCI R01CA287422.