TLF Re-zeroing: Engine Replay

Overnight and synthetic re-zeroing of the S-TLF baseline, tested inside the served nowcast engine. Replay of 27 sites, 27 April to 7 October 2026 grabs. Written 15 October 2026.

Summary

Re-zeroing improves the per-site regression arm. The best variant holds one zero per epoch, taken from the first settled day after each cleaning or install (firstday_p10), and gives it to the regression arm's TLF inputs. Regression cost changes by −24, served cost by −20.7 ± 3.1 (negative is better), and the gate serves regression on 33% of grabs instead of 20%. The XGB stack is not sensitive to the choice of zero, so it can keep the served features. Nothing has been deployed.

Re-zeroing: resetting the sensor baseline on a schedule, or across maintenance, so drift and handling steps do not leak into the E. coli features. The bench work (drift-cal, stream handling) showed that handling adds an additive optical step that relaxes over about half a day. No unit was disturbed for this test; every variant is applied to the existing field record.

The engine makes three predictions per site-hour and a gate chooses which one to serve. Calibration is the regional prior plus the site's own bias and last result, with no sensor input. The stack adds one XGBoost model pooled across sites, on 14 sensor features (9 of them TLF) plus rain and flow. The regression arm is a ridge regression per site on tlf_now, tlf_anom, ntu_now and temp_now, fitted once the site has 6 grabs with sensor data.

Field check: what a cleaning does to S-TLF

For each field cleaning with data (12 to 14 per time bin), the step is the change from the 6 to 1 h pre-event median to each post-event bin, minus the same change across the same clock hours the day before. Each event is ranked against 40 random event-free times on the same unit.

0 to 24 h after a cleaning: median step +2.4 to +3.2 S-TLF (log ratio +0.17 to +0.21) on a pre-event median of 9.0, ranking at the 91st to 100th percentile of the placebo times.
24 to 48 h: +0.15 (log ratio +0.01, 64th percentile). The step is gone by the second day.
The measured dark pedestal is 171 to 174 counts against the fixed 170 in the slope. That gap is the dark_p10 variant below.

So in the field a cleaning acts like a transient handling step, not a lasting change of scale. That is why several variants below stop resetting the baseline at cleanings, or remove the step instead.

All variants vs the served engine

Difference from the served engine (variant minus served); negative is better. Each variant replaces the zero in tlf_anom (the served zero is the trailing 14-day 10th percentile, reset at each cleaning) unless marked otherwise. The stack is retrained with 6 XGBoost seeds per variant and compared with the served engine at the same seed; served and stack columns are mean ± standard error over those 6 pairs. Green or red: better or worse at p < 0.05 (paired t-test, 5 degrees of freedom); with this many variants, about one such result is expected by chance. The regression arm has no random seed, so its columns are exact. RMSE is in log10 CFU per 100 mL on uncensored grabs.

Variant	What it does	Served cost*	Stack cost**	Regression cost**	Regression RMSE
Overnight re-zeroing
`night_last`	Zero = median S-TLF over local 01:00 to 05:00 of the most recent settled night.	−13.7 ± 5.2	−4.2 ± 3.2	−10	−0.001
`night_med7`	Zero = median of the last 7 settled nights.	+0.5 ± 4.3	+1.8 ± 2.6	−19	−0.018
`night_first`	Zero = the first settled night after each cleaning or install, then held for the whole epoch.	−2.2 ± 3.9	−3.7 ± 1.6	−7	−0.025
`night_first (all TLF features)`	As above, and the same zero also subtracted from the six TLF level features.	−6.8 ± 3.9	−4.5 ± 1.8	−12	−0.024
`night_last (all TLF features)`	night_last, applied to the level features too.	+8.3 ± 4.6	+2.5 ± 2.2	−5	+0.004
Daily and running zeros
`firstday_p10`	Zero = 10th percentile of the first full local day after each cleaning or install, held for the epoch.	−7.7 ± 3.7	−5.3 ± 3.2	−20	−0.024
`firstday_p10 (all TLF features)`	As above, and also subtracted from the six TLF level features.	−18.0 ± 4.2	+3.3 ± 1.1	−24	−0.027
`firstday_p10 (regression arm only)`	The same re-zeroed tlf_now and tlf_anom, given to the regression arm only. The stack and every other arm keep the served features.	−20.7 ± 3.1	+0.0 ± 0.0	−24	−0.027
`prevday_p10`	Zero = 10th percentile of the previous local day.	−1.2 ± 3.3	−4.0 ± 1.9	+0	−0.023
`epoch_min`	Zero = running minimum of the 1-hour median since the epoch began.	+0.2 ± 2.5	−4.3 ± 1.6	−11	+0.002
`epoch_min (all TLF features)`	epoch_min, applied to the level features too.	+4.8 ± 4.4	−3.3 ± 2.2	−3	+0.003
Synthetic re-zeroing across maintenance
`relevel`	At each cleaning, shift the post-event series so its 12 to 24 h level matches the 6 to 1 h pre-event level, and do not reset the baseline there. Served 14-day baseline on the result.	−2.3 ± 1.9	−1.3 ± 2.3	+2	−0.006
`step_decay`	Subtract the measured handling step (0.25 to 1 h after minus pre-event), decaying with the bench constants (hold 1.5 h, time constant 4 h). Served baseline on the result.	−9.0 ± 1.5	+3.3 ± 1.8	−7	+0.000
`step_field`	As step_decay, with the hold and decay fitted to the field cleaning curve: the step is held for 23 h, then dropped (time constant 0.5 h).	+1.2 ± 1.6	−5.0 ± 2.1	−2	+0.007
`mask24`	Blank all TLF features for 24 h after each cleaning or reinstall (that arm falls back to calibration).	−6.0 ± 0.7	+0.0 ± 0.0	+2	−0.004
`mask24_noepoch`	mask24, and cleanings no longer reset the baseline.	−5.2 ± 1.5	−3.8 ± 2.2	−6	+0.004
Dark subtraction
`dark_p10`	Slope recomputed with each sweep's measured dark pedestal instead of the fixed 170. Served baseline.	−3.5 ± 2.6	−0.3 ± 3.5	+3	+0.003
Controls
`p10_14d`	The served baseline rebuilt inside the test harness. Must equal the served engine exactly (it does, all 586 decisions, seed 0).	+0.0	+0.0	+0	+0.000
`p10_14d (regression-only route)`	Plumbing check for the regression-only route: the served features passed through it unchanged. Must equal the served engine exactly.	+0.0	+0.0	+0	+0.000
`p10_14d_noreset`	The served baseline with no reset at cleanings.	+1.7 ± 1.2	−0.2 ± 1.4	−2	+0.003
`level`	No zero: tlf_anom = tlf_now.	+4.5 ± 4.3	−1.7 ± 3.7	−9	−0.013

* Served cost: the cost of the alarms the public page actually showed. At each grab the gate was serving one arm at that site, and served cost scores that arm's alarm. The gate picks, per site, the arm with the lowest cost on the grabs known so far, switches only when a challenger is ahead by more than 1 point, and judges a new site on its region's pooled record until the site has 8 grabs with at least 2 exceedances and 2 clean results. That lets a site's regression arm serve early, before its own record is long. For the served engine, served cost is 231 and the gate serves regression on 20% of grabs.

** Stack cost, regression cost: that arm's cost if it had been served at every grab. For the served engine: stack 219, regression 233.

In all three, cost = 3 × missed exceedances + false alarms (a miss costs 5 on the Seine), where an alarm is the decision published at the grab's sample hour, before the result was known. Every grab is predicted only from grabs whose results were already known. Grabs made before any result from their site was known are left out, as the gate leaves them out, which leaves 533 grabs.

By region

Boulder already predicts well; Chicago and the Seine (Paris and Marne) are where the nowcast most needs to improve. This table splits the same differences by region, so a variant that gains in Boulder at the expense of Chicago or the Seine is visible. Served cost* is mean ± standard error over the 6 paired seeds, colored as above; regression cost** is exact. Other is Cleveland, Denver and DC. Served engine served cost by region: Boulder 72 (178 grabs), Chicago 37 (109 grabs), Seine 91 (147 grabs), Other 32 (99 grabs).

Variant	Served: Boulder	Served: Chicago	Served: Seine	Served: Other	Regr.: Boulder	Regr.: Chicago	Regr.: Seine
Overnight re-zeroing
`night_last`	−9.0 ± 2.7	−1.0 ± 0.6	−4.3 ± 4.1	+0.7 ± 0.6	−22	+5	+5
`night_med7`	+1.8 ± 3.4	−1.0 ± 0.6	−1.0 ± 1.9	+0.7 ± 0.6	−20	+0	−2
`night_first`	−2.0 ± 3.3	−1.0 ± 0.6	+0.7 ± 1.6	+0.2 ± 0.4	−14	+0	+4
`night_first (all TLF features)`	+0.7 ± 3.0	−1.0 ± 0.6	−4.8 ± 2.0	−1.7 ± 1.1	−14	+0	−1
`night_last (all TLF features)`	+15.8 ± 2.8	−1.0 ± 0.6	−5.7 ± 2.0	−0.8 ± 1.0	−3	+0	−7
Daily and running zeros
`firstday_p10`	−2.5 ± 3.4	−1.5 ± 0.5	−3.8 ± 2.3	+0.2 ± 0.7	−16	−5	−2
`firstday_p10 (all TLF features)`	−6.2 ± 2.8	−1.0 ± 0.6	−10.2 ± 2.1	−0.7 ± 0.8	−16	−6	−5
`firstday_p10 (regression arm only)`	−7.8 ± 3.1	+0.0 ± 0.0	−12.2 ± 0.8	−0.7 ± 0.3	−16	−6	−5
`prevday_p10`	−0.3 ± 2.4	−1.0 ± 0.6	+0.2 ± 2.0	+0.0 ± 0.6	−1	+6	−8
`epoch_min`	−0.8 ± 2.3	+0.3 ± 0.9	+0.5 ± 0.9	+0.2 ± 0.5	−9	+0	−5
`epoch_min (all TLF features)`	+11.2 ± 3.8	+1.2 ± 0.9	−8.2 ± 3.0	+0.7 ± 0.3	+2	−2	−6
Synthetic re-zeroing across maintenance
`relevel`	−1.2 ± 0.8	+0.0 ± 0.0	−1.8 ± 1.9	+0.7 ± 0.6	+0	+0	−1
`step_decay`	−0.2 ± 0.2	−0.2 ± 0.2	−8.3 ± 1.2	−0.3 ± 0.6	+0	+0	−7
`step_field`	−1.0 ± 0.5	+0.3 ± 0.3	+1.7 ± 1.7	+0.2 ± 0.2	+0	+0	−4
`mask24`	+0.0 ± 0.0	+0.0 ± 0.0	−5.3 ± 0.7	−0.7 ± 0.3	+0	+0	−1
`mask24_noepoch`	−1.0 ± 0.6	−1.0 ± 0.6	−4.3 ± 1.6	+1.2 ± 0.5	−3	+0	−6
Dark subtraction
`dark_p10`	−2.3 ± 1.6	−0.2 ± 0.7	−1.3 ± 1.0	+0.3 ± 0.8	+1	+0	+0
Controls
`p10_14d`	+0.0	+0.0	+0.0	+0.0	+0	+0	+0
`p10_14d (regression-only route)`	+0.0	+0.0	+0.0	+0.0	+0	+0	+0
`p10_14d_noreset`	−0.3 ± 0.6	−0.7 ± 0.7	+2.2 ± 1.6	+0.5 ± 0.3	+0	+0	−4
`level`	+3.2 ± 3.4	+0.3 ± 0.8	+1.5 ± 2.4	−0.5 ± 0.7	−11	+0	−1

firstday_p10 (regression arm only) does not trade one region for another: served cost improves or is unchanged in every group, and the largest and most consistent gain is on the Seine. Chicago's served cost does not move, although its regression arm improves. Two variants do make the trade: epoch_min and night_last applied to all TLF features gain on the Seine and lose in Boulder.

Why it helps the regression arm

Two of the regression arm's four inputs are TLF. The model is linear and fitted per site on few grabs, so a shift in the sensor's level moves its prediction directly. The served zero is a trailing 14-day percentile, so tlf_anom keeps moving as the baseline rises and falls with drift and handling. A held zero keeps it on one scale for the whole epoch.

The table below holds the grab set fixed: the 296 grabs where the regression arm makes a prediction under every variant listed (288 uncensored for MSE).

Regression arm inputs	Cost	MSE (log10²)
Served zero (rolling 14-day p10)	139	0.397
firstday_p10	129	0.382
firstday_p10, all TLF features	125	0.375
night_first	138	0.364
No zero (level)	133	0.360
Calibration arm on the same grabs (no sensor)	134	0.288

With the first-day zero, the regression arm's alarms beat calibration's on these grabs; with the served zero they do not. The gate serves regression more often under firstday_p10, which is consistent with the served-cost gain coming through the regression arm. The held zero also lowers the regression arm's squared error on these grabs, though calibration remains lower, so the larger gain is in where the arm places its alarms.

Why it does not help the stack

Across the 18 variants applied to the stack's features, its cost moves by at most 5.3 points (about one missed exceedance or four false alarms over the season), two changes reach p < 0.05 (one better, one worse), and its RMSE moves by at most 0.006. TLF does contribute to the stack: holding every TLF feature constant raises stack RMSE by +0.009 ± 0.001. What the stack is insensitive to is how TLF is zeroed.

We did not isolate why. One structural fact is consistent with it: the stack's training target already has each site's mean and its most recent result removed, and the trees are pooled across sites, so a slow level offset at one unit carries little information for them. A per-site linear model has no such buffer.

Proposed change and limits

Proposed: give the regression arm a held per-epoch zero (firstday_p10: 10th percentile of the first full local day after each cleaning or install) for tlf_now and tlf_anom. Leave the stack's features and the gate as they are. In regression-only form the stack's decisions are identical to the served engine on all 6 seeds, and the regression decisions are identical to the all-features version.
step_decay uses the bench time constants, which have the step mostly gone by 4 to 8 h. In the field it holds for about a day. step_field refits the hold and decay to the field curve, and it does not help: served cost +1.2 ± 1.6, stack cost −5.0 ± 2.1, regression cost −2, regression RMSE +0.007. Removing the handling step more precisely did not help the regression arm; what helped the regression arm was a zero held fixed across the epoch.
One season, 27 sites, 12 to 14 cleanings with data. The largest served-cost differences are 2 to 6 standard errors over 6 seeds of the same record. That is enough to rank the variants, not to call any of them settled.

Method: lume-nowcast engine at 1355768, driven by a replay harness that swaps only the sensor feature builder (the served-baseline rebuild, and the served features passed through the regression-only route, each match the served engine on all 586 decisions). Sensor data were pulled from Pumphaus on 14 October 2026, with the maintenance registry as of the same date. Harness scripts are not in this repository.