Generated from LUME-DIGITAL-TWIN.md by scripts/build_twin_page.mjs. Not hand-edited; the document is held to its own constants by 121 automated checks.

Lume — Digital Twin

A behavioural model of the instrument: what each channel physically measures, which direction it moves under every perturbation we have observed, what each failure mode looks like, and which claims are known to be false.

Purpose. Reasoning about Lume data goes wrong in a specific way: the fact is usually already written down somewhere, but buried in prose, or contradicted by a second note, so the wrong assumption survives. This file is the single authority. If another document disagrees with this one, this one wins and the other should be fixed.

Every entry is tagged with provenance, and provenance sets precedence:

Constants below are [CODE] as of MODEL_VERSION 2026-08-20-tlf-clean-step. Do not trust a number here over the module — regenerate with: node -e "import('./shared/ecoli-model.js').then(m=>console.log(m.SIPM_CAL, m.SLOPE_CAL, m.SLOPE_TEMP_RHO))"

---

1. The physical instrument

[CONFIRMED] Three distinct optical surfaces, which foul independently:

1. Cap interior — what the ToF IR pulse bounces off.
2. TLF lens — the fluorescence path (UV excitation, SiPM detection).
3. ToF lens — the IR emitter/detector path.

The single most consequential structural fact: ToF and TLF can foul independently. A ToF-only change says nothing about TLF health, and vice versa.

2. Channels

✅ ToF: RANGING NOW WORKS — on v0.1.20+ only, from 2026-08-22

[MEASURED] 2026-08-31, from api_samplesraw.range_status joined to Notehub firmware_host. The distance feed was fixed by host firmware v0.1.20 (built 2026-08-21 19:50 and 20:25 UTC). Fleet-wide, rows with valid ranging went 459 → 1,567 → 10,500 across 2026-08-20 / 21 / 22. Treat 2026-08-22 as the first usable day. This supersedes the 2026-08-21 entry that recorded the feed as broken.

Firmware decides it completely — there is no overlap between the two populations:

Host firmware	Sensors	Rows (3 d)	Rows with range_status = 0
v0.1.20+	31	22,083	100.0 % — all 31 sensors individually at 100 %
v0.1.18	35	142,686	0.0 % — 34 of 35 at exactly 0 %

The range_status gate is still mandatory — but it is now a firmware selector, not a noise filter

Filter range_status = 0. On v0.1.20+ it passes every row; on v0.1.18 it passes none, and distance_mm is the 70 mm sentinel on every reading. Averaging distance_mm across the fleet without the gate mixes real millimetres with 70 mm sentinels and yields a meaningless number that looks plausible.

Roughly half the fleet still cannot range. 35 of 66 reporting sensors remain on v0.1.18. Any ranging analysis today is a 31-sensor study, not a fleet study.

⚠️ The feed works. The response laws do NOT yet exist

Valid distances read 15.9–68.5 mm with per-sensor SD mostly 0.7–1.9 mm — a stable, plausible signal. But no direction has been established for any perturbation, fouling or gas included. Do not import the amplitude signs from §3: that table describes tof_sps, and §3 deliberately carries no ranging row. The single deliberate gas trapping on the bench (2026-08-30T22:34Z) moved the amplitude floor down, which is n=1 and confounded with lifting the units out of water.

Ranging is the intended fouling-vs-turbidity discriminator — a film sits at a fixed short range on the window while suspended particles attenuate over the whole optical path. Amplitude alone cannot separate the two, which is why detectTofFouling has to decorrelate against streamflow (FLOW_CORR 0.4) and why that guard has nothing to work with at the many sites with no usable gauge (Paris, Cleveland, BGS/Thames, Bow River; Chicago's main-stem CAWS reaches publish no discharge at all). Ranging is now measurable. It is not yet characterised. Those are different claims and the second one still needs an experiment.

Amplitude was never in question. tof_sps / signal_per_spad_kcps worked throughout; everything built on it stands unchanged — model tof_slope terms, the fouling detector's daily p10 floor, install_tof_baseline, and /api/field-refit's tofd / tofr / foul_off.

Do not "fix" the ambiguity by tuning a ToF coefficient. The discrimination was missing hardware information, not a modelling shortfall — a term that separates them in-sample on these grab counts is fitting noise (KNOWN-FALSE §7.8).

Calibrated operating point [CODE]: SIPM_CAL = {LED: 512, BIAS_TARGET: 3000, BIAS_LO: 2960, BIAS_HI: 3040}. Readings are only comparable at this combo. Every reading carries a full LED × bias sweep.

Pedestal [CODE] SLOPE_CAL.PED = 170 is now only a FALLBACK. fitTlfSlope measures the pedestal from each sweep's own no-gain cells (bias < PED_MAX_BIAS = 2850, median, at least PED_MIN_PTS = 3). Real pedestals run 166-175 per sensor and are quantisation-exact, so one fleet constant is wrong by up to 5 counts; 157523 of 157523 archived sweeps carry enough low-bias cells to measure it. LOW is GONE: railing is the only exclusion, every LED x bias cell is used, and the same pipeline runs for every sensor group. Below SiPM breakdown the counts sit at ~170 regardless of light. A unit reading ~180 is sitting on the pedestal — that is the readout floor, not a measurement of the water.

Clamps [CODE]: CFU_LOD = 0.1, CFU_ULOD = 24196. A value at exactly 24196 is the model railing, not a measurement. LINEAR_CEIL = 2900; saturation ceiling ~3299.

3. Response laws — THE SIGNED TABLE

The core of the twin. For each perturbation, the direction each channel moves.

3.1 THE ONE THAT KEEPS BEING GOT WRONG

Fouling ATTENUATES the fluorescence path. Cleaning therefore makes TLF step UP, and the post-cleaning reading is the CORRECT one. The sensor is not broken and the operator did nothing wrong — the calibration is now stale, because it was fit against the fouled state.

A step at a touch is not automatically an artifact. Ask which direction the physics predicts before diagnosing. Attenuating deposit → cleaning steps UP. Fluorescent biofilm → cleaning steps DOWN. Both are real; both require re-anchoring, not a lecture to the partner about technique.

⚠️ OPEN INVESTIGATION — the correct calibration frame is NOT decided. [CONFIRMED] Evan, 2026-08-21: "We don't know yet — this is still an investigation." What ships today (fouled frame + tlf_clean_step divisor) is one candidate, not settled design — do not present it as the answer, to a partner or in a report. Candidates: (1) fouled frame + step divisor [current]; (2) clean frame canonical, correcting history upward; (3) refit per cleaning epoch; (4) continuous fouling model — the TLF analogue of computeFoulingOffset. The evidence that would settle it is post-cleaning grabs, so ask partners to sample soon after any service while the instrument state is known.

Currently handled by sensors.tlf_clean_step + applyTlfCleanStep(). Whatever frame is chosen, these implementation constraints are settled — each cost a real error on 2026-08-20:

• The correction must be time-aware — applied only from the cleaning onward. Applying it retroactively deflated Bercy's whole pre-cleaning history from 47 → 3 CFU.
• The logged cleaning time must be derived from the mon2 step in the 15-min series, not guessed. A guessed time divides still-fouled readings.
• Measure the ratio on the S-TLF slope, not mon2 net — base20 drifts and goes NEGATIVE at some sites (6 of 20 Bercy/Grenelle grabs), leaving the mon2-net ratio undefined.

[MEASURED] City of Paris 2026-08-18 — the reference case: Bras Marie ×2.29, Bercy Coliminder ×1.35, Bercy ×4.31 (S-TLF slope), while five untouched sensors on the same rivers moved <3%.

4. The S-TLF bias-response slope

[CODE] SLOPE_CAL = {PED:170, PED_MAX_BIAS:2850, PED_MIN_PTS:3, SAT:3250, REFB:3000, SCALE:1000, MIN_PTS:6, MIN_SPAN:100}, SLOPE_TEMP_RHO = -0.024.

Definition. mon2 ≈ PED + LED·gain(bias)·fluor, so (mon2−PED)/LED collapses all three LEDs onto one gain curve. Fit that vs bias as a quadratic over the screened sweep and take the derivative at a fixed reference bias (3000). Evaluating at one bias is range-independent, so the LEDs agree even though each sweeps a different arc.

Always use log(slope), never raw slope, with per-sensor intercepts. Fleet gain spans ~250× (50074 ≈ 0.2, 50073 ≈ 55) — a shared coefficient on the raw slope is dimensionally incoherent. In logs, each unit's gain is an additive offset its fe absorbs.

Always normalise to 20 °C via tlfSlopeAt20C() before the log. The slope quenches less than mon2 but is not quench-free.

What it fixes / what it does not — [MEASURED] 2026-08-18…20:

Measured proof of the right-hand column: across the Paris cleaning the slope stepped ×2.29 / ×4.31 / ×1.35 — as much as or more than mon2. A uniform optical gain change moves both identically. Never offer S-TLF to a partner as the fix for a fouling event.

5. Reference methods

[CONFIRMED] (Evan, 2026-08-21) Ceiling grabs stay in the fit, censored — never dropped and never treated as exact. They are the strongest exceedance evidence available: dropping them would discard 6 of 47 Chicago and 5 of 13 Cleveland samples, most of them real exceedances, while treating >2419.6 as exactly 2419.6 biases the fit low. Tobit imputes E[y | y ≥ ceiling] each EM iteration. Both the undiluted 2419.6 and the diluted 4839.2 are treated the same way — as a censoring event at their respective ceilings.

6. Action limits (per customer — these are theirs, not ours)

Serving cuts are SPLIT from the model that computes them [MEASURED] 2026-08-27

PARIS_MODEL is fit across the Seine and the Marne (its fe carries 500127; fit_n 91 = 69 Seine + 22 Marne), so one model served both under a single cut of 250 — and that cut was wrong for both, in opposite directions. Sweeps on the served predictions:

No coefficient changed for Paris or the Marne. REGION_CUT holds the serving cut so the models keep their fitted exc_t provenance; MARNE is a threshold region only (no model of its own, predicted by PARIS_MODEL, same 900 action limit).

⚠️ DC's cut is a deliberate over-flagging choice [CONFIRMED] (Evan, 2026-08-27), now 232 after the 2026-08-31 n=26 refit moved it with the model: sensitivity 1.00 bought at ~32% of clean days raising an alert (was ~46% at 145). It is far BELOW the action limit, the mirror image of the Denver warning below.

⚠️ A cut cannot fix the Seine. Its predictions top out at 397 CFU against a 900 limit, so it structurally cannot express an exceedance. Only exceedance grabs with coincident sensor signal move that.

⚠️ These are 3-5 exceedance samples. Bootstrap 95% CIs on the retuned BAs are DC [0.64, 0.91], Seine [0.45, 0.99], Marne [0.54, 1.00]. 0.90 sits inside all three, so none of these is distinguishable from 0.90 — or from 0.65. Do not report them as achievements.

Every decision cut in the code right now [CODE]

Model exc_t as fitted, and the serving cut actually applied where REGION_CUT overrides it. Regenerate rather than trusting this table.

Levers tested 2026-08-27 and REJECTED — do not re-run these

• Rainfall (ERA5 archive, 48 h and 24 h, capped 20 mm). DC is genuinely wet over its grab window (48 h median 9.4 mm, max 21.0) and rain adds exactly nothing: BA 0.769 with and without. Marne it makes worse (0.819 → 0.792; rain-alone 0.569). Pooled Seine+Marne it appears to help (0.792 → 0.827) but that window is a drought (median 0.3-0.5 mm), so the coefficient is fit on almost no rain. rain48 is also null in the serving path for both customers, so adopting it means wiring rainWindowHours for them. Re-test after a wet spell.
• Channel swaps. At DC ln(S-TLF slope @20 °C) ties the mon2 form exactly (0.769 both) and tlf_amp does not resolve at all (0/18). At Paris slope, amp and mon2 sit within 0.015 BA of each other. There is no free win in the fluorescence estimator at either site.
• DC level-only benchmark scores 0.731 against the full model's 0.769 (§11.5 firing): the DC classifier is mostly a per-site base rate.

The action limit follows the WATER, not the model that computed the number [CONFIRMED]

Evan, 2026-08-27: "Paris/Marne is 900." modelNameFor answers which model computes the prediction and must keep mirroring predictEcoliCfu's routing (§12.1). That is a different question from which threshold the operator acts on, and for three French bathing sites the two had silently diverged:

Mechanism: each has a synthetic FIELD_MODEL.base20 entry but no PARIS_MODEL.fe entry, so first-match-wins routing dropped it to FIELD and it inherited Boulder Creek's numbers while its siblings on the same two contracts ran at 900/250. Fixed by THRESHOLD_REGION + thresholdRegionFor(), which actionLimit() and decisionCut() now consult; modelNameFor is deliberately unchanged so it still mirrors the routing.

⚠️ This fixes the THRESHOLD only, not the calibration. All three still have zero paired grabs (10 grabs between them in validation_grabs, none pairing) and still take their prediction from the FIELD fallback. When grabs pair, give them real PARIS_MODEL.fe entries and delete them from THRESHOLD_REGION — a permanent entry there is a smell.

Display bands are the partner's call — but every edge needs a source [CONFIRMED]

Evan, 2026-08-21: the safe/caution/danger bands a dashboard paints are per-partner, not standardised — a partner whose regulation carries two numbers (a geomean criterion and a single-sample max) may show both. What is not allowed is an unexplained constant. Every band edge below must name where it came from; anything marked ❓ is a number nobody has justified and is owed either a source or a correction.

The band edge is a display choice; the exceedance CALL is not. Wherever a page's safe→caution edge is also what it means by "exceedance", that edge must be decisionCut(), not a literal — chicago.html painted caution at a hard-coded 126 against a 200 limit for weeks because the literal drifted from the model and nothing caught it.

The action limit is the only threshold that means anything [CONFIRMED]

Evan, 2026-08-21: "The action limit is what matters — the tuning is ONLY so that we actually capture those. Our prediction isn't accurate enough to say that it's real — it's just an internal tuning parameter to output predicting above/below the action threshold."

COMPARE against decisionCut(). LABEL with actionLimit(). A predicted concentration is not accurate enough for exc_t to be read as a real boundary, so the two numbers differing is not a defect and not something to level-calibrate away — it is the knob doing its job.

[CODE] current exc_t: FIELD 79 (FIELD_MODEL.slope, the branch Boulder actually runs; the legacy FIELD_MODEL.exc_t = 78 is the superseded pre-slope value and is not what serving decides with) · CHICAGO 200 · CLEVELAND 235 · DC 232 · DENVER 410 · PARIS 700 (slope branch 250). Spread against the limit: Boulder 79/126, Paris 250/900, Chicago 200/200. Whenever a model is refit, re-derive exc_t — a model whose predictions top out below the action limit has zero sensitivity at that limit.

[CODE] ACTION_LIMITS / modelNameFor / actionLimit / decisionCut / riskStatusFor in shared/ecoli-model.js, emitted to the browser. Never hard-code either number in a page. A hard-coded cut goes stale silently at the next refit: chicago.html shipped || 300 from a superseded refit and a riskStatus alerting at 126 whose comment claimed to be exc_t.

⚠️ exc_t ABOVE the action limit is the one case the rule cannot cover. A cut above the limit cannot make the call track the limit — it can only under-flag. DENVER is there now: exc_t 410 against a 235 action limit (the single-sample max the page's own hero displays, next to the 126 TMDL geometric-mean standard). It is the demo-calibrated model, so nothing real depends on it, but any model that lands this way needs a refit, not a knob.

7. ⛔ KNOWN-FALSE — do not believe these

Every one of these was believed, acted on, and cost rework.

1. distance_mm is permanently unusable. ❌ RESOLVED 2026-08-22 by host firmware v0.1.20. The old 0-of-7 null measured a broken feed, not a physical fact about ranging, and must not be cited either way. On v0.1.20+ ranging returns valid millimetres on 100 % of rows; on v0.1.18 it returns the 70 mm sentinel on 100 % of rows. [MEASURED] — see the §2 banner. What remains false is the inverse: that a working feed means we know what it means. No response law for ranging has been established for any perturbation.
2. A ToF-only change means the TLF channel is dead. ❌ Separate optical surfaces. [CONFIRMED]
3. Flat absolute mon2 means a dead detector. ❌ 50074/50064 read 179–187 (CV 0.8 %/1.8 %) and were called dead; across the full bias arc 50074 reaches mon2 492 and fits a clean slope on every sweep. They are low-gain, sitting on the pedestal. [MEASURED]
4. Cleaning makes TLF go down. ❌ See §3.1.
5. S-TLF is immune to fouling artifacts. ❌ See §4.
6. Open-Meteo's forecast past_days is interchangeable with the ERA5 archive. ❌ The forecast returned a flat 0.0 mm across 07-27…07-29 at Old Woman Creek where ERA5 records 7.0 mm. Archive must overwrite forecast, never the reverse. [MEASURED]
7. A documented metric stays true. ❌ Chicago's BA 0.809 was not reproducible after a data bug was fixed (0.743). Metrics must be regenerated, not quoted.
8. mon1 and mon2 are independent detectors, so agreement between them corroborates. ❌ They are two amplifiers on the SAME SiPM (mon1 at ~3.95× gain). Do not ratio them and never read agreement as confirmation — it is the same photons twice. [CONFIRMED] (This is why the validation estimator's hybrid path can convert a mon1 cell into a mon2-equivalent through a measured gain ratio: same detector, known factor.)
9. A free ToF slope is safe to fit on a small grab set. ❌ It fits negative and then extrapolates absurdly the first time a turbidity excursion lands outside the fitted range. [MEASURED] DC at n=8: free ToF → predictions of 1e10 CFU. Tried again at n=12 and it blew up again, and again at n=18 on 2026-08-27 (1.2e6 CFU). DC now zeroes the slope rather than holding it — the old "held at the CHICAGO value" 0.0035829 had drifted ×43 from the Chicago value it named (0.0000831 today), and zero scores the same (R² 0.285 vs 0.284). ⚠️ PARIS_MODEL's mon2 fallback branch still carries 0.0035829 with that same stale comment — its amp branch is zeroed, the fallback is not. DENVER zeroes its ToF slope outright for the same reason — Bear Creek's turbidity excursions blow up any held ToF term. Hold or zero the ToF slope until the grab set actually spans the turbidity range; a negative in-sample ToF coefficient on a small n is the warning, not a finding. (Paris is the exception that proves the rule: its negative ToF slope is data-supported — negative within site at 3 of 4 sites, pooled −0.150 — which is the check to run before trusting one.)

Where rejected refits are recorded [CONFIRMED] (Evan, 2026-08-21): here, and only when the dead end is general enough to matter. No per-model rejection log, no refit journal — the point is to stop the repeat attempt, not to archive the bookkeeping. If a failed variant would fool the next person, it belongs on this list; if it was specific to one fit set, let it go.

8. Pairing rules (HARD)

• ±20 min grab↔reading window, everywhere a fit is computed, and it does not vary by site or by how fast the water moves [CONFIRMED] (Evan, 2026-08-21). [CODE] one definition: const WIN = 20*60000 in fieldRefitCompute's pairing helper (inventory-api/worker.js). Nothing else in the repo pairs a grab for a fit.

Where it comes from. [INFERRED] — no original derivation is recorded, and this is a reconstruction, not the reasoning that produced it: firmware reads every 15 min and syncs hourly, so the worst-case distance from any instant to the nearest reading is ~7.5 min and ±20 min clears one missed cycle with margin. It is deliberately not tied to the cadence in code — if the cadence ever lengthens past ~20 min, this window stops pairing rather than pairing badly, and the drop shows up in the /api/field-refit waterfall step "No in-window calibrated reading". Check that step after any firmware cadence change.

⚠️ ±60 min is a DIFFERENT thing, and it is not a fit. denver.html, dc.html and marne-seine-dashboard.html pair at ±60 min because they match against the hourly site_rollup grid, where ±20 min would drop most grabs. So a confusion matrix on those three pages is not the same measurement as the fit's: different window, resampled series. chicago.html and cleveland.html avoid this entirely by reading the /api/field-refit scatter, which is already fit-space. Do not compare a metric from a ±60 min rollup panel against one quoted from a fit.

The "7 of 16 Denver grabs have zero sensor data ±6 h" figure is from a scratch diagnostic, not a pairing window — no ±6 h window exists anywhere in this repo.

• At LED 512, use exactly one reading — the single bias nearest 3000 within [2960,3040]. Never average across biases; that mixes gains. [CONFIRMED]
• Install-window rule: a grab pairs only when its timestamp falls inside a valid site_installations window for that sensor. No covering window — including a sensor with no windows at all — means drop, never pair. [CONFIRMED]
• Pumphaus install windows are the sole authority. Do not add heuristic exclusions, "repair date" cutoffs, or stale sensors.notes on top. [CONFIRMED]
• The rule is absolute — but bad windows get FIXED in pumphaus, not worked around [CONFIRMED] (Evan, 2026-08-21). A wrong window is a data-integrity bug, not an inconvenience to route around.

✅ RESOLVED 2026-08-21 — and here is the repair mechanism. The sensor_installations mirror had drifted from pumphaus on 5 rows, including two that mattered for pairing (50064 Lume 50064/May → Wild Mile/2026-06-18, 50072 → Little Calumet/2026-06-23; also 50024, 50043, 50050). This was dangerous rather than cosmetic: rebuildSiteInstallations does DELETE … WHERE customer=? and re-inserts the mirror verbatim with no placeholder filtering, so a rebuild against a stale mirror would have renamed both sites and widened their windows back to May — admitting the pre-install grabs this rule exists to drop. PATCH /api/installations/:id triggers that rebuild as a side effect, so the obvious repair route was itself the trigger.

Repair with POST /api/installations/remirror (admin). Reads pumphaus (the SoR), pages the installation and site lists, and reports or applies the differences in site_name, installed_time and removed_time. Dry run by default — pass {"apply": true} to write; {"barcodes": [...]} scopes it; {"sample": true} returns a raw pumphaus record. It rebuilds read-models only for customers it actually touched, and reports truncated plus any mirror rows absent from pumphaus, because a truncated scan reporting "0 drifted" is a false clean — the first run hit a 60-page cap and did exactly that. Verify with scripts/audit_pipeline.js (mirror.drifted-from-pumphaus).

9. Two gates that make grabs invisible

Both have silently hidden months of data. Check these first when a customer has grabs but the model ignores them:

1. Site→barcode map (FR_CHI_SITE_BC) — an unmapped site name upserts with a NULL barcode and enters nothing. Hid 27 of 59 Chicago grabs.
2. FR_BASE20 / FR_BTOF — a barcode absent here is dropped from /api/field-refit entirely. Hid all 14 Cleveland grabs.

10. Data path

sensor ──20 min──▶ Notehub (Blues) ──POST /payloads/notehub/──▶ pumphaus ──▶ D1 mirror
 1 min sampling      retries on non-2xx      216.245.146.13          │
 full LED×bias sweep                         ≤28 req/s, 1 IP         │
                                                                    ├─ api_payload      row committed FIRST
                                                                    └─ django_q_ormq    task queued SECOND
                                                                          │             ↑ orphan window (§10.0b)
                                                                          │  4 workers, retry 4 h, max_attempts 1
                                                                          ▼
                                                                    process_payload ─▶ api_samplesraw
                                                                                       api_metric
                                                                                       api_tlfdiagnostics
      → /api/site-series  ≤48 h: LIVE pumphaus fan-out (~51 s)
                          >48 h: site_rollup (hourly, ~0.25 s)
      → shared/ecoli-model.js  → dashboards + customer API

• shared/ecoli-model.js is the single source of truth. The Pages Function functions/js/ecoli-model.js.js emits the browser IIFE and enumerates constants by hand — any new exported constant used by an emitted function must be added there or the browser throws ReferenceError. Verify by eval'ing the live bundle.
• Both deploys are required: Pages and cd inventory-api && wrangler deploy.
• site_rollup stores predictions. After any model change, rebuild it per customer in ~1-week chunks (a 2-month window hits Worker limit 1102) or dashboards keep serving the old model while the validation panel shows the new one.

10.0 Pumphaus is the system of record

Site and time attribution live in pumphaus, not in inventory. The join that attributes a raw sample to a site is time-bounded, so it does in-water clipping and multi-sensor stitching at once:

api_organization   the customer — org_name must EXACTLY equal contracts.customer
   └── api_meter                (organization_id)
        └── api_watersystem     the site: site_name, latitude, longitude, timezone
             └── api_installation   gateway_id, water_system_id,
                 │                  installed_time, removed_time  (NULL = still in water)
                 └── api_gateway     barcode
                      └── api_modem / api_samplesraw / api_tlfdiagnostics

D1 holds a bounded mirror and a flat read model, because pumphaus's REST API has no query filtering. Three tables, three jobs:

10.0b The ingest path — Notehub to pumphaus, and how it fails [MEASURED] 2026-08-31

Reconstructed while recovering a four-day fleet-wide ingest outage (2026-08-27 to 08-31). Everything here was measured on production, not read off a diagram.

Delivery: Notehub is a single well-behaved caller.

Consequence for any rate limiting: the 567/min peak is Notehub retrying after an interruption — exactly when a limiter is most likely to fire, which would turn a blip into a feedback loop. Exempt the Blues netblock outright rather than choosing a threshold. Internet scanners observed hitting this host ran at 58–70 req/s, only ~2× Notehub's peak, so no blind per-IP threshold cleanly separates them.

A payload can be stored and never queued. PayloadViewSet.notehub commits the row, then enqueues the parse task as a separate step. If the process dies in between — OOM, instance replacement, a deploy landing mid-request — the payload exists with no worker assigned. It is not marked failed, because it was never attempted, so no retry path and no alarm covers it. Detect with kind = 'tlf' AND NOT parsing_attempted: views.py stamps 'tlf' at creation and add_metadata() rewrites it to blues.<file> once parsing starts, so that predicate means exactly created but never picked up. It also excludes the rock7 / swarm / digi / waterwheel-import backlogs, which have their own parsers.

⛔ The Django-Q broker: a deep, motionless queue is usually a HOLD, not a stall. Q_CLUSTER: workers 4, timeout 3 h, retry 4 h, queue_limit 50, bulk 4, max_attempts 1. django-q stamps lock when a worker takes a message and will not re-offer it until retry seconds later. Every message held by a worker that died is therefore invisible for four hours: the queue looks deep, the workers look idle, and both readings are correct. Diagnose from django_q_ormq.lock, never from the queue count. On 2026-08-31, 1,765 messages sat frozen at locks of 00:28–02:39Z and nothing was actually broken.

• ⛔ Never re-enqueue a payload without decoding the queue first. Metric rows are written with a plain bulk_create and no conflict target, so a second parse duplicates every metric. Of those 1,765 held messages, 1,322 pointed at payloads that were already parsed — letting the hold expire would have duplicated all of them. Decode with pickle.loads(base64(payload.split(':')[0])) → {func, args}, where args[0] is the payload id, and join against parsing_attempted.
• Duplicate full-fleet jobs pile up at the HEAD of the queue. The broker pops oldest-first, so 34 queued copies of water_systems_rollup_15_min starved 6,095 process_payload messages down to ~50/min; deleting them restored ~600/min. Pausing a schedule does not clear copies already enqueued — check django_q_ormq too.
• django_q_task is capped (~250 recent rows). Never use it to measure throughput; measure the queue delta instead.

What actually caused the four-day outage. water_system_has_new_payloads used .first(), which forces ORDER BY id ASC LIMIT 1 and made Postgres walk api_payload's primary key — 9.1 M rows, 21 GB — from the oldest row. 100–175 s per call × 1,114 water systems on a 180-minute schedule meant each pass needed 31–54 h, so passes overlapped permanently and starved process_payload. Changing it to .exists() turned the same question into an index-only probe: 113 ms. Verified 2026-08-31 by re-running the task under observation — longest backend query fell from 100–175 s to 0–7 s, it wrote 213,326 rollup rows with zero duplicates on (water_system_id, time, rollup_kind_id), and fleet ingest stayed under 30 s of lag throughout.

The general lesson, worth more than the fix: in Django, .first() and .exists() ask the same business question and generate completely different plans. .first() silently imposes an ordering. On a large table with the wrong index that ordering is a full scan.

Dead sites are gated out before the expensive path opens. 90 of 1,114 water systems have no rollups at all, which would mean a full-history rebuild — but all 90 have zero payloads in the last 7 days, so water_system_has_new_payloads skips them on the cheap .exists() probe and get_df_rollup is never reached. A site count that does not move after a rollup pass is therefore expected, not a stall.

10.0c Firmware runtime configuration — Notehub env vars (v0.1.21+) [CODE] 2026-09-10

Source: SweetSenseInc/tlf_firmware production/README.md § "Notehub Environment Configuration". First shipped in tlf-main-notehub_env-v0_1_21$20260909214003.binpack ("First Notehub environment configuration build"). Builds ≤ v0.1.20 are fixed-config — their cadence/sweep is baked in per build (20-1 = 1-min sample / 20-min report, uvled = LED burn-in); changing behaviour there means pushing a different _fw. From v0.1.21 the firmware reads its runtime settings from Notehub environment variables, settable per device, per fleet, or project-wide. A key that is absent or invalid silently falls back to its default — a typo does not error, it reverts.

Mask worked examples (verified against the README's own): 101000100001 → bits 0, 5, 9, 11 → LEDs 1, 32, 512, 2048; the default 1010100000 → bits 5, 7, 9 → LEDs 32, 128, 512 — which is exactly the sweep the pre-v0.1.21 fleet reports.

Memory constraint (the binding one): sample size = sipm_bias_steps × (enabled LED levels), and samples accumulate between syncs. Raising cycles_per_sync or enabling more LED levels can overflow the sample buffer; compensate by cutting bias steps or LED levels.

Serving-side invariant: every model and dashboard reads the calibrated operating point (LED 512, bias ≈ 3000 in data units), so any led_currents_mask must keep bit 9 (LED 512) enabled unless the change is deliberately taking a unit off the calibrated point.

sipm_bias units — RESOLVED [MEASURED] 2026-09-11 (500185 config test): the data column is a DAC count with the linear map volts ≈ −1.385 + 0.01262 × counts. Measured two ways: the default 34.0–48.0 V sweep spans counts 2803–3912 (36 steps ≈ 31.7 counts ≈ the documented 400 mV), and narrowing to 40.0–44.0 V moved the span to 3278–3594, matching the fit's prediction to ±2 counts. The calibrated operating bias 3000 counts ≈ 36.5 V — safely inside the firmware's 33–48.5 V valid range.

⚠️ v0.1.21 version-metadata trap [MEASURED] 2026-09-11: the tlf-main-notehub_env-v0_1_21 binpack's internal metadata is stale — a device running it reports tlf-v0.1.20-dirty, built 2026-08-21, so DFU verification gated on the build date/version string NEVER confirms. Verify an apply behaviorally (cadence/sweep change at the DFU-completed timestamp), not from firmware_host/dfu.current.

Sweep truncation at the rail [MEASURED] 2026-09-11 (50091, puck target): the bias sweep ABORTS per-LED when mon2 hits the rail (~3050): a bright target yields fewer steps at higher LED (measured 26/10/6 steps at LED 32/128/512). Consequences: (1) sipm_bias_steps is a maximum, not a guarantee — never verify config by step count on a bright unit; (2) any top-of-arc readout pins at the rail and HIDES signal change — read comparisons/drift at a FIXED low bias (e.g. DAC ≈ 2950), never at max-bias-present. Dark units (e.g. the Chl-a batch) sweep the full configured arc.

Config-test evidence (500185, 2026-09-11 01:44–06:46 UTC): every key honored — cadence 60 s / sync 20, mask 1000000000 → LED {512} only, mask 11010100000 → {32, 128, 512, 1024} (LED 1024 sweepable), sipm_bias_steps=11 → exactly 11 biases per LED, 40–44 V window as above. Env changes take effect at the device's next sync (pickup ≤ one sync interval; 20–60 min observed).

10.1 Joining raw pumphaus data — four traps that fail SILENTLY [CONFIRMED]

Reconciled 2026-08-21 from validation.thelume.ai/data-pipeline. Every one of these returns an empty or wrong result rather than an error.

1. api_samplesraw has no gateway_id. Join through api_payload — and api_payload carries modem_id, not gateway_id: api_gateway.modem_id = api_payload.modem_id.
2. api_tlfdiagnostics keys its timestamp time, not timestamp, and its rows arrive in different payloads from the samples. Match temperature to a sample by time (±15 min); joining on payload_id returns nothing.
3. barcode comes back as a NUMBER. A JavaScript Set<string>.has(barcode) silently fails to match. Cast with ::text.
4. api_organization.org_name must EXACTLY equal contracts.customer or the chain breaks at the first join.

10.2 The mirror is the only input to the read model [CONFIRMED]

rebuildSiteInstallations reads only from sensor_installations. Writing straight to pumphaus without updating the mirror leaves the read model blind: the site exists, the sensor reports, and the dashboard is empty. Any pumphaus write must be followed by a mirror update and a rebuild (§8 carries the repair path, /api/installations/remirror).

10.3 Ground truth is public and unauthenticated [CODE]

validation_grabs is served read-only at /api/validation/grabs with no login and no key — filters source (mwater / chicago / email), method (colilert / mf), customer, barcode, since, status (active default, or all to include swept/deduped rows). Response is { grabs: [...], count: N }. An hourly cron refreshes from the sources with idempotent upserts, and manual exclusions survive the refresh.

10.4b ⚠️ The out-of-water detector is blind straight after a data gap [MEASURED]

2026-08-21. 500146 stopped producing sweeps mid-experiment at 2026-08-20 22:42Z and returned a single reading 20.9 h later, at 08-21 19:35Z, in air — ToF 141 against its own in-tank median of 55 (range 52–66), while all four peers still read 44–48. That is an unambiguous air signature and the detector did not flag it.

Why it misses. The detector compares tof_sps against the unit's own trailing 45-minute median, with a fleet-relative rule normalised by the unit's own typical ratio. After a 21-hour gap there is no trailing window to compare against and no recent ratio to normalise by, so both rules have nothing to fire on. It flagged 323 rows on 500225 and 53 on 500226 in the same file and zero here.

So a unit that goes quiet and comes back out of the water enters the record as in-water data. When a gap precedes a reading, the trailing-window test is not evidence of anything — check the absolute ToF against that unit's own established in-medium level instead.

10.5 Cleaning events — how they are recorded, and what they do to a prediction

Where they live. [CODE] D1 sensor_maintenance:

id · barcode · event_type · event_time · data_suspect_from · data_suspect_to
   · note · created_by · created_at

Served at /api/maintenance/all and /api/maintenance/active. event_type in use today is cleaning, fouling, air-exposure and fault.

Who writes one. Two very different sources, and the difference matters:

[MEASURED] 2026-08-21. Only eight cleanings have ever been recorded by a human, against 42 machine-generated events. A cleaning is recorded only if someone logs it — nothing in the device or the pipeline reports that a probe was serviced. That is the single largest gap in this mechanism, because everything below depends on knowing a cleaning happened.

The one thing the scan can infer. It closes an open fouling window when the daily ToF p10 floor returns to its install_tof_baseline, which is how a cleaning is detected without being logged — and why tof_fouling_offset resets at a cleaning.

What a cleaning does to a prediction

Two corrections, on two different channels, keyed to the same event.

1. ToF — sensors.tof_fouling_offset resets to 0. The offset is subtracted from ToF before turbidity (§12.3 step 1). A cleaning removes the film, so the accumulated offset must go; if it does not, the model keeps subtracting a film that is no longer there.
2. TLF — sensors.tlf_clean_step, applied by applyTlfCleanStep(value, step), which DIVIDES. [CODE] Set through POST /api/tlf-clean-step.

⚠️ The direction is the thing that keeps being got wrong. Fouling attenuates the fluorescence path, so a fouled probe reads LOW and cleaning steps the signal UP — the post-cleaning reading is the correct one. The sensor is fine and the operator did nothing wrong; what goes stale is the calibration, which was fit against the fouled state. See §3.1 and KNOWN-FALSE §7.4.

[MEASURED] City of Paris, 2026-08-18 — the only cleanings currently carrying a step:

Three sensors out of the fleet carry a clean step. Everything else is running with no TLF cleaning correction at all — which is correct only where nothing has been cleaned.

Four rules, each learned by breaking one

1. It must be TIME-AWARE. cleanStepAt(cleanSteps, barcode, time) returns the step only for readings at or after the cleaning, and 1 before it. Applying it retroactively deflated Bercy's entire pre-cleaning history from 47 → 3 CFU.
2. The cleaning time must be DERIVED from the mon2 step, not taken from when someone remembered to log it. The Paris times (06:45 / 08:00 / 08:45 UTC) came from the 15-minute series. A guessed time divides still-fouled readings.
3. Measure the ratio on the S-TLF slope, not mon2 net. base20 drifts and goes negative at some sites — 6 of 20 Bercy and Grenelle grabs — so the mon2-net ratio is undefined there.
4. S-TLF does NOT immunise against this. The slope stepped as much as mon2 (×2.29 / ×4.31 / ×1.35). It fixes readout problems — railing, pedestal, temperature — not optics problems. Never offer it to a partner as the fix for a fouling event.

⚠️ The correct calibration frame is an OPEN INVESTIGATION

[CONFIRMED] Evan, 2026-08-21: "We don't know yet — this is still an investigation." What ships today is candidate 1; it is not a settled design.

1. Fouled frame + step divisor — shipped. Keeps fe as fitted and divides post-cleaning readings so predictions stay continuous. Needs no new grabs, but leaves the calibration anchored to a degraded state and accumulates a factor per cleaning.
2. Clean frame canonical — treat the unattenuated reading as truth and correct history upward. Physically the more honest frame; rewrites past predictions.
3. Refit per cleaning epoch — no correction factors, but it fragments already thin grab sets.
4. Continuous fouling model — the TLF analogue of computeFoulingOffset. Most complete; needs a way to separate slow instrument drift from real water change.

What would settle it: grabs taken soon after a service, while the state is known. Paris has none after 2026-08-15. Ask partners to sample right after any cleaning.

In the lab this is modelled explicitly, and differently: the burn-in clean-water fit carries a settling term d·e^(−n/τ) where n is samples since the last clean, on a sample-count clock rather than wall-clock. A lifetime-count clock was tested and rejected — it predicted the next window at 17.1 % error, so the decay restarts at the clean rather than tracking device age.

10.4 The generic path's temperature ceiling — the arithmetic [CODE]

GENERIC_WARM_LIMIT_C = 22.6 is not a chosen constant, it is where the generic model crosses zero. Its temperature term applies to absolute T, not T − 20, and is large: at 26.5 °C it contributes −4.16 against an intercept of +3.54. So a sensor sitting exactly at its own clean-water baseline goes negative above 22.6 °C and clamps to the 0.1 CFU floor regardless of what TLF is doing — and warm water quenches TLF, which is precisely when mon2 − baseline is least able to compensate. On 2026-08-06 this had seven installed sensors reporting "0 CFU, safe", including a paying customer.

The fix was not new model structure. FIELD_MODEL already had a designed fallback for units with no grabs (fe_default + population slopes). What those sensors lacked was only the per-sensor baseline pair, which is what routes a sensor onto that path at all.

10.6 The nowcast compute path — a consumer, never a writer [CODE]

The E. coli nowcast (validation.thelume.ai/nowcast, and the customer view at thelume.ai/boulder/nowcast) is computed OUTSIDE this stack and pushed into namespaced D1 tables. Nothing in it can write to pumphaus or the inventory tables; the modeling stack itself is documented at validation.thelume.ai/global. Sourced from SweetSenseInc/lume_nowcast (nowcast_feed.py, .github/workflows/feed.yml) and lume-validation (functions/api/nowcast/[[path]].js), current as of 2026-08-31. Numbered 10.6, not 10.5 as drafted — §10.5 is the cleaning-events section.

pumphaus proxy (read-only, PIN)  ─┐
  endpoints: sipm / diagnostics / │
  tof / sweeps (S-TLF substrate)  ├─▶ hourly compute ──▶ POST /api/nowcast/ingest ──▶ D1: nowcast_latest
/api/validation/grabs (public)   ─┤    (GitHub Actions,      (Bearer token,              nowcast_series
/api/maintenance/all             ─┤     SweetSenseInc/        validated rows)             nowcast_grabs
Open-Meteo hourly rain           ─┘     lume_nowcast,                                     nowcast_meta
                                        cron :07)                                            │
                                                              /nowcast + /global + boulder/nowcast (GETs, public)

What it reads, and how. Raw sensor pulls go through the PIN-gated proxy only — sipm/diagnostics/tof incrementally, plus the full LED×bias sweeps (the S-TLF slope channel's substrate; §4). Grabs come from /api/validation/grabs per customer label — Chicago is split across "Chicago" and "Current - Chicago", query both or silently lose half (§10.3's endpoint; the split is a data fact, not a bug). Grab→sensor attribution is era-aware off installed_time: a grab collected during a unit's previous deployment never calibrates its current one (§8). The maintenance registry is re-fetched every run so a cleaning takes effect the hour it is logged (§9).

What it writes. Four tables, ~35k row-writes/day: each hourly run re-publishes a revisable 48 h window per site plus the last 30 days of grabs (a forced backfill seeds 21 days and the full grab archive); retention 400 d. Large payloads split into POSTs of ≤8,000 series rows — the function spends one subrequest per 250-statement D1 batch against the Workers budget of

1. Ingest caps: 20,000 series / 5,000 grab rows per POST.

Trust boundary. The bearer token is write-only-to-nowcast-tables: every statement in the Pages Function is hardcoded, identifiers never come from input, and stored strings are validated at ingest (closed enums for alarm/variant/channel, charset-capped text, structural JSON checks on the meta blobs) so a leaked token yields rejections, not stored XSS. GETs are public by design — nowcast output is not sensitive (same stance as §10.3). A dedicated NOWCAST_DB binding keeps it structurally off the shared DB.

Degradation is loud, never silent. Every run compares its output against a per-site expectations ledger (channel slope > mon2 > none; variant sensor+rain > sensor > none) AFTER pushing: serving below best-known turns the Actions run red and emails watchers, while the dashboard keeps serving the honest fallback with widened bands. Provenance rides every row — model_version (maps to an exact commit of pinned model state), channel, variant. This machinery caught a fouled sensor that was still transmitting but no longer measuring (BC_Eben, 2026-08-28).

⛔ yhat_log10 is FLAT at shadow sites by design. Under shadow the served value is site calibration, which only moves when a grab or prior lands. Anything plotting the served series at a shadow site draws a flat line — that is the honest number, not a bug. The sensor-informed dynamics live in shadow_yhat_log10 / shadow_p (the arm's own estimate and exceedance probability, null when its features are incomplete); dashboards should draw both, labeled. As of 2026-08-31 the whole fleet publishes shadow=1; promotion is per site, from the logged record.

This is the explanation for §15.1e. That section recorded, as a [MEASURED] finding on 2026-08-29, that twelve of 27 sites publish a number with no sensor input. The flat served series is the shadow posture working as designed, not twelve broken sites. Read §15.1e and this paragraph together, and do not treat either alone as the whole picture — the design explains the flat line, and it does not explain every open defect logged against the feed in §15 (a rain input that moves nothing at Chicago, and an always-alert site, are separate questions and remain open).

Per-customer action limits (§6) reach the nowcast via state/fleet_sites.json in the compute repo and are republished per site in nowcast_meta.thresholds_json (action_limit, alarm_on 0.25, alarm_off 0.15). The advisory prices a missed exceedance at ~3× a false alarm and holds a standing alarm when the sensor goes dark rather than clearing it.

11. Process invariants (non-negotiable)

1. Fit on the features the server actually serves. Pull them from /api/field-refit (tofd, foul_off, rain48, tlf_slope, iso). Three separate bugs came from fitting on offline recomputes.
2. Replay shipped constants against the fit set before deploying, and require an exact match. This caught coefficients transcribed from a truncated terminal read.
3. De-foul each grab with the offset in effect AT ITS OWN DATE, never today's stored offset applied to history. Sensors get cleaned.
4. Never transcribe coefficients or metrics — regenerate them.
5. Break classifier true-positives down by site, and benchmark against a level-only model. If a level-only model ties, the features are detecting nothing and the metric is a site base rate. (Paris: all 4 detections were one site; level-only matched it.)
6. node --check is not sufficient — wrangler's deploy validation catches what it misses.
7. Report in-sample metrics only. No LOOCV. No within-one-category accuracy.

---

12. Calibration

12.1 Routing — first match wins [CODE]

predictEcoliCfu tries, in order:

CHICAGO → CLEVELAND → DC → DENVER → PARIS → FIELD (base20) → generic (ECOLI_CFU_COEFS)

A barcode reaches a regional model only by having an fe entry in it. Adding a barcode to a regional model silently re-routes it — check what it was using before.

12.2 Baseline hierarchy — highest available wins

[CONFIRMED] Never synthesise a baseline when a lab capture exists. Read it as the median LED-512 / bias-nearest-3000 mon2 over the stated window — and never over a cooldown transient (the 6/02–03 batch ramps the whole way; use the cooled sub-window).

[CONFIRMED] The clean-water baseline is NOT the install window. They are separate records. Never derive a clean-water baseline from field data inside an install window.

install_tof_baseline must be the sensor's true clean floor. A fouled or turbid install window inflates it and hides real fouling — 50072 was set to 442 against a true floor of ~29, so a climb to 738 never flagged. [MEASURED]

A synthetic baseline is acceptable under a per-site fe model (the intercept absorbs a constant offset) but not on the generic path, which has no per-sensor term to absorb it.

A derived floor MAY anchor a per-sensor calibration [CONFIRMED]

Evan, 2026-08-21: a rank-2 derived floor is good enough to anchor a fit once the unit has enough in-water data. It is not a second-class baseline that has to be replaced before the sensor can carry its own coefficients.

Precedent: 50058 (Chicago Cal-Sag). No lab capture; base20 = 670 = p10 of 4,559 in-water readings across the 04-18 → 06-30 deployment. It is deliberately not in synthetic_base and carries its own fitted fe = 1.9081. That is correct, not an oversight.

No numeric bar has been set for "enough" — the judgement is made per unit at refit time, with 50058's volume and span as the reference case. What disqualifies a floor is evidence, not provenance: if the deployment had no genuinely clean period, the p10 is a fouled floor and anchoring on it biases every prediction from that unit low (the 50072 install_tof_baseline failure, one line up, is the same mistake on the ToF channel).

Not yet earned it — no lab capture, no local grabs, still in synthetic_base, all Deployed: 50041, 50055, 50060. 50056 (SDSU) has no base20 at all and falls to the generic path. A synthetic listing means not locally calibrated yet, not disqualified; when a unit accumulates in-water history or local grabs, remove it from synthetic_base.

Reading rank 2 against the rule above it. "Never derive a clean-water baseline from field data inside an install window" governs the rank-1 record — do not backfill sensor_baselines / clean_water_baseline_at from deployed data and call it a lab capture. Rank 2 is the declared fallback when rank 1 does not exist, and it is field data by definition. The two do not conflict; a derived floor must never be written into the lab baseline fields.

12.3 Correction stack — order matters

Apply in this order; each is anchored to a different event:

1. ToF fouling offset (sensors.tof_fouling_offset) — subtract before turbidity. Per-date for historical data, current stored value for live. Resets at a cleaning.
2. TLF clean step (sensors.tlf_clean_step) — divide both TLF references. Time-aware: only from the logged cleaning onward.
3. Temperature normalisation — S-TLF to 20 °C via tlfSlopeAt20C. The coefficient is now per sensor, not a fleet constant — see §12.3b.
4. Model — per-site fe + slopes.

A fifth step, offset and gain (§12.3c), converts a corrected amplitude into ppb. It is not part of this serving chain — the E. coli models consume the corrected signal directly and absorb each unit's sensitivity in its own fe — but it is required for any reading reported as a concentration, and it is the only step in the whole stack that makes two sensors comparable in absolute terms.

Fouling is called even where the flow guard cannot run [CONFIRMED]

Evan, 2026-08-21, asked whether the detector should hold back at sites with no usable gauge: keep calling it. The offset is roughly right either way — storm turbidity and fouling both inflate the ToF floor, and both should be subtracted before the floor is read as water. The decorrelation guard (FLOW_CORR 0.4) improves attribution; it is not a precondition.

Where the guard actually runs: Boulder (06730200), Denver (06711565), CRWA (01104500), and Chicago via per-site overrides (north sites → 05536085 Pulaski, Calumet sites → 05536290 South Holland; Lemont 05536890 demoted to fallback as a 30-mile-downstream proxy). Where it does not: Paris, Cleveland, BGS/Thames, Bow River — no gauge, so a rising floor is genuinely ambiguous and the detector calls fouling regardless.

The two outputs of a fouling call are not equally supported, and must not be conflated:

• The offset is defensible without the guard, per the rule above.
• The data_suspect window the scan also opens is a claim about the instrument, which a gauge-less site has not established. That is precisely why those windows are never an automatic exclusion (§14) and expire into review at 14 days (§14). 50064's window, open since 2026-07-10 from auto-detect, is the live example of the two coming apart.

⚠️ Known cost, accepted. Where the offset absorbs storm turbidity, the corrected ToF understates real turbidity during storms — which at CSO-driven sites is exactly when E. coli spikes. Chicago and Cleveland are insulated because they carry antecedent rainfall as a direct covariate, so the storm signal reaches the model through a channel the offset cannot eat. Sites with neither a gauge nor a rainfall term have no such backstop.

Revisit when ranging lands. "Roughly right either way" is a stand-in for a discriminator we do not yet have (§2 banner). A working distance_mm separates a film on the window from particles in the path directly, at which point this rule should be re-derived, not inherited.

12.3b Temperature quench is PER SENSOR [MEASURED]

shared/tlf-quench.js (generated by scripts/build_tlf_quench.mjs, never hand-edited) replaces the fleet constant SLOPE_TEMP_RHO = −0.024 with a measured per-sensor coefficient ρ = d ln(amp)/dT. Resolution order, first hit wins:

1. the unit's own measured ρ from TLF_QUENCH;
2. the batch coefficient TLF_QUENCH_BATCH_TLF = −0.0215 (−2.15 %/°C);
3. the legacy fleet constant, retained only so nothing loses a coefficient.

14 units currently carry a measured ρ. The spread is the point: −1.01 %/°C (50062) to −3.75 %/°C (500193), a 3.7× range. A pooled coefficient silently attributes one sensor's response to another.

🚨 FIT ρ ON SHORT WINDOWS. A long window absorbs the drift's curvature into ρ and can flip its sign. [MEASURED] 2026-09-08 on 50043, reproducing on a bench unit what /barrel established for the fleet: refitting on 5 h windows made an apparently unstable b_T stable. Fitted across whole multi-day holds, 50043's three clean holds give +0.83, −4.25 and +3.25 %/°C, two of them positive, and an earlier version of this section read that as "no usable ρ" and raised an alarm over every unit in TLF_QUENCH. Both are withdrawn. Refit on 5 h windows, the interval short enough that drift contributes ~0.1 % against the full daily swing, the same unit reads −1.94 %/°C (21 windows, 81 % negative) and −2.41 %/°C (36 windows, 94 % negative) in its first two holds — beside the fleet's −1.71 and the TLF batch's −1.78 — and +0.34 %/°C with 44 % of 36 windows negative in the third, which is a coin flip and means no quench is identifiable there rather than a positive one.

⚠️ What survives is a procedural rule, not an alarm. A sign gate applied to long-window fits will keep whichever window's curvature happened to come out negative. The gate is not the defect; the window length is.

The procedure, concretely. ρ is fitted by regressing ln(amp) on the sensor's own sipm_temperature. A long window contains both the daily temperature cycle and the slow drift, and where the two overlap the fit cannot separate them, so the drift's curvature is charged to ρ. Cut the window short enough that the drift is small against the temperature swing, then fit many of them:

1. slide a 5 h window by half its length across the hold;
2. drop any window spanning less than 0.5 °C — there is nothing to fit against;
3. fit ln(amp) ~ sipm_temperature inside each surviving window;
4. take the median slope across windows, and report the share with the expected negative sign as the confidence check.

The arithmetic is why 5 h works. On 50043 a 5 h window carries a median swing of 0.78–0.94 °C, worth about 2 % at −2.4 %/°C, while the drift over the same 5 h is about 0.8 % at 4 %/day: the temperature term is roughly 2.4× the nuisance. Over the whole 135 h hold the drift totals 15 % against 11 % for the temperature swing, and the two are no longer separable.

Any ρ in TLF_QUENCH fitted over a window longer than a few hours should be re-derived this way before it is trusted. §13.5m carries the worked example.

HARD RULE — fit ρ on the sensor's OWN sipm_temperature. Never a board thermistor, never an external reference. An early generator read x.temperature (the board channel) and produced a table that had to be withdrawn.

HARD RULE — ρ is fitted only where the analyte is constant and the temperature moves. Two window types qualify: the registered clean-water capture, and a labelled fixed-concentration temperature ramp. Everything else is disqualified, and the guards that enforce it are not optional — each one caught a wrong number that had already reached a shipped table:

Candidates are selected on SE(ρ), not R² — R² rewards a wide temperature span even when the fit is loose, and SE is what actually bounds the correction.

⚠️ Known weakness. 50046 and 50066 rest on a 0.7 °C span. Their ρ values (−2.98, −3.37) are the least trustworthy in the table. Sensitivity is flat enough that stopping the use of a pooled coefficient matters far more than the exact value each unit lands on — but these two should be re-fitted on a real ramp before anything leans on them.

Quench does NOT help when sensors share water [MEASURED]. At BC-13, five units in the same stream: a uniform ρ is exactly neutral (6.31 % → 6.31 % inter-sensor spread), because sensors that share water also share temperature, so a common correction cancels. Any difference between per-sensor ρ values is applied to a common T and pushes the units apart. The correction can only earn its keep where sensors sit at different temperatures. Drift correction, by contrast, is the dominant lever there: 6.31 % → 2.20 %.

12.3c Offset and gain are still required [MEASURED]

The amplitude is exp(c − c0) — a ratio to each unit's own reference sweep. That is exactly what makes it railing-proof, and it is also why it does not put two units on a common scale: every unit reads ≈1 at its own reference state regardless of how many counts per ppb it actually delivers. Converting to concentration needs a per-sensor pair:

ppb = (corrected amp − offsetC) / gainC        // slopeView[bc] in dilution_calibration.json

Fitting conc = a + b·amp per sensor on the 2026-07-16 dilution ladder, 13 sensors:

A single shared calibration is barely better than predicting the mean. Reproduce with scripts/dilution_feature_compare.mjs.

The blanks fall into two groups (≈0.52–0.64 and ≈0.96–1.12) tracking a two-tier gain structure (≈93–118 against ≈404–983). That is a hardware population difference, not something any correction removes.

Operational consequence. A new sensor needs a dilution ladder before its readings mean anything in ppb, and a repaired or re-capped one needs a fresh one. The FDOM, Chl-a and new-TLF batches carry no ladder calibration at all, so they can be compared in shape and never in concentration. 50091 has a negative fitted gain (−0.00054 at R² 0.30) — more tryptophan reading as less signal, which is not a calibration.

DERIVE THE OFFSET AND GAIN ON THE CORRECTED SERIES [CONFIRMED]

Evan, 2026-08-21: "you should be deriving the offset and gain only from the temperature and drift corrected series." They now are. build_dilution_calibration.js fits temperature and drift jointly on the clean-water window — sequentially would let whichever ran first absorb the other's effect — and the ladder step means divide out both before gainC/offsetC are fit.

This is only identifiable because of the window. The retired tap-water baseline was a monotonic cooldown with temperature and elapsed time at r = 0.88, where a time term would have been a second copy of the quench. The DI outdoor window carries real diurnal cycling: measured |corr(temp, days)| ≤ 0.69 across all 24 fitted units, over a 6.1-day span that clears the 12 h floor drift_correction.mjs requires. Effect, measured:

The fleet-mean ladder R² dips slightly even though most units improve, because a few units with a poor quench fit to begin with get worse. Per the never-drop rule those are kept and labelled, not gated out: drift lands at −22.4 %/day on 500108 and +13.1 %/day on 500226, both of which carry weak quench fits (R² 0.48 and 0.75). 17 of 24 sit inside ±5 %/day. The drift origin ships as driftOrigin so a consumer corrects on the identical basis.

⚠️ A calibration is only valid against a signal carried on the same basis it was fitted on. offsetC/gainC are fitted on the ladder's absolute corrected amplitude (blank ≈ 1.38). /barrel plots a baseline-relative amplitude sitting at ≈ 1 by construction. A row that subtracted one from the other was added on 2026-08-21 and removed the same day: it put 69 % of its points below zero (median −9.3 ppb, floor −106 ppb). /barrel carries no ppb axis and is not to be converted to ppm — the offset and gain belong to the signal.

What the amplitude does buy on the ladder is robustness, not accuracy. Against absolute mon2 it is level (pooled R² 0.9087 vs 0.9097 on the 12 sensors both fit), but it fits 13 of 14 sensors where mon2 fits 12, and beats the superseded slope on 11 of 13 with the gap concentrated on the hard units (50091 0.4835 → 0.8131, 500193 0.8713 → 0.9260, 50062 0.8232 → 0.8700). The estimator does not make good sweeps better; it stops bad ones from being wrong.

12.4 Calibration provenance

isLocallyCalibrated(barcode) is true only for sensors calibrated against real local grabs. A synthetic baseline routes a sensor to the field path but does not make it grab-calibrated. When a site gains real grabs, remove it from synthetic_base or it will keep reporting as uncalibrated.

12.4b Exogenous covariates

[CONFIRMED] (Evan, 2026-08-21) Rainfall is a legitimate model input on a customer-facing dashboard. It is not a sensor channel, but it drives the real mechanism — wet weather → CSO discharge → fecal loading — and it measurably improves the fits. The dashboard reports risk, not a raw instrument value.

Current use: Cleveland is rainfall-only (24 h, capped 5 mm) — the sensor contributes nothing beyond its per-site level. Chicago carries a 48 h rain term alongside ToF and the S-TLF slope, and rainfall is its strongest single predictor (pooled r = 0.54 against ~0.1 for ToF). Neither Paris, DC, Denver nor FIELD uses rainfall.

Sourcing rules are in §7.6: ERA5 archive must overwrite the forecast series, and the accumulation window is per model — call rainWindowHours(barcode).

12.5 Rules for any refit

1. WHEN to refit is a human decision — the audit only reports the drift [CONFIRMED] (Evan, 2026-08-21, choosing among an exceedance trigger / a percentage trigger / report-only / a schedule). There is no automatic refit trigger. On grab sets this small, refitting on every arrival chases noise; the failure mode being closed is the other one — a model sitting at its original n because nobody was told the set had grown. Every model declares fit_n / fit_date (and fit_scope where the fit's inclusion criteria are narrower than what the path now routes). scripts/audit_pipeline.js raises model.refit-drift with the gap, the growth %, and how many exceedance grabs the routed set now holds — those are the scarce ones that move a classifier, so a model that gained one is the one to look at first. model.fit-n-undeclared fires if a model omits it. [MEASURED] 2026-08-31: DC REFIT AGAIN at n=26 (the 08-27 n=18 set plus the 8 ARK grabs of 08-24/08-26). R2 0.232 → 0.343, RMSE 0.614 → 0.568 on the same rows; exc_t moved 145 → 232 and the DECISION IS UNCHANGED (identical tp7/fn0/fp6/tn13, BA 0.842). The refit bought magnitude only — it detects no extra exceedance — and leave-one-day-out CV gives R2 0.070 against the in-sample 0.343, so quote the CV number externally. The n=18 constants had actually held up out of sample (BA 0.842 on the new set, above their own 0.769 at n=18). Paris/Seine was refit the same day and REJECTED: +0.006 R2 for a forced re-tune of both serving cuts that lands on identical BA (Seine 0.773, Marne 0.758); a Seine-only model scores identically to the pooled one (R2 0.241 either way), and a free ToF slope buys +0.007 BA on the Seine for −0.062 on the Marne. Seine predictions still top out at ~397 CFU against a 900 limit, so its sensitivity AT the limit is structurally zero. [MEASURED] 2026-08-27: DC was CLOSED — refit at n=18, drift then 18 → 18. It was the live case at 8 → 18 paired (+125%) on 2026-08-21, and its own source comment had already said "Refit as ARK's twice-weekly samples accumulate." The refit moved magnitude a long way (R2 −0.984 → +0.285, RMSE 0.960 → 0.576; the n=8 fit was mis-levelled badly enough that fe['50057'] railed Kingman at CFU_ULOD) and moved the decision slightly the other way (BA 0.662 → 0.623 at the 410 cut, one extra false positive out of 13 negatives, same sensitivity). ⛔ The n=8 fit's headline numbers are RETRACTED — R2 0.633 / BA 0.93 were three grabs per sensor from fresh installs in one summer week and did not generalise. DC's ToF slope is now 0, not held: a free slope reproduced KNOWN-FALSE §7.9 on this set (1.2e6 CFU), and the "held at the CHICAGO value" provenance had gone stale (Chicago is 0.0000831 today against the 0.0035829 DC carried, ×43). Chicago is +9% (4 new). FIELD reads +150% but its fit_scope is Boulder + Chicago healthy sensors, and the path now routes BGS/CRWA/Denver-synthetic units too — scope first, then compare.

1. Fit on served features (/api/field-refit), replay before deploy (§11).
2. Re-derive exc_t. It is not a constant of nature; it moves with the model. A model whose predictions top out below the action limit has zero sensitivity there — Paris's slope branch tops out ~737 CFU against a 900 limit, so exc_t is 250.
3. exc_t is set to whatever maximises balanced accuracy [CONFIRMED] (Evan, 2026-08-21) — above or below the action limit. It is a tuning parameter, not a safety margin, and there is no rule that it must sit below the limit. Sweep the threshold, take the BA-optimal point, and report sens/spec at it. Preferred, where it costs nothing: level-calibrate the model so the BA-optimal point lands ON the action limit, so the validation panel reads cleanly with both axes at the same number (this is what Chicago does at 200). Do this when the level shift is small; do not do it when the shift would wreck the continuous fit — Paris needed +0.55 log and R2 fell to −0.27, so it keeps exc_t 250 against a 900 limit instead. Current spread is wide and legitimate: Boulder 78/126, Paris 250/900, Chicago 200/200.

The cost asymmetry is deliberately NOT modelled [CONFIRMED] (Evan, 2026-08-21, asked whether to favour sensitivity, favour specificity, or set it per partner): keep balanced accuracy. A missed exceedance and a false alarm do not cost a partner the same thing, but that is the partner's policy to set on top of the model, not something we bake into it. Tuning toward sensitivity would mean shipping a model whose stated performance is no longer the performance — the asymmetry would be hidden inside exc_t where nobody can see it.

What this obliges instead: always report sens and spec with their counts, never BA alone. BA hides degenerate arms. Cleveland reads specificity 1.00 — from 4 non-exceedances, and is a rainfall-only wet-weather detector that is blind to dry-weather events by construction; quoting its BA 0.833 without (4/4) next to it is a false impression, not a rounding. [CODE] all five validation panels (chicago, cleveland, dc, denver, marne-seine) already render (tp/tp+fn) and (tn/tn+fp) beside each rate — keep it that way.

Current operating points, none of them chosen for their asymmetry: Chicago 0.72/0.67 (n=47), Cleveland 0.67/1.00 (n=13), Paris 0.80/0.81 (n=48), DC 1.00/0.86 (n=8), Boulder 0.87/0.86 (n=58).

1. Never ship a wrong-signed term to buy a fraction of a grab. Check the within-site sign first; if it is inconsistent across sites, hold the slope or drop the term.
2. Report by site, and benchmark against a level-only model (§11.5).

---

13. Dashboard rules

13.1 Never display these as measurements

[CONFIRMED] The generic path floors above ~22.6 °C (GENERIC_WARM_LIMIT_C): its absolute temperature term drives any warm site to the floor regardless of TLF. A floored value at a warm, uncalibrated site means "we cannot tell", not "clean". This is exactly how seven installed sensors once showed a green "SAFE — 0 CFU" badge.

Provenance — this rule is historical, not from current testing. [CODE] It comes from a comment in shared/ecoli-model.js documenting a real incident on 2026-08-06: seven installed sensors (City of Paris ×5, Marne Vive, Xylem-Potomac Yards Marina) rendered a green "SAFE — 0 CFU" badge that was really predictEcoliCfu hitting its CFU_LOD clamp on the generic path.

Current exposure is small. [MEASURED] 2026-08-21: all seven of those sensors now carry base20 entries, so they route to FIELD/regional, not generic — the incident was closed by adding baselines, not by a display gate. 8 of 43 deployed sensors are still on the generic path, and they are bench/DMRV/drinking-water units (Suez Lab, AFRL, Rwanda ×2, MWA DRIP, SDSU) plus Bow River Basin Council 500193 / 50071, which are the only surface-water ones. So the risk is real but narrow, and it lands on Bow River rather than the recreational dashboards.

ecoliEstimateQuality() returns {cfu, reliable, atFloor, calibrated, reason} for exactly this and is emitted to the browser, but no dashboard calls it. That is belt-and-braces that was never wired up — not an active failure.

Uncalibrated sensors: disclose at PAGE level, not per reading [CONFIRMED]

Evan, 2026-08-21, choosing among wire-the-gate-up / page-disclaimer / withhold-the-number / leave-it: page-level disclosure. A dashboard whose sensors carry no site-specific grab calibration says so once, in the hero, and individual readings render normally. The per-reading quality gate stays unwired — deliberately, not by omission.

Why this is not a fudge. The prediction is a real estimate from the population model; it is the provenance that differs, and provenance is a property of the deployment, not of any one 15-minute reading. Marking every value would imply the numbers are individually suspect, which is a different (and unsupported) claim.

[MEASURED] 2026-08-21 — every sensor at these two customers is on a synthetic baseline with isLocallyCalibrated() === false:

crwa.html also claimed in its hero to be "validated against Colilert grab samples". It was not: no CRWA sensor carries a local calibration and the customer has 2 grabs total (2026-06-30, 2026-07-07). Corrected the same day to say grabs are being collected to build a calibration. A page may not claim validation the calibration set does not support.

When a page's sensors gain a local calibration, remove its disclaimer — a stale "no calibration" notice is as wrong as a missing one.

13.1b Stale readings show as offline, not as a colour [CONFIRMED]

Evan, 2026-08-21: past some age a site shows offline rather than a stale colour.

A dashboard paints the newest reading it can find and, until now, had no idea how old that was. Firmware reads every 15 min and syncs hourly, so a unit dark for weeks still has "a most recent reading" and rendered exactly like a live one. riskStatus's existing offline state only fired on a null CFU, and const offline = !openSeg only tested the install window.

[CODE] STATUS_STALE_HOURS = 6 + isStatusStale(lastSeenIso) in shared/ecoli-model.js, wired into all seven site dashboards (chicago, cleveland, dc, crwa, denver, marne-seine, boulder). A missing last-seen is deliberately not stale — that is a different failure, and greying a site over a gap in the status feed would be wrong.

6 h is measured, not chosen. [MEASURED] 2026-08-21, across 24 deployed recreational sensors: every reporting unit was ≤ 2.3 h stale (n=13, median 1.6 h) and the next unit up the list was 92 h. Nothing sits between 2.3 h and 92 h, so 6 h (~6 missed syncs) separates the two populations with wide margin either side. Re-measure this gap after any firmware cadence change — the threshold is only as good as the separation it was drawn from.

⚠️ [MEASURED] 2026-08-21 — 11 sensors marked Deployed had not reported in days to months and were being painted as live status: CRWA all four (50055 8.9 d, 50068 39.7 d, 50069 56 d, 50075 71.8 d), Chicago 50070 Main Stem 6.6 d and 50064 Wild Mile 16.7 d, Boulder 50059 3.9 d and 50046 9 d, Bow River 500193 29.7 d and 50071 71.9 d. (50047 CSU Spur is In Stock, so expected.) This rule is what stops that rendering as current water quality; it does not fix the sensors.

13.2 Thresholds on validation panels

• Both axes use the customer's action limit (§6). Observed ≥ limit vs predicted ≥ limit.
• When exc_t and the action limit differ (Boulder 78/126, Paris 250/900), the panel must say which number is which — a bare "predicted ≥300 vs observed ≥200" reads as an inconsistency and invites mistrust. Prefer level-calibrating so they coincide when the shift is small (§12.5.3); label the difference when it is not.
• Panels must read from the same served evaluation the dashboard uses (/api/field-refit), so the panel cannot silently disagree with what is being served.
• Label provisional fits on the page, not just in the code — n, exceedance count, and the fact that specificity on 4 negatives carries no weight.
• [CONFIRMED] (Evan, 2026-08-21) Publication is the partner's call, once they have seen the numbers. There is no minimum grab count or accuracy bar we enforce. Our duty is disclosure: show the confusion matrix, n, exceedance count and the limitations, then let the partner decide whether it goes on their public page. Cleveland ships on 13 grabs and Paris detects at only one of four sites — both legitimate under this rule, because the numbers are on the page.
• It follows that a metric shown to a partner must be a MEASUREMENT. A number that was tuned to look credible is not disclosure — it defeats the rule, because the partner cannot make their call from it.

⚠️ Denver runs a DEMONSTRATION calibration — labelled on the page since 2026-08-21. DENVER_MODEL's coefficients were chosen by constrained grid search over 6 paired lab samples to land a credible profile (sens 1.00, spec 0.50, BA 0.75, R2 0.65); the model comment says "demo-tuned, NOT predictive". Its validation panel therefore describes the calibration, not measured performance, and now says so directly beneath the figures, alongside the beta disclaimer the other partner dashboards carry. Replace with an honest fit once verified in-water pairs arrive (fallback: fe 2.540 / 2.179 / 2.593, Chicago-held slopes, sens 0.67 / spec 0.50). While a model is demo-tuned, its metrics must be labelled wherever they are displayed — otherwise the partner's-call rule above has nothing real to stand on.

13.3 Data path rules

• >48 h serves the hourly site_rollup; ≤48 h hits the live pumphaus fan-out (~51 s). If a view only needs the latest reading, ask for a window >48 h — a 48 h request costs ~50 s and buys about an hour of freshness. [MEASURED]
• site_rollup stores predictions, so after any model change rebuild it per customer in ~1-week chunks, or the dashboard serves the old model while the validation panel shows the new.
• A client-side PIN is not access control. The PIN is in the page source and the data endpoint is open; treat such gates as cosmetic and do not describe them as security.

13.4 Presentation integrity

• A cleaning, a re-install, or any handling is an instrument event, not a water event — it must not be shown as a water-quality change (§3.1).
• When a metric is corrected, retract the old number explicitly in the model comment and here; do not quietly publish a new one.
• Prose that points at a chart feature ("the purple points above") must be updated when the feature is removed.

---

13.5 ⚠️ OPEN CONFLICTS with the validation programme (read before any TLF work)

[MEASURED] 2026-08-21, from validation.thelume.ai — /findings, /burnin, /degassing, /barrel, /colilert. These are not settled; they are places where production and the validation bench state different things. Nothing here has been changed in production.

13.5a ✅ ADOPTED — the canonical TLF amplitude estimator

[CONFIRMED] Evan, 2026-08-21: "twin and production need to adopt the slope fitting approach of the experiments." Done. shared/tlf-amp.js is a bit-identical port of the validation programme's lume-validation/scripts/lib/tlf_model.js, verified against the original on 96 real production sweeps (39 exercising the censored path): every model parameter and every fitted amplitude matched to zero difference.

The estimator: amplitude c in log(mon2 − dark) = a_LED + b_overvoltage + c, fitted per sweep against that sensor's own LED/overvoltage shape.

• Support-invariant by construction — with a and b fixed, c is the weighted mean of (y − a − b), so every cell is an independent estimate of the same number and dropping one cannot move it. The old quadratic fitted a shape against the surviving cells, so which cells passed screening moved the answer. Measured effect on 96 sweeps: 96/96 yield an amplitude.
• Clipping is censoring, not missingness. A cell at the 3250 ADC ceiling enters the likelihood as a bound (>= SAT), never discarded.
• Overvoltage, not bias. ov = round((bias − 2.6·(T − 21)) / 16); Vbd moves 26 mV/K (AFBR-S4N33C013) ≈ 2.6 counts/K. Keying to raw bias smears a temperature-shifted curve.
• LOW = 200 is a screen, not a floor. Low cells are low-SNR, not censored; weights (w = v²/(v+25)) fade them out smoothly.

[CODE] per-sensor shape in shared/tlf-shape.js, built by scripts/build_tlf_shape.mjs (7-day windows; 15 sensors). Three were built from an earlier window because the unit had gone dark — 50064 → 2026-08-04, 50070 → 2026-08-14, 50046 → 2026-08-12. Rebuild the shape when a sensor's optics change (repair, cap swap, re-baseline); a shape fitted to different hardware silently rescales its amplitude.

⚠️ TWO PRODUCTION DEVIATIONS, both the estimator's own documented fallbacks, both visible:

1. No per-cell dark. Pumphaus's Django proxy serves {barcode, timestamp, led_power, sipm_bias, mon1_val, mon2_val} and does not expose mon2_dark, so every cell falls back to the PED = 170 constant. The validation archive has real per-cell dark. Exposing it on the proxy is the fix.
2. Temperature is per sweep, not per cell — the cycle's diagnostics reading is joined to every cell, falling back to TREF = 21 when absent.

✅ SWITCHED 2026-08-21, after refitting both models on it. /api/field-refit exports tlf_amp, tlf_amp_n, tlf_amp_ncens, quench, quench_src per grab, and both refits were fit on those served values, then replayed through predictEcoliCfu and required to match before shipping (CLAUDE.md §1).

Paris genuinely uses the new feature. lamp = +0.743, correctly signed, and positive within-site at 3 of 4 sites. BA 0.770 → 0.790 on identical rows.

Chicago does not, and that is the finding. Fitting the TLF term free on the canonical amplitude gives −0.248, and the within-site check agrees at 4 of 5 sites (−1.88, −1.43, −1.52, −0.49 against +0.96) — so it is not Simpson's paradox, fluorescence really does run inverse to E. coli at these turbid urban sites. Holding it at 0 also scores better (0.778 vs 0.753). So Chicago's gain over 0.745 comes from 4 more grabs, not from the feature, and CHICAGO_MODEL.rain_slope is demoted — reached only if the rain branch cannot serve.

⚠️ Chicago's sensitivity fell 0.88 → 0.67 (missed exceedances 4/33 → 11/33) while specificity rose 0.61 → 0.89. Balanced accuracy is the target and the asymmetry is the partner's policy (§12.5), so the BA-optimal point ships — but this is the direction that costs a swimmer, and it is a one-line change to trade back.

[MEASURED] two rounding traps the replay caught, both worth remembering:

• The Paris sweep first reported an optimum at "270.21". That was the rounded form of a prediction at 270.207, so >= 270.21 excluded the point by accident. exc_t is 272, mid-gap between 270.207 and 274.678.
• The first replay scored Chicago 0.697 because the harness reconstructed ToF from tofr while the fit used tofd. Fit and replay must read the same served column.

13.5i ⚠️ OPEN — net = mon2_val − mon2_dark is doctrine in two places and implemented in none

[MEASURED] 2026-08-21. Both validation.thelume.ai/data-pipeline ("the usable signal is net = mon2_val − mon2_dark; mon2_dark is ambient light and is exactly removable") and this document's §2 state dark subtraction as the rule. No production path does it. Every consumer — /api/field-refit, the serving model, all ten dashboards — reads mon2_val raw. The only code that subtracts dark is suez.html (mon2 = mon2_val − mon2_dark + 173) and db-proxy-server/dark_qc.js.

✅ There is now a worked counter-example. [MEASURED] 2026-09-08: 50043's full sweeps were pulled straight from api_samplesraw and the canonical estimator run on them with real per-cell dark, on 40,492 sweeps of 99 cells. It is the only place in the programme running §13.5a without the PED = 170 fallback, and §13.5m is what that bought: three windows that failed the scatter gate at 13.67 / 8.32 / 9.92 % came in at 1.88 / 0.29 / 1.30 %, and a 7 % apparent response below 1 ppb turned out to be noise in one cell. This is the size of the prize for exposing mon2_dark on the proxy, measured rather than argued.

The cause is structural, not an oversight: mon2_dark lives only in api_samplesraw.raw_payload and pumphaus's Django proxy does not expose it. The proxy serves {barcode, timestamp, led_power, sipm_bias, mon1_val, mon2_val}. So no live consumer can reach it, and the canonical amplitude estimator falls back to the PED = 170 constant (§13.5a deviation 1).

Consequence: every coefficient in every model was fit on un-dark-corrected TLF, and serves the same. That is self-consistent — fit and serve agree — so it is not a live error. The risk is narrower: mon2_dark measures ambient light, so a unit with a light leak reads high and nothing catches it. Suez 50043 is on record for exactly that.

Dark is also a fault signal. [MEASURED] 500124's dark channel collapsed from ~167 counts to 0 against a fleet 171–176, 95 minutes before its blackout cap was fitted; it was leaking and was withdrawn. A dark channel far off the fleet is an instrument fault, not ambient.

Exposing mon2_dark on the proxy is the fix. Until then, do not describe production TLF as dark-subtracted.

13.5j Drift IS characterisable — per sensor, once windows respect the interventions

[MEASURED] 2026-08-21 by lume-validation/scripts/characterize_drift.py, on barrel data refreshed through 17:00Z. Every window boundary is an event, opened 2 h after it so settling is excluded: clean/immersion, the UV dose (per-unit DFU times), blackout caps, basin carry + refill + 40 °C ramp, in-place shake, air pull, mains changeover, campaign close, barrel shake. Temperature is fitted jointly per unit; |corr(t,T)| > 0.9 is called not separable rather than quoted.

⚠️ Correction to an earlier pass: immersion is 16:45Z, verified from the data — ToF steps 140 → 46 kcps in all three batches simultaneously, which is the air/water marker. An earlier run read the prose local time as UTC and put six hours of AIR data in the WATER bucket. Local here is UTC−6. Any air/water split must be anchored to the ToF step, not to a quoted clock time.

The controlled comparison: the same 24 burn-in sensors, both media

Drift does not stop in air — it drops to about a quarter. That is the discriminator burn-in Stage 8 was built to answer, and the answer is both, in measurable proportion: roughly a quarter LED/electronics, the rest water-side.

By batch, in air — the only medium where all four have data

The three new batches drift 3–6× faster than the 13-day-old burn-in fleet, in the same medium, over comparable windows, all separable. FDOM is the clean case: its IQR [−1.56, −1.11] does not overlap Burn-in's [−0.81, −0.09], 10 units against 24. /findings lists sensor age as "not established"; this is the strongest evidence for it so far — though age and build are still confounded, since the new batches are also new optics.

Why /findings found only one trustworthy window

[MEASURED] 7 of 9 clean windows are separable (|corr| 0.13–0.69), and five have ≥40 % of units passing split-half. Only two are not: post-UV → caps (0.91) and the entire barrel tank phase (0.96, 40 units) — which corroborates the barrel page's own refusal to quote a rate.

So "only the constant-temperature hold is trustworthy" is a property of how those windows were cut, not of the instrument. The unusable ones in /findings (e.g. "Water · days 2–5", drift −13.56 against a half-disagreement of 20.74) straddle the UV dose, the cap fitting, the basin carry and the shake at once. A slope fitted across four interventions measures the steps.

What still stands, and what does not

• ✅ "Drift is real" — confirmed. Burn-in water −1.50, air −0.41; FDOM air tight at −1.34.
• ⚠️ "Small" — true of the fleet median, but it hides a 30× per-unit range: 50088 air −11.64, 500128 air +8.10, 500108 water −10.81, 50095 water −10.75.
• ⚠️ "Not predictable" — untouched by this analysis, and still stands. Drift here is consistently estimable across units at a point in time (FDOM's 10 units span 0.45 %/day). /findings tested forward extrapolation in time and found it worse than assuming flat. Consistent across units ≠ persistent over time. Do not cite one as evidence about the other.

⚠️ Two Chl-a units fit a positive quench (500184 +3.12, 50081 +3.15 %/°C), which is unphysical — both are low-signal units in §13.5k. A positive b_T is a fit artefact, not a measurement.

13.5l Drift is a STEP between tests, not a rate within them [MEASURED]

[CONFIRMED] Evan, 2026-08-21: "In between the first water test and the second, and the air test, drift independent of temperature happened." Confirmed, and it is the more important half of the drift picture.

Measured at MATCHED TEMPERATURE (lume-validation/scripts/water1_vs_water2_matched_temp.js): the same unit, same medium, same 1 °C bin, water-1 against water-2 across the 2.7 days spanning the air phase — 21–22 bins per unit over 9–30 °C, so no quench coefficient is used and nothing is circular.

That flatness across 21 temperature bins is the signature of a genuine temperature-independent level change, not a quench-model artefact — an artefact would vary with the bin.

⚠️ This is 5–50× larger than any within-window drift rate. At the measured −0.4 to −1.5 %/day (§13.5j) a unit should move 1–4 % over 2.7 days. Six moved 15–22 % and five moved 14–185 %. Something happens between the tests that the within-window fits never see, and it is bidirectional — not a decay.

Consequence for the correction stack: a slope · days term models drift as a smooth continuous rate. The behaviour measured here is a large per-unit step across events. A rate term cannot represent a step, which is the mechanism behind /findings' result that extrapolating a fitted slope forward is worse than assuming flat.

Three of my own errors on the way, all worth not repeating:

1. Comparing raw net counts at the calibrated combo gave "0.70× recovery" and I read it as incomplete re-wetting. That is a different quantity from the amplitude every drift number uses, and it conflates the air/water medium step.
2. Comparing tlf_amp straight out of two band files is invalid: the amplitude is anchored per cohort to that file's own reference window, so burnin_bands and ditank_bands sit on different scales. Rebuild one reference across both archives.
3. Correcting with a quench fitted across both water phases is circular — the fit absorbs the very level change being measured. Matching on temperature removes the need for it.

13.5m Suez 50043 — the whole stack run on the one unit whose dark is reachable [MEASURED]

[MEASURED] 2026-09-08. The CIRSEE bench campaign (9 workbook tabs, 6 Aug to 1 Sep, published at thelume.ai/suez) was re-derived on the canonical estimator instead of mon2 at the calibrated combo. 50043 is a bench unit, so its sweeps can be pulled straight from api_samplesraw and it is currently the only place in the programme where §13.5a runs with real per-cell dark rather than the PED = 170 fallback. Treat what follows as the existence proof for exposing mon2_dark on the proxy (§13.5i).

Shape and yield. A 50043 shape was built from its own sweeps: 3 LEDs, 69 overvoltage bins, residual σ 0.048, 1228 reference sweeps over 107,169 cells. The unit sweeps 3 LED drives (32/128/512) × 23 biases = 99 cells, and the amplitude yields on 40,492 of 40,492 sweeps (100%). 7,698 of those carry at least one cell at the ADC ceiling, which the estimator bounds rather than discards.

What the amplitude fixes, and what it correctly refuses to fix. Six windows failed the 8% within-window scatter gate on mon2. Three of them were the estimator, not the water:

The first three are the campaign's three lowest-signal windows, and their scatter was one LED/bias cell: it is gone on the full sweep. The last three get worse, which is the estimator behaving correctly, because the water genuinely changed inside those windows and a more sensitive statistic sees more of a real transition. Gate failures go 6 to 3.

✅ The same test CONFIRMS the July reading rather than overturning it, which is what makes the August result trustworthy rather than just convenient. Run on the July campaign's windows, the amplitude leaves the noisy ones noisy and makes them noisier:

So July's low-signal scatter is in the water or the optics and August's was in the estimator. The air-bubble hypothesis for the July standards, raised on /suez and independently by Suez on their own slide 6, stands, and degassing is still the cheap test for it. What does not transfer is using July's explanation for August's numbers. Median CV over all 44 windows barely moves (0.87 % to 0.80 %), and the amplitude is quieter on only 15 of 44 — it does not make good sweeps better, it stops bad ones from being wrong, exactly as §12.3c says.

Ladder. The 12 Aug tryptophan ladder (nine steps, 0–45 ppb in MilliQ) fits R² 0.9996 across all nine on the amplitude against 0.9909 on mon2, and R² 1.0000 above 1 ppb at 0.0352 amp@20 °C per ppb. Detection limits tighten with it: within-run blank SD 0.00304 amp@20 °C (CV 0.10–0.87 % over 45–72 readings) gives LOD 0.28 ppb and LOQ 0.86 ppb, against 0.43 and 1.30 on mon2.

⚠️ The low end is still flat, and the amplitude is what shows it honestly. 0 / 0.1 / 0.5 / 1 ppb read 1.356 / 1.331 / 1.339 / 1.355 amp@20 °C: no response at all. On mon2 the same steps read 449 / 482 / 480 / 506, a 7 % rise off the blank that looks like a response and is noise. Better precision did not buy detection below 1 ppb; it removed a false one.

Quench, measured on 50043's own clean water under §12.3b's guards. Windows come from the lab's own annotations, are split at gaps over 600 s to enforce lab cadence, must clear 12 h, and temperature and drift are fitted jointly. The corr(T,t) column is a magnitude, and 0.69 is §12.3c's separability ceiling:

⚠️ On the 150 used throughout this section. It is the Seine serving cut (REGION_CUT.PARIS, set 2026-08-27 by a sweep on served Paris predictions: n=69 with 3 exceedances, 150 giving sens 0.67 / spec 0.85 / BA 0.758 against BA 0.629 at the old 250), and it is applied here as a deliberate choice because the water in all five runs is Seine. shared/ecoli-model.js does not assign it to this barcode: 50043 has no PARIS_MODEL.fe and no FIELD_MODEL.base20 entry, so modelNameFor and thresholdRegionFor both return null and decisionCut('50043') === actionLimit('50043') === 200, the ACTION_LIMITS.DEFAULT fallback. Nothing here turns on it: at 200 the three prediction matrices read BA 0.761 / 0.722 / 0.722 against 0.766 / 0.700 / 0.750, same ordering.

Selected on SE(ρ) per §12.3b: ρ = −2.78 %/°C.

🚨 RE-EXAMINED TWICE, 2026-09-08. The second pass reverses the first. Regressed across whole holds, ln(amp) on the unit's own SiPM temperature gives +0.83, −4.25 and +3.25 %/°C, two of them positive, and that was read here as "50043 has no usable quench, and the sign gate cannot tell no-quench from quench". That reading is withdrawn. It is the long-window artefact /barrel identified: the zero drifts non-linearly across a multi-day hold and the curvature lands in ρ. /barrel's own remedy, refitting short, is what this applies.

Refit on 5 h windows, gated to windows spanning at least 0.5 °C:

A and B sit beside the fleet's −1.71 %/°C and the TLF batch's −1.78, so 50043 is an ordinary unit after all. C is a coin flip: no quench is identifiable in that hold, which is not the same as a positive one. A cubic does not improve on this, a thermal-lag term adds nothing, and there is no hysteresis — at matched temperature, rising against falling differs by −0.31 %, +0.11 % and −0.16 %.

⚠️ Nothing downstream turns on the choice. Across ρ = −1.71, −1.94, −2.41 and the −2.78 the page carries, the run-to-run blank spread reads 31.4 / 31.7 / 32.2 / 32.6 % against 29.6 % uncorrected, and the day-blocked prediction reads R² 0.403–0.418 at RMSE 0.689–0.693. Blank-referencing has already removed the level a quench would move.

🚨 RETRACTION. An earlier pass on this campaign reported 50043's quench as −0.87 to −1.21 %/°C and concluded the fleet correction over-corrects it by about 3×. That number was regressed on log10(mon2) at one LED/bias cell, over stretches that included Seine water, and it is wrong. This is the §13.5l error 1 failure mode again: raw counts at the calibrated combo are a different quantity from the amplitude every quench number uses.

⚠️ The conclusion that correction does not tighten the between-run zero SURVIVES the retraction, and gets stronger. Run-to-run MilliQ blanks, five runs:

The amplitude says the zero wanders twice as far as one cell suggested, and the day-to-day differences are anti-correlated with temperature (the 23.9 °C blank is the brightest of the five), so no quench coefficient can pull them together. Between-run zero is 57× the within-run SD, 4.9 ppb equivalent. A correctly measured per-sensor ρ does not rescue absolute level.

Drift here is a step, which corroborates §13.5l independently. The three clean windows disagree in sign: +10.24 %/day immediately after a lens clean, +1.02 %/day in a settled soak, −3.40 to −3.80 %/day in September. There is no campaign-wide rate to remove, and removing any of them makes the between-run zero worse. A slope · days term cannot represent this.

The zero moves as a TREND inside a hold, and the rate is what fails to persist. [MEASURED] 2026-09-08, re-derived after a first pass got this wrong. On a robust (median-based) structure function the variance of the change in ln(amp) grows as lag^2.04, lag^1.11 and lag^1.78 across the three clean holds, where a random walk gives lag^1 and a straight ramp lag^2, and a line in time fits at R² 0.948, 0.649 and 0.926.

🚨 An earlier version of this section reported lag^0.18, lag^0.11 and lag^1.06 and called it a random walk. Withdrawn. That came from a mean-squared estimator, which one or two sweeps per hold — single readings jumping 270–320 %, with normal fit residuals and no censoring change — were enough to corrupt. A 1 % trimmed mean gives 1.75 / 1.12 / 1.50, agreeing with the median estimator. The diffusion-coefficient extrapolation built on the walk reading goes with it.

What does not persist is the rate. Fitted jointly with temperature the three holds give +10.4, +1.0 and −3.9 %/day, and all three clear /barrel's separability gate (|corr(days, T)| = 0.189, 0.661, 0.179 against COLLIN 0.9) and its 12 h minimum, so these are rates and not temperature in disguise. Refit on 12 h sub-windows the rate keeps its sign in 90 %, 48 % and 71 % of them. One hold is a walk, two are ramps pointing opposite ways. That is /barrel's own forward-prediction result restated per-window: a slope is estimable inside a window and does not persist past it, so there is no campaign-wide rate to remove.

WHERE IT IS, on four tests.

🚨 THE OPTICAL WINDOW IS NOT EXCLUDED, and an earlier version of this section said it was. That argument used the rule that a film moves both channels together. /findings has since retracted that rule as a necessary condition (§13.5d, which this section had been contradicting): 500226 reached 13.4× its baseline with its ToF flat at 1.00–1.02, because the ToF and the fluorescence optics do not share an aperture. Coupling is strong evidence for a window effect; its absence is not evidence against one. The discriminator that works is reversibility on pressure release, and no Suez run has been through one.

What survives is everything the two amplifiers share and no LED or bias setting separates: the window, the LED-to-window coupling, the filter path, or the detector's own efficiency. 50043 sits in §13.5l's open class — what moves S-TLF without moving ToF — with the seven barrel-shake units and 500146.

The settling experiment is two units in the same water, both sweeping, compared on their common-mode amplitude: correlated walks put the cause in the shared optics, uncorrelated walks put it inside each unit. BC-13 points at the second and was never run with the full sweep. A pressure cycle on 50043 would settle the window question directly.

The correction is the baseline, and 50043 measures its shelf life. A 1 h old baseline is worth 0.2–0.4 % of the reading, 6 h is worth 1.4–2.1 %, and 24 h is worth 2.5–11.3 %, matching the 6–12 h fleet figure in §13.5l. These runs sit 6–7 days apart, which is why each is read against its own MilliQ blank and never against another run's.

Turbidity remains unidentifiable on this unit, on the August data as on July's. The 12 Aug ladder sits at ToF 37.0–38.4 kcps/SPAD across the whole 0–45 ppb range, so there is no turbidity gradient at known tryptophan to fit against. On the five Seine runs corr(log10 MPN, ToF) = +0.49: analyte and turbidity move together, so a TLF turbidity term fitted there is confounded with the thing it would correct. Deriving one still needs a turbidity gradient at fixed tryptophan, which no Suez run has yet provided.

The classifier does not move. Day-blocked leave-one-run-out at a cut of 150 MPN, 39 gated windows over 5 runs, 10 above the cut:

Referenced to each run's own blank the three bases rank the samples identically, so the amplitude buys nothing for the decision on this campaign. Where it does pay is the fit: the pooled per-run gain goes from R² 0.440 on mon2 to 0.482, with per-run gains of ×1.24 to ×2.17 per decade of MPN. The value of the estimator here is precision and honesty at the low end, not accuracy at the threshold.

13.5k Chl-a: the sensors are NOT broken — they are correctly reporting zero

[CONFIRMED] Evan, 2026-08-21. [MEASURED] same day, on barrel data refreshed through 2026-08-21T17:00Z.

DI water contains no chlorophyll-a. Zero is the right answer. A Chl-a unit in the tank has nothing to excite, so it reports no fluorescence above dark. That is the instrument working.

In air all ten Chl-a units read 1300–1966 — the dry-window return. Immersion removes it, and what remains is the true blank. Between dropouts the water signal is FLAT: MAD 5–39 counts. Nothing about that is a fault.

⛔ RETRACTED: "Chl-a water is unstable/unusable"

I concluded that from p90/p10 and CV — ratio metrics — on a series whose level is near zero. A ratio explodes as its denominator approaches zero even when the absolute variation is nil, so "3942× spread" and "372% residual" were artifacts of the metric, not properties of the sensor. Evan read the traces, saw them flat, and was right. Never characterise stability with a relative metric on a channel sitting at its blank — use absolute counts.

What is genuinely open

• mon2_cal exactly equals mon2_dark_cal on 51.4% of Chl-a water sweeps (FDOM 0.0%, Burn-in 0.0%, TLF-new 0.2%), and it is episodic — runs average 10–65 sweeps and reach 922 consecutive (500185, ~15 h at 1/min). Consistent with a true zero being reported exactly. Not yet explained: the same thing happens on 7.9% of Chl-a AIR sweeps, where the unit is reading ~1900. Worth understanding before it is called benign.
• 500134 is a real level step, not noise [MEASURED]: ~1950 counts with MAD ~3 through 08-20 13Z, decaying 14Z–16Z, then ~25 counts from 17Z onward, never recovering — 5.5 h after tank entry (10:45Z), not at it. 500185 is the other late unit (1315). Read as the same immersion transition the other eight completed immediately, arriving late.

Consequences for the published numbers

The Chl-a flat residual on /findings is dominated by a real level transition — immersion, plus 500134's late step — measured against each unit's own pre-immersion baseline. It is not instrument noise and must not be read as one. Any %/day or CV figure for Chl-a in water divides by a blank and is undefined; report Chl-a water in absolute counts.

Fixed on the way [CODE]

add_slopes_from_raw.js used to continue when the superseded pooled-quadratic fitSlope returned null, so a sweep it could not fit produced no canonical amplitude either — even though the two-way log model is support-invariant by construction. Measured on 17,655 Chl-a water sweeps: fitSlope 47.7%, log model 57.7%. The gate discarded 1,749 sweeps, 9.9% of the phase. Removed; Chl-a water coverage 47.7% → 57.6%, fleet 91.2%. The superseded estimator must never gate the canonical one.

13.5b ✅ ADOPTED — per-sensor quench

[CONFIRMED] Evan, 2026-08-21: "and also per sensor quench." tlfQuenchFor(barcode) resolves own measured coefficient → batch coefficient → SLOPE_TEMP_RHO, and tlfQuenchProvenance() reports which was used. SLOPE_TEMP_RHO = -0.024 survives only as the last-resort default. tlfSlopeAt20C(slope, tempC, barcode) takes a barcode now.

It must be fit in CLEAN WATER, and the failed attempt is why. [MEASURED] 2026-08-21 a first pass fitted b_T on 5 h windows of deployed data. The result was not identifiable: 50058 and 500107 came out POSITIVE, 50051 and 50059 sat on zero with IQRs straddling both signs, and 50066 read −3.56 %/°C against a bench range of −1.6…−1.8. The cause is physical — in a river temperature and analyte co-vary, so a real fluorescence rise partly cancels the quench and what is fitted is part biology. The bench can do this because "there is no analyte in DI water, so everything in the raw trace is instrument."

[CODE] shared/tlf-quench.js from scripts/build_tlf_quench.mjs: each sensor's own clean-water window, and specifically the cooldown ramp — the ramp is the temperature sweep, the cooled tail is where the baseline LEVEL is read. Same records, opposite ends. Guards: span ≥ 4 °C, ≥ 10 sweeps, R² ≥ 0.7 (a slope through noise is not a measurement — 50064 fitted −0.30 %/°C at R² 0.129 and 50069 −0.63 at 0.539; both excluded).

6 sensors measured, median −1.85 %/°C — which corroborates the method: the bench reports −1.78 (new TLF) and −1.59…−1.67 (burn-in), and the shipped fleet constant was −2.4.

⚠️ Most sensors that need it cannot be fitted from existing data. Chicago 50058/50073/50074 and every Paris unit have clean-water windows spanning only 0.3–0.7 °C — nothing is identifiable there. They fall to the batch coefficient, which is what the validation programme sanctions ("carry one coefficient per batch") and is not a fleet constant: every unit here is the TLF optics batch. A sensor on the batch value is not individually characterised, and the fix is a clean-water temperature ramp for that unit — never a better default. [MEASURED] the bench's own 08-20/21 ramp (36.7 → 9.9 °C on 44 units) is the first sweep wide enough to determine quench properly for a whole fleet at once; the field units need the equivalent.

13.5c A baseline is good for 6–12 h; base20 is months old

[MEASURED] findings: drift is "a random walk with regime changes, not a trend". Extrapolating a fitted slope is worse than assuming flat by 2–5×, and helps in as few as 1 of 24 units. ⚠️ That phrase describes forward persistence, not shape. It is asserted from the forward-prediction test, and §13.5m measures the shape directly on 50043: inside a single hold the movement is a smooth ramp (structure-function slope 1.1–2.0, linear R² up to 0.95), and what fails is the rate's survival into the next window. The two statements agree; do not read the /findings phrase as a claim that no slope exists inside a window. The only control is re-baselining: good for ~6–12 h at the few-percent level; beyond a day it is worth more than ten percent of the reading. FIELD_MODEL.base20 carries June lab captures used for August predictions. Mitigating, not resolving: base20 is a fixed offset that the per-sensor fe partly absorbs, and the drift figures come from bench DI, not deployment. The size of the gap is the point.

13.5d The coupling rule is WRONG as a necessary condition

[MEASURED] degassing, 500226: s-TLF rose 7× under a second vacuum with ToF flat at 1.00–1.02, and returned to 0.87× on release — twice, to within 0.01×. "By that rule this, the clearest bubble event in the programme, would have been rejected." ToF and the fluorescence optics do not share an aperture. Coupling stays strong evidence for a bubble; its absence is not evidence against one. Reversibility on pressure release is the better discriminator. Consequence: the barrel shake null "rules out coupled events, not bubbles" — seven units moved >5 % on s-TLF alone and were set aside on the rule this undermines.

13.5e The validation pages contradict each other on what a bubble call requires

/findings states a bubble call needs supersaturation, coupling and reversibility. Two of those three are contradicted by the programme's own data:

• Coupling — refuted by 500226 above.
• Supersaturation — burn-in Stage 5, the 08-14 in-place shake at ~3.7 °C in strongly under-saturated water: 20 of 24 units up, median +19 % (extremes +278 %, +209 %), ToF median 0.0 %, and four units down. "The handling itself delivers the scatterer; gas saturation state is not required." Most plausibly entrained surface air.

So there are two mechanisms, not one: dissolved-gas nucleation (needs supersaturation, shed by pressure release) and handling-entrained air (needs neither). Only reversibility survives as a general test.

13.5f The production drift correction passes where there is nothing to correct

[MEASURED] burn-in Stage 10, the production model ln(amp) = P_d(t_n) + (r1 + r2·t_n)·dT/dt + slope·days run over the frozen campaign: 24/24 in the cold hold and 23/24 in air — windows spanning 0.3 °C and 3.4 °C — but 7/25, 5/25 and 12/24 on the wide-swing outdoor windows, "which is the condition a deployed sensor actually lives in." Fitted rates there disagree in magnitude and sign: −4.28, −1.49, +2.88 %/day. All five new TLF units are rejected outright (2.9–11.1 % residual, median 8.5 %, on invented rates of ±30–70 %/day). Flatness on a bench ramp is necessary, not sufficient.

13.5g The thermal-lag term is not supported by the test that was run

[MEASURED] burn-in Stage 3: adding dT/dt does not close the temperature hysteresis loop — it shifts the cooling limb by roughly a constant, so "that is an offset, not a lag being absorbed", and residual SD barely moves. Not refuted either: dT/dt over a fixed window is a crude stand-in, and settling it needs a thermometer in the water, which these units lack. The production correction and the memory rule both carry a dT/dt lag term regardless.

13.5h What the Colilert page settles about features vs grabs

[MEASURED] sweep-slope vs mon2, swapped on identical rows (n=116 Boulder, 19 exceedances): a tie — ΔR² 0.002, Δbalanced accuracy 0.006, both inside run-to-run noise. The slope wins only in the no-grabs regime (saturating units, lab-calibration-only transfer). "The field E. coli residual is grab-limited, not feature-limited." Principled outlier removal (four dual-channel optical transients, diagnosed with no reference to the Colilert value) lifts in-sample R² 0.470→0.488 but does not improve out-of-sample or the classifier — keep it as instrument QC, not as a performance lever.

And the sober check on the rolling geomean, which must travel with the headline: 30-day rolling geomean reads R² 0.795 / BA 0.812, but sensor-identity alone gives R² 0.916, within-sensor r ≈ −0.42, and leave-one-sensor-out collapses to R² −3.77 / BA 0.500 — a coin flip at a new site. "The rolling geomean is the right reporting unit; it is not, by itself, a more accurate algorithm."

14. Criteria to CONSIDER before excluding a reading

There is no automatic exclusion filter, and the default is KEEP. Exclusion is a judgement made per case, with the reasoning recorded. This section is a list of questions to work through — not a rule that can be applied mechanically.

Why this section exists. [MEASURED] 2026-08-21 I wired data_suspect windows into fieldRefitCompute as an automatic exclusion, on the assumption that a suspect window means "unusable". It removed 77 grabs — 30% of the calibration set (277 → 212; Paris 48 → 26). The assumption was wrong: 51 of 55 maintenance rows carry a suspect window, because the auto-detect fouling scan opens one on every fouling flag — and fouling is already handled by the fouling offset. Reverted. data_suspect today mostly means "this sensor is currently fouled", not "this reading is unusable".

The questions

1. Is the effect already corrected by an existing mechanism? Fouling offset, TLF clean step, temperature normalisation, auto-range. If yes, keep it and correct it. [CONFIRMED] (CLAUDE.md): "Keep fouled grabs; correct them, don't drop."
2. Was the sensor measuring the intended water at all? Out of water, pre-install, post-removal, or attributed to the wrong site. If no, it is not an exclusion judgement — it is the install-window rule (§8), and the answer is drop.
3. Is the channel physically invalid, or merely extreme? Railed at the ceiling, pinned on the pedestal, or a dark-channel fault is invalid. A high reading is not.
4. What is the provenance of the flag? An auto-detect heuristic is advisory. An operator-reported event, or one corroborated by an independent signal, carries more weight. Check created_by before treating a flag as authoritative.
5. Is the window bounded and specific, or open-ended? data_suspect_to IS NULL marks everything after the start indefinitely. An open window is a live status ("this sensor is fouled right now"), not a considered exclusion boundary — and one that has stayed open for weeks has stopped being a status and become forgotten state. See "Open windows expire or get reviewed" below.
6. Is there independent corroboration? The strongest evidence available is the control group — other sensors on the same water over the same period. A step that hits only the units someone touched is instrumental; one that hits all of them is hydrological.
7. How much does it remove, and in which direction does it move the result? Removing more than ~10% of a calibration set, or an exclusion that improves the headline metric, is a decision to raise explicitly — never a default. Report the count either way.
8. Never drop a point because it hurts a metric. [CONFIRMED] Excluding a documented fault is justified; excluding an inconvenient point is not. The Bercy 1935 grab stays in the Paris fit for exactly this reason, even though removing it would lift BA 0.807 → 0.907.

What this means in practice

• Suspect windows are surfaced, not enforced: dashboards flag them (10 pages read data_suspect_*), the fit path does not filter on them.
• If a specific window genuinely marks unusable telemetry, exclude that window deliberately and say so — do not turn the whole mechanism into a filter.

Open windows expire or get reviewed

[CONFIRMED] Evan, 2026-08-21: "Open windows should expire or be reviewed."

A data_suspect window with a NULL end is only meaningful while the condition it describes is still true. Nothing closes one except the fouling scan seeing the ToF floor return to baseline, so a window whose sensor was never cleaned — or whose flag was an auto-detect false positive — stays open forever and is quietly read as current by the 10 pages that render data_suspect_*.

scripts/audit_pipeline.js reports every open window with its age and raises maintenance.stale-open-suspect-window (WARN) at ≥ 14 days. The threshold sets emphasis only; the full list is always printed, so no judgement is hidden behind the number.

Expiry is a REVIEW trigger, not an automatic close and not an automatic exclusion. The audit surfaces the window; a human closes it (data_suspect_to), confirms it is still live, or escalates it to a fault. Nothing in the fit path changes either way — §14 still applies.

Open ≥ 14 days as of 2026-08-21 (both 42 days, opened 2026-07-10):

15. Nowcast feed (opencurrent) — fleet-wide, data landed, SHADOW posture

A second model path, built by opencurrent (Mike Vlah) in the lume_nowcast repo and pushed hourly into this project. It is not a Boulder product: Boulder was the first region through a feed intended to serve every customer, and as of 2026-08-29 it serves all of them.

lume_nowcast (GitHub Actions, hourly)
      → POST /api/nowcast/ingest   (bearer, Pages secret NOWCAST_INGEST_TOKEN)
      → nowcast_latest / nowcast_series / nowcast_meta   (DEDICATED D1: NOWCAST_DB)
      → /nowcast/   (public, unauthenticated GETs)

15.0 What actually landed [MEASURED] 2026-08-29

The 2026-08-28 entry said all three tables were empty. They are not any more. Pulled from the public endpoints at 2026-08-29T11:54Z:

Regions and site counts: Boulder 6 (BC_*), Chicago 6 (CHI_*), Cleveland 2 (CLE_*), DC 3 (DC_*), Denver 3 (DEN_*), Marne 3 (MAR_*), Paris 4 (PAR_*). That answers §15.4 items 4 and 5 as they stood: the producer holds PIN access for every customer, and every region we serve is wired.

thresholds_json carries alarm_on 0.25 and alarm_off 0.15 at all 27 sites, and an action_limit that matches ACTION_LIMITS for six regions out of seven (see 15.1c).

The hysteresis is real. Over all 13,581 points, alarm disagrees with the naive p_exceed ≥ 0.25 rule at 886 points, and every one of the 886 is an alert held while p_exceed sits in the release band [0.15, 0.25). No unexplained disagreements. 2,155 points fall in that band.

15.1 It puts the decision on the probability axis, not the concentration axis

This is the substantive difference from §6 and §12.5, and the reason to care.

[MEASURED] 2026-08-29, on real data now, not the mock: p_exceed equals Φ((yhat_log10 − log10(action_limit)) / sigma) to within 8.2e-5 across all 13,581 served points. The mock agreed only to 0.022. The feed does integrate its predictive distribution above the partner's real limit, exactly as claimed.

15.1b ⛔ But the knob is NOT removed, only re-expressed [MEASURED] 2026-08-29

The argument for the probability axis was that making the spread explicit removes exc_t. Measured, it does not. alarm_on is 0.25 at every site and sigma is near-constant across the whole fleet (0.6569 to 0.7099 over 13,581 points, an 8% spread), so the probability cut inverts to a concentration cut in closed form:

c* = action_limit × 10^(sigma · Φ⁻¹(0.25))     Φ⁻¹(0.25) = −0.67449

c*/action_limit lands between 0.332 and 0.359 at all 27 sites. The feed has not removed a per-region knob; it has replaced it with a fleet-constant one set at roughly 36% of whatever the action limit is, and nobody chose that number per region.

DC is the one region where the two agree (146 vs 145), and that is a coincidence of its action limit being 410. Everywhere else the feed is far more sensitive than we are (Cleveland and Denver alert on 100% of hours, Boulder on two thirds), except Paris, where it is far less.

15.1c ⛔ Denver is scored against Boulder's action limit [MEASURED] 2026-08-29

All three DEN_* sites carry action_limit: 126. ACTION_LIMITS.DENVER is 235, the single-sample maximum denver.html itself displays. The feed is comparing Denver water against the Boulder number.

At the correct 235 the implied cut would be 78–81 rather than 43. On today's row it flips nothing (all three sites alert either way: 0.730 → 0.588, 0.700 → 0.557, 0.974 → 0.939), but it is wrong at every hour, and it is wrong in the opposite direction from our own Denver defect: §6 warns that our exc_t of 410 sits above the 235 limit and can only under-flag, while the feed sits far below it and can only over-flag. Report the limit to the producer; do not read a Denver probability from this feed until it changes.

15.1d The Paris acid test ran, and sigma did not widen [MEASURED] 2026-08-29

The 2026-08-28 entry said to watch sigma at Paris before believing any Paris number, because the Seine tops out near 397 CFU predicted against a 900 limit and a probability only rescues that if sigma is honest about how little the model knows there.

It is not. Paris median sigma is 0.6570, the lowest in the fleet, indistinguishable from Boulder's 0.6581. The consequence is exactly the predicted one: across 2,012 Paris site-hours the feed raised zero alerts, with a maximum predicted concentration of 136 CFU against an implied cut of about 322. Paris on this feed is silent by construction, not by observation. Marne survives only because its water runs higher (median 258 CFU), and it alerts on a third of hours against the same 900 limit.

15.1e Twelve of 27 sites publish a number with no sensor input [MEASURED] 2026-08-29

health splits ok 15 / sensor degraded 4 / sensor dark 8, and the split is exact: every one of the 12 non-ok sites falls back to variant: "none", and every ok site carries a sensor variant (sensor+rain 13, sensor 2). The fallback still publishes a p_exceed and an alarm with the same authority as a sensor-fed row, and 8 of those 12 are alerting right now (BC_55, BC_CU, CHI_NorthBranch, CHI_SouthBranch, CHI_WildMile, DEN_BearCreek, DEN_CSUSpur, MAR_Neuilly).

⛔ And the health call is not always right. [MEASURED] 2026-08-29, Boulder: the feed marks BC_Eben sensor dark and drops it to variant: "none", while /api/site-series carries 472 hourly points for that site over the same window with a last reading at 2026-08-29T05:00Z, the same timestamp as BC_13 and BC_30, which the feed calls ok. BC_CU (last reading 08-12) and BC_Can (08-17) really are dark and the feed is right about both. So one Boulder site in six is running sensor-free because of the producer's sensor access, not a dead unit, and its grabs test the fallback rather than the model. Judge this relative to the freshest site in the same pull, never against the wall clock: that window is served from the hourly rollup, which lags every site together, and an absolute staleness rule (even STATUS_STALE_HOURS) reads the shared lag as a dead fleet and hides the disagreement. Live cross-check: thelume.ai/boulder/nowcast.

/nowcast/ does label them: any row with health != "ok" renders the health string plus "serving site calibration" in place of the variant, so §13.1 is satisfied on our side. What that label asserts about the model is unverified (§15.4 item 4): it is opencurrent's text, not something we have read the fallback for. An alert carried by a site whose sensor is dark is a climatological alert until they tell us otherwise.

15.2 Serving posture and what is NOT enforced here

SHADOW per region as each lands [CONFIRMED] (Evan, 2026-08-28). [MEASURED] 2026-08-29: all 27 latest rows carry shadow=1 and all 27 meta rows carry live=0, so the posture is being honoured on the wire today.

⚠️ Nothing on our side enforces shadow. The page reads nowcast_latest.shadow only. nowcast_meta.live and promotion_json are ingested and stored but never read by the page. A row that lands with shadow=0 renders LIVE immediately. This is a request to the producer, not a control we hold.

⛔ [MEASURED] 2026-08-29 — the producer's own records already disagree with the posture. All six Boulder rows carry "evaluated_as": "live" and "reason": "still performing; staying live" inside promotion_json, while the row they ship says shadow=1. Boulder is being scored as a live region and served as a shadow one. Since the page reads only shadow, the gap between those two is one flag flip wide, and we would not see it coming. The other 21 sites say "evaluated_as": "shadow", consistently.

15.2b Scoring status: 9 sites scored, 2 clear zero [MEASURED] 2026-08-29

promotion_json exposes the promotion test §15.4 asked about: a gain against a baseline on brier where the site has exceedance positives and rmse where it has none, with a two-sided and a one-sided CI, a scored-grab count, and a hold / promote decision.

• All 27 decisions are hold. None has been promoted.
• 18 sites are unscored, gated at a documented minimum: "only N scored grabs; need 12", with N from 1 to 7.
• 9 sites are scored: the 6 Boulder sites (n_grabs 15–28) plus MAR_Neuilly (23), PAR_Bercy (15) and PAR_BrasMarie (13).
• Only 2 of the 9 show a gain whose one-sided CI clears zero: BC_13 (brier +0.0203, CI [+0.0047, +0.0411]) and BC_CU (brier +0.0403, CI [+0.0013, +0.0765]).
• 5 of the 9 have a negative point gain, i.e. worse than the baseline they are scored against: BC_55 −0.0307, BC_Can −0.0200, MAR_Neuilly −0.0112, PAR_Bercy −0.0056, PAR_BrasMarie −0.0445. Two of them (BC_55, BC_Can) are among the six carrying evaluated_as: "live".
• CHI_MainStem's promotion_json is the empty object {}: no record at all, not a hold with a reason. Everything else about that site is populated.

Nine scored sites, two of them positive, none promoted, is not yet evidence for or against this feed. It is a reason to leave every region in shadow and to keep the grab pairing (§8) feeding it.

15.3 Isolation and the one claim that was false

• Ingest token is a Pages secret, compared constant-time. Every statement is hardcoded and touches only nowcast_*.
• Bound to a dedicated NOWCAST_DB, so isolation is structural rather than a property of the statements staying hardcoded. Verified: the nowcast_* tables self-created there and not in the shared DB.

⛔ [MEASURED] 2026-08-28 — opencurrent's note claimed the ingest "validates every string it stores, so even a stolen token writing hostile rows yields rejections, not stored cross-site scripts." That was true for latest and series but false for meta. Those rows passed only okRow() (site_id + ts), so thresholds_json was stored as an arbitrary string, and the page interpolated JSON.parse(...).action_limit into innerHTML unescaped. Confirmed executing. Closed in 23592af: safeJsonObj() requires a size-capped JSON object with a finite action_limit, plus numeric coercion and esc() on the page. action_limit is the per-customer field, so this mattered more at fleet scope than at one region.

15.4 Still unverified — the model itself has not been read

lume_nowcast is not accessible to our GitHub account. Items 3, 4 and 5 were answered by the data itself (15.0 and 15.2b); these two were not, and nowcast output is not validated until they are:

1. Does the sensor arm apply the temperature and ToF correction (§3, §12)? Our rule is that every TLF and E. coli output is temp + ToF corrected, never raw mon2. The feed's variant field distinguishes sensor+rain / sensor / none but says nothing about what happens to the sensor value inside the first two.
2. What rain source? latest carries rain_48h and series carries rain_mm, so a 48 h antecedent window is being used, which matches Chicago rather than Cleveland's 24 h (§12). Neither field says whether the rain is ERA5 or forecast, and Cleveland established that ERA5 must beat forecast before a rain term is believed.

Two more opened by the first real data:

1. What is the baseline the promotion gain is measured against? A gain over climatology and a gain over FIELD_MODEL are different claims, and only the second would justify a promotion at a site we already serve.
2. What does a variant: "none" row actually condition on? The page calls it "site calibration", which reads as climatology, but the fallback has not been read. Twelve sites publish one hourly (15.1e) and eight of them are alerting on it.