Lume 1.2 – Colilert Field Calibration

Calibration Algorithm To-Do

Working checklist of outstanding calibration-algorithm tasks for the Lume × Colilert field model. Check items off, add new ones, or remove them; progress is saved in this browser.

Models Running on Production Dashboards

What “TLF (norm.)” means on the dashboards: it is the raw tryptophan-fluorescence signal (mon2) expressed relative to each sensor’s clean-water baseline and rescaled to a common LED‑512 reference frame, so all sensors are directly comparable — including units that auto-range (carry an elevated optical background that would saturate the detector at full power, so they are read at reduced LED power and their signal normalized back into the LED‑512 frame; the reading’s operating combo is surfaced as operating_combo). Zero is the pristine lab clean-water level (recorded at ≈21.8 °C, zero turbidity). In the field the value commonly sits slightly negative: real water carries turbidity, which suppresses fluorescence via the inner-filter effect, and typically runs cooler than the lab baseline. Negative therefore does not mean “cleaner than clean” — the meaningful signal is a rise above a sensor’s own local clean level. TLF (norm.) is a raw diagnostic; the deployed E. coli prediction applies the turbidity (ToF) and temperature corrections on top of it.

The customer dashboards (Boulder, Denver, DC, Chicago, future deployments) all load a single shared model module at /js/ecoli-model.js, served from shared/ecoli-model.js in the virridy-lume-summary repo. As of 2026-07-09 the two field/Colilert models below are what those dashboards evaluate for grab-calibrated field sensors. The continuous model is a single joint mixed-effects (REML) refit over all nine field sensors (n = 108: lab clean-water baselines, install-window-valid pairing, ToF/temperature corrected). Every sensor is treated identically — each carries its own empirical-Bayes-shrunk intercept and TLF/ToF slopes. Three repaired units (50048, 50052, 50062) were re-baselined from their lab clean-water tests and brought in on the same footing; 50062 auto-ranges (read at reduced LED power because it saturates at full power, then normalized back to the standard frame). Metrics are in-sample (out-of-sample validation deferred by design). CBT (drinking-water) and lab sensors keep their own separate models and are unchanged.

Update 2026-07-12 — Chicago recalibration, 50073 baseline & auto-range. The Boulder-tuned slopes systematically under-predicted Chicago’s hotter, frequently-censored regime: the deployed model was missing 7 of 13 Chicago exceedances (sensitivity 0.46). Since TLF does not track E. coli at these turbid urban sites, the three grab-bearing Chicago sensors (Cal-Sag 50058, South Branch 50073, North Branch 50074) were given raised per-site level anchors to their censored grab distributions — in-sample Chicago @200 CFU: balanced accuracy 0.61→0.80, sensitivity 0.46→0.85 (missed exceedances 7→2), specificity held at 0.75. These anchors are provisional (n = 9/4/4 grabs). Separately, 50073’s clean-water baseline was corrected (its stored value sat near its p90 rather than its clean floor, biasing predictions low; re-derived to the p10 in-water floor, as for 50058), and 50073 was registered for auto-range (its LED 512 channel compresses on ~14% of readings while LED 128 stays linear, so those readings are now recovered rather than clipped). A candidate streamflow term for the Boulder classifier was tested and not adopted — it gave no in-sample improvement (balanced accuracy 0.771 with vs 0.771 without).

Feature Test — TLF Sweep-Slope vs mon2 for the Field Regression

A direct test of the question “should the field Colilert regression use the full-sweep TLF slope (sTLF, the LED/SiPM bias-response derivative at bias 3000) instead of absolute mon2 as its primary fluorescence feature?” The TLF feature is swapped while everything else is held fixed: identical structure (per-sensor intercept + temperature + turbidity), fit on the same 116 install-window-valid Boulder grabs (19 exceedances at 126 CFU/100 mL). Every grab carries both features, so the swap is on identical rows. Balanced accuracy is the safe/unsafe classifier at the balanced-accuracy-optimal decision cut (how the deployed exc_t is tuned).

Result: a tie. The two features are statistically indistinguishable as the field-regression signal (ΔR² = 0.002, Δ balanced accuracy = 0.006, both inside run-to-run noise), and adding a feature×temperature interaction changes neither. When a sensor has its own field grabs (the deployed regime), the TLF feature choice does not move accuracy: there is no accuracy reason to switch, and no accuracy cost to switching. This is consistent with the field E. coli residual being grab-limited, not feature-limited (more exceedance grabs is the lever, not a better fluorescence feature).

This controlled test uses a fixed-effects OLS harness on the six Boulder dilution-calibrated units, so its absolute balanced accuracy (~0.67) sits below the deployed mixed-effects headline (~0.79); the quantity that matters here is the near-zero gap between the two features, not the absolute level. Where the slope does win is the no-grabs regime in the next section: it is decisively more robust on saturating units and transfers better to lab-calibration-only sensors. Reproduce with scripts/compare_slope_vs_mon2_field.py.

Outlier Screen Test — Optical Transients (Lume) and Reference Outliers (Colilert)

A test of whether removing outliers, on both sides of the pairing, helps the field regression. The discipline: an exclusion is only admissible if its rationale is independent of the model residual. Dropping a point because it hurts the fit is circular and inflates apparent quality; a credible drop needs a physical or measurement-validity cause established without reference to how well the point fits.

Lume (sensor) side, a credible rule exists: dual-channel optical transients. Because mon2 and the sweep slope are two independent optical read-outs of the same fluorescence at the same instant, a reading where both spike far above the sensor's own robust baseline (|robust-z| > 5 on each) while ToF stays flat (no turbidity event) and the sensor reverts to baseline immediately after is a transient artifact (a particle, bubble, or biofilm slough on the fluorescence window), diagnosed with no reference to the Colilert value. This flags four readings, all low E. coli with spuriously high fluorescence, generalizing the already-documented 50059 2026-06-05 artifact to three more:

Colilert (reference) side, no credible rule here. The site-anomalous grabs by a robust per-site z-screen are, on inspection, mostly real exceedances the sensor corroborates (50066 2026-05-18 at 1986 MPN with elevated TLF; 50066 06-25 at 1046; 50046 04-27 at 1986): removing those would delete the exact high-E. coli signal the model exists to detect. The one uncorroborated high grab (50048 2026-06-26, 866 MPN with the sensor flat) cannot be distinguished from a real event that TLF simply cannot see, absent independent context (a replicate, a rainfall or turbidity driver). So no Colilert grab is credibly removable, and trimming the uncorroborated one only removes a true exceedance.

Verdict: credible removal exists but does not move the model. Dropping the four transient artifacts lifts in-sample R² (mon2 0.470 → 0.488), which is mechanical (four high-leverage points gone), but the out-of-sample LOO R² does not follow (0.355 → 0.345) and balanced accuracy is flat (±0.006). The safe/unsafe classifier is untouched because every flagged artifact is low E. coli, so it never affected an exceedance call. Removing the lone uncorroborated Colilert grab (not credible) loses a real exceedance and makes every metric slightly worse.

The dual-channel transient screen is still worth keeping as instrument QC (it is physically correct and the deployed model already excludes the 50059 case), but it is not a performance lever: the field residual is grab-limited, not artifact-limited, so even principled outlier removal leaves the ceiling where it is. Reproduce with scripts/test_outlier_removal.py.

Weekly Moving Average — Matching the Sensor to a Weekly Geomean

A single grab is a noisy point-in-time sample of a stream whose E. coli varies a median 2.9x within a week (up to 130x). We usually have several grabs per site per week (103 of 116 grabs sit in a multi-grab week), and the EPA recreational primary criterion is itself a geometric mean, not an instantaneous value. So the operationally correct target is a weekly geomean, and because the sensor reads continuously it can produce a weekly-averaged prediction to match. This study decomposes where the grab-level error actually lives, at three resolutions using the same fitted model.

Most of the grab-level error is the grab, not the sensor. Holding the C_TLF predictions fixed and only changing the target from the individual grab (A) to that grab's weekly geomean (B) lifts R² from 0.470 to 0.634: roughly a third of the apparent residual is grab-sampling noise, not sensor error. The sensor tracks the weekly geomean, the quantity a beach manager actually acts on, far better than it tracks any single bottle. Averaging the prediction as well and reporting at weekly resolution (C) pushes the safe/unsafe classifier to balanced accuracy 0.78 at sensitivity 1.00. This reproduces the deployed model's weekly result (which lifted 0.470→0.632) on the C_TLF feature.

Caveats and next step. Weekly aggregation leaves only 27 site-weeks and 2 geomean exceedances, so the categorical sensitivity is low-powered; R² across resolutions is not directly comparable because n and the target variance change (the clean, matched comparison is A→B, same 122 predictions). This weekly prediction averages only the grab-matched readings (2–3 per week); the true moving average over the sensor's full continuous stream that week would smooth further and is the proper next step (it needs a sensor-timeseries pull). The direction is clear and motivates the rolling-geomean metric in the next section: report the sensor as a rolling geomean and judge it against the geomean standard, not bottle-by-bottle. Reproduce with scripts/study_weekly_moving_average.py.

Continuous Stream and the Rolling Geomean

The weekly study raised two questions the grab-matched data could not answer: does the sensor's full continuous stream (not just the 2–3 grab-time readings) help, and does a rolling geomean, the form the EPA criterion actually takes, tighten agreement? We built the hourly C_TLF stream for all six Boulder sites (6,571 hourly points): the full-sweep slope recomputed for each hour from raw SIPM, temperature/cycle/gain-corrected to C_TLF, joined to the hourly temperature/ToF rollup, and run through the same C_TLF mixed model. The per-sensor fixed effects absorb unit calibration, so each comparison isolates the framing, not the model.

Step 1, full stream vs grab-time: a tie. Averaging the prediction over the whole week (~167 points) does not beat using only the grab-time readings when the target is the sparse weekly grab geomean.

Both land at R² ≈ 0.6. So the weekly win seen earlier came from the geomean-reference framing, not from denser sensor sampling: at the sparse weekly resolution, the grab-time prediction already carries the signal. The continuous stream's real value is that it enables the rolling geomean below, which a few grabs cannot support.

Step 2, the rolling geomean: the payoff. The EPA recreational primary criterion is a geomean over a rolling period (commonly 30 days). We compute the sensor's trailing W-day rolling geomean and, where grabs exist, compare it to the grab geomean over the same window.

Agreement tightens sharply as the window approaches the regulatory geomean period. Pearson r climbs 0.78 → 0.90 → 0.93 and R² 0.60 → 0.80 → 0.82 from 7 to 30 to 60 days, with the safe/unsafe geomean call reaching balanced accuracy 0.812 at 30 days. Both the sensor and the grabs are converging on the same long-term mean as the point-to-point noise integrates out. Against grab-level R² = 0.47, the same sensor judged as a 30-day rolling geomean tracks the reference at R² = 0.80. This reproduces the deployed model's rolling-geomean tightening on the C_TLF feature.

Caveats. The evaluation points are per-grab trailing windows and therefore overlap heavily (consecutive grabs share almost all of a 30–60-day window), so they are autocorrelated: read the R²/r as tracking descriptors, not independent out-of-sample scores, and the effective independent count is far below the ~116 windows shown. Results are also in-sample (the sensor is calibrated on these grabs). The robust, model-agnostic finding is the monotone tightening with window length, which follows directly from averaging out noise. Takeaway: the strongest lever in this whole analysis is not a new feature or outlier rule, it is the reporting unit, report the sensor as a rolling geomean (30-day to match the standard), which it can serve continuously and a handful of grabs cannot. Reproduce with scripts/study_continuous_weekly.py and scripts/study_rolling_geomean.py.

Does that accuracy generalize? A sober check.

Before treating the rolling geomean as an algorithmic win to push further, it is worth asking what the high R² is actually made of. Two checks say it is between-sensor structure, not temporal tracking, and that it does not generalize to a new site.

The R² is between-sensor. Knowing only which sensor a window belongs to (its chronic mean level) already gives R² = 0.945 at 60 days, higher than the model. Within a single sensor, the 60-day prediction does not track the site's geomean over time (within-sensor r ≈ −0.32, i.e. no temporal signal). The model is placing each site at its chronic level, which the per-sensor fixed effect, fit on that site's own grabs, already encodes; it is not detecting a site going from safe to unsafe.

So can the algorithm be improved to raise this accuracy? Not meaningfully. Leave-one-sensor-out, a brand-new site with no local grabs, collapses the 60-day R² from +0.823 to −4.84 (far worse than guessing the mean) and balanced accuracy from 0.82 to 0.50, a coin flip. Pushing the in-sample number higher would only fit the per-sensor means better, which is circular (those effects come from the same grabs) and operationally empty: it neither helps a new site nor flags an existing site trending upward. The binding constraint is the same one throughout this analysis: repeated grabs per site across time and across exceedances, so within-site change can be calibrated and validated. The rolling geomean is the right reporting unit; it is not, by itself, a more accurate algorithm. Reproduce with scripts/study_rolling_generalization.py.

Scatter and Confusion Matrices Across Aggregations

The same C_TLF candidate model and the same reference, viewed at five aggregations, from a single instantaneous grab up to a 60-day rolling geomean. Each panel pairs a predicted-vs-observed scatter (green circle = correctly classified, red × = misclassified; dashed 1:1 line; dotted lines mark the 126 CFU action limit on the observed axis and the balanced-accuracy-optimal operating cut on the predicted axis, so the four quadrants are the confusion matrix) with the safe/unsafe confusion matrix at that operating point. Watch the cloud tighten onto the 1:1 line as the aggregation window grows. All in-sample; the rolling panels use autocorrelated overlapping windows (see the caveat above).

Overlapping CIs = the fraction of points whose sensor 95% CI and Colilert reference 95% CI overlap (|predicted−observed| ≤ 1.96·(σ_sensor + σ_ref)). The reference uses the value-dependent Quanti-Tray/2000 95% CI, and both CIs tighten with the geomean (÷√n) as the window grows — a deliberately conservative test: the sensor CI is floored at the Colilert geomean CI so an aggregated sensor estimate is never treated as more precise than the reference geomean it is judged against.

Future-Proofing — the Lab-Calibrated C_TLF Transfer Model

The deployed model above needs local Colilert grabs to fit each sensor's TLF→E. coli slope. That is fine for a well-sampled site, but it does not scale: most future deployments will get a lab gain + clean-water calibration (a serial dilution like the TLF dilutions, including clean water) but no field Colilert. The question this section answers: can we predict E. coli on such a sensor from its lab calibration alone?

The feature is C_TLF, the fully corrected tryptophan-equivalent concentration, built in per-sensor steps from the full-sweep TLF slope (sTLF, the LED/SiPM bias-response derivative at bias 3000; railing-proof because it is fit from the whole sweep rather than one 512/3000 reading, see the definition box): (1) temperature-correct the slope to 21 °C using the sensor's own fluorescence-quench ρ; (2) sample-cycle (burn-in) correct it to the calibration's cumulative sample count, because a new sensor's slope drifts over its first thousands of readings (established on the clean-water control and dilutions pages); (3) apply the sensor's lab gain and blank to map it onto a common µg/L-equivalent tryptophan scale. A single shared field model (C_TLF + temperature + turbidity → log₁₀ E. coli) then applies to any lab-calibrated sensor, no local grabs required. The cycle step is a per-sensor calibration correction baked into C_TLF, not a regression covariate.

Foundation (2026-07-23 refresh). Grabs are pulled fresh from the current mWater validation_grabs master and filtered strictly to each sensor's Pumphaus site_installations window: 125 install-window-valid grabs, 19 exceedances across the six Boulder units, of which 122 match an in-window full sweep. The window filter correctly drops pre-install grabs (50048 / 50052 / 50062 were repaired and redeployed 6/25) and the 5/19–6/03 pull-out gap (50046 / 50059 / 50066). The transfer headline is the four units that carry their own lab dilution blank+gain (50046, 50048, 50052, 50066: n = 75, 16 exceedances); each held-out sensor is treated as brand-new (lab calibration only, no field grabs, shared field coefficients). 50059 (no dilution, borrows a calibration) and 50062 (weak / saturating responder) are reported separately below because their C_TLF is not independently trustworthy.

Every field grab on these four units predates the 2026-07-16 lab calibration (the sensors were newer in the field than on the bench: cumulative-count offset ΔN runs from −7,300 to −160), so the burn-in step bridges each grab back to the calibration's cycle state. Adding it roughly doubles the calibration correlation (0.054 → 0.107) and lifts leave-one-sensor-out balanced accuracy modestly (0.596 → 0.604). The balanced-accuracy gain is small and sits within the exceedance-count noise (16 exceedances); the robust, reproducible effect is the tighter calibration correlation.

Per held-out sensor, median |predicted−observed| log₁₀ error under the shared model (trained on the other lab-calibrated units), temperature-only vs temperature + sample-cycle:

The cycle correction's field payoff is concentrated on the highest-drift unit. 50048 has the steepest burn-in coefficient (k_N ≈ 0.040 vs ≈ 0.021 for the others) and the widest gap between its lab-cal cycle state and its field grabs; correcting for it cuts its transfer error from 0.50 to 0.20 log. The three lower-drift units already sit near their mature slope and tie, and critically no clean unit is made worse. 50062 (weak responder) shows a deceptively low error only because its C_TLF barely varies, so predicting near the mean scores well on its narrow E. coli range: it carries no real fluorescence signal and is excluded from the headline. 50059 transfers worst (0.90 log): borrowing another unit's gain and blank is a fallback, not a substitute for a unit's own lab dilution.

Signature matching — calibrating a unit that has no dilution of its own

A sensor may reach the field with no lab dilution at all. The sweep still carries a signature: the shape of each LED's bias-response (∂mon2/∂bias at LED 32 / 128 / 512), which is intrinsic to that unit's SiPM and optics and independent of the water. Matching that signature to a lab-calibrated unit lets the new sensor borrow that unit's gain and blank. Worked example: 50059 (27 field grabs, never in the dilution) has a per-LED signature almost identical to 50046 (relative distance 0.11; next-closest 0.24), so it borrows 50046's calibration. On the fresh install-window-valid set that borrow transfers at 0.90 log error, noticeably worse than a unit predicted from its own blank (~0.4–0.5 log) and not improved by the cycle step (0.94): a usable fallback, not an equal substitute. The signature match is real (it lets the calibration set grow by similarity to units never bench-tested), but a unit's own lab dilution remains materially better.

As the fleet grows into lab-only calibration, C_TLF is the feature to standardize on: it is railing-proof (full-sweep slope, no combo-picking), it carries a physical temperature and burn-in correction rather than a per-site empirical fit, and it degrades gracefully (a borrowed calibration still transfers, if imperfectly). The binding constraint is not the feature but the calibration inputs: each new sensor needs a clean-water baseline, a lab gain, and a cumulative-count reference recorded at calibration time.

Reproduce: grabs from /api/validation/grabs filtered to each sensor's site_installations window (/api/site-series, PIN 4001); C_TLF and the transfer metrics via scripts/field_transfer_fresh.mjs then scripts/transfer_tables_fresh.mjs; per-sensor quench ρ, burn-in k_N and gain/blank are fit on the frozen dilution sweeps in data/raw/tlf-dilutions/.

Field Calibration Data

Served directly from the D1 validation master (validation_grabs) — the single table every source now feeds: mWater (Boulder / Denver), the Chicago H2NOW program sheet, and future email / attachment feeds. The Source column shows each grab’s origin; the Method column the reference enumeration used — Colilert (IDEXX defined-substrate MPN) or membrane filtration (MF, CFU). The Use in Calibration column flags rows unusable for the field model: mWater grabs with no /diagnostics water-temperature record within ±20 min of the sample, and Chicago sites without a grab-calibrated baseline (only Cal-Sag Channel 50058, North Branch 50074 and South Branch 50073 enter the models; Wild Mile and Main Stem are excluded). Times are shown in both local and UTC, corrected from the mWater-stored value using each grab’s recorded sample timezone (device clock Boulder, MDT = UTC−6).

From raw grabs to the calibration set

Every exclusion applied to the mWater + Chicago grabs above, in order, to reach the n = 68 samples used in the field models.

Field Colilert Calibration — Boulder, Chicago & Boston

Rebuilt 2026-06-30 from the mWater Colilert-idexx datagrid (Boulder + Denver), the Chicago / H2NOW program sheet, and CRWA (Boston / 1NBS). Each grab is paired to its sensor within ±20 min, and only where that sensor was validly installed at the site (the site_installations window). The deployed model reads mon2 at the calibrated operating point (LED 512, bias closest to 3000); the C_TLF candidate sections below instead read the full LED×bias sweep to compute the railing-proof slope s_TLF.

Install-window rule (enforced in the pairing pipeline): a Colilert grab is paired to a sensor only when the grab time falls inside a valid installation window for that sensor — Pumphaus site_installations, [installed_time, removed_time). If no install window covers the grab (including a sensor with no windows at all), the grab is dropped, never paired. Sensor telemetry that exists outside a valid deployment window — bench/lab tests, pre-install or post-removal periods, or a broken/uninstalled unit — is never matched to a grab.

Per-sensor clean-water baselines are read from each unit's lab clean-water test, using a stable cooled sub-window near 21–22 °C (avoiding hot-transient setup readings that would bias the baseline high); for saturating units the baseline is taken per operating combo. Cal-Sag (50058), which has no lab baseline, uses a derived in-water clean floor (mon2 670, ToF 103). 50048, 50052 and 50062 were repaired and redeployed on 6/25; each was re-baselined from its own lab clean-water test and is now included and calibrated identically to the rest (50048/50052 at the standard LED 512; 50062 auto-ranges to LED 128 and is normalized back to the LED 512 frame because it saturates at full power). Their grabs are paired only inside their valid post-repair install windows. Remaining exclusions are minimal: one 50059 6/05 sensor-fault spike, plus CRWA/Boston (50055) and Chicago’s Wild Mile / Main Stem, which have no lab-calibrated baseline yet. Final set: n = 68 paired grabs (60 Boulder, 8 Chicago; Denver contributes 0), with 18 exceedances at the per-customer thresholds (Boulder 126, Chicago 200 CFU/100 mL). All results below are in-sample; leave-one-out validation is deferred to a later pass.

Continuous model — log₁₀(E. coli)

These sections evaluate the C_TLF candidate model, not the deployed one. The Production Dashboards section above is the factual record of what customers run today (mon2 mixed-effects, all nine sensors). Here the fluorescence feature is swapped to C_TLF (full-sweep slope → temperature → sample-cycle → per-sensor gain/blank; see the transfer section), evaluated on the six Boulder dilution-capable units only (Chicago has no lab dilution, so it drops out). Same regression structure as deployed: per-sensor intercept + shared C_TLF/ToF slopes + temperature + C_TLF×temperature. All in-sample.

The form is log₁₀(cfu) = fe[bc] + s·CTLF + stof·tofr + btemp·(T−20) + b·CTLF·(T−20), per-sensor fixed intercept with shared slopes. The temperature terms are kept (ablation shows they still add R² even after C_TLF's optical quench correction, so the correction is not perfect and a residual, likely biological, temperature signal remains). tofr is per-sensor baseline-relative ToF.

Each Boulder unit carries its own fixed intercept (its site's E. coli baseline, fit on its grabs); the C_TLF and ToF slopes are shared across sensors. C_TLF already puts every sensor's fluorescence on a common µg/L-eq scale via its lab gain and blank, so the per-sensor intercept here calibrates the site-specific tryptophan→E. coli relationship, not the optics. Metrics are in-sample. All six Boulder units are shown; 50062 (weak/saturating responder) and 50059 (no dilution, borrows 50046's calibration) are the two units whose C_TLF is least trustworthy.

This page loads the C_TLF candidate evaluation (static, regenerated offline from the frozen raw data). Loading…

What the residual is made of — grab variability, not sensor or method error

The field prediction error (0.38 log₁₀) decomposes into three independently-grounded measurement components plus an unresolved term. The sensor term is measured from lab reproducibility (three co-located units on identical bucket water); the grab and Colilert terms come from the water-quality literature and the IDEXX assay interval. The remainder is unresolved error (short-term representativeness plus any model error), which we do not assign a value.

The unresolved term is dominated by grab representativeness: same-day grabs at a near-identical sensor reading return E. coli differing by 3–9× (up to ~1 log in 30–80 min): e.g. 50066 on 06-09 predicted 91/88 for actual 51/260; 50046 on 06-29 predicted 24/25 for actual 29/272. The sensor integrates the water continuously, so a single instantaneous grab is a noisy target it cannot be expected to match point-for-point. Because we lack high-frequency data to measure it, we leave this as unresolved error rather than assigning it a value [6,7].

Sensor vs Colilert by site

C_TLF-predicted vs Colilert-observed log₁₀(E. coli), one color per Boulder site (n = 122, in-sample; r = 0.69). Each point carries two measurement CIs: a horizontal bar for the reference (grab ±0.10 log ⊕ the Colilert/Quanti-Tray 95% CI) and a vertical bar for the sensor (±1.96·σ_sensor = ±0.31 log, lab reproducibility). Green rings mark points that agree with the 1:1 line within the combined 95% CI, |predicted−observed| ≤ 1.96·√(σ_sensor²+σ_ref²); red rings do not. 82% agree. The in-sample regression slope is 1.00 by construction (fixed-effects OLS fitted values), so unlike the deployed mixed-effects fit this scatter is read for spread, not attenuation.

Colilert CI source: the IDEXX Quanti-Tray/2000 MPN table with 95% confidence limits (doc 2593-00; ±≈0.18 log₁₀ near 126–200 MPN).

A note on bias

The scatter measures precision (how tightly predictions track the grabs: 82% agree within the combined 95% CI in quadrature; 88% have overlapping error bars). Bias — a systematic offset — cannot be read from the in-sample fit: the per-sensor fixed intercepts absorb the mean residual, so an in-sample bias test is uninformative. A genuine bias / equivalence test (e.g. TOST against the Quanti-Tray/2000 CI margin) requires held-out data and is deferred.

Measurement-uncertainty framework & references

Every uncertainty term above is a measurement or a published value, not an assumption:

• Sensor precision (σ = 0.16 log₁₀): reproducibility of three co-located Lume units reading identical bucket water (see the Colilert-Lab page), the instrument-precision concept of ISO 15839 / ISO 5725 [1,2].
• Grab collection (σ ≈ 0.10 log₁₀): field-duplicate reproducibility of E. coli grabs (McCarthy et al. 2008, ±33%; Harmel et al. 2016 sample-collection term) [3,4].
• Colilert assay (σ ≈ 0.10 log₁₀): the Quanti-Tray/2000 95% counting CI [5] and the laboratory-analysis term of Harmel et al. [3].
• Agreement test: a point agrees when |predicted−observed| ≤ 1.96·√(σ_sensor²+σ_ref²), combining standard errors in quadrature (bar-overlap would overstate agreement): a difference test in the spirit of Bland & Altman limits of agreement [8] under the GUM uncertainty framework [9].
• Attenuated slope (0.65): because the reference grab carries error, ordinary least squares under-estimates the slope; an errors-in-variables (Deming) fit would correct it [10].
• Unresolved representativeness: short-term temporal/spatial E. coli variability is real and large but site-specific, so it is left unquantified pending high-frequency data [6,7].

1. ISO 15839:2003, Water quality — On-line sensors/analysing equipment — Specifications and performance tests. iso.org
2. ISO 5725, Accuracy (trueness and precision) of measurement methods and results.
3. Harmel et al. 2016, Uncertainty in monitoring E. coli concentrations in streams and stormwater runoff, J. Hydrology. link
4. McCarthy et al. 2008, uncertainty in field-collected E. coli data (±33%).
5. IDEXX Laboratories, Quanti-Tray/2000 MPN table with 95% confidence limits (doc 2593-00). PDF
6. Leecaster & Weisberg 2001, Effect of sampling frequency on shoreline microbiology assessments, Mar. Pollut. Bull. PubMed
7. Boehm et al. 2007, Enterococci concentrations in diverse coastal environments exhibit extreme variability, Environ. Sci. Technol. link
8. Bland & Altman 1986, Statistical methods for assessing agreement between two methods of clinical measurement, Lancet.
9. JCGM 100:2008, Evaluation of measurement data — Guide to the expression of uncertainty in measurement (GUM).
10. Deming / errors-in-variables regression for method comparison (e.g. Linnet 1998, Clin. Chem.).

Predictive classifier (in-sample)

The exceedance classifier thresholds the C_TLF continuous prediction at the balanced-accuracy-optimal operating point; three-level probabilities bin the prediction at 126 and 1000 CFU. n = 122. In-sample. This is the C_TLF candidate classifier (Boulder dilution units), not the deployed detector: its headline balanced accuracy (~0.67 on all six units, ~0.74 on the four cleanly-calibrated ones) sits below the deployed mon2 detector's ~0.79 because C_TLF is Boulder-only and includes the weak 50062 / borrowed 50059 units.

Three-category — Low (<126) · Mid (126–1000) · High (≥1000)

Read each accuracy as a range, not a point — for three independent reasons. (1) Few exceedances. Balanced accuracy rests on only ~19 exceedance grabs out of —, so its bootstrap 95% CI (shown in the summary above) is wide; a single decimal implies precision this many grabs cannot support. (2) The reference is itself uncertain. Colilert is an MPN with a published confidence interval, and ~20% of grabs sit close enough to the 126/1000 boundaries that their true category is ambiguous — scoring against a fuzzy truth turns a point into a band (propagating the Colilert CIs moves the headline by ≈±0.03–0.04). (3) Floor vs ceiling. How you score those boundary grabs sets a floor (the hard number: counted wrong if off) and a ceiling (the “up to … within Colilert CI” line under each card — the highest score you reach once every prediction the reference cannot prove wrong is credited). The operational value lives between them. So a C_TLF headline like “balanced accuracy ≈ 0.67” is better read as “credibly ~0.52–0.77, depending on sampling and reference ambiguity” — and the three uncertainty sources are distinct (sampling noise, fuzzy labels, and scoring convention), so they widen the range for different reasons rather than double-counting.

Overall — exceedance detection (safe vs unsafe, BA-optimal operating point)

Collapsing to a single safe/unsafe call at the BA-optimal operating point balances sensitivity and specificity; live numbers load below.

Confidence intervals load with the live numbers above.

Because this is a three-category model, sensitivity and specificity are computed one-vs-rest (each category against the other two). Sensitivity (recall): of the grabs truly in a category, the fraction the model assigns to it, TP / (TP+FN). Specificity: of the grabs not in a category, the fraction the model correctly keeps out, TN / (TN+FP). Precision: of the grabs predicted in a category, the fraction truly in it, TP / (TP+FP). F1: harmonic mean of sensitivity and precision. Macro average is the unweighted mean across the three categories. Of the two headline cards above, overall accuracy is the 3-level (Low/Mid/High) fraction of grabs whose category is correct (the confusion-matrix diagonal ÷ n), and balanced accuracy is the two-level (safe/unsafe) figure at the model’s chosen BA-optimal operating point — (sensitivity + specificity)/2, where truth is <126 vs ≥126 (the action limit) and the prediction flags unsafe at ≥88 (a buffer below 126). Overall accuracy is inflated by the dominant Low class, so balanced accuracy is the better single number; it matches the safe/unsafe summary near the top of this section. The small “up to … within Colilert CI” line under each card is a CI-tolerant ceiling: a prediction is credited whenever the grab’s Colilert 95% CI cannot call it wrong — its confidence interval overlaps the predicted category (3-level) or straddles the 126 line so the true class is unresolved (safe/unsafe). Because ~20% of grabs sit close enough to a boundary for this to apply, it is a ceiling (the reference’s own resolving power), with the hard number as the floor.

Thresholding the continuous model at an BA-optimal operating point balances the two errors (live sensitivity and specificity in the cards above) — versus the old multinomial's specificity ~0.52. Raising the threshold toward 126 buys specificity; lowering it buys sensitivity. (The 3-level macro-recall is lower because the continuous fit rounds some ≥1000 grabs down to Mid; they remain flagged as exceedances.)