The barrel — four batches in shared DI water
Twenty-five units of the same Lume v1.2 design tested together in air: ten built with FDOM optics, ten with chlorophyll-a optics, and five TLF units, the established channel, running alongside them — same room, same bench, same hours, so a difference between the columns reflects the optics rather than the conditions (channel colours throughout follow the house convention — gold = FDOM, green = chlorophyll-a, blue = TLF). This page qualifies them on the two properties that must hold before any wet calibration is worth doing: does the signal drift, and how does it respond to temperature. The fleet starts in air — a dry window cannot hold a bubble or a conditioning film, so whatever drift appears here belongs to the LED, the detector or the electronics, not to the water — then goes into shared DI water on 2026-08-20 and stays there for the next nine days.
variant, ditank, degas,
burnin) are marked frozen in scripts/export_burnin_sweeps.js, so no
further data enters this page and every number on it is stable. variant and
ditank also carry a hard stop at that instant, so even a re-pull given
--thaw can only reproduce this window — it cannot quietly extend a published
record.
500196 (08-21), 500128 (08-22), 500156 (08-24),
500228 (08-25), 500179 (08-26) and 500208 (08-27) each fall
silent and produce nothing afterwards — checked against the database directly, not inferred
from a gap in this file. Only two of the six are explained by power:
500179 and 500208 were on battery and read 3.42 and 3.36 V at their
last sample, below this page's 3.6 V red line. The other four had usable packs, and
500196 and 500156 were on mains and charging at a healthy
4.38 V, sampling normally, right up to the instant they stopped. That is not a discharge
curve, so something else ends those two records and this page does not know what it is.
Stage 1Fleet status
The signal is sFDOM and sChl-a — the identical methodology the
tryptophan fleet runs on as sTLF, applied to these optics with nothing changed. For each
sweep, every dark-subtracted cell of the LED×bias grid enters
log(mon2 − dark) = aLED + bovervoltage + c
and c is the channel amplitude. The estimator is wavelength-agnostic —
it never assumes what the LED emits, only that one emitter is swept across powers and biases — so it
carries over unchanged; it is support-invariant (railed cells enter as censored bounds, so
which cells survive screening cannot move the answer), which is what makes it robust when a bright dry
window pushes the top of the sweep into the ADC ceiling.
Two consequences worth stating. The name follows the channel, not the history:
the same estimator is sTLF on tryptophan units, sFDOM here on FDOM units and
sChl-a on chlorophyll-a units; the stored column keeps the historical name
tlf_amp so one code path serves every fleet. Absolute levels do not transfer:
different emitters, filters and detector efficiency put the three channels on different scales, so nothing
here is comparable to the tryptophan fleet in absolute terms — every series is normalised to its own
unit, and only shape (drift rate, temperature response) is compared across channels.
Drift is quoted per phase, not pooled. The air phase and the water phase are different experiments — bubbles and wetted-window effects exist only in one of them — so each rate is fitted on its own window and reported with the number of units behind it. The headline figure is the water phase, since that is the current condition; the air rate sits beneath it. A unit contributes to a phase’s number once it has 12 h inside that phase. No sensor is excluded for fitting badly. The 12 h floor is a statement about the window — whether drift and temperature are separable in it at all — while scatter is a property of the sensor, and is reported alongside each rate instead of used to suppress it. So the counts differ between phases only because the phases have different lengths. A rate is only quoted where time and temperature are separable. Where |corr(t,T)| > 0.9 inside a phase, the drift term simply absorbs the quench, so the tile reads not separable rather than a number. On the two-day version of this page the water phase was 22 h dominated by a 27 °C ramp — exactly that situation — and every column read not separable. The soak that followed is nine days at room temperature, so the correlation falls and the tiles below now carry rates. A second guard applies underneath it: a unit whose every candidate model implied an implausible correction has no fitted rate at all, and is counted out of its column rather than entered as a zero — which is why the chlorophyll-a tile reads no believable fit. DI water contains no chlorophyll, so those units are tracking their own noise floor and there is nothing there to fit.
Stage 2Drift — uncorrected and temperature-corrected
sFDOM / sChl-a — uncorrected
Normalised to each unit’s first settled hour, so the two optics are comparable in shape even though their absolute levels are not. Both panels share one y-axis. Whatever the building’s temperature does, you see here.
sFDOM / sChl-a — temperature-corrected
Each series divided by its own fitted response
e^(bT(T−25)). Differences between the columns are the point of the
experiment: the emitters and filters need not age or respond alike.
This panel applies the production correction — the same form as the
canonical drift_correction.mjs, so what is plotted is what a deployed sensor would
do: ln(amp) = Pd(tn) +
(r1+r2tn)·dT/dt + slope·days, evaluated back
to T = 20 °C and dT/dt = 0. Every term comes from the
sensor’s own sipm_temp_c; a correction needing a reference instrument
beside it could be validated on this bench and could never ship.
The model is only used where the data supports it. This window now spans the air
phase, the UV dose, DI entry, shakes and two temperature ramps — and a cubic asked to
describe all of that absorbs the non-thermal steps as temperature curvature. Unguarded it implied
corrections of ~2400×. So each fit is checked against the correction it
implies and falls back until one is physically plausible: cubic + lag →
cubic → linear + lag → linear → none, with an 8× ceiling.
On the current record 30 units take cubic + lag, 4 cubic, 8 a linear form, and
2 fail every model and are shown uncorrected rather than plotted with a
correction we do not believe. dT/dt is a least-squares slope over ±10 min, because a
two-point difference on 1-min data differentiates temperature noise into 20–30 °C/hr
artefacts on a barrel that never ramped faster than ~5.
Flatness here is still not evidence — the fit is in-sample. Scored on
held-out 15 min blocks across the up-then-down ramp, where |corr(t,T)| falls to 0.41 and the
terms are finally separable, the production form gave 0.397 % against
4.693 % uncorrected and 0.742 % for the linear fit this page used
before.
Fully corrected — quench, lag and drift removed
The complete production correction, referenced to the start of the fit window:
ln(amp) = Pd(tn) +
(r1+r2tn)·dT/dt + slope·days with every term
divided back out. This is what a deployed sensor would report after correction, so
flat is the target and whatever is left is what the model does not explain.
Read it against the panel above: that one removes temperature and leaves drift, so the
units fan apart at their individual rates. Here the drift is removed too, and the fan should
close. Where it does not, the drift is not a constant per-day rate over this window — which
is the substantive question, since a rate that will not extrapolate cannot be corrected forward
(see findings: extrapolating a fitted slope was worse than assuming
flat in every configuration tested).
In-sample, necessarily. The drift term is fitted on this window, so removing it
here is close to circular — a flat line proves the fit converged, not that the rate
generalises. Judge this panel by the spread, never by the slope: subtracting a fitted
ramp always levels a trace, so the slope is guaranteed and only the scatter is informative.
Every unit is plotted. Nothing is withheld for fitting badly. An earlier version
of this page dropped units whose corrected scatter exceeded a fixed threshold, which removed all
five new TLF units. That was a mistake in the metric, not a property of those sensors. A single
residual number cannot separate a bad fit from a good fit on a noisy sensor,
and the remedies are opposite — refit the first, average the second. So the residual is now
split against each unit’s own measurement-noise floor, estimated from successive
differences, and the correction is reported at the 15-minute cadence the firmware
actually reports at rather than the 1-minute bench cadence.
Out-of-water rows are now excluded, and that mattered more than the model.
The exporter flags readings where the ToF says the sensor is not submerged; this page never
applied the flag. Because lag-1 noise squares each difference and an outlier enters it twice,
three such rows inside 1,291 took one unit’s apparent noise from
0.75 % to 21.5 %. That contamination is what made the TLF batch look ~100×
noisier than the burn-in fleet and drove their rejection. On clean data the gap is
11× (1.68 % against 0.15 %), not 100×.
The TLF batch is still the worst of the four, and not only from noise.
Its fully corrected residual at the reporting cadence sits well above the burn-in fleet's
against a 1.68 % noise floor — so most of that residual is unexplained
structure, not scatter that averaging will remove. The per-batch figures are measured
below and are not repeated here, because they moved when the window was extended from
20 h to nine days and a second hand-typed copy would go stale again. One of the five,
500156, is a hardware fault on four independent axes and is listed for
rework in findings; it also stopped reporting on 2026-08-24
and contributes nothing after that.
Measuring the fully corrected residual…
Offset and gain applied — every unit on one scale
Everything above is relative: each unit divided by its own baseline, so those panels show shape and never level. This row removes each unit’s own offset and gain instead, which is the step that makes two units comparable rather than merely similarly-shaped. It is not a concentration and needs no dilution ladder.
The shared water is the reference. Every unit in a batch sits in the same DI, so at any instant they are all measuring one thing and any difference between them is the instrument. That is what is fitted, on the fully corrected series — quench, lag and drift already removed:
gaini =
spread(si) / spread(fleet) ·
offseti = med(si) − gaini·med(fleet)
→ plotted: (si(t) − offseti) /
gaini (fleet = median across the batch; spread = median absolute
deviation, so one excursion cannot set a unit’s scale)
A unit that differs from its peers only by an offset and a gain collapses onto the fleet line. Whatever is left over is the part offset and gain cannot explain — which is the whole point of the panel, and the reason it belongs next to the corrected row above rather than instead of it.
An earlier estimator was rejected here. Regressing each unit on the fleet (
si ~ offset + gain·fleet) is the textbook form and is wrong
for this record: once quench, lag and drift are removed the shared water is nearly flat, so
there is almost no common dynamic range left to regress against and the fitted gain lands on
noise. It returned a non-positive gain on 11 of 19 burn-in units, which
would have silently dropped more than half the batch. The spread-ratio form above is always
defined, so every unit is drawn.
Binned to the reporting cadence before fitting, so a unit sampling faster cannot weight the fit, and the reference is taken only over bins where the whole batch reported, so the fleet cannot shift as units drop in and out. A unit whose fitted gain is not positive — moving opposite its own batch in shared water — is left out rather than drawn through an inverted scale.
SiPM temperature
The ambient record each correction is built on. Both variants sit in the same room, so these panels should track each other — if they do not, the two groups are not in the same thermal environment and the comparison is confounded.
ToF signal
In air the IR return reflects off the cap interior — a fixed target, and shared hardware across both variants. A wandering ToF means something on the window, not turbidity.
Battery
Mains were restored 2026-08-20 at 10:23 local (teal dashed line): right of it these are charge curves, not discharge, and the supply change itself can shift the optical signal — so drift windows should sit on one side of that line, not span it.
Pack voltage over the run. The fleet was unplugged before that, so those are discharge curves — the slope is the useful part, since it sets how long a unit can keep measuring. Dotted traces are the two units on mains (500117, 500210); they rise because they are charging. A unit that falls away from its group is heading for an early end to its own record.
Per-unit drift and temperature response
Event registry & reading notes
Everything that was done to this fleet, and the rules for reading the panels above. The charts and the table are the record; this is what happened to produce it.
There is no temperature control: the fleet rides the building’s ambient swing. So the fit estimates
each unit’s own temperature coefficient jointly with its time slope,
ln(amp) = a + bT(T−25) + slope·days — the same structure the
tryptophan fleet’s air test uses. Until a full diurnal cycle has passed, time and temperature are
collinear and the split between them is not identifiable; those fits are flagged rather than reported as
fact.
The first 2 h after each unit’s first reading are excluded as power-on settling, and a unit needs 12 h of settled record before it gets a rate — the floor established by the drift-correction validation, below which fits are coin flips.
The air phase ended 2026-08-20 at 10:45 local, when the variant batch went into DI water together (blue dashed line); mains had been restored 22 min earlier at 10:23 (teal line). Everything right of the blue line is a water record, so the drift and temperature rates on this page describe the air phase only unless a window is explicitly stated otherwise — a fit that spans those lines mixes two media, and briefly two power states, and is not interpretable as drift.
The whole barrel was shaken at 14:19 local (purple dashed line), with all 44 units submerged in it. This is the bubble test at fleet scale. The degassing rig established the mechanism on a small sample: pulling a vacuum grew something on the optical window, and shaking shed it — 500226 ran to 13.4× its baseline under vacuum and returned to 0.93× the moment it was shaken, with ToF moving in step both ways. A shake here should therefore move any unit carrying a bubble and leave a clean window alone. Because every unit is shaken at the same instant and shares the same water, the spread of step sizes across the batches is the measurement, not any single unit — and sTLF and ToF must move together for a step to count as a window effect.
The barrel is being heated from 14:18 local (red dashed line), ramping at about 4.6 °C/h from 27.0 °C, to be held at temperature for several hours. The heater came on in the same minute as the shake, so the two are not separable at that instant — the shake figures below are quoted after dividing out the quench, and the correction moves them by under a percent, so the null survives it.
Why this run matters more than the ones before it. Across all three experiments the limiting problem has been that temperature and elapsed time move together, which makes a drift number unidentifiable: of six windows tested, five had split-half disagreements larger than the drift being reported. A ramp followed by a hold breaks that, because the ramp moves temperature a long way in a short time (identifying the quench with little drift in it) and the hold moves time at fixed temperature (identifying the drift with no quench in it). How long the hold runs decides what the drift number means. Measured on the only previously reliable hold, successive 6 h blocks gave −1.32, −1.12, −0.76 then −0.53 %/day — so a 2–4 h hold returns roughly −1.4 %/day where the settled value is nearer −0.5. A short hold measures a settling transient sitting on top of the drift, not the drift.
Result: no unit shows the bubble signature. On 37 of the 44 units reporting across the ±15 min window, every batch median sits within about a percent of zero — FDOM −0.20 %, Chl-a +0.91 %, TLF +0.52 %, Burn-in −1.15 % — and zero units move more than 5 % on both channels at once. Seven units move more than 5 % on sTLF (largest 500109 at −19.0 %), but each has a ToF response of 3 % or less, so by the rule set in advance none of them is a window effect.
That null is informative, not a failure. It is the control the degassing rig needed: the same agitation that stripped a 13.4× excursion off 500226 does essentially nothing here. The difference is the gas. These units went into fresh DI a few hours earlier at ambient pressure, so there is no supersaturation to nucleate from and nothing on the window to shed — whereas the degassing units had bubbles deliberately grown on them by lowering the pressure first. Read together, the two results say bubbles cause large coupled steps when the water is supersaturated, and that a shake in ordinary, freshly-poured DI does not produce them. It also puts a boundary on the fleet’s historic +14 % in-water handling steps: those units had been submerged for days, not hours, which is the variable this test did not vary.
The seven sTLF-only movers are the same pattern as 500146 on the degassing rig — a real signal move with no optical corroboration. Something is moving sTLF that is not on the window, and it is still unexplained.
Columns are batches. The FDOM, Chl-a and TLF batches are the three optics builds of the Aug-2026 variant run; the Burn-in batch is the older fleet. Both the TLF batch and the Burn-in batch are TLF sensors — same channel, same sTLF estimator, so the two blue columns are directly comparable on the y-axis. What differs is their history, which is exactly the contrast worth reading. Grouping this way keeps each column a set of instruments with a shared build and a shared history, which is the only basis on which a column median means anything.
The Burn-in batch arrived at 11:40 local: 19 units (navy dashed line) joined the same DI tank, 55 min after the variant batch. Their air test closed at 09:00 and its record stays on the burn-in page; from 11:40 they share this water. They are deliberately not pooled with the TLF batch next to them, because they are not comparable instruments — roughly 15 days of burn-in, a 46 h air phase and, for most, a UV dose spike sit behind them, so their drift state entering the tank is nothing like a new unit's. Same water, same instant onward, different history: that contrast is the reason to show them together, and it only survives if the two stay separable. Note the two entry times differ, so “hours in water” is quoted per column, never once for the page.
The shaded band is the UV-LED burn-in on 2026-08-20. Every unit was put on the UV-LED
high-power build and taken back off it, to spike the cumulative dose on the new optics the way the
tryptophan fleet was spiked on 08-11. Firmware is pushed as a request, so each unit flashed on
its own next sync and carries its own window — the band is the fleet envelope, from the first unit
flashing (14:08 UTC) to the last confirmed back on the normal build (15:02 UTC). The intended
hold was 20 min per sensor; the revert fires on the first poll past that mark, so the doses actually
delivered run 26–33 min (median 30). Per-unit start, end and dose are recorded
in data/fw_experiment_variant.json — read the dose per unit from there rather than
assuming 20 min.
The dose is visible in the record itself: on the UV build a unit stops emitting normal sweeps, so each one leaves an outage. 24 of the 25 show that outage (median 32.6 min, range 29.8–38.1), and the time each resumed is exact. The outage slightly overstates the UV runtime, because samples taken since the last sync are lost when the unit reboots to flash. 500156 (TLF) shows no outage at all — it sampled continuously every minute right through the window, so despite its firmware-update record reporting success it never actually ran the UV build and took no dose. Treat it as an unplanned within-batch control for the TLF column rather than as a dosed unit.
ReferenceThree In-Situ AquaTroll 500 in the same barrel
The sondes are not a fifth batch — they are a reference frame. A batch answers “do these instruments agree with each other?”; the sondes answer “does any of this correspond to the water?” Only some of their channels have a Lume counterpart, so each one is sorted into a tier and the tier decides whether it is compared at all.
Tier 1 — the same quantity we measure by proxy
Water temperature against sipm_temp_c, which is a
thermistor on the electronics, not in the water — every temperature coefficient on this
site is currently fit against that proxy. And turbidity in NTU against
tof_sps, which was never calibrated to NTU. These are comparable in absolute terms, and they
are where the real payoff is.
Water temperature vs board thermistor
Turbidity (NTU) vs ToF
Tier 2 — same analyte, incomparable units
The sonde reports FDOM and Chl-a in RFU; we report a sweep amplitude. The scales are unrelated, so these are compared as shape, never level — both normalised to a common baseline — and strictly batch-matched: sonde FDOM against the FDOM batch only, sonde Chl-a against the Chl-a batch only. An AquaTroll has no tryptophan channel, so the TLF and Burn-in batches have no reference here at all. That gap is real and is stated rather than hidden.
FDOM — sonde vs FDOM batch
Chl-a — sonde vs Chl-a batch
Tier 3 — barrel state, no Lume counterpart
Conductivity, salinity, TDS, resistivity, density, pressure, depth and supply voltage. Not decoration: if conductivity steps, the water was changed, and that reframes every drift number either side of it.
Method, and what this stage cannot tell you
Provenance. These 20 units were provisioned in Notehub on 2026-08-14 and 08-18 but had
no api_gateway / api_modem records in pumphaus, so every reading they sent was
discarded on arrival. Those records were created 2026-08-19; no data exists before that date,
and the record here begins at each unit’s first successfully stored sweep.
Resolved — channel separation. Confirmed against the first real sweeps (2026-08-19): each unit drives a single emitter across the same grid the tryptophan fleet uses — LED powers 32/128/512 over a ~2800–3300 bias sweep, 30–105 cells per reading. No unit alternates two emitters inside one sweep, so the pooled amplitude is the right estimator and each unit yields one s value per sweep: sFDOM on FDOM units, sChl-a on chlorophyll-a units, sTLF on the TLF units. No per-wavelength split is needed.
Note on the first hour: the amplitude is fit against a per-unit reference shape that needs at least 200 dark-subtracted cells before it will return a value, so a unit shows temperature and ToF immediately but no s value until roughly its fourth sweep (~1–1.5 h at 20-minute reporting). Empty amplitude columns early on are the estimator refusing to guess, not a fault.
What the TLF column is for. The five TLF units are the established channel measured under identical conditions. Their air behaviour is already characterised on the tryptophan burn-in fleet (Stage 8), so they give this page a reference point: if they behave here as they do there, the bench, the room and this page’s processing are consistent with the existing campaign, and an FDOM or chlorophyll-a difference can be read as the optics. If they disagree, the setup is the more likely explanation than the new builds, and the other two columns should wait.
What a flat line here does and does not prove. Flat in air means the emitter, detector and electronics are stable over the window measured. It says nothing about wet behaviour: fouling, bubbles and conditioning films are water-side effects this stage deliberately excludes, and the tryptophan fleet showed those dominate the fast drift. Air is the clean baseline the wet phase is later compared against.
The burn-in campaign (2026-08-05 → 08-20, frozen)
Merged in from the burn-in page 2026-08-21. The 25-unit tryptophan fleet that preceded this barrel and supplied 19 of its units. The campaign is closed and its data frozen, so every number below is stable.
Burn-in fleet — validation, baseline & calibration
A single place to carefully qualify the burn-in fleet through the campaign:
dilution calibration → the clean-water record → thermal lag → ToF → baseline/QC.
Every stage reads one signal: tlf_amp, the two-way log-model amplitude
— per sensor, every dark-subtracted cell of the LED×bias sweep enters
log(mon2 − dark) = aLED + bovervoltage + c
and c is the amplitude (support-invariant: censored/railed cells enter as bounds, so which cells
survive screening cannot move the answer). It replaced the pooled quadratic
d/d(bias) slope on 2026-08-13. Because it pools every LED and bias there is no operating point to pick and no
per-sensor band to track, which is why absolute mon2 is no longer used anywhere on this page.
Campaign & window
The window is fixed in the page, not adjustable:
it opens at 2026-08-07 13:15 local, the moment all 25 units were cleaned and put into fresh
DI water outdoors, and closes with the campaign on 2026-08-20 09:00. Every stage reads that one window, so nothing
on this page depends on a control someone may have left in an odd state. Extending it is a code change, in
BASELINE_START, which scripts/build_dilution_calibration.js mirrors as
COOL_FROM.
The full stored record
The complete record for every unit, start of campaign to its close, straight from data/burnin_bands.csv.gz. Unfiltered —
the retired tap baseline, the ToF handling window, the dilution ladder and the dosed bath are all included, so
nothing the later stages leave out is hidden here.
TLF slope (sTLF) — all units
SiPM temperature — all units
ToF turbidity (signal per SPAD)
Internal absolute humidity (leak signal)
Stage 2Clean-water record — DI, outdoors
From 13:15 on 2026-08-07 all 25 sensors were cleaned and placed in fresh DI water, outdoors. This is now the clean-water reference for the whole page: the baseline block, the quench, the ToF calibration and the end-to-end test all read this window and nothing earlier. Outdoors it sweeps a real diurnal range rather than a bath cooldown, which is the condition a deployed sensor actually meets — and, unlike the retired tap window, it breaks the collinearity between temperature and elapsed time.
There is no analyte in DI water, so everything in the raw trace above is instrument. Three terms account for it, fit per sensor on this window:
ln s = a + cubic(T − 21) + c · dT/dt + d · e−n/τ
— a temperature response, a thermal-lag term in the rate of change (the thermistor is on the electronics, not the water), and a settling term where n is samples since the last clean. The clock is sample count, not wall-clock, so it is independent of calendar and cadence. The only disturbance modelled is cleaning; the one event here is 2026-08-07 13:15 local.
Finding the offset: it is the fit intercept.
Set T = 21, dT/dt = 0 and n = ∞ and every term
but a drops out, so ea is the settled blank at 21 °C —
measured directly in clean water, which is what this window is for. Applying it: subtract it,
then divide by the ladder gain mapped into the same frame. The temperature term is multiplicative on the whole
signal, analyte included, so gain@21 = gainraw ÷ ecubic(Tladder−21)
— no blank enters, so there is no circularity. The mapping factor is 0.984 median.
Fit on train, reported on test. Every coefficient, τ included, is fit on the first two thirds of the samples since the clean (—). The last third is withheld and is shaded on the panels below. The split is temporal rather than random, because neighbouring samples are minutes apart and a random split would leak; and because the settling term is a function of samples since the clean, holding out the tail forces it to extrapolate the decay rather than interpolate inside its own training data.
On the held-out third the full model leaves — against — for temperature alone, so the settling term is doing real work out of sample, on — sensors. In-sample it leaves —, and the gap between those two is the honest cost of having fit the thing. A lifetime-count clock was tested and rejected: fit on the earlier window it predicts this one at 17.1 %, so the decay restarts at the clean rather than tracking device age.
mon2 at the calibrated combo
The single-combo reading — LED 512, SiPM bias nearest 3000, one sample, never averaged across biases. Shown first, for comparison only: the rest of the page runs on the full-sweep slope precisely so there is no operating point to track. Watch what the two channels do side by side under the same outdoor temperature swing, and note that mon2 approaches its 3250 saturation screen where the slope does not.
Full-sweep TLF slope
The same samples read as the full-sweep slope: one number per sweep, pooled across every LED and bias that passed the screen, so no single operating point can carry the reading. Raw here means raw — no temperature term, no offset, no gain. Every wiggle below is instrument, because the water is DI and holds no analyte. This window is the clean-water baseline the rest of the page reads.
Fully corrected sTLF — should be flat
The three terms above divided out of the raw trace: temperature, thermal lag and settling, each fit per sensor on this window. What is left should be a flat line, because clean water has nothing in it that varies. The lines are stacked — each sensor settles at its own level, and that level is exactly the offset the next panel removes. The spread between them is per-unit, not drift.
Baseline subtracted — should sit on zero
The same series with each sensor’s settled blank subtracted, that blank being the fit intercept rather than anything read off this chart. Still in sTLF units, not concentration: dividing by gain comes later. The water is clean, so the truth is zero — every line should lie on it and all 24 should overlay. Departure from zero is the honest residual of the whole correction.
Against temperature, before and after
The same data on a temperature axis rather than a time axis, which
is where a quench is easiest to read. Left: raw
ln sTLF against SiPM temperature, with each sensor’s fitted temperature
term drawn through it. Right: the fully corrected signal — temperature, thermal lag and
settling all removed — which should be a horizontal band for every sensor. Any residual
slope or curve is temperature the model did not capture. Both use the model applied in this stage, not a
separate fit.
Excluded sensors
The point of the DI run is to find sensors needing rework and to correct the rest to a flat zero line. These are the ones that cannot be corrected. Withdrawn units are out of every fit and every fleet chart; rework units still report and still hold data, but are kept out of fleet statistics and hidden on the charts above — click them in any legend to bring them back. Ranking is from a clean window with the full 36 °C range and no handling (08-12 14:00 to 08-13 08:40 MT), so it measures the sensor rather than the disturbance.
The same four views as Stage 2, excluded units only
Identical panels to the stage above, so these sit on the same axes as the fleet they were removed from. A withdrawn unit’s trace ends where it stopped reporting, and units without a usable calibration are absent from the two corrected panels — which is itself the reason several of them are here.
mon2 at the calibrated combo
Full-sweep TLF slope
Fully corrected sTLF — should be flat
Baseline subtracted — should sit on zero
Stage 3Thermal lag — what temperature alone misses
Correcting on temperature alone leaves a hysteresis: at the same measured SiPM temperature the signal does not read the same coming down as it did going up. Across — that gap averages —, and it did not shrink when the post-clean decline stopped, so it is not a settling transient. Warming and cooling are separated per sample from the sign of dT/dt, not from a fixed peak time, and the gap is measured at every whole degree the fleet covered rather than at one chosen temperature. That second point matters: quoting a single bin was hiding the result below.
The likely reason is that the thermistor is on the
electronics and there is no thermometer on the water or the optical window. Outdoors the enclosure
follows air quickly while the water body lags it, so at the same measured temperature the medium the
light passes through is not at the same temperature on the two limbs. A lag shows up as a dependence on the
rate of change, so the test is to add a dT/dt term and see whether the loop closes.
(The UV LED reports its own temperature, but it is the same thermistor —
r = 1.0000 against the SiPM — so it adds nothing.)
dT/dt actually does here is shift the cooling limb up against the warming limb by roughly the
same amount everywhere, so whether it looks like a fix depends entirely on which temperature you read it at:
at the bins where the ratio already sat just under 1.000 it lands near 1.000, and at the bins where it sat
above, it moves further away. That is an offset, not a lag being absorbed. The two panels below therefore
look almost identical, and they should — the binned curves differ by well under a percent on an axis
spanning about thirty. The thermal-lag hypothesis is not supported by this test. It is not
refuted either: dT/dt over a fixed 30-minute window is a crude stand-in for a real thermal time
constant, and settling it properly needs a thermometer in the water, which these units do not have.Fleet loop: temperature term only
Every sensor corrected on temperature alone, then pooled into half-degree bins and split by limb. If measured SiPM temperature told the whole story the two curves would lie on top of each other, because the same temperature is the same temperature whichever way it is heading. They do not: the cooling limb sits below the warming limb across the range.
Same, with a dT/dt term added
The identical construction with one extra regressor: the rate of temperature change over a centred 30-minute window. If the gap were a lag between the thermistor and the water this should absorb it and the limbs should close onto one curve. Read the figure under each panel rather than the picture: the change is well under a percent, which is why the two look the same.
Per sensor, before and after
The fleet curves can hide a term that helps some units and
hurts others, so this is one point per sensor: mean |hysteresis − 1| over every
whole-degree bin that sensor covered, without the term on the x axis and with it on the y. The origin is no
hysteresis. Points below the dashed diagonal were helped, points above were made worse.
SiPM temperature over the same window
The driver behind the other three panels: outdoors the sensors sweep a real diurnal range rather than a bath cooldown. A cooldown is monotonic, so temperature and elapsed time are collinear and cannot be told apart. A day/night cycle breaks that, because the same temperature recurs at different times and on both limbs, which is what makes a hysteresis test possible.
Honest limits: the term is fit and tested on the same cycles, so any variance it removes is in-sample and flattered. It does not buy accuracy either — residual SD goes —, which is nothing. Nothing in this stage feeds the correction applied in Stage 2; it is a diagnostic, and on this evidence the honest reading is that the remaining hysteresis is still unexplained.
Stage 4ToF turbidity calibration
The clean-water period is 0 NTU by construction, so every bit of variation in
signal_per_spad_kcps over it is instrument, not water. That makes it a ground truth: fit a
correction there, and the corrected series must flatline on zero.
spscorr = sps − g · (T − 21),
then ΔSPS = spscorr − sps₀.
The axis is ΔSPS, not NTU. No formazin reference was run on these 25 units, so there is
no scale factor to apply. The pooled k ≈ 1.59 on
turbidity comes from two other sensors whose own slopes were 2.05 and 1.38
— a 1.5× disagreement between just those two — so transferring it here would print a unit we
have not earned. Everything below is still a valid relative turbidity signal; converting it to NTU
needs a formazin ladder on this fleet. g is per sensor, not fleet-wide: measured
r(SPS, T) runs from −0.69 to +0.91 across the 24 units, so it does not agree even in sign
and a single coefficient cannot be justified. The fit uses the central 90 % of each sensor’s
clean-water SPS, because three units (500155, 500230, 500228) show sustained excursions there
— 500155 runs 181 consecutive readings above twice its median — and those are events in the
optical path, not drift to be absorbed into a baseline.
Calibrated ΔSPS over clean water
The corrected ToF signal across the clean-water window, with each sensor’s own SPS zero and its linear-in-temperature coefficient g taken out. The water is 0 NTU throughout, so every line should sit on zero; what is left is the turbidity noise floor, in the instrument’s own units.
Calibrated ΔSPS at each dosed level
Same treatment as the corrected sTLF chart in Stage 1: mean ΔSPS per dosed level, per sensor, 95 % CI hashes, OLS regression with R² in the legend. The tryptophan ladder adds no particles, so every line should be flat at 0 — any slope against concentration is the ToF channel responding to something it should not.
Stage 5Indoor bath control — 50059 & 500214
Test ended 2026-08-17 07:44 local (5.9 days). Data below is the complete record. Headline outcomes: 50059’s 08-14 basin-move step (×2.6 sTLF, −15 % ToF) survived two deliberate shakes — only its ToF reverted — while 500214 in the same water stayed within a few percent throughout: handling artifacts are unit-local, shake-resistant, and sit on the LED/SiPM optical path, not the ToF window.
Two units ran from 2026-08-11 10:00 local in a water bath indoors, tracked separately from the 25-unit outdoor fleet. They are a control on the temperature confound: indoors the SiPM sits at —, against the fleet’s 21–40 °C diurnal swing. Any drift here therefore needs no temperature model to see, and cannot be a temperature artifact.
Three things make them not directly comparable to the fleet on absolute level or rate. They are the ToF firmware-swap pair, so the two may be on different builds from each other. Their cadence is far slower — roughly 4 sweeps an hour against the fleet’s 60 — so per-sweep dose accumulates about 15× more slowly, which matters for any comparison framed in dose rather than calendar time. Both start from the same instant here, so the two records are directly comparable in elapsed time.
sTLF over time
The LED-128 matched-filter amplitude, each unit normalised to its own first readings. No temperature correction is applied and none is needed — the bath holds temperature to a fraction of a degree, so this is the raw drift with the largest confound physically removed rather than modelled away.
ToF signal
The IR time-of-flight return (signal per SPAD, kcps), reflected off
the cap interior. The two units run different firmware builds, so absolute levels are not comparable between
them — what matters is each unit’s own steps. A ToF step at a handling event with
distance_mm unchanged means something entered the optical near field (a bubble, debris), not that
the cap moved.
SiPM temperature
Shown to prove the control holds. If this line is flat, the panel beside it is measuring drift and nothing else. Compare with Stage 3, where the same axis spans about 16 °C and temperature dominates everything.
Read with care while the record is short. So far 50059 has moved — and 500214 —. At roughly four sweeps an hour a day of running is only ~100 points, so these rates are indicative until the record is several days long.
18:37 shake outcome (the bubble kill test): a split verdict. Shaking 50059 in place did not collapse its ×2.6 basin-move step — sTLF ticked up (+9 % first sample, settling +5 %) — but its ToF snapped fully back 41 → 35 → 41, recovering its pre-move value. So the IR obstruction was loose (dislodged by the shake) while the TLF-boosting change is shake-resistant: either a bubble held in a crevice at the window, or the unit’s position/reflection geometry, which an in-place shake does not reset. 500214, shaken the same way, stepped up +2.4 % — consistent with the fleet-wide finding that vigorous shaking adds signal. Next discriminator: lift 50059 out and re-insert slowly at a tilt (the manufacturer-prescribed bubble-shedding move), or wipe its face — a positional cause survives both; a crevice bubble does not.
Basin carry — 2026-08-13, the fleet’s own handling step (±1 h)
The outdoor basin was carried to the DI supply at 11:52 local and the tank refilled from 12:02. During the carry itself — before any water changed — 21 of 24 units stepped UP in raw sTLF (median +3.3 %; 500179 +303 %, 500108 +166 %, 50084 +159 %, 50088 +133 %) while the bath temperature fell smoothly, so the jump is not thermal, and only 1 % of the window’s rows are flagged out-of-water. This is the same signature as the indoor pair’s 08-14 basin move (50059: sTLF ×2.6 up, ToF −15 %), consistent with sloshing depositing bubbles in the optical near field. The refill and the 40 °C ramp that followed then drove the level down — the −35 % headline step is that composite, not the carry itself.
sTLF, normalised to each unit’s pre-carry level
Raw full-sweep amplitude, each unit divided by its own 10:52–11:45 median. Shaded band = the carry (11:52–12:02); after it the refill begins. Out-of-water rows are excluded, as everywhere on this page.
ToF signal, normalised the same way
Signal per SPAD, each unit divided by its own pre-carry median — absolute SPS differs unit to unit, so the step is what is comparable. A ToF step coincident with the sTLF step, with no cap change, is the near-field (bubble) signature.
In-place shake — 2026-08-14 19:12, all 24 units (±1 h)
The companion test: the same 24 units shaken in place at 19:12 local, with no transport, no water change, and the bath at ~3.7 °C after a ~48 h cold hold — strongly undersaturated water, so any up-step here cannot be a supersaturation-grown bubble. With an hour of post-shake data: 20 of 24 units up in sTLF, median +19 % (500103 +278 %, 50091 +209 %), while ToF barely moved (median 0.0 %) — unlike the indoor basin move, the fleet’s TLF steps came without a ToF signature. Four units went down (500179 −45 % with ToF +17 %, 50084 −29 %, 500226 −21 %, 500108 −12 %), so shaking removes material from some faces while adding it — most plausibly entrained surface air — to most. The handling itself delivers the scatterer; gas saturation state is not required.
sTLF, normalised to each unit’s pre-shake level
Each unit divided by its own 18:12–19:10 median. Shaded band = the shake window. Out-of-water rows excluded.
ToF signal, normalised the same way
A coincident ToF step with no cap change is the near-field (bubble) signature, as on the indoor pair.
Stage 6Blackout caps — 4 units, fitted 2026-08-12 12:35
New blackout caps went on 500146, 50095, 500226 and 500124 at 12:35 local. 500124 has since been withdrawn — it was leaking, so this stage shows the remaining three. Its dark channel had collapsed from ~167 counts to 0 against a fleet 171–172 95 minutes before the cap was fitted, so the cap did not cause the failure and the earlier “capped versus control” numbers that included it were carrying a leaking unit. The question is what the cap interior does to sTLF — both whether it shifts a unit’s level, and whether a common cap makes units alike. Every unit’s blank differs, and if the inside of the cap is part of that, fitting the same cap to several units should pull their levels together.
Showing only the four means there is no control group to difference against, so the level figure is a before-versus-after and still contains the bath’s temperature swing. The spread panel is the one that survives this: it asks whether the four agree with each other more closely than they did, which does not need a control. Each unit is normalised on a window 8–4 h before the change, deliberately separate from the 2 h “before” window, so the pre-change spread is free rather than forced to zero.
Level — sTLF against each unit’s own earlier value
Each unit normalised to its own median over the window 8–4 h before the change, so the lines share a scale without being pinned to 1.0 at the moment the caps went on. All four are under test; there is no control series. Temperature moves all four together, so a common step at the dashed line is as likely to be the bath as the caps — what the caps would show, and the bath would not, is the four converging on each other.
Agreement — spread across the four
The point of the test. Each line is the spread of normalised sTLF across the units in that group, so lower means the units agree with each other more closely. If a common cap interior is what makes units alike, the capped line should fall below the control line and stay there.
—
Stage 7Baseline, validation & QC
One row per sensor. The baseline block is the median sTLF and
signal_per_spad_kcps (ToF) over the baseline sub-window, with the window mean temperature —
baselines are temperature-dependent, so it is recorded alongside. These become each sensor's canonical
clean_water offsets; a stable baseline has a low CV, a high CV means the window is not clean or
settled yet. The QC block is readiness over the same window: stability of TLF / ToF /
temperature judged over the last 4 h, the leak signal (internal absolute-humidity trend — rising
means ingress) and battery. Click any header to sort.
blank ÷ gain is the ppb-equivalent of each sensor’s blank, taken from the raw ladder fit. It converts blank instability into the units that matter: a unit at 36 ppb loses 0.4 ppb for every 1 % its blank moves, one at 1790 ppb loses 18. It is the acceptance criterion that decides how stable a given sensor has to be. Green ≤ 60, amber > 300.
Per-sensor baseline & QC
Stage 8Air drift test — all 24 units, 08-18 11:00 to 08-20 09:00
Started 2026-08-18 11:00 local. The whole fleet was pulled from the ~29.3 °C bath and left in air. This is the bubble discriminator: a dry window cannot hold or grow a bubble, and the LED still fires every sweep against a fixed target. If the residual burn-in drift (fleet median −0.66 %/day over the final 46 h stable cold hold in water) continues in air, it is LED/electronics aging and the cycle-count correction stands. If it stops, it was water-side — bubbles or films — and window maintenance, not modelling, is the fix. In between: both, in measurable proportion.
This stage is closed. The air test ran 2026-08-18 11:00 → 2026-08-20 09:00 local, about 46 h, and every series, drift fit, fleet median and table entry here is cut at 09:00 — not merely hidden from the charts, so no in-water reading can leak into an air-phase result. One shared end for all 24 also means every unit’s rate covers the same window. Afterwards the fleet went into water in two groups: five units into the vacuum container for the degassing test at 09:29, and the other 19 into the shared DI tank at 11:40, where they join the FDOM/chlorophyll-a batch — that record lives on the variant page, not here.
Reading rules. The pull itself is a handling event — expect a step and ignore it; windows came out wet and cool by evaporation, so the first 2 h are settling, excluded from the fit. Per-unit rates appear once a unit has ≥12 h of settled air data (the same floor the drift-correction validation set; shorter fits are coin flips). There is no temperature control in air — the fleet rides the building’s ambient diurnal — so the drift fit estimates each unit’s own temperature coefficient jointly with the time slope (ln amp = a + bT(T−25) + slope·days, the same structure as the canonical drift correction). The table shows the temperature-corrected rate, the fitted coefficient, and the raw slope for comparison; until a full diurnal cycle decorrelates temperature from time the two are confounded, and such fits carry a ⚠ instead of a verdict. These charts deliberately include the rows the out-of-water detector excludes from every water stage above.
sTLF in air — uncorrected
Full-sweep amplitude (tlf_amp), each unit normalised
to its own median over the first settled hour in air. Whatever wiggle the building’s temperature puts in,
you see here.
sTLF in air — temperature-corrected
The same series divided by each unit’s own fitted temperature response (ebT(T−25), from the joint fit). What remains is drift and noise. The coefficient is provisional until a unit clears the 12 h floor and firms up as diurnal cycles accumulate — early on this panel can over- or under-correct. Flat lines here = no drift dry.
ToF signal
In air the IR return comes off the cap interior instead of water — the step at the pull is the air signature, and each unit’s level should then hold. A wandering ToF in air means something on the window (droplet drying, debris), not turbidity.
SiPM temperature
Air-phase temperature record. The drop at the pull is evaporative cooling off the wet housing; after that the fleet rides the ambient diurnal.
Battery
Pack voltage. The fleet came off mains at 13:53 local on 08-19 (teal marker), so everything right of that line is a discharge curve. This firmware samples every minute and flattened a pack in about three weeks, so the slope here is effectively the clock on the air test — a unit that dies truncates its own drift record.
Per-unit air drift vs the water reference
Stage 9Degassing test — moved to its own page
The five units in the vacuum container — 500220, 500225, 500226, 500108, 500146 — now have their own page: Degassing test →. It reads a dataset scoped to start at the test itself (2026-08-20 09:29 local), so nothing on it is diluted by this campaign's history.
Stage 10Drift correction — the production model on this campaign
The same correction a deployed sensor would apply, run over the frozen campaign:
ln(amp) = Pd(tn) + (r1+r2tn)·dT/dt
+ slope·days, fitted per sensor on its own sipm_temp_c and
evaluated back to 20 °C with dT/dt = 0. Precomputed by
scripts/build_burnin_drift.py, because this archive is 507,660 rows and the cohort is closed.
Each unit gets the richest model its data supports, falling back cubic + lag → cubic → linear + lag → linear → none when a fit implies an implausible correction (>8×) or leaves more than 8 % residual scatter. The pass rate is the finding, not a footnote.
What the correction leaves behind
Fleet median with interquartile band, normalised to each unit’s own first settled window. Blue removes temperature and leaves drift; green removes drift as well — the complete production correction, which should be flat. Shown for the two windows where the model passes fleet-wide; on the outdoor windows too few units pass to draw a fleet band honestly.
Cold hold
Air phase
In-sample, and it matters here. The drift term is fitted on the same window it is removed from, so a flat green trace shows the fit converged, not that the rate would hold tomorrow. The forward test on findings is the honest one, and there extrapolating a fitted slope was worse than assuming the last baseline holds.
The instrument, and what each correction is for
Part numbers are from
SweetSenseInc/tlf_hardware (v0.1/tlf-bom.csv,
v0.1-led/tlf-led-bom.csv); wavelengths from the TLF method documents. Everything in the
measured column comes from this campaign's own data, and is stated separately from
the datasheet because in one important case the two differ by a factor of two.
Optical chain
| Stage | Part | Ref | Note |
|---|---|---|---|
| Excitation | CUD7GF1B ×4 | D1–D4 | SETi / Seoul Viosys, ~273 nm UV-C |
| LED drive | TPS92390 | — | constant-current, so drive is controlled not assumed |
| Emission | — | — | ~350 nm, tryptophan-like fluorescence |
| Detection | AFBR-S4N33C013 | U11 | Broadcom NUV-HD SiPM, 3×3 mm, 9815 microcells |
| Range / turbidity | VL53L4CD | — | ToF; ranges off the cap interior, not the water |
| Humidity / temp | HDC2080 | U3 | enclosure humidity — the leak signal |
The SiPM's peak sensitivity is 420 nm and its quoted PDE (43–54%) is specified there. Emission at ~350 nm sits on the falling shoulder of that curve, so absolute sensitivity is below spec — though this does not affect the temperature coefficient, which depends on breakdown voltage rather than on PDE magnitude.
Detector: datasheet against measurement
| Quantity | Datasheet | Measured here |
|---|---|---|
| Breakdown voltage Vbd | 26.9 V | 28.82 V (fleet median, from each unit's own bias sweep) |
| Vbd uniformity | 180 mV (3σ) | 0.72 V spread across 20 units — 4× the spec |
| dVbd/dT | 26 mV/K | 2.6 DAC counts/K, recovered by scanning the shape-alignment shift — the same number at ~100 counts/V |
| Overvoltage in service | 3 V “typical” | 1.18 V (bias 3000 counts = 30.00 V) |
| Gain tempco (relative) | 0.69 %/°C at 3 V 2.21 %/°C at 1.18 V |
1.73 %/°C (fixed-bias minus overvoltage-keyed) |
The datasheet's gain tempco is conditional on operating point: it is dVbd/dT ÷ Vov, so it doubles at half the overvoltage. Taking the 3 V figure at face value disagreed with measurement by 2×; using the measured 1.18 V it agrees to ~25%, the remainder being gain non-linearity near breakdown plus smaller PDE and crosstalk terms. Two consequences: running at 1.18 V makes the detector roughly 2.5× more temperature-sensitive than it would be at 3 V — a firmware setting, not a physics limit — and a Vbd spread of 4× the specification is why per-sensor calibration is unavoidable.
The correction chain
- Sweep → amplitude. The reading is a product,
mon2 = dark + G(Vov)·L(LED)·Φ·η, so subtracting the measured dark and taking
logs makes a sweep additive:
y = aLED + bovervoltage + c. Fitting a free offset per LED drive assumes no linearity of the emitter; keying the detector shape to overvoltage rather than raw bias follows the physics and was confirmed by the data (scanning the shift minimises scatter at exactly the datasheet's 2.6 counts/K). Clipped cells enter as censored bounds, not discards. Because every cell independently estimates the samec, the estimate cannot move when one is lost — which is what fixed a sensor that was alternating between two discrete values. - Out-of-water screening. A sensor clear of the water reads a higher ToF signal-per-SPAD. Flagged against the unit's own trailing 45 min median (>1.15×) and against the fleet at the same instant (>1.3×); both are needed, because a trailing baseline cannot see a sustained excursion and a fleet comparison cannot see a brief one. ~1% of rows, dropped before any fit reads them.
- Temperature: a single exponential, ρ ≈ 2 %/°C, not a polynomial. d(ln signal)/dT is constant at −1.87 %/°C from 3 to 39 °C, varying ±3% across the interior — there is no curvature for a cubic to fit. It is also what the physics predicts: LED output and fluorescence quench are both exponential in temperature and they multiply. Held out by temperature (train on the middle, predict the ends, which is what a deployment does): exponential 1.28%, quadratic 1.81%, cubic 2.17%, quintic 15.03%.
- Every temperature counts once, not once per second. The bath holds one temperature for hours and crosses the rest in minutes, so a unit's training samples run 929 in the 5 °C bin against 55 at 20 °C. Ordinary least squares weights samples, which silently hands the slope to whichever temperature the bath sat at longest, and ρ came out steep at 2.51 %/°C. Weighting by inverse temperature density instead — each 1 °C bin carrying equal total weight — gives ρ = 2.29 and cuts the clean-window residual from 3.175% to 1.354% on identical data, model form and training span. It improved 16 of 20 units; three of the four that did not were already under 1%.
- Thermal lag, a dT/dt term, measured under that weighting: 1.730% → 1.356% on a clean window. Under sample weighting the same term bought nothing (3.147% → 3.176%) — it could not show its value while the fit was being dominated by an isothermal hold, which is exactly where a lag term has nothing to say.
- Drift: currently not applied. It is fitted and reported, not emitted. On a clean window a log-time drift term made things worse, and the metric that says otherwise is computed on a window containing a handling step — where a flexible term will always look good. Per-unit step offsets at automatically detected changepoints were also tried, and were worse again (1.611% vs 0.961%): the detector finds 1–5 segments per unit and mostly fits noise. The handling log, not a more flexible model, is what closes this.
What is still open
Removing the detector term leaves 1.85 %/°C for LED output and tryptophan quench combined. Splitting them needs the CUD7GF1B optical-power-versus-temperature curve, which is not in the BOM or the repo — so the fluorescence coefficient, the one that actually belongs in a concentration model, is currently absorbed into ρ rather than measured. Separately, the handling steps of 2026-08-13 are not modelled: they are per-sensor multiplicative changes in optical throughput, invisible to every diagnostic channel, and no temperature or drift correction can remove them.