Forecasts that admit how wrong they can be
A fuel-consumption model for one vessel class, served through a small web console. The work that mattered most was the audit that came after the first version.
- Physics baselinesets the shape
- Learned correctionsloading and trim
- Weather termsscaled by speed
- Hull conditiona filter that adapts
- Calibrated intervalchecked on unseen data
- Predict daily consumption from speed, loading, trim and weather, with ranges, from a couple of years of noon reports: a few hundred clean rows.
- Very little data, and no real examples of slow steaming.
- Every number in the app is shown with an interval, so the interval has to be honest.
Decisions.
Physics sets the shape, data sets the level.
Learned directly from daily reports, the speed curve would badly misjudge slow steaming, a regime the data never contains. A hull-resistance baseline carries the shape, and a robust regression learns the loading and weather corrections.
Every score uses only earlier data.
Replay is walk-forward, and the final holdout is touched once.
Audit your own claims.
A from-scratch audit found the “90%” intervals covered far less than 90%, because calibration used residuals from inside the training set. I replaced it with a disjoint split-conformal calibration and re-measured coverage until it matched its label.
Test the direction, not just the score.
The physics-based part had a head-wind response that flipped sign in most conditions. Cross-validated R² could not see it. A counterfactual test can: hold everything fixed, change only the wind direction.
Say what the data cannot tell you.
The write-up states plainly that the model is good inside the observed envelope and that slow-steaming claims are a data gap, not a modelling gap.
- Interval coverage now matches the label printed on it.
- Errors are reported per day and per voyage, and the model states its noise floor: the underlying readings disagree by a few percent day to day, so error cannot reach zero.
- I would run the coverage check before showing the first interval, not after.