Calibrate on data the model never saw
Why an interval labelled 90% can cover far less, and the small change that fixes it.
Conformal prediction is wonderfully simple: take the model's errors on a calibration set, find the error size that covers 90% of them, and add that band around every prediction. The guarantee, though, depends on one condition that is easy to break: the calibration errors must come from data the model did not train on.
Flexible models nearly memorise their training data, so their in-sample errors are tiny. Calibrate on those and the band is far too narrow. The interval still says 90%. It just no longer means it, and no aggregate accuracy metric will tell you.
# split-conformal, the honest version
train, calib = split(data) # disjoint
model.fit(train)
scores = abs(calib.y - model.predict(calib.x))
q_hat = quantile(scores, ceil((n + 1) * 0.9) / n)
interval = (pred - q_hat, pred + q_hat)The check is one more measurement, not one more model: replay the data forward in time, and count how often the true value falls inside the interval. If the label says 90% and the count says something else, the label is the bug. Run it before you show the first interval to anyone.