In studies of extreme‐event forecasts, researchers found that the true sample size is the number of exceedances—days when the outcome falls beyond the predicted interval—rather than the total number of observations. For prediction intervals set to a miscoverage level α, the number of data points needed to compare forecasts grows ≈1/α, while the number of exceedances stays constant. An anytime‑valid confidence sequence built with an empirical‑Bernstein bound reaches this optimal 1/α rate without assuming any distribution.

The study also examined the Diebold‑Mariano (DM) test, a common tool for comparing forecasts. When exceedances are scarce, the DM test frequently declares the less conservative forecaster superior with a probability at least as high as the chance of observing no exceedance, even if that forecaster is not truly better. Small‐sample corrections do not fix this bias, though a bootstrap‑t variant largely mitigates it.

These findings suggest that researchers should count exceedances rather than total observations when assessing tail‑forecast performance and be cautious of the DM test’s tendency to favor less conservative models in sparse‑exceedance settings.