Probability calibration explorer
See what a stated probability promises at this sample size — reliability bands, base rates and honest uncertainty.
What does one probability promise?
- Expected yes outcomes in the sample
- 50.0
- Observed yes share, about 19 samples in 20 (normal approximation)
- 40.2% – 59.8%
- With p really at 50% and 100 independent events, the share landed here is typical — not promised. With only 100 events, an honest 50% forecaster waits many samples before their calibration can be judged.
Calibration, illustrated
| Forecast band | Rows | Mean forecast | Mean outcome |
|---|---|---|---|
| 0–20% | 40 | 0.10 | 0.12 |
| 20–40% | 60 | 0.30 | 0.28 |
| 40–60% | 120 | 0.50 | 0.52 |
| 60–80% | 90 | 0.70 | 0.66 |
| 80–100% | 30 | 0.90 | 0.86 |
Synthetic example — not a real forecaster and not Tollbridge data.
Calibration compares the mean forecast inside a band with the mean outcome inside the same band. A calibrated record keeps those two columns close across the whole table. The gap between them — not the raw accuracy — is what makes probabilities reliable.
Sample size pushes the other way: ten events tell you almost nothing about calibration, which is why a published record must carry its sample sizes (340 rows here) rather than a bare percentage.
Frequency is one sample; probability is a claim
A probability says: out of many like situations, roughly this share would come out yes. A single outcome can never confirm or refute it — a 90% claim fails one time in ten, exactly as stated. Calibration is what you check instead: whether outcomes match stated rates across many samples.
How the band is computed
sd = √(p(1−p) / n) · band = p ± 1.96 × sd
This normal approximation is the honest quick estimate for sizable n; for tiny samples exact binomial intervals are wider and one-sided results dominate. The explorer states its formula and its limits rather than implying more precision than the arithmetic carries.
Base rates and honesty
Calibration has to be judged against the base rate of outcomes in the sample: on questions that turn out yes 80% of the time, “80%” answers everywhere is a calibrated but useless record. That is why scoring against a reference — the share of yes outcomes in the sample — is standard, and why a record that hides its mix of easy and hard questions cannot show real skill. Tollbridge grades every decision against its published record with the refusals in place, and states sample sizes wherever a rate is claimed.
Where calibration turns into a decision you can price
If calibration is the standard, the next questions are concrete: what probability does the current market state deserve, and what does that probability cost? The live perp list states the inputs, the Brier calculator grades any record — yours or Tollbridge's published one on the track record, where every number carries its sample size. Nothing on this page is financial advice.