Build a Forecast Calibration Workflow to Track Prediction Accuracy
Most people remember the predictions they got right and rationalize the ones they got wrong. This workflow forces every meaningful forecast into a structured record so you can measure not only whether you were right, but also whether your confidence was justified. The objective is not simply to make more correct predictions. It is to make your stated confidence increasingly match reality. For every forecast, record the following: Step-by-step: 1. State exactly what you believe will happen. The claim must be specific enough to resolve later. Bad: “AI stocks look vulnerable.” Better: “The Nasdaq-100 will fall at least 15% from its current level within six months.” 2. Assign a numeric probability from 0–100%. Avoid vague terms such as “likely” or “probably” without attaching a percentage. 3. Define when the prediction should resolve, such as within 30 days, by year-end, or within 12 months. 4. Identify the one to three factors most responsible for the forecast being correct. This makes the causal thesis explicit. 5. Specify what evidence would materially weaken or invalidate the forecast. This prevents moving the goalposts later. 6. Define in advance what counts as correct, incorrect, or ambiguous. Do not wait until the outcome is known to decide how success will be measured. 7. Freeze the forecast by preserving the original forecast exactly. If new information materially changes your view, create a new forecast rather than editing the old one. This preserves the forecasting record. 8. At the end of the time horizon, record what actually happened. Then distinguish between a thesis error, in which the underlying reasoning was wrong; a timing error, in which the thesis may have been right but the forecast horizon was wrong; and a driver error, in which the expected outcome occurred but for different reasons. 9. Review calibration by comparing probability bands with actual results over time. If forecasts assigned 70% confidence are correct about 70% of the time, your confidence is well calibrated. If high-confidence forecasts fail too often, you are overconfident. If lower-confidence forecasts succeed too often, you may be underconfident. Do not overinterpret tiny sample sizes; calibration becomes more meaningful as forecasts accumulate. 10. Periodically review error patterns, including overconfidence, underconfidence, timing errors, weak assumptions, ignored counterevidence, wrong key drivers, poor resolution criteria, and repeated mistakes by forecasting domain. Use those patterns to improve future forecasts. Copy-and-paste prompt Act as a forecasting and calibration analyst. Whenever I make a meaningful prediction, create a forecast record using: CLAIM: PROBABILITY: TIME HORIZON: KEY DRIVERS: DISCONFIRMING SIGNALS: RESOLUTION CRITERIA: Make the prediction specific enough to evaluate later. Require a numeric probability and a defined time horizon. Preserve the original forecast exactly. If new information materially changes the forecast, create a new version rather than rewriting the original. When the forecast resolves, add: OUTCOME: RESULT: THESIS ACCURACY: TIMING ACCURACY: DRIVER ACCURACY: WHAT WAS RIGHT: WHAT WAS WRONG: CALIBRATION REVIEW: LESSON: Periodically review resolved forecasts for: Accuracy by probability band Overconfidence Underconfidence Timing errors Repeated bad assumptions Strongest forecasting domains Weakest forecasting domains Never rewrite history to make an old forecast look better. The objective is not to maximize correct predictions. The objective is to make confidence increasingly match reality.
