Changelog

2026-05-20

Estimate question difficulty for market questions on the Baseline leaderboard via two-way fixed effects

We use the market Brier to estimate question difficulty on the Tournament leaderboard. But, for the Baseline leaderboard, models face additional difficulty in that they have absolutely no context with which to forecast beyond their training cutoff. Hence use the two-way fixed effects model to estimate question difficulty in this context. More at Issue #190. See 6564a98.

2026-03-05

Require a minimum of 50 resolved market questions

Following the change in sampling method to account for end-of-year effects on forecasting platform questions (mentioned in the Changelog on 2026-02-19) , we have introduced a minimum requirement of $50$ resolved market questions per forecasting round for models to be included on the leaderboard. See ea79cf3.

2026-03-04

Use the Brier Index to display results

While our scoring method remains unchanged, we have started using the Brier Index to make leaderboard results more interpretable. See 3007a70.

2026-02-19

Improve sampling

Our sampling method for questions pulled from market sources suffered from end of year effects, where many questions were resolved at the end of the year. We now use stratified sampling to have a better distribution of questions across time. See 890d263.

2025-11-19

Limit submissions per round

For all future rounds, each team is limited to $3$ submissions per round. If more than $3$ forecast sets are uploaded, we will only consider the first $3$ in alphabetical order by filename.

2025-10-17

Removal of combination questions

Combination questions have been removed from the question set because they were not being scored on the leaderboard. We plan to include them in a future version of the benchmark. Consequently, there are now $500$ questions to forecast in the question set as opposed to $1,000$. The first question set without combination questions will be released on 2025-10-26.

2025-10-07

Scoring method: difficulty-adjusted Brier score

Since we released the paper, we have been working on how best to score forecasts across multiple datasets. The paper used the Brier score, which is a standard scoring mechanism in the forecasting literature. However, it does not control for variability in question set difficulty across time.

We looked at several scoring methods, including the Peer Score, the Brier Skill Score, and Elo. In the end, however, we settled on the two-way fixed effects model to separate forecaster ability from question difficulty, and thus generate a difficulty-adjusted Brier score. Read more about the difficulty-adjusted Brier score in the technical report.

50-day delay before a forecaster is included on the leaderboard

Our initial hope was to release forecaster performance 7 days after the forecast due date. After running a series of stability tests, we found that rankings stabilized after the 50-day forecasting horizon had passed. Accordingly, we are increasing the delay to 50 days after the forecast due date. The stability analysis is available in the technical report.

Technical documentation forthcoming.

Only score resolved market questions

We found that, for market questions, many unresolved markets did not move much. This dampened the signal we were getting from looking at resolved markets, so we have decided to no longer score unresolved market questions; we now wait until resolution to update a forecaster's performance on market questions.

Keep combo questions for a later version of the benchmark

Combination questions are currently excluded from the leaderboard rankings, consistent with the approach followed in the original paper. Our initial analysis found that LLMs performed very poorly on these questions, leading us to exclude them from the main leaderboard. While these questions do not contribute to rankings at present, we require all participating teams to submit forecasts for combination questions as we plan to incorporate them in future iterations of ForecastBench.

Forecasters must provide forecasts on ≥95% of all (standard and combo) questions in a forecasting round

In the paper we offered to impute values to missing forecasts. Though we still impute values, we prefer to add a requirement on the number of forecasts a team should produce within a forecasting round as this is, in the end, a leaderboard for determining forecaster ability.

Forecasters must provide forecasts on ≥95% of standard dataset and ≥95% of standard market questions

A further participation requirement is that forecasters must provide forecasts on ≥95% of standard (i.e., non-combination) dataset questions and ≥95% of standard (i.e., non-combination) market questions. Again, in its current iteration, we want to judge

Impute 0.5 to missing forecasts

We were imputing the market value on the due date to missing forecasts of market questions, but this was overly generous and made forecasters seem better than they actually were.

Baseline forecasters

We have created several comparison models that forecast on every round. They are the:

• Naive Forecaster: forecasts using Prophet to forecast dataset questions and the market value the day before the forecast due date to forecast market questions.
• Imputed Forecaster: forecasts $0.5$ on dataset questions and the market value on the forecast due date to forecast market questions.
• Always 1: Always forecasts $1$.
• Always 0.5: Always forecasts $0.5$.
• Always 0: Always forecasts $0$.
• Random Uniform: forecasts are drawn uniformly from the interval $[0,1]$.

2025-02-28

The arXiv v5 version of the paper describes the Benchmark until 2025-02-28.