Brief
Competence gate improves hybrid forecasting by weighting language models per domain
A new arXiv preprint describes a method that estimates when a language model adds value beyond an existing forecast. The evidence is a single preprint with no implementation or availability details.
The paper introduces a competence gate that estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast. Across 2,357 resolved binary questions and five language models, it improves the main external baseline from 0.0771 to 0.0732 Brier and significantly outperforms global forecast combinations. The useful detail for practitioners: outcome-estimated competence supports better abstention decisions, while verbal confidence does not reliably identify when the model outperforms the external forecast. However, the gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market.
Our reading
Our reading is that this offers a practical way to decide when to trust a model's forecast, but its value depends on having resolved outcomes for the specific domain.
What to do or watch
The precise unresolved question is whether the competence gate's gains generalize beyond the studied questions and models, since it gave no significant improvement on the official ForecastBench market subset and no implementation or availability details are provided. Watch for evidence on whether outcome-estimated competence supports better abstention decisions in domains where resolved outcomes are unavailable, as verbal confidence did not reliably identify when the model outperforms the exter…
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- Across 2,357 resolved binary questions and five language models, the gate improves the main external baseline from 0.0771 to 0.0732 Brier.
- The gate significantly outperforms global forecast combinations.
- The gate gives no significant improvement on the official ForecastBench market subset, where it largely defers to the market.
- Verbal confidence does not reliably identify when the model outperforms the external forecast, while outcome-estimated competence supports better abstention decisions.
- The gate estimates domain-level source weights from resolved outcomes, shrinks uncertain estimates toward a global weight, and recalibrates the pooled forecast.
Sources
- arXivText stored 14 September 2026
How this story was checked. Written from the 1 page listed above, stored 14 September 2026; claims checked against that stored text on 14 September 2026.
What that means
- 5 of 5 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.