Research
Intuition Calibration Drift: Check Whether Yesterday's Confidence Still Means the Same Thing
2026-09-10
Calibration drift occurs when the relationship between stated confidence and observed outcomes changes across time or conditions. In an intuitive research workflow, examine existing dated predictions with stable outcome definitions, report uncertainty and separate changes in task difficulty from changes in performance. The analysis concerns recorded prediction behavior, not a diagnosis of the researcher or proof of a bodily mechanism.
Confidence needs a reference population
A statement such as eighty percent confident is only useful if it refers to a defined outcome and can be compared with similar recorded forecasts. Confidence in a general idea, confidence in a precise sign prediction and confidence that a dataset is usable are different quantities.
Andrei Ursachi's exploratory framework may include a confidence rating alongside an intuitive candidate. That rating is a research variable, not evidence that the candidate is correct. Its meaning should be assessed using the external outcomes attached to the same prediction records.
A change in the relationship can arise because the questions became harder, the dataset source changed or the analysis workflow was revised. It should not immediately be attributed to a change in intuitive ability.
An illustrative shift between two archived task groups
Imagine two existing batches of twenty binary predictions, each recorded at eighty percent confidence. In the first batch, sixteen outcomes match the predictions. In the second, only ten do. These invented counts illustrate a possible discrepancy, not a performed calibration study.
The first batch happens to match its average stated confidence, while the second falls below it. Twenty observations per batch are too few to support a sweeping claim about a stable trait. The comparison should include uncertainty and the composition of each batch.
Perhaps the first batch concerned familiar equipment logs and the second concerned unfamiliar climate-model outputs. Pooling them as one ability score would obscure a plausible task difference. Breakdowns should follow meaningful, preferably preselected categories rather than whichever split looks dramatic.
| Hypothetical batch | Stated confidence | Correct outcomes |
|---|---|---|
| Familiar equipment questions | 80% each | 16 of 20 |
| Unfamiliar model questions | 80% each | 10 of 20 |
Score the probabilities, not just the winning labels
Proper scoring rules evaluate probabilistic forecasts against observed outcomes. For a binary event, the Brier score uses the squared difference between the predicted probability and the outcome coded as zero or one. Lower values are better under this convention.
A prediction of 0.8 receives a squared error of 0.04 when the event occurs and 0.64 when it does not. This original arithmetic shows why strongly confident errors matter. Report an appropriate baseline so the score has context.
Gneiting and Raftery discuss the foundation of proper scoring. A single average score is not a complete calibration analysis, however. It mixes several performance properties, so also inspect how stated probabilities correspond to observed frequencies and whether the task mix changed.
Sources: Gneiting and Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation.
Locate the change before inventing its explanation
Use the dated archive to mark workflow revisions, new data sources, altered outcome definitions and changes in who resolved the results. A difference that begins exactly when the scoring rule changed may be a measurement problem rather than a performance problem.
Ovadia and colleagues studied predictive uncertainty under dataset shift in machine learning. Their results concern models, not human intuition. The relevant caution for this proposed audit is that performance under one distribution should not be assumed to remain unchanged under another.
Do not infer neurological, psychological or medical causes from a research log. The archive is unlikely to identify them. Keep the conclusion at the level the evidence supports: confidence was less reliable for this task group under this workflow.
Sources: Ovadia et al. (2019), Can You Trust Your Model's Uncertainty?.
Use drift as a reason to narrow the next brief
If the existing record suggests poorer calibration in an unfamiliar domain, the practical response may be a more conservative shortlist, a stronger ordinary baseline or an explicit abstention option. It is not necessarily to abandon cross-disciplinary inquiry.
For a consulting discussion, a non-confidential summary of the task groups, timestamps and outcome definitions can establish whether a meaningful audit is possible. A scoped review can assess comparability before anyone pays for a sophisticated model of an inconsistent archive.
The practical takeaway is to recalibrate the claim as the work changes. Confidence should earn its meaning from relevant outcomes, and a historical success rate should never travel into a new scientific domain without its limitations.
Questions this raises
Does a change in confidence accuracy diagnose a health problem?
No. This analysis concerns prediction records and task conditions. It cannot establish a medical or psychological diagnosis.
Can two small batches prove calibration drift?
They can flag a question worth examining, but small counts and changing task composition limit the strength of any conclusion.
Sources and their limits
- Gneiting and Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation. Proper scoring rules assess probabilistic predictions against observed outcomes.
- Ovadia et al. (2019), Can You Trust Your Model's Uncertainty?. The study evaluates predictive accuracy and calibration under dataset shift in machine learning, not human intuition.
Prepared with AI assistance. The linked sources support the specified technical points; they do not validate applied psionics as a whole or guarantee a result for a client.
Read the editorial and evidence standard.
Continue reading
Explore Scientific Oracle consultingfor a scoped review of an existing-data research decision. Start with a non-confidential outline of the question, available evidence and the decision it needs to inform.