Research

The Intuition Base-Rate Trap: Why a Strong Signal Can Still Be Mostly Wrong

2026-09-10

An intuitive alert must be interpreted against how often the target event occurs and how frequently the method raises false alarms. Even a sensitive method can produce mostly false alerts when the event is rare. Use counts, not just a success percentage, and compare the cost of checking an alert with the cost of missing a real case.

The missing number in a persuasive claim

Someone says a method catches most important anomalies. Before treating that as useful, ask how many ordinary cases it also flags. Sensitivity describes the fraction of real targets detected. It does not tell a buyer how often an alert is correct.

This matters for exploratory psionics because unusual, vivid impressions may feel especially informative. Andrei's framework can use those impressions to propose checks, but their practical value still depends on the actual pattern of true and false alerts in a defined task.

Start with the target event in plain language. A corrupted timestamp, a mislabeled unit and a scientifically important anomaly are different targets. Changing the definition after inspection changes the base rate and invalidates a comparison made under the original definition.

A hypothetical archive of 10,000 records

Suppose an archive contains 10,000 sensor records and exactly 100 have a particular verified labeling defect. Imagine a screening rule detects 80 of those defects and incorrectly flags 5% of the remaining 9,900 records. These are invented numbers for illustration, not performance claims.

The rule produces 80 true alerts and 495 false alerts. Among the 575 alerts, approximately 13.9% correspond to the defect. An 80% detection rate therefore coexists with an alert queue that is mostly wrong. Both statements can be true at once.

Gigerenzer and Hoffrage studied how frequency formats can help people reason about Bayesian problems. Counting records this way is useful because the denominators stay visible. It prevents the persuasive number, eighty percent, from replacing the decision-relevant number.

Sources: Gigerenzer and Hoffrage (1995), How to Improve Bayesian Reasoning Without Instruction.

A low-precision alert can still have a bounded use

If an automated metadata check costs almost nothing, a queue with many false alerts might still save effort compared with manually examining every record. If each alert triggers weeks of specialist work, the same queue may be commercially unacceptable.

Suppose each verification takes two minutes. Reviewing 575 alerts would take roughly nineteen hours, compared with roughly 333 hours for all 10,000 records. That arithmetic describes workload only. It does not price the twenty missed defects or establish that the screening rule is appropriate.

The buyer needs a cost model with both error types. A research triage system can be useful without being a truth oracle, provided its limited role is explicit and its downstream verification is real.

Keep confidence separate from the operating threshold

An analyst's subjective confidence is not automatically a calibrated probability. To evaluate probability statements, compare them with recorded outcomes using a stated scoring rule. Gneiting and Raftery provide the statistical foundation for proper scoring of probabilistic forecasts.

A threshold then determines which cases become alerts. Raising it may reduce workload while missing more targets. Choose that trade-off from the intended use and a separate development set, not by searching the final evaluation data for the most flattering chart.

Report the threshold, target prevalence, precision, sensitivity and number of evaluated cases together. If the task changes from common defects to rare discoveries, the old alert precision should not be assumed to transfer.

Sources: Gneiting and Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation.

Specify what the next review is buying

For a computational consulting brief, describe the candidate event, its approximate frequency and the consequence of a false alarm. If the frequency is unknown, say so. Estimating a credible range may be the first useful piece of analysis.

A suitable initial review can ask whether an unconventional screening idea deserves a bounded comparison on existing labeled records. The outcome may be a useful queue, a narrower application or a decision not to proceed. It should not promise that every intuitive alert corresponds to a discovery.

The practical takeaway is that rarity changes the meaning of accuracy. An alert becomes valuable when it improves a real decision under transparent costs, not merely when it sounds unusually confident.

Questions this raises

Is an alert with low precision always useless?

No. It may help when verification is cheap and missed cases are acceptable, but the costs and limits must be explicit.

Does this numerical example describe Andrei's performance?

No. It is a hypothetical calculation showing how prevalence and false alarms affect any proposed screening method.

Sources and their limits

• Gigerenzer and Hoffrage (1995), How to Improve Bayesian Reasoning Without Instruction. Frequency representations can make Bayesian reasoning more accessible; the numerical example here is original and hypothetical.
• Gneiting and Raftery (2007), Strictly Proper Scoring Rules, Prediction, and Estimation. Proper scoring rules assess probabilistic predictions against observed outcomes.

Prepared with AI assistance. The linked sources support the specified technical points; they do not validate applied psionics as a whole or guarantee a result for a client.

Read the editorial and evidence standard.

Continue reading

• Why subjective confidence needs calibration
• Selecting a useful scientific triage question
• Explore the topic library

Explore Scientific Oracle consultingfor a scoped review of an existing-data research decision. Start with a non-confidential outline of the question, available evidence and the decision it needs to inform.