Scientific Discovery

Build a Psionics Counterexample Library That Makes the Next Idea Better

2026-09-10

A counterexample library preserves failed intuitive hypotheses together with the conditions, analysis and ordinary explanation that defeated them. Organize entries by failure mechanism, not by embarrassment. Before funding a similar idea, check whether it repeats a known error. This can improve search discipline without claiming that learning from past failures proves the intuitive method's effectiveness.

A failure is useful only if its structure survives

A folder called rejected ideas is not enough. Without the original prediction and the reason it failed, later readers cannot tell whether an idea was contradicted, technically untestable or simply abandoned. Those outcomes should lead to different decisions.

In Andrei Ursachi's exploratory psionics framework, an intuition may supply a candidate connection. A counterexample library records where the translated connection did not survive an external check. It is a method-development asset, not a museum of selectively reinterpreted successes.

Store the candidate, its frozen test, the result and the narrow lesson. Avoid conclusions such as intuition does not work in physics after one failed physics question. Equally, avoid declaring the failure irrelevant simply because a different intuitive story can now be told.

A hypothetical apparent periodicity that vanishes

Imagine a candidate claiming that public equipment logs contain a periodic cycle. The pattern looks compelling after an intuitive cue directs attention to a particular interval. On review, the same cycle appears in the data-collection heartbeat and disappears when missing observations are handled correctly.

The useful counterexample is not merely periodicity failed. It is that the acquisition schedule produced an apparent signal at the nominated period. Save the original analysis and the corrected version, including the sampling pattern and the diagnostic plot specification.

A future idea about periodicity can then trigger a precise warning: inspect the collection schedule before interpreting a spectral peak. The library has converted one failure into a cheaper first check, without pretending the original prediction was correct.

Use failure categories that change the next action

Some failures concern the hypothesis, others concern the test. A simulation that never represented the proposed mechanism cannot decisively reject that mechanism. A clean test that returns the opposite predicted sign is a different kind of result.

Create a small taxonomy with categories such as data artifact, semantic ambiguity, ordinary baseline explains result, unavailable discriminator and reproducible contradiction. Keep a free-text field for details so that categories do not flatten important differences.

Kapoor and Narayanan document how information leakage can inflate scientific prediction performance. Leakage deserves its own category because the response is usually to repair the evaluation separation and reevaluate, not to argue more confidently for the original story.

  • Data artifact: inspect provenance and preprocessing.
  • Ambiguous claim: fix the interpretation before another test.
  • Baseline explanation: retain the simpler account unless new evidence discriminates.
  • Unavailable discriminator: label the question unresolved.
  • Contradicted prediction: preserve the adverse result and limit the claim.

Sources: Kapoor and Narayanan (2023), Leakage and the reproducibility crisis in machine-learning-based science.

Search the library without turning it into another bias

Before a new analysis, retrieve similar failures by observable, dataset structure and proposed mechanism. Similar language alone is insufficient. A repeated word such as coherence may refer to entirely different quantities in two fields.

Do not use the library to remove difficult cases from a final evaluation after seeing their outcomes. Improvements inspired by previous failures belong to development. The resulting procedure must face a separate, frozen evaluation if performance is being claimed.

Simmons and colleagues show why undisclosed flexibility in analysis and reporting is dangerous. The corresponding safeguard here is to disclose which historical failures changed the method and to preserve the original versions they replaced.

Sources: Simmons, Nelson and Simonsohn (2011), False-Positive Psychology.

Make the next research brief more discriminating

A useful brief can include a short account of routes already tried, what specifically failed and which observations remain unexplained. This is often more valuable than another broad literature summary because it identifies the exact alternatives a new hypothesis must beat.

Share a non-confidential description first. A scoped computational review can turn existing failure records into a shortlist of decisive checks and, when warranted, a proposal for deeper investigation. It may also conclude that the attractive new idea repeats an already understood artifact.

The practical takeaway is to treat counterexamples as constraints on the search space. Fast progress can mean recognizing a familiar dead end early, not producing a larger volume of novel-sounding ideas.

Questions this raises

Should failed ideas be published with private client details?

No. Preserve the evidence privately under agreed handling rules. Public examples should be authorized, anonymized appropriately or clearly hypothetical.

Does a counterexample always kill a theory?

It constrains the specific claim tested. Broader conclusions depend on whether the test represented the theory and whether its assumptions held.

Sources and their limits

Prepared with AI assistance. The linked sources support the specified technical points; they do not validate applied psionics as a whole or guarantee a result for a client.

Read the editorial and evidence standard.

Continue reading

Explore Scientific Oracle consultingfor a scoped review of an existing-data research decision. Start with a non-confidential outline of the question, available evidence and the decision it needs to inform.