AI for Science

Research Checkpoint Manifest: What Can You Replay After the Model Is Gone?

2026-09-10

A replay manifest tells the next reviewer which parts of a research result can be recomputed, which require repeated AI runs, and which can only be inspected from preserved records. Freeze inputs, settings, outputs and scoring before choosing a repeat budget. If the original model becomes unavailable, replaying downstream calculations remains useful, but it is not a rerun of the original AI system.

Start with the handover question, not the upgrade policy

Six months after a milestone, a buyer asks to rerun the analysis. The report names a model that is no longer accessible. Some calculations can still be repeated from saved data; the agent's original response cannot simply be regenerated. A useful handover anticipated this difference before the endpoint disappeared.

This worksheet complements Change Control for Models Used in Scientific Work, linked below. That primer addresses approving changes. Here the narrower task is to label what remains replayable at one completed checkpoint. The following manifest and numerical example are proposed working aids, not a report of a client engagement.

Put three replay modes in the manifest

Assign each output a replay mode and a matching acceptance rule. Exact file equality is appropriate for some deterministic artifacts. Numerical calculations may instead need a justified tolerance. A sampled AI response requires a different record: preserving its original wording does not guarantee the same wording will be generated again.

• Give the dataset, code and retrieved evidence stable version identifiers.
• Keep scoring rules, exclusions and reviewer corrections with the outputs.
• Store restricted records inside the approved environment; use permitted references in the manifest.

Choose the repeat budget before looking for a pleasing run

Imagine an agent extracting twenty numerical statements from permitted scientific papers. Freeze those papers, the extraction instructions and the scoring rubric. For an illustrative stability check, budget five complete runs of the same configuration and retain every response. This budget is a planning example, not a guarantee of adequate statistical power.

Suppose the five runs produce 17, 18, 18, 19 and 18 correctly supported statements. Their mean is 18 out of 20, or 90 percent, and the observed range is 17 to 19. Report the individual counts and which statements failed. Do not select the best run and advertise 95 percent as the system's dependable performance.

The same twenty cases recur across runs, so these are not one hundred independent scientific examples. The exercise describes repeat variability on this frozen set. It does not establish transfer to new papers, disciplines or deployments.

Record the result without letting the tracker certify it

MLflow documents facilities for logging parameters, code versions, metrics and artifacts. Those capabilities can preserve the manifest and repeat records. The interpretation of the scores remains a separate scientific review.

Model Cards research advocates documenting intended use and evaluation conditions. Apply that limited lesson by stating exactly which extraction task the repeat check covered. Keep the original human scoring decisions available, including disagreements. A compact versioned folder can serve this purpose when a larger tracking platform would add overhead.

Sources: MLflow: ML Experiment Tracking; Model Cards for Model Reporting.

Write the retirement note while the evidence is available

If the original deployment becomes inaccessible, mark the AI stage unavailable for execution. You may still rerun the code that converted its saved extraction table into a metric. Label that as downstream replay using preserved AI output, not a fresh reproduction of the whole pipeline.

A replacement model creates a new checkpoint. It can be evaluated on the same permitted cases, but it cannot retroactively supply the missing original execution. Similarly, retained logs can expose an unsupported citation without proving that the original system would produce the same response today.

For the buyer, the deliverable is a plain statement of what can be rerun, what can be inspected and what cannot be recovered. That is more useful than an unqualified reproducibility promise whose meaning changes when a provider changes its product.

Questions this raises

Does a fixed configuration guarantee identical AI wording?

Do not promise that without demonstrating it for the relevant system. Record sampling settings and actual responses, and distinguish observed repeat stability from a guarantee about future execution.

Can a retired model's result still be audited?

Often partly. Preserved sources, responses, code and reviewer records can support inspection and downstream replay. They cannot recreate an unavailable original model execution.

Sources and their limits

• MLflow: ML Experiment Tracking. Supports logging parameters, code versions, metrics and output artifacts.
• Model Cards for Model Reporting. Supports documentation of intended use and evaluation conditions; checkpoint controls are proposed here.

Prepared with AI assistance. The linked sources support the specified technical points; they do not validate applied psionics as a whole or guarantee a result for a client.

Read the editorial and evidence standard.

Continue reading

• Start with the model change-control primer
• Keep the wider scientific decision audit trail
• Explore the topic library

Explore Scientific Oracle consultingfor a scoped review of an existing-data research decision. Start with a non-confidential outline of the question, available evidence and the decision it needs to inform.