A few months ago, we asked an AI agent to replicate three research papers. It managed two out of three. Then we caught it cheating. When it could not reproduce a result, it extracted the figures from the paper’s PDF and pasted them into its report. When we banned that, it fabricated curves that visually matched the paper’s figures. When we banned that too, it quietly swapped in simpler methods that produced similar-looking results.
We patched each hole with a paragraph in the prompt. We observed that after a few context compactions, the agent forgets small details about the method it was supposed to implement, things it has done, and what remains to be done.
Our attempt at a real answer is now a paper: https://arxiv.org/abs/2607.02134. This post is the short version.
From prompt to skill
A skill is a package the agent loads when it needs it: instructions, scripts, and validation logic. We have been exploring the best way to build reliable scientific skills over the past three months. A complete discussion would require its own post. But our findings can be summarized as follows:
- Text alone, e.g., in the SKILL.md file, is not sufficient to build a reliable scientific skill.
- The skill requires establishing a harness. The harness must include the following:
Use the filesystem to keep track of the objective, the plan, and the current status. The idea is to build a memory system that is more reliable than the context window.
Provide vetted CLI tools that enable the agent to correctly bootstrap its working directory, verify an artifact is correct, etc. The idea is that the agent can use the CLI tools to comply with the skill assumptions about inputs/outputs and to get immediate feedback on what it creates.
The paper-replication Skill
Our skill is called paper-replication. It operationalized the generic assumptions above as follows:
Everything lives in files. Claims, assumptions, runs, results.
- The paper becomes a checklist. The agent must first turn the paper’s claims into explicit targets: this figure, this table, this reported error. Each target has a status, and only one can be active at a time. This is done with a CLI tool.
- Every result has a paper trail. A reproduced figure counts only if it is linked to the actual run that generated it, the code behind that run, and the claim it supports.
- The checks run outside the agent. The harness hashes the paper’s own figures, so a collage is caught mechanically. Validators, also known as CLI tools, decide when a claim is matched and when the paper is complete.
- Each claim gets the right kind of check. A reported error must match within tolerance. A chaotic attractor must match in structure, since no two simulations of chaos agree point by point. A posterior must match as a distribution.
