I repurposed Karpathy’s autoresearch idea for scientific machine learning
It’s been a while since I last posted. But this post took me about 16 hours to put together. And I’m not counting the more than 80 hours I spent during spring break experimenting with agentic patterns… You can find my git repo to replicate this work here
Coding agents like Codex and Claude Code can already accelerate research and development.
However, the default way of using them requires a human supervisor to constantly review and nudge the agent to keep it on the right track. We have run many paper replication experiments, as well as more targeted scientific machine learning studies (e.g., building a computationally inexpensive surrogate for an expensive PDE solver). A recurring problem is that, over time, the agent’s context drifts. What was in the prompt at the beginning of the run tends to be forgotten; the agent stops, asks a stupid question, declares victory after a small improvement, or just takes shortcuts.
The idea of a harness
A harness is an infrastructure around the agentic loop that can endow an agent with long-term memory and keep it on track. OpenClaw’s harness is exactly one of the reasons behind its success. There are two ways you can build a harness: prompt injection and code scaffolding.
Harness from prompt injection
The first time I heard about a harness based on prompt injection was Ryan Lopopolo’s post Harness engineering: leveraging Codex in an agent-first world.
The idea is straightforward: you create a Markdown file with instructions the agent must follow no matter what, and you inject it into the prompt at every step of the loop.
With Codex, this can be done by using the AGENTS.md file (Claude Code uses CLAUDE.md). The file AGENTS.md is a simple text file that you put in a work directory. Once Codex enters that directory, it automatically reads the file. Then it keeps it in its context. If the context is compacted, it may read it again. The idea is that Codex tries to satisfy what is in both the prompt and the AGENTS.md (if possible).
So, what do I put in this file? For our workflows, I typically give them some basic rules to follow that aim to enforce the scientific method, set up a simple memory system, and guide when they can consider a task done.
It looks like this:
# AGENTS.md
Build for correctness, reproducibility, traceability, and continuity.
## Rules:
- Don’t guess silently. Mark assumptions explicitly.
- Inspect relevant files before changing code.
- For nontrivial work, make a short plan first.
- Prefer small, focused changes over broad refactors.
- Write artifact-backed outputs, not just prose.
- Validate before claiming success.
- If blocked, say exactly why.
## Memory:
- Do not rely on conversational memory across sessions.
- Write important decisions, lessons, pitfalls, and next steps to repo-local notes.
- Prefer short dated notes in `notes/memory.md`, `notes/runs.md`, and `notes/todo.md`.
## A task is done only when:
- the requested artifact/code exists,
- the main checks were run,
- assumptions and limitations are stated,
- and outputs are easy to trace and replay.
When I use Codex to build a big system, with lots of files, and so much code that it doesn’t fit in the context, I try to do two things. First, I explain how it can organize memory hierarchically so it can find things quickly.
I may write something like this:
## Memory:
- Do not rely on conversational memory across sessions.
- Write important decisions, lessons, pitfalls, and next steps to repo-local notes.
- Prefer short dated notes in `notes/memory.md`, and `notes/todo.md`.
- Use a hierarchical structure to organize the notes. For example, you can have a
`notes/` directory with subdirectories for different topics, and within each
subdirectory, you can have dated markdown files that capture the relevant information.
Second, I may enforce some basic systems engineering principles, e.g.,
## Rules:
- Use `design/` for high-level design documents and architecture decisions.
Organize design hierarchically in system, subsystem, etc. Each system has
requirements, conceptual design which decomposes into subsystems.
- Use `src/` for implementation code.
- Use `tests/` for test cases and validation scripts.
- A subsystem is complete when it has a design document, an implementation,
and a test that validates the implementation against the design.
- A system is complete when all its subsystems are complete and it has an
integration test that validates the system against the requirements.
- Always rerun the relevant tests after making changes to ensure that the
system remains correct and that you have not introduced regressions.
That said, I’m not entirely sure that this is indeed the best way to design a complex system, but that’s another discussion.
