How to Build a Self-Evolving Coding Harness

The harness speaks several model dialects

In the early public API era, model access felt almost suspiciously simple. OpenAI's first Completions API arrived in June 2020: send a freeform prompt and receive text. In March 2023, Chat Completions made the conversation structural. Instructions, user input, and assistant output became messages with roles instead of pieces hand-spliced into one giant string. Function calling followed that June, giving the client a JSON-shaped way to connect a model to external actions.

In March 2025, OpenAI introduced the Responses API. Its item-based design could represent messages, function calls, function results, and built-in tools inside a more agent-shaped interface. In April 2026, OpenAI added WebSocket mode for Responses. The request stayed familiar, but a tool-heavy coding loop could keep one connection open and reuse recent response state instead of rebuilding everything after every turn.

That history is a useful miniature of harness engineering. The model became easier to reach, then the conversation became structured, then tools became first-class, then the transport adapted to long agent loops. Each improvement gave the harness a better primitive. Each new wire format also became something the harness had to speak correctly.

A real harness may talk to several providers at once. Many expose OpenAI-compatible chat endpoints. Anthropic uses its own Messages shape. Some return tool calls as content blocks, some as role-bearing messages, and Responses uses typed items and events. Authentication headers, context limits, stop reasons, retry hints, usage records, and streaming fragments vary too.

The model may be asking for the same thing on every provider: “please run this test.”

The bytes do not agree.

My multi-provider harness, CodeGraff, makes that translation embarrassingly concrete. Its provider layer currently distinguishes anthropic, openai, and responses wire kinds. The same tool result becomes an Anthropic tool_result content block, an OpenAI tool role message, or a Responses function_call_output item. Kimi can even switch protocol and authentication style according to the live model catalog. The adapter is not glamorous, but a mistake there can make a perfectly capable model appear forgetful, mute, or bizarre. Tiny protocol details become visible as personality. The implementation is in the provider table and message normalizer.

Context is a decision

Once a provider can hear us, the harness still has to decide what to say. A repository is too large, too redundant, and too alive to paste blindly into every turn. The harness chooses instructions, files, symbols, diffs, test output, and earlier observations under a token budget. A larger context window only gives it a larger room to make messy.

This is why retrieval is not an accessory. Show the model the wrong file and every later action can be locally sensible and globally useless. Show it the definition, callers, and relevant tests at the right moment and the same weights take a different path.

Tools are the action language

A model does not execute grep, edit a file, or run a test merely because it can describe those actions. The harness exposes a grammar of things that may actually happen. Tool names, descriptions, input schemas, permissions, timeouts, and result shapes determine what the agent can express and what the runtime will permit.

Make every tool return fifty pages of vaguely relevant text and the model learns patience. Make the contract narrow and the observation structurally useful and suddenly the next action becomes easier to choose.

State turns calls into a workflow

The loop is simple to draw: send context, receive an action, execute it, return the observation, repeat. Keeping it healthy is less simple. The harness has to preserve call identifiers, partial tool arguments, plans, failures, and enough recent evidence to continue after compaction or interruption. It must also know what can be thrown out and how to recover the rest.

This is where the difference between a transcript and a trajectory begins. A transcript helps continue this conversation. A trajectory records enough about the path, environment, cost, and outcome to compare this run with another one later.

Evaluation gives the loop an ending

Imagine the request is “fix the flaky cache test.” The harness assembles the relevant test and symbols, validates each tool call, applies the edit, runs the tests, and loops when a new failure appears. Then a separate evaluator checks the result against the task's tests, permissions, and budget. The model can propose “done.” The harness decides whether “done” is true.

How I ended up here

I did not begin with a grand plan to build a self-evolving intelligence. I was making odd little coding tools for fun and following whatever made the loop faster, cheaper, or easier to inspect.

The evolutionary part did not come out of nowhere. I have been fascinated by evolutionary algorithms for years, especially the possibility that an archive can preserve odd, temporarily weaker stepping stones that a greedy optimizer would throw away. Lineages, novelty, selection, and the occasional magnificent failure are a wonderfully strange way to make progress.

I was lucky to contribute to OMNI-EPIC and Automated Design of Agentic Systems, or ADAS, through the discussions and feedback we traded as friends exploring the same questions. These were not distant papers I discovered later. I got to watch the ideas take shape, argue about them, and become even more obsessed with the strange little evolutionary systems I was already building for fun.

It is also why I am amazed by what the team at Recursive is doing now. Their automated research system proposes ideas, implements and validates experiments, runs many long-horizon threads, preserves useful context, combines promising branches, and checks for reward hacks and variance before calling something progress. Seeing that rhythm applied to model training and GPU kernel optimization feels like watching a long-running fascination become a real research engine.

At AI Engineer Singapore, I tried to tell this story from the bottom up: first make inference cheap, then make context cheap, then preserve the trajectory, then let several agents try, and only then ask what should survive. By the CodeGraff slide I had reduced the whole thing to one slightly aggressive sentence: “The harness IS the fitness function.” More precisely, the harness defines the process for variation, measurement, and promotion; its evaluator supplies the fitness signal.

The recording shows the work in its natural habitat: a trajectory store, parallel agent fan-out, and a selector fed by DevSwarm telemetry. It was not a polished theory. It was me showing the strange machinery I had managed to make run.

I built each part as a separate toy: CodeDB for cheap repository sensing, DevSwarm for comparing parallel runs, CodeGraff2 for carrying personas and trajectories across processes, and the current Zig CodeGraff for selection and promotion. The names matter less than the order of the questions.

When I read Lilian Weng's “Harness Engineering for Self-Improvement”, it clicked. She had a much cleaner vocabulary and a much broader research map for this pile of implementation experiments. I borrow and cite that map throughout the post. What follows is a lab notebook from someone who kept giving agents strange tools, watching what they did, and saving the funny failures because they were often the useful part.

The rest of the post follows the same route as the talk: first the economics of a loop, then context, then trajectories, and only then the question of what a later run should inherit.

The six demos in the talk were one loop-economics story: make inference and repository sensing cheap, preserve the trajectory, then spend the saved budget on selection. But throughput only helps after the objective is credible. A fast loop with a flattering judge simply explores the wrong mountain more efficiently.

I was also influenced by Yichao “Peak” Ji's Manus post, “Context Engineering for AI Agents”. Its filesystem section describes a form of restorable compaction: remove bulky content from the active context, but keep the URL, path, or other handle needed to recover it later. That is a much better mental model than “summarize until it fits.” Compaction should make evidence cheap to revisit, not erase where it came from.

That Smithery collaboration also shaped the implementation. Thanks, Henry, for exploring where the server ends and the harness begins, and for running far too many interface benchmarks with me. Those conversations are a large part of why CodeDB became MCP-based: MCP gives the harness a structured retrieval surface it can search, budget, and compose instead of forcing every agent through raw shell output.

The current context composer is intentionally fussy. It starts with at most five identifiers or concepts, admits a bounded number of ranked results for each, keeps only the strongest files and lines, and inlines a few short symbol definitions. When the symbol set is small, at most three definitions, it can augment them with callers, callees, and nearby tests while they still fit. When the budget is tight it falls back to a leaner representation and says what it omitted. The exact caps are implementation details. The important idea is that a context budget should force prioritization, not silent truncation.

This matters even more in a swarm. One worker rummaging through the wrong files wastes one context window. Five workers repeating the same rummage turn bad retrieval into a budget multiplier.

Together, structural retrieval, durable use traces, and executable ranking benchmarks change the harness in three ways.

1. Retrieval becomes part of policy

If a tool returns the definition, its call sites, and the relevant tests in one bounded result, the model follows a different trajectory than it would after fifty grep hits. The context composer is therefore not passive storage. It is code that chooses which evidence can affect the next action.

CodeGraff makes that policy explicit at the tool boundary. When a CodeDB index exists for a concrete source file, its codedbGuard steers broad grep, sed, and whole-file reads toward structural CodeDB operations first. It still allows non-source files, globs, and unindexed paths. The point is not to ban shell tools. It is to make the cheap, typed, structurally informed path the default before an agent reaches for the fire hose.

2. Use becomes feedback

CodeDB's feedback is narrower than that phrase may suggest. For fuzzy filename lookup, codedb_find records a query and a read or outline that follows within five seconds. That durable trace can boost the opened file in later fuzzy filename results. Separately, restart warm-up replays frequent codedb_search queries to prefill caches; it does not change ranking. This is intentionally modest: neither mechanism trains a model. But the first demonstrates an important shape:

The harness can adapt from behavior without touching a model weight. Whether that adaptation improves retrieval still has to be measured.

3. Failure becomes a first-class artifact

The more interesting lesson came from the ranking experiments. I built a reproducible fitness loop from git history: use a commit subject as a query, treat the changed file as gold, and score where that file appears using mean reciprocal rank (MRR).

Four plausible single-signal changes, including graph distance, two centrality variants, and a file-size prior, failed to improve their target benchmarks. The centrality variants and size prior were flat; graph distance regressed aggregate MRR from 0.125 to 0.119 without lifting any of the buried-gold ranks. I kept those null results in experiments/ranking/failed.md.

One signal derived from engram, my learned-reranking experiment, did improve a small exploratory SWE-lite retrieval slice from 0.833 to 1.000 MRR. Its LexFreqPenalty downweights registries and changelogs saturated with the query term, allowing the eponymous implementation file to surface. That small result is a useful lead, not a general claim about retrieval quality.

The repository keeps the evaluator executable rather than burying it in prose. Build CodeDB, replay the git-derived query set, and compare the ranking score.

One implementation detail matters here: CodeDB is CodeGraff's token-efficient index and navigation layer. CodeGraff does not maintain a second secret index. It exposes CodeDB as a harness tool and steers indexed source exploration through structural slices before raw grep or whole-file reads. CodeGraff is the larger runtime that composes that layer with models, tools, subagents, budgets, scoring, and learned policy. The CodeGraff README and its codedbGuard make that relationship explicit.

The first concrete version of that idea in DevSwarm was a small MAP-Elites-style archive. Each role had an 8 by 8 grid. One axis represented token efficiency and the other thoroughness. A cell accepted a new prompt variant only when its composite fitness was strictly better than the current occupant. The score was 50 percent task success, 20 percent token efficiency, 15 percent speed, and 15 percent low error rate. Selection used a softmax across occupied cells, so the system could revisit several different behavioral niches instead of cloning the current global winner forever.

There is an important historical limit: DevSwarm implemented the archive, fitness calculation, persistence, and selection foundation. Its source did not yet contain the autonomous mutation operator imagined in the design notes, and the evolver was not the fully closed production loop. That is still useful. It taught me that before an agent can “evolve,” the harness needs a place to put variation and a fair receipt for what happened.

CodeGraff2 made the receipt more literal. Its trajectory schema records structured start and end events. AgentRun binds identity, model, and version; AgentRunEnd records success, turns, tokens, tool calls, errors, and wall time. Fitness is derived afterward instead of being trusted as part of the agent's own event. Its experimental darwincode protocol works across two process runs:

1. Pass A invents two to five read-only persona files and exits.
2. The next process reloads those personas from the registry.
3. Pass B fans work out across them, records the resulting trajectories, and synthesizes the result.
4. A separate workflow scorer derives per-persona fitness from that trace evidence.
5. The persona body hash acts as its fitness identity. _archive.jsonl preserves birth and lineage rows, while the optional backend archive carries mean_reward, child counts, and selection probability.

The parent sampler favors strong personas but discounts prolific families with a child-count term. Low-scoring variants can remain as stepping stones rather than disappearing from history. Again, this is best understood as a runnable harness protocol, not a magical daemon endlessly rewriting itself in the background.

OMNI-EPIC and ADAS were two earlier pieces in the published research lineage that preceded the Darwin Gödel Machine, but in different ways. OMNI-EPIC used a growing archive of learned and failed tasks to propose the next interesting, learnable environment in code. ADAS used a fixed meta-agent to write, evaluate, and archive new agents in code.

The DGM paper cites OMNI-EPIC as part of its open-ended archive and stepping-stone lineage, and it explicitly contrasts its self-referential loop with ADAS's fixed meta-agent. DGM then makes the coding agent modify its own implementation. That is a much clearer historical path than pretending self-evolving harnesses arrived fully formed. I was close enough to those earlier conversations to become even more curious, then went off and built my own smaller, weirder versions for fun.

In her section on harness optimization, Weng describes a useful progression in the object being optimized:

instruction → structured context → workflow → harness code → optimizer code

The farther right we move, the more important the outer constraints become. A prompt optimizer can overfit a prompt. A harness optimizer can alter the process that produces and judges an answer. An optimizer optimizer can alter the search itself. Each step increases the design space and the number of ways a metric can lie.

The talk's biology shorthand

In the talk I used an intentionally over-literal analogy. The model is the seed. Compute is the sunlight. A task and its repository are the environment. The trajectory records what the organism did. The harness supplies the selection pressure.

The simplest picture of model learning is next-word prediction. Show a model many sentences such as “the cat sat on the mat,” “the bird sat in the tree,” and “the dog slept on the rug.” Training changes the model's internal weights. Later, when it sees “the cat sat on the ...”, words such as “mat” receive more probability than unrelated words. Context can make “sofa” or “tree” plausible too. The model is learning a distribution over what could come next.

The team can learn too

The learned route policy is intentionally conservative. Route resolution respects a precedence chain: an explicit user pin wins, then a persona-specific choice, then a learned phase policy, then ordinary workflow and session defaults, and finally the automatic ladder. An unpinned image task can also take the capability path to a vision-capable model. For a learned reroute, both the ladder baseline and a challenger need at least two observations in their own (shape, role, tier, model) cells. The learner first looks for exact workflow-shape and role evidence. If that is sparse, it may use the wider workflow-shape pool. Silence is a valid result.

Before UltraCode has enough local trajectories, it is not starting from a blank table. CodeGraff first looks for a project benchmark sheet, then a personal one, then falls back to a shipped DeepSWE score-and-cost snapshot. It folds those model and effort measurements into a provider-local Pareto front. The frontier rung is the capability ceiling. The small rung is the score-per-dollar knee below it. A mid rung exists only when the measurements contain a real survivor between those two. If another model matches or beats one on quality for strictly less money, the dominated model is removed before anyone is seated. The first automatic worker usually starts one rung below the root on this measured ladder. The exact derivation lives in the benchmark priors.

That benchmark sheet is the router's initialization, not its verdict. The analogy to initial model weights is useful at the harness level: it gives UltraCode a starting bias about which model should occupy which role. It does not change any model's neural weights. The current sheet score enters a route cell with the weight of three lived runs, while cost stays anchored to the benchmark instrument.

Trajectory evidence touches that policy at two damped layers. One recipe-linked outcome can nudge the global sheet and rederive its provider ladders. Then, for a particular workflow shape and role, the current sheet score acts as a pseudo-count of three and is averaged with the local cell's outcomes. The role-specific router waits until both the ladder baseline and a challenger have at least two observations. A couple of runs can nudge the prior. A season of runs can own it. The challenger replaces the ladder answer only when its blended quality is no worse and its benchmark-anchored effective cost is strictly lower. Otherwise the router keeps the prior or returns no opinion. The update and gate are visible in the route policy and phase-level seating.

That is the part I find most interesting. Model price and relative benchmark performance behave a little like initialization for the team around the models. The benchmark sheet initializes the routing policy. The trajectory archive teaches it where that initialization is wrong for this repository, task shape, and role. This is closer to a cautious compiler optimization than a model popularity contest. Here price means average benchmark dollars per task. The separate orchestration learner below uses observed model-call count and p90 budget instead, because deciding which model fills a seat and deciding how much team to buy are different control problems.

The orchestration policy learns a different decision. Its arms are roughly: R0 solo, R0d solo with an evidence retry, R1 with a scout, R2 with a fleet, and R3 with a fleet plus judges. Evidence is stratified by task class, budget band, and resolved-model group because a swarm that makes sense for an unlimited refactor may be absurd inside a 15-call bug fix.

This learner waits until both the baseline arm and a challenger have at least three observations inside the same task-class, budget-band, and model-stratum cell. Trading up requires a meaningful quality gain, currently 0.05 in mean quality, and enough reserve for the challenger's observed p90 call use. Trading down is allowed only when the cheaper arm is not worse. One compiled policy example is wonderfully unromantic: under a 30-call bug-fix budget, solo scored 1.0 in 10 calls, while a fleet scored 0 and exhausted all 30. The same family of fleet behavior could still be useful in the unlimited stratum, where it had room to spend 48 calls and score 0.8. “Use more agents” is not a policy. “Use more agents when their measured upside fits this budget” is.

The early CodeGraff experiment made that loop concrete with a small Darwin Gödel Machine-style driver. Calling the system prompt a genome is an analogy: it is a text policy that can be copied, changed slightly, run, and scored. Its replay suite is the primary gate. Deterministic failures remain below 0.9, while full passers are ordered by tool-call efficiency. The script prints cost for inspection, but cost is not part of that score, and this early demo does not perform parent-relative regression testing.

Parent selection follows one plain rule: favor prompts that scored well, but reduce the chance of choosing a family that has already produced many children. That keeps one successful lineage from consuming the whole search budget. It encourages exploration across lineages; it is not a measure of semantic novelty. Each iteration records the prompt fingerprint, child tool sequence, score, and parent lineage. When a score key is configured, the score record is HMAC-signed before it returns to the archive.

The loop also produced a very useful stupid failure. One version could spend three judge calls even when all three runs used the same prompt genome. We paid three judges to discover there was no tournament. The current scorer first requires at least two distinct genomes and refuses to compare variants resolved to different models. Before buying opinions, the harness now checks that there is actually an axis to rank.

The demo is tiny on purpose: pin the outside judge, try three generations, and keep the trace.

The demo is fun because the whole mechanism fits in one small script. It is also deliberately crude. Pool. Score. Mutate. Repeat. Then stare suspiciously at whatever won.

This looser DGM loop is useful for understanding the mechanism, but it is not the system I would trust to rewrite the root policy unattended. For that, CodeGraff now has an experimental, much more conservative loop. The manual path is four steps: initialize, run, inspect the evidence, and promote only an eligible result.

graff learn init can also enable automatic learning, but background trials require separate runtime gates and invoke learn run --auto. Either way, this is prompt-policy hill climbing, not magic recursive self-rewriting. It proposes a handful of children, chooses at most one, and makes that child clear the same gates before it can become the next session's root policy. It does not evolve model weights or arbitrary harness code, and UltraCode's routing and orchestration learners remain separate feedback paths.

The mutator does not ask a model to rewrite the full prompt. It asks for one strict JSON object containing one clause. Local code inserts that clause at a fixed target, counts changed bytes, enforces the total prompt size, and checks that protected substrings remain unchanged.

Incumbent and challenger run on the same cases and seeds in fresh workspaces. Python reports the evidence, but Zig checks the hashes, groups related cases, runs the paired comparisons, and rejects critical regressions.

The tournament ordering is deliberately legible. It compares critical failures first, then critical regressions, pass rate, raw passes, measured tool calls, measured behavior score, cost, genome size, and finally a stable identifier. Correctness gets to veto economy. A cheap challenger does not beat a correct incumbent by failing quickly. The paired primary comparison also accounts for repeated looks through multiplicity correction instead of treating every noisy difference as a discovery.

Then only a primary-eligible winner is evaluated on the holdout. A primary rejection is final, and trying to attach holdout evidence to a loser is itself an error. The hidden assertions and resulting evidence are not fed back into candidate generation. This winner-only rule spends the hidden set once per tournament and makes it harder for the search to learn the holdout by repeated exposure.

If the winner still qualifies, the run is written immutably. Genomes, evidence, runs, and promotion transactions live under .graff/learn as SHA-256-addressed objects. The only mutable policy-activation point is active.json, which names the trusted generation. Checkpoint and automatic-cadence state can also change, but neither can activate a policy. Promotion takes a concurrency guard, appends a transaction to the chain, and replaces the active pointer atomically. Rollback uses the same mechanism. On the next ordinary session, startup loads the promoted prompt as the root policy; an explicit --system-prompt still wins.

That storage design is not glamorous, which is exactly why I like it. The challenger can produce a persuasive answer. The store keeps a receipt that local code can verify later.

There is an important limit to that boundary. It is governance and integrity checking, not an OS sandbox: adapters run with the invoking user's authority and can access whatever that user can. The holdout is withheld by the orchestrator from ordinary challengers, not secret from a malicious or colluding adapter. Stronger isolation requires an external sandbox.

The list is boring on purpose. Most of the value is simply making tampering, ambiguity, and regressions easy to catch.

The tiny-script pattern Weng highlights

Weng uses Andrej Karpathy's autoresearch as a clean example of workflow automation. Three files explain almost the entire loop: program.md supplies the operating policy, train.py is the experiment the agent may change, and prepare.py keeps data preparation and evaluation fixed.

The run has a fixed five-minute budget and one within-platform metric, val_bpb (lower is better); autoresearch explicitly warns against comparing results across different compute platforms. The agent changes train.py, evaluates it, records the outcome, keeps improvements, and discards regressions. The most interesting “script” is program.md: a plain-language program for the outer loop.

This is exactly the pattern worth reusing. The particular training code is not the point. The useful abstraction is a small set of explicit roles: policy, mutable experiment, protected evaluator, durable results, and one pointer to the version currently trusted.

CodeGraff's version is more defensive because it is optimizing the harness that performs real work. But the bones are the same. A file can be a better agent primitive than a baroque framework when every file has one clear authority.

This is also why Weng's filesystem-as-memory pattern resonates with my experience. CodeDB keeps ranking traces and a durable failed.md; CodeGraff keeps immutable run records, rejected mutations, and lineage. That state does not belong in every model context. Store it durably, retrieve the slice needed for the next decision, and do not make “remember everything” a prompt-length problem.

Weak evaluators

Software is unusually friendly because tests and benchmarks often exist. Even so, CodeGraff's replay judge can only reward what its cases and assertions express; research taste, maintainability, architectural fit, and whether a surprising result is interesting remain much harder to encode. A fast, precise evaluator makes evolution work; a fuzzy evaluator turns it into confident hill climbing on the wrong mountain.

Diversity collapse

Purely selecting the current top score produces many cousins of the same idea. The early DGM loop's child-count penalty, DevSwarm's 8 by 8 MAP-Elites-style archive, and current CodeGraff's named niche champions preserve different forms of exploration. CodeGraff's governed root learner is intentionally narrower: it is an active-parent hill climber, not a full population search. That is safer for a default policy, but less open-ended.

Reward hacking

If the evaluator is editable, the fastest “improvement” is often disabling it. If the task suite is visible, the harness can memorize its quirks. If cost is rewarded too early, a cheap failure can beat an expensive success. That is why CodeGraff pins evaluator and suite hashes, verifies them after use, and keeps promotion authority outside the bounded clause mutation.

Long-term repo health

A benchmark can tell me whether today's patch passes. It is much worse at pricing the migration, ownership, compatibility, and debugging burden imposed six months from now. This is especially important when the challenger is a root policy that will shape every later session. Humans need to move up the stack, as Weng argues: less step-by-step steering, more ownership of goals, boundaries, and irreversible decisions.