Self-Optimizing Harnesses: Searching the Harness, Not Just the Prompt

When a language model gives a weak answer, the reflex is to blame the model or rewrite the prompt. But most of what determines whether a model succeeds in a real system is not the model at all. It is the harness: the ordinary code that decides what the model is shown, what it can retrieve, which tools it may call, and how its output is checked. That harness is almost always built by hand and tuned by intuition. A newer idea flips this around: treat the harness itself as something a search process can improve automatically, the same way we already tune models and prompts.

The harness, not the prompt, is where the leverage is

A large language model (LLM) is a system trained to predict and generate text. On its own it does not know your documents, cannot call your tools, and keeps no memory between turns. The harness is the surrounding software that supplies all of that: it assembles the context the model sees, retrieves supporting evidence, selects and invokes tools, and validates the result before anything acts on it.

Because the harness decides what information reaches the model and what happens to its output, small harness choices often move accuracy, cost, and latency more than swapping in a bigger model does. Yet harnesses are typically hand-written and hand-tuned, one judgment call at a time, which means their quality depends on how much time an engineer happened to spend.

📌

Harness. The code around a model that prepares its input, gives it tools and retrieval, and checks its output; the model is one component inside it.

A harness is really a search space

Once you list the decisions a harness makes, it stops looking like fixed plumbing and starts looking like a space of options. What context to include and what to leave out. How to retrieve evidence and how much. Which tools to expose and in what order. How to structure the control flow, and what to keep in memory across steps.

Each of those knobs has many settings, and the settings interact. That is the definition of a search space: a large set of candidate configurations where the best combination is not obvious and cannot be found by changing one thing at a time. Seeing the harness this way is the first step to improving it systematically instead of by intuition.

The harness is a set of interacting design choices, which together form a large space to search.

Turning hand-tuning into an outer loop

If the harness is a search space, then improving it can be automated as an outer loop that runs around the normal system. The loop proposes a change to the harness, runs the changed harness on real tasks, scores how well it did, and uses what it learned to propose the next change. Repeat, and the harness gets better without a person editing it by hand each time.

This is the same shape as other optimization we already accept: we tune model weights against a training signal and tune prompts against examples. A self-optimizing harness simply applies that discipline one level up, to the code that surrounds the model.

An automated loop proposes a harness change, runs it, scores it, and feeds the result into the next proposal.

The scorer: evaluations as the objective

An optimization loop is only as good as the number it optimizes. For a harness, that number comes from evaluation, meaning a repeatable way to run the system on a fixed set of tasks and measure the outcome. Public evaluation frameworks such as EleutherAI's Language Model Evaluation Harness already do this for models, running many tasks with configurable prompts, scoring, and standard benchmarks.

The same idea provides the objective for harness search: give each candidate harness the same tasks, measure quality with an evaluation, and prefer the candidate that scores higher. A weak or narrow evaluation will reward the wrong thing, so the evaluation set deserves as much care as the search itself.

⚠️

Garbage objective, garbage result. A harness search only optimizes what the evaluation measures, so a shallow or biased eval will produce a harness that games it.

Learning from every past attempt

The strongest recent result on this idea comes from research on end-to-end harness optimization. The reported system runs an outer loop that searches over harness code, and its key move is giving the proposer rich access to prior experience: the source code, the scores, and the full execution traces of every earlier candidate, rather than a compressed summary of feedback.

That detail matters. Earlier text-based optimizers tended to squeeze feedback down to a short note, which throws away exactly the information a proposer needs to understand why a candidate failed. Keeping the full history of what was tried, and what happened, is what lets the search make informed changes instead of guessing.

What the results suggest

In the reported experiments, an automatically discovered harness beat a strong hand-built baseline on online text classification by 7.7 points while using four times fewer context tokens, meaning better accuracy at lower cost. A single discovered harness also raised accuracy on 200 competition-level math problems by 4.7 points on average across five models it had not been tuned for, and discovered harnesses surpassed the best hand-engineered baselines on an agentic coding benchmark.

The through-line is not that any single number is large. It is that a search process, given enough access to prior experience, can find harnesses that humans did not, across very different kinds of tasks. That points to where a durable advantage may come from next: not only from the most capable model, but from better, partly automated engineering of the system around it.

Across different task types, an automatically discovered harness matched or beat carefully hand-built baselines.

Keeping the search inside the guardrails

A search that optimizes only for a score will happily do dangerous things to raise it. Left unconstrained, harness search could widen a tool's permissions, pull in more private context than a task needs, or quietly remove an output check, because any of those might nudge the number up. In an agent that can take real actions, that is a security problem, not just a quality one.

The fix is to make safety a hard constraint on the search rather than part of the score. Every candidate harness must pass fixed guardrails first, such as least-privilege tool scopes, context boundaries, and mandatory output validation, and only candidates that keep those guardrails are eligible to be scored. A higher score never buys an exemption. Self-optimization then improves the harness within a fence the search is not allowed to move.

A candidate harness is only allowed into scoring if it preserves every safety guardrail.

Key takeaways

The harness, the code that feeds, tools, and checks a model, often decides success more than the model choice does.
Listing a harness's decisions reveals a large search space, not fixed plumbing.
Harness improvement can be an outer loop: propose a change, run it, score it, learn, repeat.
Evaluations supply the objective, so a weak eval yields a harness that games it.
Keeping full traces and scores of past candidates, not compressed feedback, is what makes the search informed.
For agents that act, safety must be a hard constraint on the search, never something a higher score can override.

Glossary

Harness: The software around a model that prepares its input, gives it retrieval and tools, and validates its output.
Large language model (LLM): A system trained to predict and generate text, used here as one component inside a harness.
Search space: The full set of possible configurations of a system, where the best combination must be found rather than guessed.
Evaluation (eval): A repeatable way to run a system on fixed tasks and measure the quality of its results.
Execution trace: A recorded step-by-step account of what a run actually did, used to understand why it succeeded or failed.
Guardrail: A fixed safety constraint, such as a tool permission limit or a required output check, that a system must always respect.