Inside HarnessPro: The Prompt Rules That Keep a Self-Evolving Agent Honest (Nov 2026)
TL;DRHarnessPro freezes the model and lets a search loop rewrite the harness around it: instructions, tools, hooks and budgets. The reusable part is the prompt contract. A test-writing agent replaced an LLM judge that caught only 41.7% of failures, and its frozen tests caught 83.3%. Three rules keep the search from grading itself into a better score. The 53% saving is a deployment cost per task on SWE-Bench Pro, not the cost of running the evolution.
By Cameron Voss and West Mao
On this page
- What does HarnessPro actually change?
- Why can't an LLM judge grade the work?
- What are the three rules for writing tests an agent cannot game?
- How is this different from RHO?
- How does one epoch work?
- What do the three instruction templates forbid?
- Does recursion help, or is one epoch enough?
- How much cheaper is it, and against what?
- What does the 53% figure actually measure?
- What breaks when you remove a piece?
- What can a small team copy without running the loop?
- Frequently asked
What does HarnessPro actually change?
HarnessPro starts from a plain split: an agent is a model plus a harness. The model weights stay frozen. The harness is everything around them, including the instruction file, the tools the model can call, and the runtime policy that decides when a run should stop. The seed harness is the stock configuration of Codex CLI or Claude Code, with no code changes.
Each candidate harness is scored on two numbers: pass rate on an evaluation set and mean cost per task, converted from token prices. Latency is recorded but never enters the objective. Pass rate and cost are not folded into one score. That decision is what makes a Pareto frontier possible, and it is why the system returns three operating points instead of one winner.
The paper appeared on arXiv on October 19, 2026 as 2610.26770. It lists nine authors from Tsinghua, Z.AI, Shanghai Jiao Tong University, the University of Warwick and Kuaishou. Zhuoshi Pan is first author and Jie Tang is on the author list. The paper runs 41 pages and is marked under review.
The public repository is MIT licensed and ships nine evolved harness artifacts, one for each benchmark and tier. It does not ship the automatic evolution loop. That distinction matters if you plan to reproduce the results: the repository contains the resulting harnesses, not the search loop that produced them.
Why can't an LLM judge grade the work?
RHO, the strongest earlier method in this line, has a model compare two trajectories and pick the better one. HarnessPro ran a pre-experiment on training tasks from Terminal-Bench 2 and SWE-Bench Pro and found a specific bias: the judge was too optimistic. It called flawed implementations successful.
On failed cases, the LLM judge's recall was 41.7%. More than half of the failures passed review. A synthesized test suite raised recall on the same tasks to 83.3%. The paper reports the pair directly: failure recall increasing from 41.7% to 83.3%.
TestSynth, the replacement, is a separate agent. For each task it reads the public task description and the seed harness's first trajectory. It writes 10 executable tests and freezes them before evolution begins. A validation tool runs the suite, and the same agent reviews its own work. No second, independent judge is described.
Freezing the suite is the important design choice. If the tests are regenerated each epoch, the scorer drifts. A candidate can optimize against the new tests, and scores from different epochs stop being comparable. The cost of freezing is just as clear: test quality depends on the model writing the tests. A loose suite lets bad candidates through. A strict suite rejects correct work. The paper does not report agreement with the official test suites, so this part of the method remains unaudited.
What are the three rules for writing tests an agent cannot game?
Appendix C publishes the test-writing prompt. Three clauses carry most of the weight, and each one closes a known attack surface.
Scroll horizontally to see all columns.
| Rule | What it requires | What it blocks |
|---|---|---|
| Evidence boundary | The task description and public files define the contract. A frozen reference implementation is diagnostic context, never an answer key. The agent may not look for hidden official tests, scorers or solutions, locally or online. | Benchmark leakage |
| Oracle strength | Rank the evidence for a correct output: a compatible public reference implementation, a known answer, necessary content properties, a discriminative metamorphic relation. File existence, a successful process, non-empty output, or a value inside a plausible range do not prove behavioral correctness on their own. Do not reverse-engineer expected constants, hashes, tolerances or execution paths from the reference. | Tests that mirror the implementation and cannot tell a real difference |
| Verdict semantics | status=ok with passed=false means the execution was valid and the product failed. Making the artifact pass is not the purpose of the suite. infra_error is indeterminate. Distinguish a defective test from a product exception under a legal call, and fix a test only when there is evidence. | False passes and false failures |
The three clauses are summarized from the test-writing prompt printed in the paper's appendix. They are not the benchmark results, which are self-reported and not independently replicated.
These rules travel further than the benchmark numbers. Any team that asks a model to write its own regression tests faces the same three choices: where the expected behavior comes from, how strong the evidence is, and what a failed result means.
How is this different from RHO?
Scroll horizontally to see all columns.
| Method | Feedback | Objective | Output |
|---|---|---|---|
| RHO | An LLM compares trajectories | Pass rate | One improved harness |
| Balance | RHO plus a cost-aware instruction | Pass rate with a cost hint | One harness |
| HarnessPro | Frozen synthesized tests | Pass rate and cost as separate axes | A Pareto set of three |
Balance and RHO+Iteration exist as controls, not as products. Balance changes only the wording of RHO's optimization instruction. RHO+Iteration gives Balance the same number of epochs.
Those controls answer a fair question: did HarnessPro win because it searched longer or spent more? The results below suggest it did not.
How does one epoch work?
Stage 1, diagnosis. Each candidate runs three times on every training task. The analyzer reads all three traces plus a metric vector of pass, cost and latency, along with token counts and tool calls. Its output is structured: the problem, its effect on performance and cost, and the evidence. Evidence includes a trace excerpt, a file path and an event range. Diagnoses above 64K tokens are summarized, and later agents can follow the path back to the raw trace.
Stage 2, planning and editing. Each candidate receives three instructions in parallel: cost-first, balanced and performance-first. The planner sees the full implementation, the diagnosis, the metrics, and cross-candidate evidence: diffs, diagnoses and metrics from the other candidates. A plan has to name the component that changes, describe the change, and estimate the effect on pass rate and cost. An editor applies each plan in an isolated workspace and commits the result to Git.
Stage 3, evaluation and selection. Surviving candidates from the previous epoch and new candidates from this one run once per validation task. Two filters apply. A candidate below the floor is dropped; the floor is the seed harness's pass rate on the validation set, measured with the frozen tests before evolution starts. A dominated candidate is dropped when another candidate is no better on either objective and strictly better on at least one. What remains sits on the performance-cost frontier.
Selection uses a normalized weighted squared distance. The pass-rate gap and the cost excess are each scaled by the range in the current filtered set. Each tier picks the candidate that minimizes J = α·d_pass² + β·d_cost². Lite weights cost, Heavy weights pass rate, and Medium sits between them. The three representatives become the starting set for the next epoch. Terminal-Bench 2 and SWE-Bench Pro run three epochs; ProgramBench runs four. Held-out tests are touched only after the harnesses are frozen.
Scroll horizontally to see the full figure.
What do the three instruction templates forbid?
The appendix also publishes the three optimization instructions. Their wording is specific, and most of it is defensive.
- Cost-first: a lower bill from skipping required work, abandoning hard tasks or weakening validation does not count. Fewer steps or shorter output are not improvements by themselves. Total cost includes failed attempts and the overhead introduced by the change.
- Balanced: the plan improves the tradeoff itself. It is not an average of the other two directions. Peer candidates can inform a plan, but their mechanisms cannot be copied wholesale, and the overhead of combining changes has to be counted.
- Performance-first: extra compute goes where it resolves meaningful failures. It is not spread evenly across every task. A more complex harness is not assumed to be more capable. Added work has to justify its expected quality, cost and residual risk.
All three share a rule about provenance. The prompt supplies no routing rubric, fixed thresholds, profile taxonomy or per-task rules. Every mechanism has to be derived from evidence. All three also separate predicted savings from measured results. Whether quality held and cost fell is decided by the fixed external evaluator, not by the plan's own claim.
Validation and test sets stay hidden from the planner. Plans are asked to stay focused and generalizable. Read separately, these clauses look ordinary. Together they place a gate in front of each place where the system could fool itself.
Does recursion help, or is one epoch enough?
Terminal-Bench 2 Heavy starts at 74.6% and $0.91 per task in epoch 1 and ends at 80.8% and $0.72 in epoch 3. Both numbers improve.
SWE-Bench Pro Medium is the more interesting path. Epoch 1 reaches 70.0% at $1.57. Epoch 2 reaches 77.0% at $1.66: more score and more cost, a move along the existing frontier. Epoch 3 reaches 79.0% at $1.26. The extra spending in epoch 2 paid for a mechanism that made epoch 3 cheaper.
A single epoch can only move near mechanisms it has already found. The planner's evidence does not change much within one round. Multi-round recursion carries diffs and diagnoses forward, so the next planner knows which mechanisms have been tried and which bottlenecks remain. That is how the frontier moves outward instead of only along itself.
Test-set numbers from intermediate epochs appear in plots only. They never participate in selection.
How much cheaper is it, and against what?
Two experiment suites are reported: GPT-5.5 with Codex CLI 0.147.0, and GLM-5.2 with Claude Code 2.1.231. Cost is computed from token prices. The main tables report the mean of three runs; ProgramBench uses two.
Scroll horizontally to see all columns.
| Method | Terminal-Bench 2 | SWE-Bench Pro | ProgramBench |
|---|---|---|---|
| Vanilla | 69.5% / $0.89 | 59.0% / $1.88 | 60.3% / $2.65 |
| RHO | 76.3% / $1.09 | 80.0% / $2.68 | 68.8% / $3.59 |
| Balance | 74.6% / $0.79 | 78.0% / $2.08 | 65.7% / $3.29 |
| RHO+Iteration | 76.3% / $0.82 | 80.0% / $1.90 | 68.3% / $3.13 |
| HarnessPro Lite | 72.9% / $0.43 | 72.3% / $0.80 | 64.3% / $2.18 |
| HarnessPro Medium | 76.3% / $0.56 | 79.0% / $1.26 | 67.9% / $2.90 |
| HarnessPro Heavy | 80.8% / $0.72 | 80.7% / $1.97 | 69.6% / $2.95 |
GPT-5.5 with Codex CLI. Each cell is pass rate / mean cost per task. HarnessPro rows are means with sample standard deviations in the paper; the deviations are omitted here.
On GPT-5.5 and Codex, Heavy beats RHO on all three benchmarks while costing less. On Terminal-Bench 2 it adds 4.5 points and cuts cost by 33.9%, from $1.09 to $0.72. On SWE-Bench Pro it adds 0.7 points and cuts cost by 26.5%. On ProgramBench it adds 0.8 points and cuts cost by 17.8%. The three-benchmark average is +2.0 points and 26.1% lower cost.
Lite is the cheaper claim. It beats the unmodified harness on all three benchmarks: 72.9% against 69.5%, 72.3% against 59.0%, and 64.3% against 60.3%. Cost falls by 17.7% to 57.4%. On SWE-Bench Pro the gap is widest: 13.3 more points at $0.80 against $1.88.
The two fair controls narrow the claim. Medium ties RHO+Iteration on Terminal-Bench 2 at 76.3% while costing 31.7% less. Heavy adds 1.3 points on ProgramBench while costing 5.8% less. On SWE-Bench Pro, Medium costs 33.7% less than RHO+Iteration at one point lower pass rate. The win is not explained by a larger search budget.
Scroll horizontally to see the full figure.
On GLM-5.2 and Claude Code, Heavy beats RHO on all three benchmarks while costing less: 81.4% against 78.0% at $0.62 against $1.00, 77.3% against 75.0% at $1.01 against $2.17, and 64.4% against 63.3% at $14.95 against $17.00. Medium costs less than RHO but gives up 1.7 points on Terminal-Bench 2 and 0.3 points on ProgramBench, and ties on SWE-Bench Pro at 75.0% for $0.87. Lite costs the least and gives up pass rate on all three.
Scroll horizontally to see all columns.
| Method | Terminal-Bench 2 | SWE-Bench Pro | ProgramBench |
|---|---|---|---|
| Vanilla | 74.6% / $0.92 | 56.0% / $1.17 | 59.7% / $11.34 |
| RHO | 78.0% / $1.00 | 75.0% / $2.17 | 63.3% / $17.00 |
| HarnessPro Lite | 75.1% / $0.43 | 70.7% / $0.70 | 60.1% / $11.10 |
| HarnessPro Medium | 76.3% / $0.50 | 75.0% / $0.87 | 63.0% / $13.74 |
| HarnessPro Heavy | 81.4% / $0.62 | 77.3% / $1.01 | 64.4% / $14.95 |
GLM-5.2 with Claude Code. Balance and RHO+Iteration are not reported for this pair in the main table.
Latency is not a target, but the SWE-Bench Pro Heavy run drops from RHO's 520 seconds to 412 seconds. The paper logs latency without optimizing it, and the three metrics move independently.
What does the 53% figure actually measure?
The number comes from SWE-Bench Pro with GPT-5.5 and Codex. Medium reaches 79.0% at $1.26 per task. RHO reaches 80.0% at $2.68. Medium costs 47% of RHO, a 53% reduction, for one point less pass rate.
That is a deployment saving. It is the cost of running tasks after the harness exists. The paper does not report a single figure for the cost of evolution itself: diagnosis, three plans per candidate, repeated evaluation, and the agent calls around them. Those costs are likely more than an order of magnitude larger than the per-task figure. Treating the 53% as the cost of building the harness turns a cheaper run into a cheaper project.
The number is also local to one comparison. Lite saves more and gives up score. Heavy gives up some saving to beat RHO. The right operating point depends on what a failed task costs.
What breaks when you remove a piece?
Removing the test synthesizer hurts most: 3.4 to 10.0 points lost across two benchmarks and three tiers. That matches the 41.7% to 83.3% recall gap. The evaluation layer is the foundation.
Removing the planner is second. SWE-Bench Pro Medium falls from 79.0% to 73.0%, and cost rises from $1.26 to $1.42. Reading cross-candidate evidence is not decoration.
The reverse experiment is the most useful one. Take a single strategy from HarnessPro's evolved artifacts, such as dynamic routing or budget hooks, and bolt it onto RHO. The result is cheaper but lower scoring, and HarnessPro's own artifacts dominate it. Force the reasoning tier to low and Terminal-Bench 2 drops to 62.7%. The strategies work as a set: routing decides where budget goes, hooks enforce it, and compression controls what enters the context.
Four limits remain. The test writer is an LLM, and the paper does not report agreement with official tests. The training set has 10 tasks, so representativeness is open. The test writer sees the seed harness's first trajectory, which fixes its bias in place rather than removing it. And the evolution cost has no unified accounting.
What can a small team copy without running the loop?
Most teams will not run a full harness-evolution loop. Five pieces are cheaper to test.
- Budget hooks with stages. In the Terminal-Bench 2 Medium artifact, a reminder appears after tool call 14 asking the agent to batch calls and avoid duplicate checks. At 26 it asks the agent to focus on completion and verification. At 44 it blocks further tool use and asks for a submission. The reminder is appended to tool results rather than inserted into the system prompt, which keeps the prefix cache intact.
- A frozen regression suite. Before changing a prompt or a config, have the model write tests for typical tasks from public contracts. Freeze the suite, run it before and after, and treat the delta as evidence. Apply the three rules above: no hidden tests, graded oracle strength, and a clear meaning for a failed result.
- A point on the frontier. Ask for the budget first, then choose the tier. Medium often does most of the work at about half the cost of the top-scoring configuration.
- Cross-candidate evidence. When several variants run in parallel, give the editor the diffs and failure diagnoses from the others. Knowing which traps others hit is cheaper than rediscovering them.
- Anti-cheat lines in your own instructions. Two sentences from the paper are worth copying: fewer steps or shorter output is not an improvement, and quality and cost are decided by a fixed external check rather than the agent's own claim.
A note on the repository: it ships nine evolved harness artifacts, one per benchmark and tier, with instruction files and runtime policy. It does not ship the automatic evolution loop. The repository is a research artifact, not a one-command reproduction.
Frequently asked
What is HarnessPro?
HarnessPro is a research system that evolves the harness around a frozen model. It searches instruction files, tool wiring and runtime policies, then returns three harnesses on a performance-cost frontier: Lite, Medium and Heavy.
What is the 53% cost cut?
It is the difference between HarnessPro Medium and RHO on SWE-Bench Pro with GPT-5.5 and Codex: $1.26 against $2.68 per task, at 79.0% against 80.0% pass rate. It measures deployment cost, not the cost of running the evolution.
Does HarnessPro need labeled tests?
No. It never uses official tests or reference answers during evolution. A separate agent writes 10 executable tests from the task description and the seed harness's first trajectory, then freezes them for the run.
What is TestSynth?
TestSynth is the test-writing agent. In the paper's pre-experiment, synthesized tests raised failure recall from 41.7% with an LLM judge to 83.3%.
Is HarnessPro open source?
The repository is MIT licensed and publishes nine evolved harness artifacts. The automatic evolution loop is not part of the release.
How is HarnessPro different from RHO?
RHO uses an LLM to compare trajectories and optimizes pass rate. HarnessPro uses frozen synthesized tests, treats pass rate and cost as separate objectives, and returns a Pareto set.
What is the catch?
The 53% is one comparison on one benchmark. Test quality is not audited against official tests, the training set has 10 tasks, and the cost of the evolution itself is not reported.