Install 0.8.0Umbra 0.8.0 evals
61 graded tasks,every result
We build Umbra against an eval of 61 coding tasks. Each task has a hidden check, and every change to the prompt, the loop or the defaults is run against them before it's kept. This page has the method, the results of Umbra 0.8.0 and where the eval falls short.
- Tasks
- 61
- Categories
- 9
- Runs for each agent
- 183
- Final run
- 14 Nov 2026
Hard set20
Context fills9
Sessions7
Bug fixes6
Precise edits5
Finding code4
New files4
Large output3
Instructions3
Each square is one run of Umbra 0.8.0 in run final-8df86c3. Tap, hover or focus a square for the task.
How we use it
A change stayswhen the numbers improve
Every idea for Umbra is an experiment with a target measure, written in the ledger before it runs. It's kept only when its target improves by more than Pi's own spread between its best and worst of three runs, and no other measure gets worse by more than its spread.
Otherwise it is reverted, and the ledger says so. Of 27 experiments so far, 6 were turned off or reverted, among them an edit tool beside bash and saving long output to a file.
What is measured
The same tasks,the same model
The tasks.
61 tasks in 9 categories, written by Black Rainbow Labs. Each has a hidden check that the agent never sees, which passes or fails the result.
The runs.
Each agent ran every task 3 times, 183 runs for each agent, one at a time.
The agents.
Umbra 0.8.0 and Pi 1.3.0, each as shipped, with their own default prompts.
The model.
ninfer-qwen3.8-27b-coletti-v9, served on one RTX 5090 with 32 GB of memory, with the same server and sampling settings for both agents.
The proxy.
A measuring proxy sits between each agent and the model, so every token count and every time comes from the model server’s own responses.
A tie.
A difference counts only when it is larger than the spread between Pi’s own best and worst of its three runs.
Umbra 0.8.0
Results forthis release
| Runs passed | 91.3% |
|---|---|
| Mean score of the hidden checks | 98.7% |
| Hard set, runs passed | 78.3% |
| Tasks that fill the context window, runs passed | 92.6% |
| Time for each task | 69.4 s |
| Time to the first reply text | 382 ms |
| Prompt tokens served from the cache | 95.8% |
| Cache hit rate right after a compaction | 74.2% |
| New prompt tokens for each passed task | 14,694 |
| Requests for each passed task | 24 |
| Size of the first request | 1,835 tokens |
| Start time when resuming a long session | 36 ms |
| Session store bytes for each step | 763 |
| Category | Tasks | Passed |
|---|---|---|
| Hard set | 20 | 78.3% |
| Context fills | 9 | 92.6% |
| Sessions | 7 | 100% |
| Bug fixes | 6 | 100% |
| Precise edits | 5 | 93.3% |
| Finding code | 4 | 100% |
| New files | 4 | 100% |
| Large output | 3 | 100% |
| Instructions | 3 | 100% |
The 0.5.2 and 0.6.2 baselines ran 33 tasks once, so they aren’t shown next to this result.
Tested on local models,on purpose
We run the evals on a 27B model on one PC. A smaller model makes more mistakes, so a weak prompt or a wasteful loop shows up in the pass rate straight away, and we can run every task three times at no cost.
We expect what the harness gains there to carry forward to frontier models, because the gains come from sending less, caching more and handling long sessions better. We haven’t measured that yet.
The model: ninfer-qwen3.8-27b-coletti-v9, served on one RTX 5090 with 32 GB of memory.
Next to Pi
Next toPi 1.3.0
We ran Pi 1.3.0 as shipped on the same tasks, the same model and the same settings. "Pi's spread" is the difference between Pi's best and worst of its three runs.
| Measure | Umbra 0.8.0 | Pi 1.3.0 | Pi’s spread | Result |
|---|---|---|---|---|
| Runs passed | 91.3% | 80.3% | 6.6 points | Umbra |
| Mean score | 98.7% | 90.7% | 5.4 points | Umbra |
| Hard set, runs passed | 78.3% | 66.7% | 10.0 points | Umbra |
| Tasks that fill the context window | 92.6% | 44.4% | 22.2 points | Umbra |
| Time for each task | 69.4 s | 117.5 s | 22.1 s | Umbra, 69% faster |
| Time to the first reply text | 382 ms | 1,095 ms | 27 ms | Umbra, 187% faster |
| Total model seconds | 3,841 | 6,092 | 130 | Umbra, 59% faster |
| Prompt tokens, total | 19.6 million | 21.9 million | 0.7 million | Umbra, 10% fewer |
| New prompt tokens for each passed task | 14,694 | 78,944 | 7,646 | Umbra, 81% fewer |
| Completion tokens, total | 606,332 | 848,601 | 17,840 | Umbra, 29% fewer |
| Requests, total | 1,327 | 1,631 | 41 | Umbra, 19% fewer |
| Prompt cache hit rate | 95.8% | 82.4% | 1.1 points | Umbra |
| Cache hit rate right after a compaction | 74.2% | 32.4% | 1.7 points | Umbra |
| Start time when resuming a long session | 36 ms | 498 ms | 248 ms | Umbra |
| Session store bytes for each step | 763 | 1,094 | 29 | Umbra, 30% fewer |
| Reasoning tokens, total | 338,858 | 315,434 | 25,289 | tie inside the band |
A paired sign test over the 61 tasks finds Umbra better on 15, Pi better on 5 and 41 tied (p = 0.041).
Where Pi does better
- Precise edits: Pi passed 100% of runs and Umbra 93.3%.
- No Pi run reached the eval's own task files. One Umbra run of 183 did: it found the reference solution in the checkout and ran it to check its own output, and the eval failed that run.
Ties
- Reasoning tokens, 338,858 against 315,434, inside Pi's spread.
- Runs that hit a limit, 0 against 1, and runs with a nonzero exit, 0 against 1.
- Resume after a crash, 3 of 3 against 2 of 3.
What this does not show
The limitsof this eval
- One model, one PC. Nothing here says how the two agents compare on Claude or GPT.
- The tasks and checks were written by the maker of Umbra.
- Umbra’s prompt, defaults and code were changed while watching this eval. Pi was run as shipped and was not tuned.
- Some categories are small: tasks that fill the context window have 9 tasks, precise edits 5 and large output 3.
Run it yourself
Run the evalon your machine
The suite needs Node 24, Python 3.13, git, Git Bash, Umbra on the PATH, Pi, and a model server on port 8080. These are the commands from evals/README.md.