AI Research Reports
Tests we ran on AI models and on our own builds, each with the method, the numbers and the sources.
Latest findings
All reports- ×2Owner rules written into a spreadsheet roughly doubled AI answer scores.
- 2.5%A cheaper model directed by a stronger one hit 98% of the quality on 2.5% of the frontier tokens.
- 94.4%Recall on Claude Sonnet 5 at 730,000 tokens, down from 100% at 34,000.
- ×2.5Two AI judges read the same document from the same brief. One raised 15 numeric claims, the other 6.
- 23%Fewer context tokens on the dev set, and 18% on hidden queries, after an agent ran 60 retrieval experiments overnight, with zero lost answers.
- $0.14Per coding task for Kimi K3, against $0.44 for Claude Fable 5 and $1.62 for GPT-5.6-Sol.
All Reports
Is Firecrawl Worth It? 252 Pages, 12 Pipelines, 20 Claude Code Tasks
Firecrawl's markdown used 94% fewer tokens than raw HTML and 2.91 times trafilatura's. In Claude Code its MCP went 19 for 20 against WebFetch's 12, at 4 times the tokens.
Read more →Owner Rules Written Into a Spreadsheet Roughly Doubled AI Answer Scores on 12 Synthetic Businesses
A pre-registered, blind-judged test of Sheet Geek, our open-source spreadsheet skill, on 12 new synthetic businesses: 5.25 to 10.32 of 15, and every limit we found.
Read more →98% of the Quality for 2.5% of the Frontier Tokens
81 runs. A weaker model directed by a stronger one scored 96.8 against 98.8, on 2.5% of the frontier tokens. Add a codebase map and its best run ties outright.
Read more →Over-Engineering
We audited our own harness for three days and got 123 true issues. Arrivals tracked attention at r = 0.826. The failure was no budget, not false positives.
Read more →AI Benchmarks
Two 2026 audits found broken answer keys and a grader that passed 62.81% of wrong answers. The one check that tells you if a score is real.
Read more →Context Rot: We Measured Recall at 700,000 Tokens
128 trials against Claude Sonnet 5. Accuracy fell from 100 percent at 34,000 input tokens to 94.4 percent at 730,000, on a task whose difficulty never changed.
Read more →Two AI Judges, One Document, Two Different Reviews
LLM as a judge, measured: 15 numeric claims against 6, 31 bullets against 88, and the four rules that made adversarial review terminate.
Read more →We Let an Agent Run 60 Experiments on Our RAG Stack While We Slept
A worker-supervisor loop ran 60 controlled experiments on our hybrid retriever overnight and cut context tokens 23% with no accuracy loss. The waste was inside the chunks, not the ranking.
Read more →Does an LLM Understand Credible Punishment?
We put a frontier model against a grim-trigger pricing rival and swept the odds the game continues. It cooperates exactly when game theory says it should, then leaves money on the table.
Read more →Kimi K3 vs the Frontier: A Private Coding Eval
Kimi K3 launched, so we skipped the leaderboard and ran it against Claude Fable 5 and GPT-5.6-Sol on our own coding gauntlet. K3 was the cheapest per task.
Read more →We Ran 154 Controlled Trials of Claude vs GPT. Both Scored Perfect.
A blocked full-factorial experiment pitting Claude Fable 5 against GPT-5.6 on nine proven-solvable coding tasks, blind dual-judged, with a Wilson interval on every pass rate.
Read more →Claude Fable 5 vs. GPT-5.6 Sol: A Practitioner's Field Report
Claude Fable 5 and GPT-5.
Read more →Why AI Agents Fail on Tasks They Can Actually Do
AI agents fail on real work less from weak models and more from a missing harness.
Read more →Running AI Coding on Matt Pocock's Method, in Production
We run our AI coding pipeline on Matt Pocock's method: grill the plan, lock a spec, cut tickets, TDD, two-axis review.
Read more →Token Efficiency: Spending an AI Context Budget Wisely
Token efficiency means treating context as a budget: progressive disclosure, subagent delegation, and prompt caching cut cost per task, n...
Read more →Harness Engineering
Our operating doctrine: when an AI agent fails, fix the harness, not the model.
Read more →AI Agent Evaluation
Public benchmarks measure models. They tell you almost nothing about your system. How we test agents like software.
Read more →
Want this thinking applied to your business?
Building What's Next.