Custom benchmarks
Benchmarks with answers you can check
Professional societies, research organizations and public agencies want to know how well AI models handle the work in their field. We design task sets with verifiable answers, run models side by side and keep the benchmark current as models change. We take on funded RFP and grant work.
What we do
From task design to re-tests
- Task sets with checkable answers.
Each task states its inputs and the method to apply, so exactly one answer is right. Expected answers are computed in code with exact arithmetic, then recomputed by a second program that shares no code with the first. A task goes into the suite only when the two agree. We include clean cases where nothing is wrong, so a model that flags everything is marked down.
- Models run side by side.
Every model works through the same suite version under the same instructions, three runs per task by default. We connect through OpenAI- or Anthropic-compatible APIs, or a command that takes each task as JSON and returns JSON. Our runners record the model ID the provider actually served next to the one requested, so each result can be traced to the model that produced it.
- Objective scoring.
Responses are scored by code against the key, never by another language model, so the same answer always gets the same score. Results come by task and by task family, with consistency across runs and a 95% confidence interval on the overall pass rate.
- Versioned re-tests.
Suites carry version numbers. When new models come out, we re-run the same version and compare against the saved baseline, so changes in the results reflect the models and not the test. If we find an error in an answer key, the fix is logged.
- Working with volunteer subject experts.
Many organizations have members willing to review a benchmark. We set that up so their time goes on judgment, not arithmetic: each task reaches them with its inputs, method, computed answer and independent check, and they tell us whether it is realistic, fair and unambiguous. Their changes go into the next suite version.
What we have built
Suites built with this method
All tasks are synthetic, and every answer key has been recomputed by an independently written checker.
Private credit
40 tasks across six families: maintenance covenants, EBITDA add-backs, borrowing base, NAV facilities, amendment chains and reporting deadlines. 15 contain no problem. Read more
Insurance and actuarial
A starter set of 5 tasks: chain-ladder, Bornhuetter-Ferguson, a rate indication with on-level premium and trend, credibility and increased limits. 2 contain no problem. Read more
CRE lending
8 tasks on DSCR, debt yield, loan sizing, ended interest-only periods and lease rollover, including 3 clean cases. Read more
Data audits
9 tasks checking extracted spreads, rent rolls, covenant terms and loan tapes against the source, including 3 clean cases. Read more
Funded work
We bid on RFPs, grants and research contracts
If your organization has funding to evaluate AI in its field, we can respond to a request for proposals or join a grant application.
A concrete plan
Proposals set out each task family, how its answers are computed and checked, and how models will be run and scored.
A clear budget
An itemized budget, including model API costs, with milestones tied to deliverables.
Outputs you can reuse
Task sets, scoring code and scorecards in plain formats, plus a written report. We can keep evaluation data private and release the rest openly if your project calls for it.
Independence applies here too. Our fee does not depend on how any model scores, and we take no fees, referrals or commissions from the AI providers we test.
Contact
Talk to us about a benchmark
Send us the RFP or a short description of the field and the questions you want answered. We will reply with how we would approach it.