Six topics, checked the same way every time
Each seeded topic runs through the whole crew with the scripted stand-in model, then the finished brief is measured: all six sections, a citation for every figure, enough citations, the Critic's loops and the token cap. No model grades these checks, so the same code gives the same score. These runs say how well the pipeline keeps its rules. They do not say how well a real model writes.
- topics pass, run 22 Oct 2026, 17:20 UTC
- 5 of 6
- mean score (pass at 0.8)
- 0.91
- Rubric judge
- skippedthe model provider answered 429 (daily allowance spent); the deterministic score stands alone
| Topic | Sections | Citations per 120 words | Uncited figures | Key facts | Critic loops | Tokens | Score | Judge |
|---|---|---|---|---|---|---|---|---|
| Electric scooter rental in LisbonPasses545 words | 6/6meets | 6.2meets | 0meets | 4/8below target | 1meets | 24,674meets | 0.90$0.0176 at list price | skipped |
| Meal-kit delivery in SingaporePasses549 words | 6/6meets | 6.1meets | 0meets | 4/8below target | 1meets | 21,130meets | 0.90$0.0152 at list price | skipped |
| Podcast hosting platformsPasses551 words | 6/6meets | 5.7meets | 0meets | 2/8below target | 1meets | 21,346meets | 0.85$0.0156 at list price | skipped |
| Compliance software for small law firmsPasses479 words | 6/6meets | 6.5meets | 0meets | 6/8meets | 2meets | 31,792meets | 0.95$0.0221 at list price | skipped |
| Pet insurance in GermanyFails447 words | 6/6meets | 6.4meets | 0meets | 6/8meets | 1meets | 20,093meets | 0.95$0.0142 at list price | skipped |
| Home battery storage in AustraliaPasses562 words | 6/6meets | 6.4meets | 0meets | 5/8meets | 1meets | 25,097meets | 0.93$0.0180 at list price | skipped |
What each check found
The measured value for every check, per topic. A topic that fails opens by default.
Electric scooter rental in LisbonPasses at 0.90
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets28 citations in 545 words is 6.17 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 24 sentences with a figure carry a citation.
- Key facts coveredbelow target3 of 5 key facts and 1 of 3 named players. Missing: Ride Lisboa holds 38% of trips; Ride Lisboa unlock fee and per-minute rate; Names Ride Lisboa; Names Alfama Volt.
- Critic loops within limitmeets1 follow-up round (at most 2 allowed).
- Cost within the capmeets24,674 of 120,000 tokens (21% of the cap).
- Length in rangemeets545 words (450 to 1500 wanted).
- Distinct sourcesmeets6 distinct sources cited (at least 4 wanted).
Judge skipped: the model provider answered 429 (daily allowance spent); the deterministic score stands alone.
Meal-kit delivery in SingaporePasses at 0.90
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets28 citations in 549 words is 6.12 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 24 sentences with a figure carry a citation.
- Key facts coveredbelow target2 of 5 key facts and 2 of 3 named players. Missing: 62% of new subscribers cancel within 12 weeks; S$68 acquisition cost; Packaging producer responsibility fee; Names Wok & Whisk.
- Critic loops within limitmeets1 follow-up round (at most 2 allowed).
- Cost within the capmeets21,130 of 120,000 tokens (18% of the cap).
- Length in rangemeets549 words (450 to 1500 wanted).
- Distinct sourcesmeets6 distinct sources cited (at least 4 wanted).
Judge skipped: an earlier topic got a 429, so the rest were not tried.
Podcast hosting platformsPasses at 0.85
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets26 citations in 551 words is 5.66 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 22 sentences with a figure carry a citation.
- Key facts coveredbelow target1 of 5 key facts and 1 of 3 named players. Missing: 3.9 million active shows; Soundshelf share of active shows; Median entry plan price fell 14%; 31% of new shows are video-first; Names Castwell; Names Reedwave.
- Critic loops within limitmeets1 follow-up round (at most 2 allowed).
- Cost within the capmeets21,346 of 120,000 tokens (18% of the cap).
- Length in rangemeets551 words (450 to 1500 wanted).
- Distinct sourcesmeets7 distinct sources cited (at least 4 wanted).
Judge skipped: an earlier topic got a 429, so the rest were not tried.
Compliance software for small law firmsPasses at 0.95
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets26 citations in 479 words is 6.51 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 22 sentences with a figure carry a citation.
- Key facts coveredmeets4 of 5 key facts and 2 of 3 named players. Missing: 38% of sanctions involve trust accounts; Names LedgerLaw Shield.
- Critic loops within limitmeets2 follow-up rounds (at most 2 allowed).
- Cost within the capmeets31,792 of 120,000 tokens (26% of the cap).
- Length in rangemeets479 words (450 to 1500 wanted).
- Distinct sourcesmeets7 distinct sources cited (at least 4 wanted).
Judge skipped: an earlier topic got a 429, so the rest were not tried.
Pet insurance in GermanyFails at 0.95
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets24 citations in 447 words is 6.44 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 20 sentences with a figure carry a citation.
- Key facts coveredmeets4 of 5 key facts and 2 of 3 named players. Missing: Average claim up 9.3% to EUR 412; Names Tierwohl Direkt.
- Critic loops within limitmeets1 follow-up round (at most 2 allowed).
- Cost within the capmeets20,093 of 120,000 tokens (17% of the cap).
- Length in rangefails447 words (450 to 1500 wanted).
- Distinct sourcesmeets5 distinct sources cited (at least 4 wanted).
Judge skipped: an earlier topic got a 429, so the rest were not tried.
Home battery storage in AustraliaPasses at 0.93
- Required sectionsmeetsAll 6 sections are present and say something.
- Citation densitymeets30 citations in 562 words is 6.41 per 120 words (at least 1 wanted).
- No uncited numbersmeetsAll 26 sentences with a figure carry a citation.
- Key facts coveredmeets3 of 5 key facts and 2 of 3 named players. Missing: AUD 300 per kWh rebate up to 14 kWh; Payback 9.8 years, 7.4 with a VPP; Names Kookaburra Energy.
- Critic loops within limitmeets1 follow-up round (at most 2 allowed).
- Cost within the capmeets25,097 of 120,000 tokens (21% of the cap).
- Length in rangemeets562 words (450 to 1500 wanted).
- Distinct sourcesmeets8 distinct sources cited (at least 4 wanted).
Judge skipped: an earlier topic got a 429, so the rest were not tried.
How the score is made
- Required sections
- All six are written and say something: market snapshot, players, pricing, differentiators, risks, recommended angle. A section that only admits the evidence does not cover it counts as missing.
- Citation density
- At least 1 citation per 120 words, and every citation points at a listed source.
- No uncited numbers
- Every sentence that contains a figure carries a citation.
- Critic loops
- No more than 2 rounds of follow-up research.
- Cost
- Tokens spent stay under the run cap (at most 120,000).
- Length and sources
- 450 to 1500 words, citing at least 4 different sources.
- Key facts
- How many of a topic's checklist facts and named players appear. It lowers the score but does not fail a brief by itself.
- Score
- A weighted mean of the checks (sections 0.2, citations 0.2, figures 0.2, key facts 0.2, loops 0.1, cost 0.1). A topic passes with at least 0.8 and none of the gating checks failed.
- Rubric judge
- When a real model is configured it also reads each brief against the facts it was written from and scores faithfulness, usefulness, balance from 1 to 5. If the provider refuses (a 429 once the free allowance is spent), the judge is skipped and the card says so.
Run it yourself
1 evaluation saved. The command runs the six topics in seconds with no key and no network, prints this card, and saves it when a database is configured. It exits with an error if a topic fails. The same data is at /api/eval.
pnpm eval