Eval scorecard
The numbers below come from pnpm eval run on a fixed set of 40 shopper questions (32 product searches and 8 that need a specific tool), then committed. They are not worked out when you load this page.
Last run4bef2aeconcierge-2026-10-22.1
Product search
Does the right product appear near the top? hit@3 is the share of questions where an acceptable product is among the first three results. Raw sentence gives the search the shopper’s own words; tool arguments gives it what a good model would pass after reading them.
- hit@1, raw sentence
- 100.0%
- hit@3, raw sentence
- 100.0%
- gate: 85% or more
- hit@3, tool arguments
- 100.0%
- gate: 85% or more
- Wrongly confident
- 0 of 8
- things the shop does not sell
| Mode | Questions | hit@1 | hit@3 | hit@5 | MRR | p50 | p95 |
|---|---|---|---|---|---|---|---|
| Raw sentence | 32 | 100.0% | 100.0% | 100.0% | 1.000 | 538 ms | 771 ms |
| Tool arguments | 32 | 100.0% | 100.0% | 100.0% | 1.000 | 541 ms | 809 ms |
30 of 32 answerable questions came back with a confident match; the rest were found but flagged as a weak match. Latency is one catalogue query from the sandbox to Neon in Singapore, so it is mostly the network.
Tool selection and faithfulness
The 8 tool questions, plus four product searches, run through the real concierge: the same route, tools, cart and database, with a throwaway copy of the catalogue. Right tool means the expected tool was called at some step; right arguments means it was called with the right order, product or variant (for add to cart: the cart ended up holding the right item in the right quantity). Grounded means every price, measurement and order number in the answer appears in what the tools returned. The judge is a second model that reads each answer against the same evidence.
No language model has been measured yet.
Workers AI’s free daily allowance was used up before the first run could finish, so the only run recorded below is the scripted stand-in. Run pnpm eval once the allowance resets (00:00 UTC), or with an Anthropic key, and the result appears here.
Scripted stand-in
The model behind the browser tests and the handler tests: a keyword planner that reads the same prompt a model would. Its score checks that the harness and the tools work. It says nothing about how a language model performs.
- Right tool
- 8 of 8
- first tool 7 of 8
- Right arguments
- 8 of 8
- Grounded answers
- 12 of 12
- Judge
- not run
- needs a language model
History
One point per run. A flat line at the top means nothing has slipped; a drop shows which change to look at, because every run records its commit and prompt version.
Search hit@3
- Raw sentence
- 100.0%
- Tool arguments
- 100.0%
One run so far. Scale 0 to 100%.
Right tool called
Out of the 8 tool questions.
- Scripted stand-in
- 100.0%
One run so far. Scale 0 to 100%.
Answers with every figure grounded
- Scripted stand-in
- 100.0%
One run so far. Scale 0 to 100%.
Judge: fully supported answers
Needs a language model to judge.
Not measured yet.
Every run, as a table
| Run (UTC) | Commit | hit@3 raw | hit@3 args | Model | Right tool | Grounded |
|---|---|---|---|---|---|---|
| 22 October 2026 at 17:24 | 4bef2ae | 100.0% | 100.0% | scripted, router | 8 of 8 | 12 of 12 |
What this does not tell you
- The 40 questions were written by the people who built the search. 100% here means nothing on this set has regressed, not that every shopper’s question will work.
- Eight tool questions is a small sample: one miss moves the score by 12.5 points.
- The faithfulness judge is a language model and can be wrong in either direction. The grounded check is stricter about numbers and blind to everything else.
- Nothing here measures how the answers read. The concierge is the place to judge that.