Evaluating small models honestly

Leaderboards rank models on tasks nobody has. They tell you which model is best in general and almost nothing about whether a small model is good enough for the one job you have. For that, you need to measure your job.

Here is the method we use, in the order we use it.

1. Write the task down narrowly

“Summarise text” is not a task. “Turn a two-sentence lesson note into a single quiz question with four options” is. A narrow task has a right answer you can check, which is the whole point.

2. Build a small, honest test set

Fifty real examples beat five thousand synthetic ones. Pull them from the actual content the model will see, and include the awkward cases: the short ones, the messy ones, the ones you are not sure about. If your test set is easy, your evaluation is theatre.

3. Decide the bar before you look

Write down what “good enough” means in advance: the pass rate you need, and the failure mode you cannot tolerate. Deciding after you see the numbers is how teams talk themselves into a model that quietly breaks in production.

4. Test upward from the smallest

Start with the tiniest model that could plausibly work and move up only when it fails. This is the opposite of the instinct to start at the top, and it is how you find the cheapest thing that clears the bar.

• Measure accuracy on your set, not on a leaderboard.
• Measure latency on the hardware you will actually ship on.
• Measure cost per thousand, including the ones you will run forever.

5. Watch the failure modes, not just the score

A model at 92% that fails confidently and plausibly is worse than one at 85% that fails obviously. In a learning tool, a wrong answer that sounds right is the expensive kind of wrong.

Benchmarks are a starting point for a shortlist. Your own fifty examples are the decision.