smevals — a small eval suite for evaluating models, prompts, and harnesses

Blog · Read

• New eval framework built with Jesse Vincent’s Prime Radiant lab (github: prime-radiant-inc/smevals); defines clear vocabulary — evals (challenge collections), tasks, configs (model+params), runs, graders, grades, and checkers (custom validation scripts).
• CLI commands: uvx smevals run, uvx smevals grade, uvx smevals serve; tested against gpt-5.5 and claude-opus-4.6, with results visualized through an interactive dashboard (example: a Haiku-evaluation dashboard at static.simonwillison.net).
• Takeaway: YAML-defined eval suites plus both simple string-match and model-based grading make it easy to stand up repeatable, comparable evals across models — part of Willison’s ongoing push for rigorous, first-hand model testing over vibes-based comparison.