Stop Guessing Which LLM to Use: Test It With Promptfoo

I made a video on how to stop picking models on vibes and start testing them on your own workload. Here is the gist and my take, with enough of the how-to that you can run it yourself without watching.

Test on your prompts, not a benchmark

A public benchmark measures a model against someone else's questions. Your app has its own questions. In the video I use Promptfoo, an open-source runner recently acquired by OpenAI. You give it a YAML file of your prompts and test cases, it runs them against multiple models, and it reports real pass and fail results on those exact prompts. It works with OpenAI, Claude, and local models through Ollama, so a paid API and a free local model can sit in the same comparison. You commit the file and rerun it like code.

Getting started is promptfoo init, which drops a config with three sections: prompts, providers, and tests. You swap in your own prompts and expected outputs, add your API keys, run it, and open the HTML report to read the results as one matrix.

Cost per correct answer is the real number

Two models can have the same price per token and very different real costs. If one passes 95 percent of your cases and the other passes 60, the cheaper-looking one costs more once you count retries and cleanup. Cost per correct answer is what surfaces the cheapest model that still clears your bar.

I ran this on my own site, markdown.me. I pulled real production inputs from two tools and compared the small model already in production against a premium small model, using a stronger model as the judge. The result talked me out of the upgrade. The current model won the strict JSON tool because the premium wrapped its JSON in code fences. On the other tool they tied on normal inputs, and the premium only helped on extremely short, underspecified cases. The better fix was keeping the cheap model and adding a UI hint that asks the user for more detail.

The five dimensions that actually decide it

• Task quality on your own data, not someone else's leaderboard.
• Cost per correct answer, not cost per token.
• Latency at the 95th percentile, not the median. A median of 800 milliseconds can hide a P95 of nine seconds.
• Privacy and data boundary. Cloud API, in-tenant cloud like Bedrock or Vertex, or fully local like Ollama. Decide this first and it rules out most candidates.
• Tool use and structured output. At 95 percent tool-call reliability across an eight-step agent, end-to-end success drops to about 66 percent. At 99 percent you are still only near 92.

Two ways to grade

Deterministic checks are exact rules, like valid JSON, a required field, or a cost and latency ceiling, run with no model in the loop. LLM-as-a-judge scores answers that have no single correct form, like summaries, against a plain-English rubric. In the demo I compared Nova Micro against Haiku 4.5 with Sonnet as the judge. Nova Micro cleared the quality bar at roughly a twenty-eighth of the cost, so it won on the requirement that mattered to me.

Build a representative set from production logs, and mix it. Ten to twenty cases track small changes, but confident model selection wants 100 to 200, around 60 percent happy path, 30 percent edge cases, and 10 percent adversarial. A set that is all happy path is lying to you. Start with one prompt, three models, and ten test cases, where one of the three is your current default. Then keep the file and rerun the decision whenever a prompt changes or a cheaper model shows up.

If you want a hand