The biggest bottleneck in AI development isn’t the model—it’s evaluation. How do you know if your prompt change actually made things better?
The Problem with Vibe Checks
“It feels better” is not a metric. When you change a prompt to fix one edge case, you often break three others. This is the regression trap.
Levels of Evaluation
Level 1: Deterministic Tests
For structural constraints.
-
Is the output valid JSON?
-
Does it contain the required keys?
-
Is the response under 500 characters?
Tools: Zod, Regex, JSON Schema.
Level 2: Model-Graded Eval (LLM-as-a-Judge)
Using a stronger model (e.g., GPT-4) to evaluate the output of a faster model (e.g., GPT-3.5 or Llama 2).
“Rate the following summary on a scale of 1-5 for accuracy and conciseness based on the original text.”
Level 3: Embedding Similarity
checking if the semantic meaning of the output matches a “gold standard” reference answer.
Implementing a Test Suite
-
Golden Dataset: Curate 50-100 examples of inputs and ideal outputs.
-
Runner: A script that runs your prompt against all inputs.
-
Scorer: A function that grades the outputs.
// Pseudocode for a simple eval
const results = await runPrompt(dataset);
const scores = results.map(result => evaluate(result, criteria));
console.log(`Average Score: ${average(scores)}`);
Conclusion
Stop guessing. Start measuring. Your confidence in shipping AI features will skyrocket once you have a green test suite.