The Honest Guide to LLM Evaluation

Why LLM evaluation is uniquely hard

For most ML systems, evaluation is well-understood. Classification: precision, recall, F1. Regression: RMSE, MAE. Recommendation: NDCG, hit rate. You compute a number, you know if you’re improving.

LLMs produce open-ended text. The same question can have ten equally correct answers. The same answer can be good for one use case and terrible for another. And the most common failure mode — confident hallucination — is not captured by any automated metric because the output looks perfectly fluent and well-structured.

What automated metrics actually measure

BLEU and ROUGE

These measure n-gram overlap between generated text and a reference answer. The problem: they assume there’s one correct answer (or a small set). For open-ended generation, an answer can be completely correct and score near zero if it uses different words from the reference. They correlate reasonably with quality for translation tasks. For general LLM evaluation, they’re close to meaningless.

Perplexity

Measures how “surprised” the model is by a piece of text. Lower perplexity = more fluent output. The problem: hallucinations are fluent. A model that confidently states the wrong capital city with perfect grammar has low perplexity. Perplexity measures fluency, not factual accuracy.

Building a human evaluation framework

This doesn’t have to be expensive or slow. A robust minimum:

1. Collect 50–100 representative examples — actual questions from real users, or written by domain experts to cover the space of likely queries. Not cherry-picked easy ones.
2. Define a 3-point rubric specific to your use case — not “good/ok/bad” (too subjective) but task-specific. For a technical QA system: 1 = incorrect or misleading, 2 = correct but incomplete or unclear, 3 = correct, complete, and clear.
3. Have at least two raters per example — compute inter-rater agreement (Cohen’s kappa). If kappa is below 0.6, your rubric is ambiguous. Refine it before continuing.
4. Track this score over time — every time you change the model, prompt, or retrieval system, re-run eval on the same set. Treat any regression as a blocking issue.

LLM-as-judge: when it works and when it doesn’t

Using a strong LLM (GPT-4 level) to judge outputs from a weaker model is increasingly popular. It works well for: relative comparison (“which of these two answers is better?”), checking for factual consistency against a provided reference, and detecting obvious refusals or off-topic responses.

It doesn’t work well for: detecting subtle factual errors in specialized domains the judge model doesn’t know well, absolute quality scoring (LLM judges are inconsistent about what “7/10” means), and catching the specific failure modes of the model being judged (similar models have similar blind spots).

JUDGE_PROMPT = """Compare these two responses to the question.
Question: {question}
Response A: {response_a}  
Response B: {response_b}

Which is better? Consider: factual accuracy, completeness, clarity.
Answer with just "A" or "B" and a one-sentence reason."""

Evaluating for the failure modes that matter most

Build specific test cases for each failure mode relevant to your application:

• Hallucination — ask about things not in the context. The model should say “I don’t know.” If it answers, that’s a hallucination.
• Boundary cases — ambiguous questions, edge cases, questions where the correct answer is “it depends.” How does the model handle uncertainty?
• Adversarial inputs — questions designed to confuse, leading questions, questions that assume a false premise.
• Consistency — ask the same question 5 times with minor rephrasing. Correct answers should be consistent. Contradictory answers on the same factual question is a serious reliability problem.