LLMs as a Judge: How to Know if Your LLM is Healthy

Why you should evaluate every AI agent before shipping to production — and how (Sponsored)

If you’re shipping AI agents without offline validation, your users are the ones doing the testing. Get the practical framework for evaluating production-grade AI agents before they hit production.

Get the guide to learn how to:

• Build annotated test datasets that cover core use cases, edge cases, and adversarial inputs

• Design deterministic and LLM-as-a-judge evaluators that reflect real business impact

• Trace multi-agent workflows end-to-end during experimentation to catch failures before users do

• Prevent model drift by keeping your offline test environment aligned with production

---

Large Language Models (LLMs) are also software systems just like any other software system we may have encountered. But we cannot test an LLM the same way as an ordinary software system.

For example, a normal function in a software application can receive two numbers and always return the same number as the total. However, when we ask the same question twice to an LLM, it will most likely produce two differently worded answers. Both of those answers may be acceptable. But it makes evaluation tricky. To evaluate an LLM, we have to measure whether the application continues to behave properly across many situations.

“LLM-as-a-Judge” is one part of this evaluation process. It involves using one language model to assess the output generated by another language model. But a judge model isn’t enough on its own. A healthy LLM evaluation system combines several ingredients such as conventional software tests, carefully curated examples, automated checks, model-based judging, human review, and production monitoring.

In this article, we are going to look at the process of LLM evaluation in detail. Here’s what we will cover:

• What does it mean for an LLM application to be healthy?

• Why are ordinary tests not sufficient for LLMs?

• The basic LLM evaluation loop

• Golden datasets for repeatable tests of LLM behavior.

• Automated metrics: Fast but limited checks

• What does LLM-as-a-judge mean?

• Different ways a judge model can evaluate an answer

• Human evaluation and calibration

• The evaluation stack

What Does it Mean for an LLM Application to be Healthy?

When can we call an LLM healthy?

The answer is quite simple. An LLM is deemed healthy if it consistently generates useful results while remaining within acceptable limits for accuracy, safety, speed, reliability, and cost. For example, consider a customer-support assistant. We cannot say it is healthy by a single question such as “Did it return the correct output?”

We need to consider several different questions:

• Did it understand what the customer was asking?

• Was the answer factually correct based on the company documentation?

• Did it answer the entire question?

• Did it follow the required tone and format?

• Did it avoid inventing policies that do not exist?

• Did it refuse requests that it should not answer?

• Did it respond within an acceptable amount of time?

• Did the request cost an acceptable amount to process?

As we can see, these questions are all related to different dimensions of quality, where each dimension is important. We can have an assistant that is friendly and relevant, but it might generate factually incorrect answers. It might be accurate but so verbose that users can’t simply find the answer. It might produce excellent answers but take 30 seconds for every request. Therefore, while evaluating an LLM, we must measure multiple aspects of the system.

We also need to differentiate between the health of the LLM and the application. We can’t simply label the model as the source of all problems. The problems can be caused by various sources. For example, we might have an incorrect prompt, missing documents, poor retrieval logic, incorrect tool calls, stale data, or a change in the surrounding code.

Why Ordinary Tests Are Not Sufficient for LLMs?

Traditional software tests usually depend on deterministic behavior. If a function receives a known input, the test expects a specific output. For example:

Input: add(2, 3)
Expected output: 5

This type of test works only because there is one exact correct answer. However, an LLM’s task is different. For example, consider the question “Explain why a password reset link may expire”

One possible response might begin by talking about security. Another response might first explain the expiration period. The way the sentences are written may be totally different even though the underlying answers are correct.

Three properties make LLM evaluation tricky.

First, the outputs from an LLM are not deterministic. This is because LLMs generate answers using a probability-based approach. We can use the same prompt and model, but the answers may be different. Increasing the model’s temperature only increases the variation. But even if we set a low temperature, it doesn’t make every model and infra combination perfectly repeatable.

Even an exact string comparison can mark many valid answers as failures. For example, the sentences “The payment was declined because the card expired” and “The card’s expiration date caused the payment to fail” convey the same information.

Second, quality is multidimensional and partly subjective. We can’t have a universally correct level of detail, tone, or organization of the answer. A response appropriate for a developer may simply confuse the customer. A short answer may work well in a support chat. But a longer answer might be more suitable in documentation.

This doesn’t imply that quality is impossible to measure. It just means that we need to first define the desired qualities. For example, having a goal like “give a good answer” is too vague to test. A much better option could be something like “Answer the question directly, use only the supplied policy, and explain all required steps”.

Third, the correctness also depends on the context. For example, the answer to “Can this order be refunded?” depends on multiple data points such as order date, product type, account status, region, and current refund policy. A particular response might be correct for one customer and wrong for another.

To evaluate properly, we must therefore include the context available to the model. We must be able to test whether the answer is correct relative to that context.

Of course, traditional testing is still needed for the deterministic parts of an LLM application. For example, things like JSON parsing, permission checks, calculations, database operations, API contracts, and tool execution should continue to rely on the usual unit and integration tests.

The Basic Evaluation Loop

A practical LLM evaluation system is built around a repeated process that works as follows:

• Collect representative test cases.

• Run the application on those cases.

• Inspect the resulting answers with several evaluation methods.

• Compare the results with the current production version.

• Block or investigate changes that cause an important regression.

• Monitor real production traffic for problems that the test set missed.

• Add newly discovered failures back into the test set.

See the diagram below:

The last step is super important. We need to evolve the useful evaluation dataset by adding more examples. Such a dataset grows as real users reveal unexpected questions, ambiguous instructions, document formats, and new ways the system can fail.

Golden Datasets for Repeatable Tests for LLM Behavior

A golden dataset is a carefully curated collection of possible inputs and information about what a good response to such an input should contain. Think of it as something that plays a role similar to a unit-testing suite. But unlike unit tests, we don’t always check the answers based on exact equality.

For example, a test case for a refund assistant might contain the following details:

This type of approach is much more useful than providing one perfectly written reference answer. This is because there may be many reasonable ways to express an answer. The important point is that the LLM makes the correct decision while following the stated constraints.

We should create a strong golden dataset with more than just easy and ordinary requests. It should ideally include the following:

• Common requests that represent most real traffic.

• Important cases where an incorrect answer would cause significant harm.

• Ambiguous questions that require clarification.

• Questions for which the answer is absent from the supplied information.

• Malicious or irrelevant instructions embedded inside retrieved documents.

• Very short, very long, poorly written, and multilingual inputs where relevant.

• Previous production failures.

• Boundary cases, such as a return made exactly on the final allowed day.

We should also divide the dataset into groups. A development set can be used while prompts are in the improvement phase. However, a separate holdout set should remain less visible during the development phase. Otherwise, the prompt may gradually be tailored to known examples without becoming better for real users.

We should not consider golden datasets as static input-output pairs. Depending on the application, a test case may contain an input, source documents, expected facts, forbidden claims, acceptable tool calls, a scoring rubric, and an optional reference response.

Automated Metrics: Fast but Limited Checks

Despite the non-deterministic nature, automated metrics are also useful in testing LLMs. However, automated metrics use code rather than another person or model to inspect an answer. They are also cheap, fast, and repeatable. Therefore, they are useful for running large numbers of tests.

Exact match checks help where the output has to precisely match an expected value. It works well for constrained extraction tasks such as returning a country code, classification label, or database identifier. But it works poorly for open-ended natural-language answers.

Regular expressions and schema validators help check whether an output follows a required structure. For example, an application may require valid JSON containing fields such as customer_id, issue_type, and priority. In such a case, a validator can reliably check whether those fields exist and whether their values use the expected types.

Programmatic checks can help verify URLs, citations, numerical ranges, required phrases, banned phrases, word limits, or whether specific product IDs exist in a database.

BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation) compare the words or word sequences in a generated answer with those in a reference answer. BLEU became popular for machine translation, while ROUGE has often been used for summarization. They can be useful for certain large-scale comparisons. But they mainly deal with measuring textual overlap. We can have a response with a different arrangement of words that still preserves the meaning.

Other semantic similarity metrics compare the meanings of responses through embeddings or specialized evaluation models. These metrics are more flexible than word overlap, but we still cannot treat similarity the same as correctness. An incorrect answer about the right subject can still be semantically similar to the reference answer.

The main limitation is that automated metrics generally measure a narrow, observable property. They should be used for things that code can check reliably.

What Does LLM-as-a-Judge Mean?

LLM-as-a-Judge involves sending one language model’s answer to another language model and asking that model to evaluate it based on specific criteria.

For example, consider an RAG assistant that answers a question using retrieved company documents. The judge may receive:

• The original question.

• The documents supplied to the assistant.

• The assistant’s answer.

• The evaluation rubric.

• An optional reference answer.

The judge LLM can then assess qualities such as relevance, factual support, completeness, clarity, and compliance with instructions. For reference, a simplified judge prompt might say:

Evaluate the response using only the supplied policy.

Score each category from 1 to 5:

Accuracy:
Does every factual claim agree with the policy?

Completeness:
Does the response answer every part of the question?

Relevance:
Does it remain focused on the customer’s request?

Instruction compliance:
Does it avoid making promises not supported by the policy?

Return JSON with the scores, a short explanation, and the exact unsupported claims, if any.

We can store the output from the judge model to compare with previous evaluation runs.

This approach is far more flexible than exact matching because the judge can recognize that differently worded responses express the same idea. It can also identify subtle failures such as answering only half of a question or introducing a claim that might not be present in the evidence.

Different Ways a Judge Model Can Evaluate an Answer

There are several common methods for judging an answer. Let us look at each of them in detail.

Point-based Scoring

In this approach, the judge model assigns scores such as 1 to 5 for different quality dimensions. This results in numbers that are easier to track over time as we refine the model under testing.

The main drawback of this approach is that the meaning of a score may be inconsistent. We need to define every score and its rubric properly. For example, consider the following scores:

• 5: Every factual claim is supported by the provided documents.

• 4: The main answer is supported, with one insignificant unsupported detail.

• 3: The answer is mostly correct but contains a meaningful unsupported claim.

• 2: Several important claims are unsupported or incorrect.

• 1: The central conclusion contradicts the documents.

Pass or Fail Classification

The judge model can also decide whether an output meets a minimum standard. We can use it for deployment gates, especially when the requirement is crystal clear. For example, “the answer must not contradict policy, expose personal information, or produce invalid instructions.”

A pass-or-fail result is simple, but it hides smaller changes. A score can decline from excellent to barely acceptable without crossing the failure boundary.

Pairwise Comparison

In this method, the judge model receives two answers and decides which one is better. For example, it may compare the response from the current production prompt with the response from a proposed prompt.

Pairwise comparison is often easier than assigning an absolute score. We can decide whether answer A is more useful than answer B more consistently than deciding whether one answer deserves a score of 3 or 4 out of 5.

To make this method better, the order of the answers should sometimes be reversed. Judge models can have a position bias and may start favoring whichever response appears first or second.

Error Identification

Instead of merely producing a score, the judge model can also identify specific problems.

The judge model can list unsupported claims, unanswered parts of the question, irrelevant sections, or violated instructions. This is quite useful during the development phase because it explains why a score has changed from one value to another.

Human Evaluation and Calibration

Despite all the advancements in testing, the ultimate test of a model’s ability depends on human evaluation.

Human evaluation involves people reviewing model outputs using the same rubric. This is why domain experts are really important when the correctness of an answer depends on legal, medical, financial, scientific, or company-specific knowledge.

However, since humans are expensive and slow, they cannot inspect every answer. Their best role is often to calibrate things. To make this work, human reviewers rate a representative sample. The results are compared with the ratings of a judge LLM.

For example, let’s say experts mark 100 answers as pass or fail. The same answers are evaluated by the judge model. The exercise reveals how often the judge agrees with the experts, which types of failure it misses, and whether its passing threshold needs adjustment.

Human reviewers can also disagree with one another. This usually means that the rubric is a little ambiguous or that the task is genuinely subjective. We should, therefore, measure reviewer agreement before treating human scores as a perfect reference point.

Human evaluation for LLMs remains highly important for:

• Creating and validating the first rubric.

• Reviewing high-risk failures.

• Evaluating new kinds of requests.

• Checking whether automated scores reflect real usefulness.

• Investigating disagreement between different evaluators.

• Periodically auditing the judge for drift.

The Evaluation Stack

Having gone through all the methods of LLM evaluation, we can now look at the overall evaluation stack.

We should understand this proposed evaluation stack as a set of complementary layers rather than a strict ladder where one layer replaces another.

• At the bottom are conventional software tests. These are used to verify deterministic components such as permissions, tool schemas, database writes, and calculations.

• Golden datasets provide repeatable scenarios on which the entire LLM application can be tested.

• Automated checks inspect properties that can be measured reliably, including valid JSON, required fields, exact extracted values, citation structure, and latency.

• LLM judges evaluate qualities that require the model to interpret meaning. We can use it to assess whether an answer is relevant, complete, supported by evidence, and compliant with a detailed rubric.

• Human reviewers calibrate the judge and assess cases where the consequences or subjectivity are too great to rely entirely on automation.

• Production monitoring completes the stack. A test set can never anticipate every real request, so production signals are needed to reveal emerging failures.

Conclusion

We should not treat LLM evaluation as a search for a perfect accuracy number. It is a system for building confidence in a model.

Golden datasets make critical situations repeatable. Automated metrics quickly check narrow and objective properties. LLM judges assess meaning according to a rubric. Human reviewers calibrate those judges and handle difficult decisions.

The most reliable question is therefore not simply, “Is the model healthy?”

It is the idea that across the situations that matter, does the complete application stay accurate, useful, safe, reliable, fast, and affordable? Also, is there enough evidence to detect when that changes?

This is the real purpose of LLM evaluation.