Building evals that don't lie

Building an impressive AI demo takes an afternoon. Building an AI product that holds deterministic reliability, low hallucination rates and accurate tool selection for thousands of daily users requires an evaluation architecture.

In ordinary software, regression testing is binary — assertions pass or they fail. In generative AI the outputs are probabilistic, non-deterministic, and prone to silent degradation whenever prompts, retrieval parameters or model weights change underneath you.

Four pillars for evaluating production RAG

Evaluating a retrieval-augmented pipeline as a single score tells you nothing actionable. Four independent metrics tell you which half of the system broke:

• Context precision — were the retrieved chunks actually relevant? This measures noise in vector search.
• Context recall — did the retriever fetch every ground-truth fact needed to answer at all?
• Faithfulness — is every claim in the response supported by the retrieved context? This is your hallucination rate.
• Answer relevancy — does the output address the question without evasive fluff?

Splitting precision and recall from faithfulness is what separates "the model hallucinated" from "the retriever handed it the wrong three paragraphs and it summarised them faithfully". Those two failures look identical from the outside and have completely different fixes.

Quantifying faithfulness

An automated eval pipeline

The runner below scores a ground-truth dataset with an LLM-as-judge at temperature zero and returns a pass/fail gate. Temperature matters: a judge that varies its own scoring turns your regression signal into noise.

import os
import json
from typing import List, Dict
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

EVAL_PROMPT = """
You are an expert AI evaluation auditor. Evaluate faithfulness and relevancy.

[Query]: {query}
[Context]: {context}
[Output]: {output}
[Ground truth]: {ground_truth}

Evaluate:
1. Faithfulness (0.0-1.0): is the output strictly supported by the context?
2. Relevancy (0.0-1.0): does the output answer the query?

Return JSON: {{"faithfulness_score": float, "relevancy_score": float, "reasoning": "str"}}
"""


def evaluate_rag_pipeline(test_dataset: List[Dict]) -> Dict[str, float]:
    total_faithfulness = 0.0
    total_relevancy = 0.0
    total_tests = len(test_dataset)

    for item in test_dataset:
        prompt = EVAL_PROMPT.format(
            query=item["query"],
            context=item["context"],
            output=item["generated_output"],
            ground_truth=item["ground_truth"],
        )

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            response_format={"type": "json_object"},
            temperature=0.0,
        )

        result = json.loads(response.choices[0].message.content)
        total_faithfulness += result["faithfulness_score"]
        total_relevancy += result["relevancy_score"]

    avg_faithfulness = total_faithfulness / total_tests
    avg_relevancy = total_relevancy / total_tests

    return {
        "mean_faithfulness": round(avg_faithfulness, 3),
        "mean_relevancy": round(avg_relevancy, 3),
        "passed_gate": avg_faithfulness >= 0.95 and avg_relevancy >= 0.90,
    }


if __name__ == "__main__":
    results = evaluate_rag_pipeline(load_dataset())
    print(json.dumps(results, indent=2))

    if not results["passed_gate"]:
        raise ValueError("CI gate failed: evaluation scores below threshold")

Running it on every pull request

Wire the runner into GitHub Actions and the gate does the work: a prompt tweak, a chunking parameter change or a dependency bump cannot silently degrade production, because the pull request fails before anyone merges it.

That is the whole return on an eval suite. Not a dashboard — the ability to change something on a Friday and know by Monday whether it helped.