Evaluation & Quality Monitoring

A RAG support answer can cite a valid document and return valid JSON while changing a refund window from 14 days to 30. Check the extracted value, citation, response structure, and abstention separately so a regression report identifies the broken behavior.

This chapter builds an offline Python comparison that reports individual failures and exits unsuccessfully when a previously passing case regresses. Every policy, input, and output below is synthetic teaching data: it is neither JR Academy refund policy nor a measured model result. No model calls, network connection, or API key are required.

1. Define a task you can grade

The example application extracts a refund window from a supplied policy and returns refund_days, citations, and abstain. Its contract requires a null value and abstention when no policy is supplied. That is a rule for this source-bound extraction task, not a universal requirement to refuse whenever a chatbot has no retrieval results.

These three cases demonstrate mechanics, not coverage. A real dataset needs your user tasks, languages, document versions, conflicting sources, and observed failures. Reaching 50 examples does not establish adequacy. Keep a stable regression set, add a challenge set, and reserve cases that were not used to tune the prompt.

2. Run a regression check that fails

Save this as eval_demo.py. It requires Python 3.8 or later and only the standard library. BASELINE and CANDIDATE are handwritten outputs for testing the grader and gate, not for measuring model capability.

import argparse
import json
import sys

# Synthetic fixtures: these are NOT model responses or a real refund policy.
CASES = [
    {"id": "answer", "days": 14, "source": "policy-v1"},
    {"id": "changed-policy", "days": 7, "source": "policy-v2"},
    {"id": "no-context", "days": None, "source": None},
]
BASELINE = {
    "answer": {"refund_days": 14, "citations": ["policy-v1"], "abstain": False},
    "changed-policy": {"refund_days": 7, "citations": ["policy-v2"], "abstain": False},
    "no-context": {"refund_days": None, "citations": [], "abstain": True},
}
CANDIDATE = {
    "answer": {"refund_days": 30, "citations": ["policy-v1"], "abstain": False},
    "changed-policy": {"refund_days": 7, "citations": ["missing-doc"], "abstain": False},
    "no-context": {"refund_days": 14, "citations": [], "abstain": False},
}


def grade(case, output):
    required = {"refund_days", "citations", "abstain"}
    if not isinstance(output, dict) or set(output) != required:
        return ["schema"]
    days = output["refund_days"]
    citations = output["citations"]
    if (days is not None and type(days) is not int) or (
        not isinstance(citations, list)
        or any(not isinstance(item, str) for item in citations)
        or type(output["abstain"]) is not bool
    ):
        return ["schema"]
    errors = []
    if days != case["days"]:
        errors.append("value")
    expected_citations = [] if case["source"] is None else [case["source"]]
    if citations != expected_citations:
        errors.append("citations")
    if output["abstain"] != (case["days"] is None):
        errors.append("abstention")
    return errors


def evaluate(outputs):
    return {case["id"]: grade(case, outputs.get(case["id"])) for case in CASES}


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--candidate", choices=["broken", "fixed"], default="broken")
    args = parser.parse_args()
    baseline = evaluate(BASELINE)
    candidate = evaluate(BASELINE if args.candidate == "fixed" else CANDIDATE)
    regressions = [key for key in baseline if not baseline[key] and candidate[key]]
    print(json.dumps({"baseline": baseline, "candidate": candidate,
                      "regressions": regressions}, indent=2))
    # This demo gate protects known passing cases; it is not a safety certification.
    return 1 if regressions else 0


if __name__ == "__main__":
    sys.exit(main())

Run the deliberately broken version:

python3 eval_demo.py --candidate broken
echo $?

In a macOS or Linux shell, the final line should be 1. All three baseline entries are []. The candidate reports ["value"] for answer, ["citations"] for changed-policy, and ["value", "abstention"] for no-context. All three IDs appear in regressions.

Check the success path too:

python3 eval_demo.py --candidate fixed
echo $?

Now regressions is [] and the exit code is 0. The fixed option reuses the known-good fixture; it does not repair or call a model. The broken demonstration should fail a CI step. Connect real candidate outputs before interpreting that step as evidence about an actual application change.

3. Test the grader, not just the application

grade() rejects missing fields, extra fields, and incorrect types. It uses type(days) is int because Python otherwise accepts booleans as integers. The citation rule permits only the document ID assigned to the case, catching nonexistent sources.

An existing citation ID does not establish that the source supports the answer. This task has an exact integer label, so a direct comparison can catch a false value. Free-text answers require checking factual claims against the source. Multiple valid sources, legitimate paraphrases, and richer JSON structures need different grading rules; the exact comparisons here can reject valid alternatives.

Passing three fixtures establishes neither injection resistance nor privacy protection nor production reliability. When connecting a model, make exceptions, timeouts, and missing outputs visible failures. Preserve actual outputs and version information instead of hiding errors behind empty strings. Repeat stochastic tasks and compare baseline and candidate on the same inputs.

4. Calibrate a model judge before trusting it

Use code for format checks. For open-ended factual support or task completion, add human review or a model grader. Compare judge decisions with human labels, inspecting both incorrect answers it accepts and correct answers it rejects. Recalibrate after changing the judge or rubric and record the version; a monthly calendar event is not a reason to change the scoring standard. OpenAI's evaluation guide covers human calibration, explicit rubrics, and model-grading biases.

A factual-support rubric might use supported for claims grounded in the supplied sources, unsupported when at least one claim lacks support, and uncertain for insufficient or conflicting evidence requiring review. Ask for the relevant answer and source excerpts, then verify those excerpts actually occur in the inputs. Treat evaluated answers and retrieved documents as data, not instructions that may override the rubric.

An arbitrary score of 4/5 is not a universal release criterion. Define unacceptable failures, then choose gates using task risk, human judgments, and historical baselines. A cheaper judge is useful only if its decisions remain adequate for your task. Anthropic's engineering discussion describes the limitations of code-based, model-based, and human graders.

5. Connect offline reports to production monitoring

Make outcomes traceable to an application version, prompt revision, model identifier, and retrieval-index version. Latency, token usage, retries, and error categories reveal engineering problems. Task completion and reviewed failure examples help assess answer quality. Neither HTTP 200 nor a thumbs-up rate establishes factual correctness.

Prefer necessary identifiers and aggregates in logs. Before retaining questions, retrieved passages, or responses, handle personal and sensitive information and define access controls, retention periods, and the applicable consent boundary. Do not copy raw customer data into a public evaluation repository.

Before a canary release, specify an observation window, stop conditions, and a rollback version. Compare equivalent task categories: a shift toward easier questions can raise an aggregate score without improving the application. Treat timeout and cost limits as separate gates; a higher average quality score does not cancel an engineering failure.

6. Practice and acceptance

1. Pass missing fields, refund_days=True, duplicate citations, and a non-object output to grade(). Confirm visible failures rather than crashes.
2. Add a conflicting-sources case. Define the product expectation before writing a fixture or grader; explain when answering or human escalation is appropriate.
3. Connect your own redacted output snapshots. Record dataset, application, prompt, model, and grader versions. Report failure categories and examples, not only an overall score.
4. Ask another reviewer to inspect labels and failure explanations. If a valid answer is rejected, repair the grading rule before concluding that the candidate regressed.

You have completed the exercise when you can reproduce both exit codes, explain each failure, and identify risks the grader does not cover. This chapter provides no model ranking or production safety guarantee.