Evaluating AI Agents in Production: LLM-as-a-Judge, Its Biases, and the Open-Source Observability Stack

Your AI agent passes every demo you run it through. Then it ships, and three weeks later a customer reports it confidently gave the wrong refund policy, or looped through the same tool call four times before giving up. Nobody caught it because nobody was actually measuring anything after launch, they were just watching it work in a conference room and calling that validation. The gap between "it worked when I tried it" and "it works reliably in production" is where most agent evaluation programs are supposed to live, and it's worth understanding both how that measurement actually happens and where it quietly lies to you.

Why "Did It Work" Isn't a Metric

Traditional software has deterministic outputs, so you write an assertion and it either passes or fails. Agent outputs are open-ended text, tool-call sequences, and multi-step reasoning chains, so there's rarely a single correct string to match against. The industry's answer to this is LLM-as-a-judge: you use a second (usually stronger) language model to score the first model's output against a rubric, instead of a human doing it by hand. It's cheap and fast enough to run on every production trace, which is exactly what made online evaluation at scale possible in the first place.

How LLM-as-a-Judge Actually Works Under the Hood

There are two common setups. Pointwise (or "single output") judging feeds the judge model one response plus a rubric, a description of what "helpful," "correct," or "on-brand" means for your use case, and asks it to output a score, usually 1-5 or pass/fail. Pairwise judging shows the judge two candidate responses side by side and asks which one is better, which tends to be more stable because relative comparisons are an easier task than absolute scoring. Either way, the mechanism is the same: a prompt containing the rubric and the output-to-be-judged goes to an LLM, and the LLM's own next-token prediction becomes your evaluation metric. That's the part worth sitting with, because it means every bias baked into how that judge model generates text also shows up in how it judges.

The 80% Number Everyone Quotes, and What It's Hiding

The evaluation field's founding claim, from the original 2023 LLM-as-a-judge research, was that GPT-4 agreed with human evaluators more than 80% of the time on open-ended questions. That number is why the whole approach took off. But it was measured on relatively straightforward comparisons, and newer, harder benchmarks built specifically to stress-test judges tell a rougher story. Adaline's analysis of judge reliability reports that on JudgeBiasBench, a benchmark purpose-built around adversarial bias tests, frontier models exceeded 50% error rates. The RAND Corporation ran its own Judge Reliability Harness across multiple models and concluded, in Adaline's summary, that no judge was uniformly reliable across benchmarks. The 80% figure and the 50%+ error rate aren't contradicting each other so much as measuring different things: agreement on easy cases versus failure under conditions designed to expose a judge's blind spots. Production traffic contains both kinds of cases, and you only find out which one you're looking at by testing for it directly.

Four Biases That Show Up in Real Judge Pipelines

Four specific failure modes recur across the research, and each has a concrete production symptom:

• **Verbosity bias**: judges systematically favor longer responses as a proxy for thoroughness, even when the extra length adds nothing. If your agent's scores went up right after you told it to "explain your reasoning more," check whether it actually got better or just got wordier.
• **Position bias**: in pairwise setups, the judge favors whichever response appears first (or second) more often than chance alone explains. A study by Lin Shi covering roughly 150,000 evaluation instances across 15 judges confirmed this isn't random noise, it's structural. The fix is trivial but frequently skipped: run every pairwise comparison twice with the order flipped and check for disagreement.
• **Self-preference and family bias**: judges rate outputs from their own model family more favorably, documented at EMNLP 2025. If you're judging a Claude-generated agent response with a Claude-based judge, or a GPT-based agent with a GPT-based judge, you have a built-in thumb on the scale.
• **Domain misalignment**: a judge tuned on general chat quality doesn't automatically transfer to judging, say, whether a refund amount is arithmetically correct. Adaline's recommendation is to treat a divergence rate above 20-25% between the judge and spot-checked human review as the trigger to recalibrate the rubric for that specific domain, not to keep trusting the automated score.

None of this means LLM-as-a-judge is unusable, it means it needs the same skepticism you'd apply to any single automated metric: useful in aggregate, dangerous to trust blindly on any individual case.

The Open-Source Observability Stack: Langfuse and Arize Phoenix

Judging a single output only matters if you can see the full trace it came from, every tool call, retrieval, and intermediate reasoning step in an agent's run, which is what observability platforms are for. Paid platforms like Braintrust, Maxim, and LangSmith bundle this well, but two credible open-source options avoid vendor lock-in entirely.

Langfuse is MIT-licensed at its core (the enterprise `ee` folders are the exception), self-hosts via Docker Compose in a few minutes, and covers tracing, prompt version management, and evaluation, including LLM-as-a-judge, code-based evaluators, and manual labeling, in one system with 20+ framework integrations. Arize Phoenix is source-available under the Elastic License 2.0 rather than a fully permissive license, and its real distinction is architectural: it's built directly on OpenTelemetry, using the OpenInference span format, so it works with LangChain, LlamaIndex, DSPy, CrewAI, or a hand-rolled agent loop without vendor-specific instrumentation. If you're already emitting OpenTelemetry traces elsewhere in your stack, Phoenix slots in as another consumer of the same data rather than a separate logging system.

Either one gives you what a pure LLM-as-a-judge script running in isolation can't: the ability to click from a bad score back into the exact sequence of tool calls and retrieved chunks that produced it. That trace-level visibility is the same discipline we cover for the retrieval side of an agent's pipeline in RAG vs fine-tuning: how to actually decide, where hit-rate-at-k is the retrieval-specific version of the same "don't trust the aggregate number, look at the failure" principle.

Building a Pipeline That Doesn't Lie to You

Start with a small, hand-labeled test set, 50 to 100 real production traces scored by an actual human, before you trust any judge model's verdict on the rest of your traffic. Run every pairwise judgment with the response order flipped and flag disagreements. Use a judge model from a different provider or family than the one generating your agent's outputs, to avoid self-preference bias by construction rather than by hoping it doesn't happen. Track the divergence rate between judge and human spot-checks on an ongoing basis, and treat anything above 20-25% as a signal that the rubric, not the agent, needs work. None of this is exotic engineering, it's closer to the discipline of unit-testing a nondeterministic system: you can't eliminate the noise, but you can measure it, bound it, and stop treating a single automated score as ground truth. We build this evaluation discipline into every agent project in AI Product Development Bootcamp, because an agent that looks good in a demo and an agent that survives three months of real traffic are graded on entirely different rubrics.