Agent Evaluation: LLM-as-Judge and Trace Analysis

Lesson 10 — how to measure agent quality, catch failures, and prove it works

Why Agent Evaluation Is Hard

Traditional software tests use exact assertions: assert result == expected. Agents produce nondeterministic output — the same input can produce different (but equally valid) answers. You can't assert exact text.

Microsoft's guidance is explicit: "Agent outputs are nondeterministic, so use scoring rubrics or language-model-as-judge evaluations rather than exact-match assertions."

Source: Microsoft Azure — AI Agent Orchestration Patterns

LLM-as-Judge

You use one LLM to evaluate another LLM's output. This is the standard approach in production AI:

# LLM-as-Judge for your BA Work Agent
from langchain_openai import ChatOpenAI
import os

judge_llm = ChatOpenAI(model="deepseek-chat", api_key=os.getenv("DEEPSEEK_API_KEY"),
                       base_url="https://api.deepseek.com", temperature=0)

def evaluate_agent_output(goal: str, agent_output: str) -> dict:
    """Use LLM to evaluate agent output quality."""
    judge_prompt = f"""You are evaluating an AI agent's output.

Goal: {goal}
Agent output: {agent_output}

Score each criterion 1-5:
1. ACCURACY: Are the facts correct? No hallucinations?
2. COMPLETENESS: Did it address the full goal?
3. TOOL USAGE: Did it use appropriate tools (not just guess)?
4. CLARITY: Is the output clear and actionable?

Respond in JSON:
{{"accuracy": N, "completeness": N, "tool_usage": N, "clarity": N, 
  "overall": N, "issues": "description of problems"}}"""
    
    response = judge_llm.invoke(judge_prompt)
    return response.content

# Run evaluation on your agent's output
result = evaluate_agent_output(
    goal="Read JIRA export and suggest top 5 priorities",
    agent_output="Based on the tickets..."  # paste your agent's actual output
)
print(result)

The 4 Failure Modes to Detect

Trace Analysis

Every agent run produces a trace — the full sequence of Thought/Action/Observation steps. LangSmith (LangChain's observability platform) captures these automatically. Without LangSmith, use verbose=True in AgentExecutor.

Practice Drill

1. Run your ba-work-agent and save the full verbose output to a file
2. Use the evaluate_agent_output function above to score the output
3. Check the trace for any of the 4 failure modes. Is your agent healthy?
4. Run the agent 3 times with the same input. Are the outputs different? (They should be — nondeterminism is normal)