I built an evaluation suite for an MCP server — the kind that hands a product's data to an AI agent as a set of tools. This is the shape it ended up with and the reasoning behind each piece, with the product specifics stripped out. Not a framework you can install; a structure you can copy.
The short version: MCP eval is messier than agent eval, and it is messier for a reason you can design around once you name it.
The simplest way to think about the suite is: deterministic checks test whether the server works; agent runs test whether a model can navigate it; an LLM judge reviews whether the server's design helped or hindered that navigation.
Agent eval is well-trodden. Fixed task, known answer, score the output, run it enough times to average out the noise.
MCP breaks that in a specific way: the server is deterministic and the caller isn't yours.
Your MCP server is ordinary software: same arguments in, same bytes out. Most of its behavior can be tested deterministically, though protocol and deployment checks still need their own layer. All the nondeterminism lives in the model on the other end — a model you did not train, cannot pin, and which the user replaces the day they switch clients.
So when an agent gives a bad answer through your tools, at least four different things could be true: Your tool returned the wrong data. Your bug. The agent never found the right tool. Your discovery surface. The agent found it and called it wrong. Maybe your names, maybe their model. The agent had good data and reasoned badly. Not your product at all.
One pass/fail score collapses all four into a number you cannot act on. Worse, the number moves when the vendor ships a new model and you change nothing.
Two consequences drove everything else I built: Separate the layers by what they need to run, not by what they're called. Grade properties that name a repair site. "The agent scored 7/10" is a vanity metric. "The agent couldn't tell from the name what this tool returns" is a rename.
Four layers. They fail for different reasons and cost different amounts, which is exactly why they stay separate.
| Layer | Asks | Needs | Can it fail a build? | |---|---|---|---| | L0 unit | does the code do what it says | nothing | yes | | L1 contract | does a deployed server return the right thing | a server + a token | yes | | L2 navigation | does an agent reach for the right tool | a server, an agent CLI, money | yes, grudgingly | | L3 answer review | did our design let the agent reach a defensible answer | a stored run + a judge | no |
L0 is normal testing. Nothing interesting, and that's the point — most of what people try to catch with expensive agent runs is a unit test.
L1 is the one teams skip and shouldn't. It speaks the protocol to a deployed server with no agent involved: does tools/list match in both directions, do IDs come from discovery rather than fixtures, do windowed tools echo the window they used, and do errors stay errors? It's free, deterministic, and catches contract drift before an agent run.
L2 is where an actual agent gets an English question and you grade the path it took. This is the expensive, flaky layer.
L3 reads a run that already happened. That offline property is the best decision in the whole suite: changing a rubric re-scores all the history you already have instead of costing another sweep, and judging a stored run costs cents against the sweep's dollars. Iterate on the grader for the price of coffee.
Here's the mistake I nearly made: grade L2 by parsing what the agent CLI says it did.
Don't. The agent is the thing under test. Its summary of its own behaviour is a generated artifact like any other — it drops calls it made and regretted, compresses retries, and occasionally describes a call it never issued.
Your server already knows. Log one row per tool call and read the trajectory back out of that.