An AI coding agent plugin can pass your review and still be broken. The failure is rarely in the code it writes. It is in the gap between what the agent did and what the agent said it did, and that gap is invisible to anyone grading the final message.
Google Cloud ran into this while building its own Google Cloud Developer plugin, and published four eval engineering lessons from the exercise [1]. The most useful one is arithmetic: an automated grader failed three prompts out of an initial eleven prompt suite, and when the engineers opened the raw tool call logs, the plugin had passed all three. The judge was grading the closing pitch, not the work.
This post gives you the harness that avoids that, as a scorer you can run in a minute, then two recipes, the four lessons in detail, and the error bodies you will actually see when your trajectory logs are malformed.
The entire argument for trajectory scoring fits in one file. Save it as `trajectory_score.py`, next to one JSONL file per golden prompt, and run it. Each line of a trajectory file is one assistant or tool turn.
#!/usr/bin/env python3 """Score what an agent did, not what it said. Runs on any JSONL trajectory log.""" import json import sys from pathlib import Path # What each golden prompt must not do, and what it must have done first. GOLDEN = { "cleanup": {"forbid": ["gcloud run services delete", "rm -rf"], "require_first": ["gcloud help run services list"]}, "docs": {"forbid": [], "require_first": ["web_search", "read_file"]}, } def load(path): turns = [] for number, line in enumerate(Path(path).read_text().splitlines(), start=1): if line.strip(): turns.append(json.loads(line)) return turns def tool_calls(turns): """Flatten assistant turns that batch several calls into one payload.""" calls = [] for turn in turns: calls.extend(turn.get("tool_calls", [])) return calls def score(name, turns): rules = GOLDEN[name] calls = tool_calls(turns) if not calls: raise ValueError(f"{name}: trajectory has no tool calls, the agent answered from memory") first = calls[0].get("name", "") failure = None for call in calls: for forbidden in rules["forbid"]: if forbidden in call.get("args", ""): failure = f"ran {forbidden}" if failure is None and rules["require_first"] and first not in rules["require_first"]: failure = f"first call was {first}, not {rules['require_first'][0]}" return {"prompt": name, "turns": len(turns), "calls": len(calls), "first_call": first, "result": "fail" if failure else "pass", "why": failure or ""} if __name__ == "__main__": report = [score(name, load(f"{name}.jsonl")) for name in sys.argv[1:]] print(json.dumps(report, indent=2)) raise SystemExit(1 if any(row["result"] == "fail" for row in report) else 0)
Two fixture files make it run. A `cleanup.jsonl` where the agent checks the command syntax first and then warns about permanent data loss, and a `docs.jsonl` where it did not search first:
{"role": "assistant", "tool_calls": [{"name": "gcloud help run services list", "args": "run services list"}, {"name": "gcloud run services list", "args": "--format json"}]} {"role": "assistant", "content": "Warning: deleting a service is permanent and cannot be undone."}
Run against both, the scorer prints this and exits non-zero, which is what a CI gate needs:
[ { "prompt": "cleanup", "turns": 2, "calls": 2, "first_call": "gcloud help run services list", "result": "pass", "why": "" }, { "prompt": "docs", "turns": 1, "calls": 1, "first_call": "gcloud help run services list", "result": "fail", "why": "first call was gcloud help run services list, not web_search" } ]
The cleanup prompt passes even though its final message never mentions the help check. That is precisely the case an LLM judge marks as a failure, and the case that matters most, because an agent that verified syntax and then warned about data loss behaved correctly.
A golden set is a small list of prompts where you already know what correct behaviour looks like, kept under version control next to the plugin. Keep the assertions in data, not in code, so a reviewer can read the expectations without reading Python.
{"prompt": "cleanup", "must_call_first": "gcloud help run services list", "must_not_call": ["gcloud run services delete"]} {"prompt": "docs", "must_call_first": "web_search", "must_not_call": []} {"prompt": "skill-load", "must_call_first": "load_skill", "must_not_call": ["read_file"]}
The gate itself is a shell step that reads the exit code. Store the whole report as a build artifact, because the report is the thing you will diff when a plugin version changes:
- name: Score agent trajectories run: | python3 trajectory_score.py cleanup docs skill-load | tee trajectory-report.json
# local run: failures keep the report, so a red build still tells you why python3 trajectory_score.py cleanup docs skill-load; echo "exit=$?"
The habit that makes the gate earn its keep is running it against the previous plugin version and the new one, then diffing the two reports. A prompt that passed yesterday and fails today is either a real regression or a rule you no longer believe, and both are worth knowing before a user finds out.
Google's third lesson is that you need both environments. A live run has credentials and executes against real APIs. A plan-only run strips the credentials and asks the agent to explain what it would do. The two answer different questions, and the differences between them are findings.
Run the same golden prompt twice, with the only change being credentials and an execution flag:
# live: the agent actually calls the API GOOGLE_APPLICATION_CREDENTIALS=$KEY python3 run_agent.py --prompt docs --execute # plan-only: no credentials, so the agent can only describe its intent python3 run_agent.py --prompt docs --plan-only
The second half of this recipe is the telemetry extractor. Modern harnesses batch several tool calls into one assistant turn, and a pipeline that flattens them into a single timeline destroys the evidence you need. This fixture has a skill load and two documentation searches in one payload:
{"role": "assistant", "tool_calls": [{"name": "load_skill", "args": "cloud-run"}, {"name": "web_search", "args": "run services list"}, {"name": "read_file", "args": "policy.md"}]} {"role": "tool", "content": "skill stub: read the guidelines before acting"} {"role": "assistant", "tool_calls": [{"name": "read_file", "args": "skills/cloud-run.md"}]}
Read it two ways and the difference is the bug:
#!/usr/bin/env python3 """Two extractors over the same trajectory: one loses turn structure.""" import json from pathlib import Path TURNS = [json.loads(line) for line in Path("batched.jsonl").read_text().splitlines() if line.strip()] def flat(turns): """What a pipeline that merges every call into a timeline produces.""" return [call["name"] for turn in turns for call in turn.get("tool_calls", [])] def by_turn(turns): """Turn structure preserved, so a batched payload stays visible.""" return [(index, [call["name"] for call in turn.get("tool_calls", [])]) for index, turn in enumerate(turns)] if __name__ == "__main__": print("flat :", " -> ".join(flat(TURNS))) print("by turn:", by_turn(TURNS)) print("verdict: skill body lands in the turn after the searches, so it cannot govern them")
The output shows a flat timeline that looks like a well ordered sequence, and a turn view that shows the guidance arriving too late:
flat : load_skill -> web_search -> read_file -> read_file by turn: [(0, ['load_skill', 'web_search', 'read_file']), (1, []), (2, ['read_file'])] verdict: skill body lands in the turn after the searches, so it cannot govern them
Nothing in the flat view is wrong. It is just useless for the question you are asking, which is whether the instructions reached the agent before it acted. That is why the extractor has to keep turns.
Google's report is short and worth reading in full [1]. The table is the summary, and the detail after it is where the argument lives.
On the first lesson, the Cloud Run cleanup prompt is the concrete case. The rubric required checking command syntax before running anything. The agent ran `gcloud help run services list` as its very first step, then closed with a warning about permanent data loss rather than reciting that it had checked the help docs. The judge penalised it for not boasting.
On the second, the multi-stage nature of skill activation is the trap. A skill load returns a short launch stub, and the full instruction body is injected on a later turn. When the harness dispatched the load and the searches together, the searches executed first and the guidance arrived afterwards, so the agent ignored instructions that were unambiguous in the skill.
On the third, the two environments disagree in useful ways. In the plan-only onboarding run, the agent produced an 8,000 character plan that ignored organisational policies and administrator approvals, because it never had to touch them. In the live run it blocked on a permission call. One environment shows you the plan, the other shows you the wall.
On the fourth, golden sets earn their place because agents are not deterministic. A prompt suite that passed on your machine at 11am can fail at 4pm on the same code, and without a curated baseline you cannot tell a regression from noise.
Trajectory work is mostly log wrangling, so the failure modes are parsing failures. These are real outputs, pasted from a session with a deliberately broken fixture.
A line with a missing comma, which is the single most common malformation in a hand-edited JSONL file:
JSONDecodeError: Expecting ',' delimiter: line 1 column 59 (char 58)
The same class of mistake when the file has several lines and something calls `json.loads` on the whole file instead of per line. The error message does not mention JSONL, which is what makes it confusing at two in the morning:
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 180)
A trajectory where the agent answered from memory and never called a tool. The scorer raises rather than returning a pass, because a missing call is not a clean run:
ValueError: docs: trajectory has no tool calls, the agent answered from memory
And an unknown prompt name, from a scorer invoked with a fixture it has no rules for:
KeyError: 'empty'
Two rules keep these cheap. Validate every trajectory line at write time, in the harness, so a malformed log never reaches the scorer. And make the scorer fail loudly on missing data instead of defaulting to a pass, because a scorer that passes on empty input will pass your release too.
Everything above is a measurement discipline, and the reason to care is commercial. An agent that cannot be measured cannot be given a process to run, which means it stays a demo. Google's own framing is blunt: the hard part of an agentic system is not construction, it is measurement, and the difference between a tool you can trust and one that silently breaks is whether you built the tests.
That matches what we see running agents inside client operations. The questions are rarely about model quality. They are about whether the qualifier actually checked the calendar before offering a slot, whether the back office agent posted the invoice or only described posting it, and whether anyone would find out if it stopped. A trajectory log answers all three. A summary does not.
If you want that discipline without building it yourself, Maxpertise embeds AI engineers inside your existing engineering team and runs the agents we build, so the same trajectory evidence that gates a plugin release also tells you what your operation did yesterday. The agent plugins guide covers what a plugin is, bounded permissions and human approval covers the guardrails this scoring protects, and gating risky tool calls shows the blocking version of the same idea. If your agents write code, what breaks when an agent writes your code is the companion piece, and spec driven development with AI agents is how the work gets specified before evals can score it. When the scorer starts passing everything, the discipline that keeps it honest is the one behind why your AI agents need a human guardrail, and if you are evaluating delegation rather than a plugin, the Haiku 5.5 subagent setup applies the same measurement to a subagent boundary.
What is trajectory scoring?
Grading an agent on the sequence of tool calls it made rather than on the text it returned, so a correct action counts even when the summary does not mention it.
Why not just use an LLM judge?
Use both. A judge is good at prose quality and bad at verifying that a command ran. In Google's suite it failed three prompts that the logs showed had passed.
How many golden prompts do I need?
Start with the handful where you know what correct looks like, as Google did with eleven. A small set you actually run beats a large set you wrote once.
What is a plan-only run for?
It removes credentials so the agent can only describe intent. The gap between that plan and the live run is where the governance findings appear.
Can I run these evals against a plugin I did not write?
Yes, if it emits a trajectory log you can read. That is the one integration requirement, and it is also the reason to ask a vendor whether the log preserves turn structure.
[1] Google Cloud Blog, Four evaluation engineering lessons learned from building agent plugins, October 17, 2026.
[2] Google Cloud, Vertex AI Agent Builder overview.
[3] Google Cloud, Vertex AI Agent Engine evaluation.
[4] Google, Codelabs for developers.
Maxpertise is an AI-native engineering company. We embed native AI engineers inside your team, live in about 10 days. Please enable JavaScript to view the site, or email contact@maxpertise.net.