Contract-driven · plugin-scored
A deterministic evaluation system for LLM-driven agent workflows. It checks a trace against a YAML-defined contract through six independent plugins — skills, context, tools, output, trace health, usage — and produces pass/fail evidence instead of an LLM judge's opinion.
$uv run run-verify --contract contracts/binding/binding_with_widget.yaml --trace-id c4739a28...
Before
Someone still has to read the trace, decide whether the agent actually satisfied the task, and repeat that judgment call for every model, every run, every regression — subjective, expensive, and rarely the same answer twice.
After
A Langfuse trace is normalized into a structured snapshot, then run through six plugins that check exactly what the contract declared: skills loaded, resources read, tools called, files changed — and nothing it didn't.
CONTRACT-DRIVEN
Expected skills, tools, resources, and outputs declared once, in YAML — versioned like code, not tribal knowledge.
EVIDENCE, NOT OPINION
Every check resolves against a specific tool call or file diff in the trace, never a model's say-so.
SIX INDEPENDENT PLUGINS
Skills, input context, tool calls, output, trace health, and resource usage — each with its own pass/fail breakdown.
BINARY SCORING
SWE-bench style: every declared check must pass. No partial credit to soften a violation.
Pipeline
run-verify --contract <yaml> --trace-id <id> runs all four and prints the breakdown.
01
A workflow contract defines expected skills, tools, resources, and outputs in YAML.
02
The trace is fetched from Langfuse and normalized into a structured snapshot.
03
Six plugins check the snapshot against the contract, each with its own evidence.
04
Plugin outputs aggregate into one score and a verification report.
Where it fits
Each command is independently useful — chain them for a full benchmark run, or drop in wherever you already are.
| Plugin | Checks |
|---|---|
| skills_loaded | Required skill(s) loaded and succeeded; no extra skills beyond required/optional. |
| input_context | Each declared resource was actually read, with its terms present in some tool call. |
| tool_calls | Every required tool appears in the trace; no forbidden tool does. |
| output | Each declared output was created/updated/deleted as specified; no unrelated file changes. |
| trace_health | No error state, no error spans, build validation passes when a Java service is in scope. |
| resource_usage | Duration, tokens, and cost reported for observability — never scored or failed. |
Worth knowing
generate-contract infers structure from one observed run — review the match clauses before trusting it.Quick start
01
cd wm_agents_validator uv sync --extra dev cp .env.example .env # fill LANGFUSE_* values
02
uv run run-verify \ --contract contracts/binding/binding_with_widget.yaml \ --trace-id c4739a2868e2b7aca6430aeae2f7ea0a
03
uv run compare-traces \ --contract contracts/binding/binding_with_widget.yaml \ --trace-ids <trace-gpt4>,<trace-claude>,<trace-gemini> \ --out comparison.html
One self-contained HTML file — sortable table, per-plugin heatmap, no server required.
04
uv run generate-contract \ --trace-id c4739a2868e2b7aca6430aeae2f7ea0a \ --workflow screenshot_to_code \ --out contracts/new_workflow.yaml