Alex Voloshin — Agentic-dev tooling, evals & tri-vendor parity

How I evaluate AI coding agents: the rubric taxonomy

Alex Voloshin — Sat, 06 Jun 2026 02:00:53 GMT

If you ship an AI coding agent into a team's workflow, "did this output look right?" is not enough. You need rubrics: scorable properties that say what good and bad mean before you grade anything. I keep 48 of them, public. Here's how I organize them.

This is the hub post for the "Evaluating AI coding agents" series. Each later post goes deep on one bucket or one rubric.

What is each rubric trying to catch?

A rubric is a property an output should satisfy (see Post B for the model-checking framing). Each one has a score scale and six paired calibration samples: three "good" examples that should pass and three "bad" ones that should fail. The pair keeps the judge honest in production.

The 48 rubrics fall into three buckets: cross-cutting checks, harness checks, and per-workflow checks.

The three buckets

Base rubrics: cross-cutting

These check fundamentals that apply across many outputs: evidence-backed claims, schema-conformance, spec-following. One bad calibration sample is named opinion-without-evidence (score 1.0 / 5.0); it catches an output that asserts a position with no supporting code, doc link, or data. Most outputs are scored against several base rubrics, not just one.

Meta-tool rubrics: harness checks

These check the harness itself, not the developer-facing output. When a workflow spawns a sub-agent, did the spawn payload conform to the schema? When a hook fired, did it produce the expected event log? Small bucket, but the rubrics that catch silent breakage in the agent machinery.

Per-workflow rubrics: one per major workflow

One rubric per major workflow: analyze, code-review, spike, feature-design, develop, bugfix, and so on. Each is tailored to what that workflow should produce. A bad sample for code-review is mixed-with-security-scan.score-1.4.md: a review that drifted into a security audit and lost focus. A good spike is auth-openidconnect-vs-oauth.score-4.4.md: a clean, decisive go/no-go.

How I use the taxonomy

When I add a new workflow, I ask three questions. Which base rubrics already apply? Does the harness call any new sub-agents that need a meta-tool rubric? What is the one per-workflow rubric tailored to this workflow's output? If a workflow cannot answer the third, it is not ready to ship.

FAQ

Why three buckets, not themes like security or style? Themes cross all three buckets. A rubric belongs to exactly one bucket by what it scores against: the harness, a workflow, or any output.

Where do the six paired calibration samples come in? Post D walks one rubric in full: its score scale and the boundary each pair draws.

The 48 rubrics live in the public repo: github.com/alex-voloshin-dev/ai-skills/tree/main/plugin/eval.

What model checking taught me about evaluating AI coding agents

Alex Voloshin — Mon, 25 May 2026 20:01:23 GMT

A unit test asks one question: did this run pass? That works when code is deterministic. An LLM coding agent is not. The same prompt produces different code each time, so one passing run proves almost nothing.

Model checking, a technique from formal verification, offers a better lens, and it changed how I write eval rubrics.

What does model checking actually do?

It verifies a system by checking properties across all of its possible behaviors, not by running one example. The hard part is never the checking. It is specifying the property: stating precisely what "correct" means before you look at any output.

Two kinds of property matter. A safety property says something bad never happens, and a violation is one concrete trace you can point at. A liveness property says something good eventually happens. Name the property first, and the rest follows.

Why does this fit AI coding agents?

A nondeterministic agent behaves like a system with a huge space of possible runs, not a function with one output. A pass/fail test samples a single run and tells you nothing about the others.

So here is the reframe: an eval rubric is a property. Writing a good rubric is specifying a property precisely, the same hard part as in model checking. Safety properties become "never" rubrics: never commit a secret, never edit a file outside scope. Liveness properties become "eventually" rubrics: the spec eventually gets fully handled.

A worked example

Take the property "the agent never modifies a file outside the task's scope." That is a safety property: a bad thing that must never happen, and a violation is a single trace.

As an eval rubric it becomes scope-adherence: score 0 if any out-of-scope file was touched, full score if not. The calibration "bad" sample is the counterexample, a recorded run where the agent "helpfully" fixed an unrelated file. The "good" sample is a run that stayed in bounds.

Notice what changed. The rubric is no longer a vague "did it stay focused?" It is a precise property with a sharp violation condition. [ALEX: swap in a real rubric from ai-skills here if you have a sharper one.]

What to take from this

Name the property before you read the output. Without a stated property you are not evaluating, only reacting.
Sort each property into safety or liveness. "Never" and "eventually" score differently.
Treat your worst calibration sample as a counterexample. It shows the exact trace that breaks the property.

FAQ

Do I need formal methods to use this? No. You need the discipline, not the notation: state the property precisely, before grading.

Isn't a rubric still just one check? A unit test checks one run. A property holds, or fails, across many. That is the shift: from "did this pass?" to "does this property hold?"

The eval rubrics I build on this idea are public, in the repo: github.com/alex-voloshin-dev/ai-skills.

Three things that don't port between Claude Code, Codex, and Windsurf

Alex Voloshin — Sat, 23 May 2026 22:51:04 GMT

Claude Code, Codex, and Windsurf now look alike: all three have skills, hooks, and plugins. So it's natural to assume your setup ports between them. It mostly doesn't.

I maintain a repo that targets all three runtimes and keeps a parity matrix. Here are the three gaps that cost the most time, and what to do about each.

Feature	Claude Code	Codex	Windsurf
Hooks	`PreToolUse` runs before a tool and can block it	command hooks run; prompt/agent hooks parsed but skipped	Cascade Hooks fire after the action
Skill triggering	model-invoked from the description	implicit invocation is a toggle	rules and workflows
Distribution	`plugin.json` plus a marketplace git repo	its own plugin format	rules/workflows plus MDM

Will my hooks still block what they blocked before?

Not always. On Claude Code, a PreToolUse hook runs before the tool and can stop it. That is a real guardrail. On Windsurf, Cascade Hooks fire after the agent acts, so they can flag or revert, but they never prevent. Codex runs command hooks, but currently parses and skips prompt and agent hooks. A hook that blocks a dangerous command on one runtime only logs it on another.

Will my skills trigger the same way?

The SKILL.md body ports. The trigger does not. Claude Code auto-invokes a skill from its description. Codex makes implicit invocation a switch: turn allow_implicit_invocation off and the skill runs only when you type $skill yourself. Codex also reads skills from a different path. The skill still works. It just may never fire on its own.

Can I publish a plugin once and install it everywhere?

No. Claude Code, Codex, and Windsurf each have their own manifest, catalog, and versioning. One plugin means three manifests to maintain and three install paths to test. "Publish once, install everywhere" is a slogan, not a workflow. Budget distribution as a per-runtime cost.

What to do

Keep a parity matrix. Write down what you have verified on each runtime. Treat the rest as unknown, not assumed.
Re-test on every runtime, especially any hook that enforces a security rule.
Budget distribution per runtime, not once.

FAQ

Do skills written for Claude Code work in Codex? The SKILL.md body usually does. The discovery path and the invocation policy need adjusting per runtime.

Which runtime has the strongest hook guarantee? Claude Code. Its PreToolUse hook can block a tool call before it runs. Treat hooks on the other runtimes as observe-and-react.

The parity matrix I keep is public, in the repo: github.com/alex-voloshin-dev/ai-skills. If you run more than one of these runtimes, start your own.