# How I evaluate AI coding agents: the rubric taxonomy

If you ship an AI coding agent into a team's workflow, "did this output look right?" is not enough. You need rubrics: scorable properties that say what good and bad mean before you grade anything. I keep 48 of them, public. Here's how I organize them.

This is the hub post for the "Evaluating AI coding agents" series. Each later post goes deep on one bucket or one rubric.

## What is each rubric trying to catch?

A rubric is a property an output should satisfy (see [Post B](https://voloshin.net/what-model-checking-taught-me-evaluating-ai-coding-agents) for the model-checking framing). Each one has a score scale and six paired calibration samples: three "good" examples that should pass and three "bad" ones that should fail. The pair keeps the judge honest in production.

The 48 rubrics fall into three buckets: cross-cutting checks, harness checks, and per-workflow checks.

## The three buckets

### Base rubrics: cross-cutting

These check fundamentals that apply across many outputs: evidence-backed claims, schema-conformance, spec-following. One bad calibration sample is named `opinion-without-evidence` (score 1.0 / 5.0); it catches an output that asserts a position with no supporting code, doc link, or data. Most outputs are scored against several base rubrics, not just one.

### Meta-tool rubrics: harness checks

These check the harness itself, not the developer-facing output. When a workflow spawns a sub-agent, did the spawn payload conform to the schema? When a hook fired, did it produce the expected event log? Small bucket, but the rubrics that catch silent breakage in the agent machinery.

### Per-workflow rubrics: one per major workflow

One rubric per major workflow: `analyze`, `code-review`, `spike`, `feature-design`, `develop`, `bugfix`, and so on. Each is tailored to what that workflow should produce. A bad sample for `code-review` is `mixed-with-security-scan.score-1.4.md`: a review that drifted into a security audit and lost focus. A good `spike` is `auth-openidconnect-vs-oauth.score-4.4.md`: a clean, decisive go/no-go.

## How I use the taxonomy

When I add a new workflow, I ask three questions. Which base rubrics already apply? Does the harness call any new sub-agents that need a meta-tool rubric? What is the one per-workflow rubric tailored to this workflow's output? If a workflow cannot answer the third, it is not ready to ship.

## FAQ

**Why three buckets, not themes like security or style?** Themes cross all three buckets. A rubric belongs to exactly one bucket by what it scores against: the harness, a workflow, or any output.

**Where do the six paired calibration samples come in?** Post D walks one rubric in full: its score scale and the boundary each pair draws.

The 48 rubrics live in the public repo: [github.com/alex-voloshin-dev/ai-skills/tree/main/plugin/eval](https://github.com/alex-voloshin-dev/ai-skills/tree/main/plugin/eval).