agent-tool-pr-reviewer

CLI that reviews the current git branch’s diff against a base ref using a single Pydantic AI call. Emits a typed findings.json plus a human-readable review-output.md under <repo>/.ai-review/runs/<timestamp>/.

Pairs with the pr-review skill in the agent-skills Claude Code plugin marketplace, which runs this CLI and surfaces blocker/high findings to the user.

Install

uv tool install --editable D:/agent-tool-pr-reviewer

Verify:

agent-tool-pr-reviewer --version    # 0.5.3

The default model is openrouter:google/gemini-2.5-pro, which expects OPENROUTER_API_KEY in the environment. See “Recommended models” below for the rationale and alternatives.

Quick start

From inside any repo with at least one rule file:

mkdir .ai-review
cat > .ai-review/no-bare-fences.md <<'EOF'
---
description: All fenced markdown code blocks must specify a language.
---

# no-bare-fences

Reason: bare ``` blocks render unstyled and lose semantic info.
EOF

git checkout -b feature/x
# make some changes ...
git commit -am "test"

agent-tool-pr-reviewer review

Output goes to .ai-review/runs/<UTC-timestamp>/. Read findings.json for the typed contract or review-output.md for a quick human read.

How it works

git diff --merge-base   →   .ai-review/*.md   →   single Pydantic AI call   →   typed Report
                                                                              ↓
                                                            findings.json + review-output.md

Deterministic everywhere except the one LLM call. The schema is the contract — every finding has a stable category (bug or project_rule), a severity (blocker | high | medium | low), a file/line range, an evidence quote (verbatim diff lines, 1–500 chars), and (for project_rule) a rule_id matching the .md filename of the violated rule.

See D:/ai-agents/docs/superpowers/specs/2026-05-07-agent-tool-pr-reviewer-design.md for full design rationale.

CLI reference

review

Reviews HEAD against the resolved base ref.

rules list

Prints discovered rules (<rule_id>\t<description>) without making any LLM call. Useful for sanity-checking which rules will be applied.

Exit codes

Excluding files from review

Some files generate false positives because they aren’t really “your code” — generated SQL, vendored deps, translated artifacts, lockfiles. Two ways to skip them:

.pr-review-ignore (recommended)

A gitignore-style file at the repo root. Same syntax as .gitignore:

# tests/fixtures/precheckawd.translated.sql is generated by the translator under review
tests/fixtures/*.translated.sql

# Don't review vendored deps
vendored/**

# But DO review this one security-critical vendored package
!vendored/security-critical/**

# Skip lockfiles
*.lock

The reviewer logs Filtered N file(s) from diff: ... to stderr whenever the filter is active.

Note: the reviewer does NOT auto-respect .gitignore. The intent is different — .gitignore is “don’t commit”; .pr-review-ignore is “don’t review.” Most repos copy their .gitignore patterns to .pr-review-ignore as a starting point and prune from there.

--exclude <glob> flag

Per-invocation overrides on the review subcommand:

agent-tool-pr-reviewer review --exclude 'tests/fixtures/**' --exclude '*.lock'

The --exclude flag is additive to .pr-review-ignore patterns — both sources merge into one PathSpec.

How it works

• Layer 1 (pre-LLM): matching files are dropped from the diff before the LLM sees them. The model never reads them, never emits findings about them. This is the primary mechanism.
• Layer 2 (post-LLM): if a finding’s file path matches an exclude pattern, it’s dropped before findings.json is written. Defense in depth — catches the rare case where the LLM hallucinates a finding for a path that wasn’t in its input. Each Layer-2 drop logs Dropped finding for excluded path: <path> to stderr.

Pattern syntax

Gitignore-style via the pathspec library:

• * matches within a single path segment, ** matches any number of segments
• Leading / anchors to the repo root (/build/** only matches the top-level build dir)
• Leading ! re-includes a file that a prior pattern excluded
• # for line comments
• Empty lines are ignored

Rule file format

Rules live in <repo>/.ai-review/<rule-id>.md (flat, no subdirectories). The filename — minus .md — is the rule_id. Each file:

---
description: One-line summary used by `rules list` and surfaced to the model.
---

# rule-id-here

Free-form prose: explain why, give examples, anti-examples, edge cases.
The model receives the full body verbatim.

The frontmatter description: field is required — the CLI exits 2 with a clear message on missing or non-string descriptions. Subdirectories of .ai-review/ (notably runs/) are skipped during discovery.

Output layout

<repo>/.ai-review/
  <rule>.md                      ← committed; rules
  runs/                          ← gitignored; one subdir per run
    2026-05-08T03-55-23Z/
      findings.json              ← typed Report (Pydantic-validated)
      review-output.md           ← human-readable rendering
    latest.txt                   ← single line: basename of the freshest run

findings.json round-trips through pr_reviewer.schema.Report.model_validate_json cleanly; downstream consumers (e.g., the pr-review skill) can rely on the schema without defensive re-validation.

Configuration

No config file in v1. Everything is via flags + env.

Using OpenRouter

--model openrouter:<model-name> wraps an OpenAIChatModel with Pydantic AI’s OpenRouterProvider. Model names follow OpenRouter’s slash convention:

agent-tool-pr-reviewer review --model openrouter:anthropic/claude-sonnet-4
agent-tool-pr-reviewer review --model openrouter:openai/gpt-4o
agent-tool-pr-reviewer review --model openrouter:google/gemini-2.5-pro

A single OPENROUTER_API_KEY covers all of them. The model name shows up in findings.json’s metadata.model exactly as you typed it (e.g., openrouter:anthropic/claude-sonnet-4), so runs across providers stay distinguishable.

Multi-model consensus mode

For higher-precision reviews, run N models in parallel and keep only findings that two or more models flag independently. Convergence is the strongest TP signal we have without a verifier-pass LLM call (see the v0.2.x trial retrospective).

Default basket

agent-tool-pr-reviewer review --models default

default expands to the empirically-Pareto-optimal 3-model basket:

1. openrouter:google/gemini-2.5-pro — breadth
2. openrouter:moonshotai/kimi-k2.6 — precision
3. openrouter:deepseek/deepseek-chat-v3.1 — quietness sentinel

Default threshold is --consensus 2 (a finding must be flagged by ≥2 of the 3 models). Output annotates each surviving finding with agreement_count and agreed_by in findings.json and a **Agreement:** N/M (flagged by: ...) line in review-output.md.

Custom basket

agent-tool-pr-reviewer review \
    --models openrouter:google/gemini-2.5-pro,openrouter:anthropic/claude-sonnet-4-6 \
    --consensus 2

--consensus 1 keeps every finding (no filtering); --consensus N requires unanimous flagging across N models.

Trial debugging

agent-tool-pr-reviewer review --models default --include-uncorroborated

Writes below-threshold findings to <run-dir>/uncorroborated.json alongside findings.json. Useful for inspecting what each model flagged uniquely.

Behavior notes

• All N models run in parallel via asyncio.gather. Different OpenRouter upstreams = no shared rate limit.
• If one model fails (timeout, validation, rate limit, network), the run continues with the surviving models; the failure is recorded in metadata.per_model_usage. Hard fails only if 0 models succeed.
• Per-model timeout is 30 minutes (Kimi K2.6 observed at 22 min on 50K-token diffs).
• --model and --models are mutually exclusive. The single-model --model path is unchanged.
• .pr-review-ignore and --exclude apply ONCE before dispatch — all N models see the same filtered diff.

Date-FP guard (v0.5.3)

agent-tool-pr-reviewer v0.5.3 ships a deterministic post-LLM filter that drops “future date / typo” findings whose evidence contains a recent ISO date AND whose description contains a known future-date/typo keyword. This eliminates the Gemini-style training-cutoff false-positive class — model says “this 2026 date is a typo” about a date the model can’t yet know is real — without paying --verifier default’s ~$0.05/run.

Two-signal AND gate:

1. Date signal: evidence contains an ISO-8601 date in the half-open interval (today - 730 days, today). Ancient dates (e.g. 1979) pass through; genuine future dates pass through (the model may be flagging a real typo).
2. Keyword signal: description (case-insensitive) contains one of: "future date", "future-date", "date in the future", "likely a typo", "appears to be a typo".

Both required → drop. Either alone → keep.

The guard runs once after consensus.merge_reports() (or directly after the agent run in single-model mode), BEFORE the verifier. Drops are final; the verifier never re-evaluates a guard-dropped finding. --include-uncorroborated’s uncorroborated.json passes through unfiltered by design.

# Default: guard ON
agent-tool-pr-reviewer review

# Opt-out: keep date findings (for users who want to see Gemini's full output)
agent-tool-pr-reviewer review --no-date-guard

Drops are written to <run-dir>/dropped-by-date-guard.json:

[
  {
    "finding": { "...full Finding..." },
    "drop_reason": "model_knowledge_cutoff",
    "matched_keyword": "future date",
    "matched_date": "2026-05-09"
  }
]

RunMetadata.date_guard_dropped: int records the count. The stdout summary grows a _Date-FP guard: dropped N_ row when N > 0; omitted when zero.

Known FN: consensus._merge_group keeps the longest evidence quote. If the longest-evidence cluster member lacked the ISO date but a shorter sibling had it, the merged finding loses the date signal — the guard misses. Pinned by test_asymmetric_merge_fn_pinned; revisit in v0.5.4+ if real-world data shows this class is meaningful.

Verifier pass (v0.5.0)

--verifier MODEL enables a Layer-3 precision filter that runs after consensus + scope filter. It catches false-positive classes that prompt-side guards and convergence don’t fully address:

Default verifier model is openrouter:anthropic/claude-sonnet-4-6 — cross-family from the Gemini-led consensus basket for bias resistance. Same OPENROUTER_API_KEY as the reviewer; no extra credential.

# Single-model + verifier
agent-tool-pr-reviewer review --model openrouter:google/gemini-2.5-pro --verifier default

# Multi-model consensus + verifier (stacked precision)
agent-tool-pr-reviewer review --models default --verifier default

# Custom verifier model
agent-tool-pr-reviewer review --models default --verifier openrouter:google/gemini-2.5-flash

When the verifier drops findings, they are written to <run-dir>/dropped-by-verifier.json with per-finding drop_stage ("deterministic" or "judge") and drop_reason. The kept findings go to findings.json as usual.

project_rule category findings skip the LLM judge stage (deterministic gate still applies). The verifier’s system prompt does not include rule bodies; judging rule violations as “scope drift” without the rule context would be unsound.

Cost: one Sonnet 4.6 call per run on the surviving bug-category findings. Typical 5–15 finding survivor count: ~1–2k input tokens + ~500 output tokens ≈ $0.02. Add this to your per-run reviewer cost when running with --verifier default.

--verifier is off by default. Drop-only verdicts in v0.5.0 (downgrade and description-rewrite verdicts deferred to v0.5.x).

Troubleshooting: model tool-use support (v0.5.1)

Some OpenRouter model IDs get routed to provider backends that do not support function/tool calling. Pydantic-AI’s structured output requires tools, so those routings fail with 404 — "No endpoints found that support tool use." mid-run.

agent-tool-pr-reviewer v0.5.1 auto-runs a tiny tool-use probe on each resolved model (reviewer + every basket member + optional verifier) before the real review dispatches. Detected incompatibilities exit 2 with an actionable error; the reviewer never runs and no tokens are wasted.

Known-incompatible model IDs are denylisted (zero-token short-circuit):

• openrouter:meta-llama/llama-4-maverick
• openrouter:deepseek/deepseek-r1-distill-qwen-32b

Newly-discovered bad routings get caught by the live probe — about $0.001 × N models per run. Add the known-bad ones to the denylist constant in src/pr_reviewer/compat.py when a new one is identified.

Probe outcomes:

To opt out of the precheck (deliberate experimentation with a borderline model, or if the probe itself is flaky):

agent-tool-pr-reviewer review --skip-precheck --model openrouter:experimental/new-model

The probe adds ~1-3 s of latency on a clean run (probes are parallel via asyncio.gather).

Recommended models

Two trials (16 distinct models, 39 successful runs across 4 chorus-sqlserver PRs) produced this preference order — both for single-model use and for the eventual Tier 2 consensus mode:

1. openrouter:google/gemini-2.5-pro — default. Caught both real bugs across the trials (:r regex in trial 1, error-message wording in trial 1) at ~$0.06/run. Has one known FP class (scope-misalignment on generated fixtures) that the deferred Tier 2 scope filter will eliminate.
2. openrouter:moonshotai/kimi-k2.6 — precision pick. 1 TP, 0 FPs across 4 PRs at ~$0.06/run. Slow on large diffs (up to ~22 min on a 50K-token diff), so a poor fit for interactive use but well-suited to CI and consensus mode.
3. openrouter:deepseek/deepseek-chat-v3.1 — quietness sentinel. 0 TPs, 0 FPs at ~$0.006/run. Useless as a primary reviewer, valuable in a basket: when DeepSeek does emit a finding, it’s worth a closer look because it almost never speaks.

The retrospective with full data is in D:/ai-agents/CONTRIBUTING.md under the agent-tool-pr-reviewer section. Models that did NOT make the cut despite costing more or being marketed for code: Claude Sonnet 4.6, Claude Opus 4.7, GPT-5, GPT-5-mini, Codestral 2508, Qwen3 Coder 480B, GLM 4.6, Grok Code Fast 1, MiniMax M2.7, Llama 4 Maverick (architecturally unusable), DeepSeek R1 Distill (architecturally unusable).

Troubleshooting

UnicodeDecodeError: 'charmap' codec ... on Windows. Already fixed in v0.1.0 — _git forces encoding="utf-8" for subprocess output. If you see it on a non-Windows platform, file an issue.

ImportError: Please install anthropic to use the Anthropic model. The anthropic SDK lower-bound floats; recent 0.100+ versions removed types that pydantic-ai 0.8.x still imports. The pin in pyproject.toml (anthropic>=0.61,<0.70) addresses this. Run uv tool install --editable . --reinstall after pulling.

error: Could not determine base ref. Pass --base <ref> explicitly. Your repo has no origin/HEAD, no main, and no master branch. Pass --base <whatever-your-default-is>.

error: prompt exceeds --budget N tokens. The diff is too big. Either split the PR, or pass --budget with a higher value if you trust your model’s context window.

Calibration notes

Phase-1 trial (2026-05-08, v0.1.0): 4 models (Claude Sonnet 4.6, GPT-5, Gemini 2.5 Pro, DeepSeek V3.1) via OpenRouter on a single Docker/Flyway diff in chorus-sqlserver. 4 findings, 0 true positives. Two failure modes named: external-tool-hallucination (GPT-5 confidently flagged a valid sqlcmd -No flag as invalid) and speculative-downstream-consequences (GPT-5 chained “X fails → Y fails → Z blocked” without grounding). v0.2.0’s required evidence field, system-prompt exclusions, and hedging-word guard on blockers are the targeted response.

Bug-vs-rule FP rate: in phase-1, bug-category findings (no rule_id) had a higher false-positive rate than project_rule findings. The skill body’s calibration note recommends surfacing the evidence quote prominently and asking the user to verify before acting on bug findings. Re-evaluate this once phase-2+ data accumulates.

What’s NOT in v1

Deferred deliberately:

• GitHub PR mode (--pr <num>, gh integration) — local branch only.
• Verifier / evaluator-optimizer pass — single LLM call, no second-pass grounding.
• Security findings — covered by Anthropic’s /security-review slash command. Out of scope here.
• API/contract-breaking-change, doc-drift, test-coverage categories.
• Auto-chunking for oversized diffs — refuse with exit 2.
• Per-rule scoping (glob applies_to) — all rules apply to all changed files.
• Token cost estimation in dollars.
• MCP server wrapping the same logic.
• Auto-apply for suggested fixes (--fix mode).

See the spec’s “Out of scope” section for the full list.

Development

cd D:/agent-tool-pr-reviewer
uv sync --extra dev
uv run pytest -v

63 tests across 8 modules: schema, paths, rules, diff, prompt, render, agent, CLI smoke. Tests use Pydantic AI’s TestModel for deterministic LLM stubbing.

Architecture

Small, focused modules in src/pr_reviewer/:

Implementation plan: D:/ai-agents/docs/superpowers/plans/2026-05-07-agent-tool-pr-reviewer.md.