Claude Code ships plugin evals: prove your skill actually helps
v2.1.269 runs your plugin against a no-plugin baseline to show the delta it adds — six graders and a CI score gate. Plus Copilot code review now self-resolves.
Copy markdown
What `claude plugin eval` actually measures
Point it at a plugin and each realistic prompt runs in a fresh, isolated headless session; graders then score the result — was the right tool called, does the reply match, did your skill even fire. `claude plugin eval init` writes the whole suite for you by interviewing Claude about the plugin, so you are testing behavior, not just checking that files parse.
The no-plugin baseline is the whole point
Every case runs three times with your plugin and three times without, giving you WITH, W/OUT, and a Δ. If a case scores 1.0 in both arms, your plugin isn't why it passed. The most common first finding is a Δ near zero with the tool_used:Skill grader failing — Claude isn't choosing your skill on natural phrasing, which points you straight at the skill description to fix.
Six graders, four of them free, plus a CI gate
regex, tool_used, tool_order, and file_exists are computed from the transcript for free; llm and baseline call a judge model and cost tokens. For CI, `--threshold 0.8 --trust-plugin --max-cost-usd 20` fails the build on a regression and caps spend, and every run drops a self-contained HTML report you can attach to the job. Requires Claude Code v2.1.269.
Elsewhere: Copilot code review cleans up after itself
GitHub's code-review bot now auto-resolves its own comments once you address them in a later commit, so open threads mean work still left to do. Its Lite tier moved to ensemble reviews — several agents, one merged verdict — landing 47% more high-severity fixes at roughly 8% lower cost. Rolling out to all users, no toggle to flip.
Elsewhere: a US 'duty of care' bill could gate model releases
Senate negotiators are drafting a bill that would make mitigating known major risks a legal duty for AI developers and let the government block models deemed unsafe, per Reuters. Still pre-text and pre-vote, but if it lands it reshapes who can ship frontier models and when — worth tracking if your stack rides the newest weights.