Claude Code can now test plugins before they break your workflowsClaude Code can now test plugins before they break your workflows

The hard part was never writing a skill

A Claude Code plugin can look useful after a couple of manual tests and still fail when someone phrases the request differently. That happens with skills for Laravel reviews, deployment routines, or homelab documentation: behaviour depends on the prompt, the model, and the tools available. Until now, checking that behaviour was mostly manual and difficult to repeat.

The new claude plugin eval command gives that problem a concrete shape: turn real situations into test cases. Each case contains a prompt and one or more graders that check whether the outcome meets expectations. This is not about validating that a plugin manifest is syntactically correct; it is about checking whether the plugin actually steers the agent toward a useful result.

What an eval actually measures

Claude Code runs each case in an isolated, non-interactive session with the plugin loaded. It can then inspect the final response, the tools used during the run, or files created in the workspace. It can also use a judge model to decide whether an answer satisfies a rubric written in natural language.

The official documentation defines six grader types: regex, tool_used, tool_order, file_exists, llm, and baseline. The first four inspect execution data or generated files, while llm and baseline make extra calls to a judge model. I would start with deterministic checks whenever possible because they are more stable and much easier to debug.

The no-plugin comparison is the useful bit

A passing case with the plugin enabled does not prove the plugin contributed anything. Claude may have completed the task just as well without extra instructions, especially for small or generic requests. That is why evals run every case with the plugin and, by default, without it too.

The summary includes WITH, W/OUT, and Δ. That delta is the difference between the plugin run and the no-plugin run, which makes it more useful than an isolated pass result. If both scores are the same, it is worth asking whether the skill contains practical project knowledge or merely restates what the model already knows.

A small Laravel plugin case

I would begin with a request that a teammate might write naturally, without naming the skill. For a plugin that enforces conventions in a Laravel application, the case could verify that Claude activates the skill and gives specific architectural guidance. The files live under evals/ in the plugin, and a starting template can be created with claude plugin eval init --bare case-name.

my-plugin/
├── .claude-plugin/plugin.json
├── skills/
│   └── laravel-review/
│       └── SKILL.md
└── evals/
    └── review-service/
        ├── prompt.md
        └── graders/
            ├── criteria.md
            └── skill-fired.md

The prompt.md file accepts frontmatter for turn limits and the read-only tools a case needs. Its body should be the prompt a real user would send, not an artificial instruction written only to make the test pass. If a scenario needs code, files, or a prepared repository, add fixtures through case.yaml rather than relying on state from the local machine.

---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill]
---
Review this Laravel service and explain what you would change
so it does not mix validation, data access, and business logic.

Check both the answer and the route taken

An llm rubric works well when the wording can vary but answer quality matters. For an important rule, I would define passing and failing conditions as precisely as possible. If the result can be checked with a pattern, a file, or a tool invocation, I would use one of those non-model graders first.

---
type: llm
---
PASS if the response clearly separates validation, persistence,
and business logic, and proposes Laravel-specific changes.
FAIL if it only gives generic advice unrelated to the service structure.

It is also useful to verify how the agent reached the result. A tool_used grader can confirm that a skill was invoked, while an outcome grader checks that the invocation was worthwhile. The documentation notes that checks that only make sense with the plugin, such as verifying Skill usage, are treated as indicators so they do not artificially inflate the delta against the no-plugin run.

From a homelab routine to a repeatable test

The same approach fits a homelab. A plugin containing procedures for Proxmox, backups, Docker, or self-hosted services can be tested with everyday prompts: restore a container, review a backup policy, or plan a network change. The important part is that these tests should not hit real infrastructure by default.

Claude Code supports mocks for MCP servers and tool responses. That lets you verify which tool call the agent would attempt and which data it would send, without starting the real service. For workflows that generate files, an eval can inspect a file in the isolated workspace or assert that it was created with file_exists.

Making it a CI gate

Once the suite is stable enough, it makes sense to run it in CI. The command can write results as JSON, keep the HTML report local, and return an error when a case misses the configured threshold. I would also pin both the agent model and the judge model, so a model rollout is not mistaken for a plugin regression.

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

The HTML report is saved alongside the eval results and helps identify the failing grader in each run. The JSON output can be archived as a CI artifact or processed by another tool. I would not treat a single run as final proof: these agents are non-deterministic, and the documentation itself recommends confirming changes using the default repeated runs.

The fine print before automating it

An eval does not make a plugin safe. Claude Code loads the plugin’s skills and hooks on your machine, and the isolation protects the agent being evaluated rather than acting as a security boundary around the plugin’s own code, hooks, or real MCP servers. I would only use --trust-plugin, --scaffold, --allow-real-servers, or --mocks off with code I would review and run myself.

It is also not a free or instant unit test. Runs and judge-based graders consume model calls, and --max-cost-usd caps a list-price estimate rather than guaranteeing exact usage. If that limit is reached, results can be partial, and I would not mix them blindly into quality trends.

Finally, an eval only measures what you wrote into the cases, not everything the plugin can do. Overly friendly prompts, vague rubrics, or unrealistic mocks can produce a passing suite while real-world use remains poor. The value of claude plugin eval is that it forces us to preserve difficult, repeatable examples from daily work instead of adding another green badge to the repository.

Source: MarkTechPost

Official documentation: Test plugins with evals

Release notes: Claude Code v2.1.269