Member-only story
I ran claude plugin eval against my own skill. I’d tell you to do it too.
I’ve had Claude Code write me a couple of skills at this point. One strips AI-sounding tells out of a draft. One runs an entire blog pipeline end to end. I still use both, and until a few days ago I had no actual way to know whether either one was pulling its weight, or just riding along on the fact that Claude tends to do a decent job anyway.
That’s the honest reason I tried claude plugin eval. Not curiosity about a new command. I wanted to know if a thing I’d built and kept trusting without checking was doing anything.
The setup is almost insultingly simple once you’re in front of it. You sit at the plugin’s root, the folder with a plugin.json (or, for a plain skills folder like mine, the equivalent), and run claude plugin eval init. That opens an interactive session where Claude reads your own skill back to you, proposes prompts a real user might type, designs a grader for each one, tries them once to check they behave, and writes the whole thing to an evals/ directory it creates on the spot. You don’t write a single test by hand unless you want to.
Then you run claude plugin eval ., and this is the part I actually cared about. Each case runs three times with your skill loaded and three times with it switched off. Six runs, for one case. You get two scores out the…
