Evals for Agent Skills: From Gut Feel to Regression
Most agent skills start as a prompt someone tweaked until it “looked good.” That works until the skill breaks …
Evals: The Missing Piece That Turns a Skill Prompt Into a Reliable Asset
Most agent skills start as a prompt someone tweaked until it “looked good.” That works until the skill breaks six months later because of a model update or a small wording change—and nobody notices until a client complains. Evals (evaluation sets) change that. They turn a skill from a one‑person gut‑feeling into a maintainable asset that any team member can test in minutes.
An evals set is simply a collection of real‑world inputs paired with what the skill should produce, plus clear pass/fail rules. When you update the skill, you re‑run the evals. If the skill starts failing cases that passed before, you’ve caught a regression before it hits production. For operator‑run businesses, this means the AI that writes your outbound emails or qualifies leads doesn’t silently get worse while you focus on growth.
What an Evals Set Actually Looks Like
At its core, it’s a table of examples—nothing more complex than that. You don’t need a database or a fancy testing framework. A spreadsheet works.
Here’s a sample structure for a skill that drafts LinkedIn connection notes:
| Scenario | Input (prospect data) | Expected Output Contains | Pass? | Notes |
|---|---|---|---|---|
| Normal case | Name: Sarah, Role: CMO, Company: Acme, Industry: SaaS | “Sarah”, “CMO at Acme”, reference to SaaS | Yes | |
| Missing role | Name: Tom, Role: (blank), Company: BoxCo | “Tom” and “BoxCo”; no role‑based personalization | Yes | |
| Empty input | Name: (blank), Company: (blank) | Polite refusal to draft, no hallucination | No | Skill returned a generic template instead |
Each row is one test. The pass criterion is not “looks good” but a concrete check: does the output contain specific strings? Is the format correct? Are placeholders filled? This removes ambiguity—the kind of ambiguity that made the “gut feel” method unreliable.
Step-by-Step: Building Your First Evals Set
You don’t need 500 examples. Start with 10‑20 that cover what you actually run into, then grow the set each time something breaks.
- Pull from real tasks. Go back to your last month of usage. Pick a mix of typical inputs, edge cases (missing fields, very long names, non‑English characters), and a few where the skill once produced something wrong. Don’t cherry‑pick only clean successes—evals must include the messy stuff.
- Define pass criteria that a script could check. For a cold‑email skill, that might be: subject line contains the recipient’s first name; body references the company’s industry; no unfilled placeholders like [CompanyName]; reply‑to line is set. If you can write a checklist, you can make it regressable. Avoid “reads naturally” as a stand‑alone check.
- Run the full evals set on every skill change, even a small tweak. If the pass rate drops, something degraded. Don’t release. This tiny habit is what separates “experimental” from “production‑grade.”
- When a failure slips through to a real user, add that exact case to the evals. The set gets stronger over time. After 3 months, you’ll have a regression suite that reflects your actual operating environment—no theory, all practice.
Our open‑source agent skills library at /en/products/365skill follows this flow exactly. Skills ship with an evals file that’s required to pass before any release. The governance is deny‑by‑default: a skill that can’t pass its own tests never reaches users. You can browse the repository to see the evals for skills like five‑step‑dev or trade‑email drafts.
A Worked Example: Cold‑Email Skill Evals
Let’s walk through a real‑life scenario. Suppose you maintain a skill that drafts a short cold outreach email given a prospect’s name, company, and website. Your evals table might start like this:
| Test # | Input | Expected Outcome | Pass? | Reason |
|---|---|---|---|---|
| 1 | “John, SupplyChain Inc., e‑commerce” | Email subject includes “John” and “SupplyChain”; body mentions “e‑commerce logistics” | Yes | |
| 2 | “Maria, (no company), health tech” | Generic subject; body avoids specific company claim | No | Skill hallucinated a company name from industry |
| 3 | “李华, EastTrade Co., cross‑border” | Name correct in Chinese; body references cross‑border trading | Yes | |
| 4 | “, Inc.,” (missing name) | Skill returns error message or asks for name | Yes (after fix) | Previously it would insert “Dear ,” |
After you change the prompt or model, you run these four tests again. Test 2 still fails—good, you haven’t fixed that bug yet. Test 4 now passes because you added a guard clause. The evals log becomes your history of what works and what still needs attention. No guessing, no “I think it’s fine.”
The Pitfalls That Make Evals Useless
Most teams that start with evals abandon them after a month because of three common traps:
- Too few samples, all sunny. If you only test the easiest cases, the evals give false confidence. A skill that handles “John from Acme” perfectly but breaks on a prospect with a 40‑character company name will still break in the field. Include boundary cases and past failures.
- Vague pass criteria. Saying “the output should be good” cannot be checked automatically. Even if you review manually, ambiguity leads to inconsistency. Spell it out: “must include at least two personalization hooks” or “no placeholders surrounded by brackets.”
- Not running evals after every change. If you tweak the skill and release without re‑testing, you’ve lost the entire point. Regression doesn’t announce itself. Make it part of your update checklist, just like you wouldn’t deploy a website without checking if the homepage still loads.
Evals don’t need to be perfect on day one. A small, brutally honest set that you actually use beats a massive, polished set that sits in a shared drive.
FAQ
How many examples do I need to start?
10‑20 real cases that include at least 3 edge scenarios (missing input, extremely long text, non‑English names). You can expand later as the skill matures.
Can I automate the pass/fail checks?
Yes, and you should. A simple script that checks for keyword presence, string length, or format compliance works in 90% of cases. At /en/products/365skill we use lightweight checks that don’t require an LLM call, keeping evals fast and cheap.
What if I don’t have historical tasks to draw from?
Use a handful of realistic made‑up examples, but then refine them as soon as you run the skill in production. The first time a user reports a bad output, add that input to the evals set. You’ll have a representative sample within days.
Do evals replace human review?
No. Evals catch obvious regressions—missing fields, format breakage, safety issues. They don’t measure tone or subtle persuasion. Human spot‑checks still matter, but evals free you from re‑checking the same basic things every time you change a prompt.
---
If you’re building agent skills for your own operations, start with a tiny evals file tomorrow. Or skip the setup and use skills that already come with production‑proven evals—check the library at /en/products/365skill and see what else we run in‑house across our tools at /en/products. Need help wiring evals into your workflow? Reach out for a free consult.