Guides / How we build and test our skills

How we build and test our skills

Most prompts are written once and never tested. We treat a flagship skill like something to be trained: we try it on real tasks, score it, change one thing, and keep the change only if a held-back test improves. We publish the scores, including the attempts that failed.

Published 5 October 2026.

The loop, in nine steps

  1. Problem definition. Pick one job and say what a good result looks like.
  2. Representative tasks. Collect real tasks for that job. Split them into train, selection and test.
  3. Baseline. Score the model with no skill, so we know where it starts.
  4. Several candidate versions. Write more than one version of the skill to compare.
  5. Automatic scoring. A program checks every answer, so the score is not our opinion.
  6. Failure analysis. Read the recordings of what went wrong and look for patterns that repeat.
  7. Optimisation. Change a few things at a time, then run again.
  8. Regression test. Check on tasks that did not shape the edit that nothing got worse.
  9. Final benchmark. Run the held-back test tasks once, at the end, and publish the result.

What it found

  • The Opus version of Agent Browse. On 11 unseen tasks (2 runs each), every condition got 22 of 22 right. The trained skill used 29.4k tokens per task, against 34.2k with no skill and 109.4k with the tool vendor's own skill.
  • The Qwen version. On the selection split (8 tasks, 6 runs each) it got 48 of 48 correct at 36.4k tokens per task, against 21 of 24 at 68.7k with no skill. Its held-out test is not finished.
  • A rule that had to be reversed. "Never guess selectors on a page you have not seen" was rejected in Part 1, because Opus guessed right and the rule only added a step. Qwen guesses wrong often, and the accepted Qwen skill contains a version of it.
  • Edit sets that looked sensible and were rejected. Two of three edit sets read well and made things worse. Only a score on unseen tasks told us. For Qwen, three of four candidates were rejected.

The tables, the edits and the attempts that failed are below.

Which products this covers

Today the full training loop has been run on our flagship browsing skill, Agent Browse, in two versions: one for Claude Opus and one for Qwen3.8-27B. It is where we apply the method first. Our other products are tested on real tasks before release: we run each one on real examples and read the result against what we gave it, and each product page says what its own tests were. The full loop comes to more flagships over time. The recordings, every candidate and a starter you can run are in the public repo.

What we measure on our other skills

We also ran two of our shipped skills against typing one line, on 10 invented cases each, run twice, scored by a script. Nothing was tuned or retrained for this test, so it is not the full method above. The numbers are outputs fully right out of 20.

SkillOne line, strong modelWith the skill, strong modelOne line, cheapest modelWith the skill, cheapest model
Review reply drafter4 of 2020 of 200 of 2019 of 20
Product description rewriter1 of 2020 of 200 of 2012 of 20

One skill we tested did not beat a one-line prompt on our checks: the late-invoice chaser. On the strong model the one-line email got 18 of 20 fully right and the skill 13 of 20, and on the cheapest model the skill got 1 of 20, with threats and wrong figures. It is being fixed, and we show no comparison for it until it is.

What we learned about models, Claude models only, on small invented test sets: a careful prompt on the strongest model is close to the ceiling on the client email (31 of 32 outputs passed every check). On the mid-level models most free users get, the gap is large: for the recruiter note, 9% of outputs passed every check with one line and 59% with a careful prompt (the strongest model: 22% and 62%).

The technical detail

The problem with hand-written skills

Most skills are written by hand. Someone writes advice that sounds sensible, tries it on a few examples, and ships it.

The trouble is that sensible advice can make results worse. Nothing tells you. The skill reads well, the three examples worked, and the extra cost or the new mistake never shows up. We wanted a way to find out.

How the loop ran in this case: six steps

We treat a skill like something you tune. The model stays the same. Only the skill file changes, and only when a score says it should. The method is a small version of Microsoft's SkillOpt paper (arXiv 2605.23904).

The training loopSix steps in order: tasks, rollout, reflect, bounded edit, gate, keep or reject.1Taskswith a checker2Rolloutmodel does them3Reflectread the failures4Bounded edita few changes only5Gatescore unseen tasks6Keep or rejectkeep only if higherKept? The next round starts from the new skill.
  1. Tasks. A set of jobs where a program can check the answer. Split into train, selection and test.
  2. Rollout. The model does the train tasks with the current skill. We record every step and the score.
  3. Reflect. A second model reads the recordings and looks for patterns that repeat across tasks, not one-off fixes.
  4. Bounded edit. Only a few changes per round: 4, then 3, then 2. If a round fails, we know where to look.
  5. Gate. The new skill runs on the selection tasks, twice. Those tasks did not shape the edit.
  6. Keep or reject. The new skill stays only if its score is strictly higher. A rejected idea is written down so it is not proposed again. The test tasks are used once, at the end.

The case: the Agent Browse skill

Browse the web with an AI agent in fewer steps teaches an AI assistant to browse the web with the agent-browser command line tool. We trained it on 32 browsing tasks (13 train, 8 selection, 11 test), using Claude Opus 5.5 as the model doing the work.

The model got every task right even with no skill. So accuracy gave the gate nothing to measure. We scored cost as well: a wrong answer is 0, a right answer is 0.7 plus up to 0.3 for using fewer tokens.

Result 1: training, on the selection tasks (8 tasks, 2 runs each)

StepEditsScoreTokens per taskTurnsDecision
0, starting skill150 words0.943537.7k5.2start
140.968720.8k3.0kept
23 more0.964423.8k3.2rejected
31 more0.963224.5k3.0rejected

Every run was correct, so the score differences are all about cost.

Result 2: unseen test tasks (11 tasks, 2 runs each)

ConditionCorrectTokens per taskTurns per task
No skill22/2234.2k5.0
The tool vendor's own skill22/22109.4k5.5
Starting skill22/2233.3k5.1
Trained skill22/2229.4k4.1

The vendor's skill is about 9k tokens and we put all of it in the prompt, which is harsher than loading it only when needed. The trained skill is about 700 tokens. With two runs per task, the test figures are a guide, not a precise number.

The four edits that helped

All four removed waste the model could not have known about.

  • Put several commands in one call, and close the browser in the last one.
  • Do not chain work to the page-open command with &&. It can time out on a slow page that is still usable.
  • A short recipe for JavaScript dialogs and right-click. The model had been searching the help text for these.
  • A short command list, so the model stops asking for help text at all.

The one edit that hurt

The rule was: "Never guess selectors on a page you have not seen." It sounds careful. It made the model take a page snapshot on tasks where guessing already worked, and that cost more than the failures it prevented. It came in the second set of edits. The gate rejected that set. Without the gate, we would have shipped it.

Part 2: the same skill on a local model

Status: scored on the selection split only. The held-out test is not finished. Those runs were still in progress when we wrote this.

Part 1 ended on a plain point: a strong model leaves little to train. So we swapped in a weaker one, Qwen3.8-27B, running on a local server and driven by the pi coding agent, and asked how much more the method gives.

The setup

RolePart 1Part 2
Student (does the browsing, never edits the skill)Opus 5.5 through claude -pQwen3.8-27B on a local OpenAI-compatible server, through pi -p
Optimizer (reads the recordings, writes edits)Claude Fable 5.1Claude Fable 5.1
JudgeA check in the test harnessThe same
Starting skillA 150-word seeds1, the skill accepted in Part 1

The harness is the same one, with the student swapped. Same tasks, prompt, splits and score. Pi forced three differences. It has no turn limit, so each run gets 15 minutes and a timeout scores 0. It cannot block curl, so a run that uses it is scored wrong. And turns are counted as assistant messages, which is close to the Claude figure but not identical.

Results on the selection split

Runs on a local model are free, so each candidate got 6 runs per task (48 runs) instead of 2.

SkillEditsCorrectScoreTokens per taskTurnsDecision
No skill (24 runs)21/240.788768.7k13.9
s1 (trained on Opus)47/480.917742.3k9.2start
q1443/480.836742.0k8.3rejected
q2q1 with one edit swapped48/480.945436.4k8.5accepted
q34 more47/480.921539.4k8.1rejected
q42 of the q3 edits44/480.854247.0k9.1see note

About q4: three of its four failures were 15-minute timeouts on the 50-page catalogue crawl, after the practice site began refusing requests under four crawls at once, so that is the site throttling, not the skill. A one-at-a-time re-run is pending.

For reference, Opus with s1 on the same split got 16 of 16, a score of 0.9687, 20.8k tokens and 3.0 turns. The tuned skill brings Qwen level with Opus on accuracy here, but Qwen still needs nearly three times the turns and about 75% more tokens. On the train split q2 and s1 were level (0.9436 against 0.9484, both 39 of 39), so the selection gain is not confirmed there.

What the edits were

Kept in q2, all on top of s1:

  • Never invent what you have not seen. Qwen made up deep addresses that returned 404, and ids and classes that did not exist. The rule: start from the given address, follow the site's own links, and look at the page before writing a selector.
  • Selector syntax. It kept using a style the tool rejects. The skill now names the one that works.
  • Longer scripts on stdin. Shell quoting cost two or three turns in several runs.
  • A paging check. The one recurring wrong answer: Qwen looked for a next-page button with a guessed selector, found nothing, and reported a 26-book category as 20. The rule: compare what you collected with the page's own total.
  • Do not close the browser early. It closed it before it had read the output it needed.

Rejected:

  • "When the output contains the answer, reply at once." It was meant to stop re-checking. It failed the paging task more often.
  • Retry the page-open command in a loop. It made the 50-page crawl about 50% more expensive.
  • Custom dropdowns need a different command. Qwen hit this on every run of one train task, but no selection task uses it, so the gate could not reward it.
  • Always quote selectors, wrap scripts in a function. Both prevent real errors seen in the recordings. Neither moved the selection score.

Edits were gated in bundles, so which edit did what is our reading of the recordings, not a measured effect.

What we noticed

  • A skill is tuned to the model that reads it.One rule changed sign. "Never guess selectors on a page you have not seen" was rejected in Part 1, because Opus guessed right and the rule only added a step. Qwen guesses wrong often, and the accepted Qwen skill contains a version of it.
  • The failures differ in kind.Every kept Opus edit removed waste from runs that already worked. The Qwen edits prevent errors: wrong selector style, broken quoting, a list called complete when it was not.
  • Most of the value transferred.The Opus-trained skill alone took Qwen from 87.5% to 97.9% correct and cut tokens by 38%, much more than it did for Opus. The Qwen step added a smaller gain. That points to a shared core plus a thin layer for each model.
  • A skill helps a weaker model more.Training moved Opus's score by 0.025. For Qwen, no skill to tuned skill is 0.157 (0.789 to 0.945), and accuracy went from 87.5% to 100%. The starting points differ, so this is rough.
  • Noise limits how far you can tune.Three of four Qwen candidates were rejected. The q2 margin over s1 is one failed run in 48 plus a 14% token saving, and a second batch of s1 runs alone scored 0.941.
  • The test sites were the bottleneck.Qwen answers in under a second per turn. Nearly all the time went on page loads and waits. Four runs at once made the practice sites return errors, so the selection numbers carry that noise.

Limits of Part 2

  • The held-out test for Qwen is pending. All the Part 2 figures are on the selection split, the one the gate used to pick the skill.
  • The cross-check on Opus was not run: the Qwen skill on Opus. If it made Opus slower than s1 does, that would be direct evidence for model-specific tuning. Until then the sign change above rests on two separately gated bundles.
  • One local model, one quantisation, one harness. Thinking was left at pi's default.
  • The Qwen skill is q2 plus the same hand-added Setup section, which did not go through the gate.

Browse the web with a local Qwen model, fewer wrong turns is the free skill. The recordings and every candidate are in the research repo.

Another case: the starter example

To let anyone try this, we built a small loop that needs no browser: the starter. Its 18 toy tasks ask for a "house style" (date formats, invoice ids, file names) that the task never states. Here the model did not start at 100%, so accuracy itself could be the score. The model was Claude Haiku. One reflection step, one candidate.

Skill given to the modelSelection (6 tasks, 2 runs)Test (6 tasks, 2 runs)
No skill0 of 6 (one run)0 of 12
Seed: two lines, no rules0 of 120 of 12
Trained candidate (215 words, one step)10 of 12, kept10 of 12

The whole run was 84 model runs and one reflection call, about $1.50. A local 27B model (Qwen 3.8 via Ollama) scored 0 of 18 with no skill. The skill trained with Haiku then got 5 of 6 unseen tasks right on that local model, in one run.

Read it with care. The seed was too thin to show a climb: it went from nothing to a rule set in one step. The 10 of 12 comes from small numbers, and one task is worth 8 points. Four of the 24 unseen runs still slipped on small details. The tasks are toys. They show that the loop works, not that it works on your job.

What we learned

  • Sensible is not the same as better.Two of three edit sets read well and made things worse. Only a score on unseen tasks told us.
  • Small edits are easier to judge.With few changes per round, a rejection points at one idea.
  • Good skills are short.The trained skill is about 700 tokens. The vendor's is about 9k and cost three times as much to use.
  • Fix the test setup first.Our first score looked awful, and the cause was the setup, not the skill: browsers that crashed each other and a test site that failed under load.

Where each skill stands today

Trained and gatedAgent Browse (for Claude Opus) and its variant for Qwen3.8-27B

This is one skill in two versions. Scored on tasks, changed in small steps, kept only on a higher score. The Opus version has a held-out test. The Qwen version is scored on the selection split so far; its held-out test is pending.

Tested by handEvery other skill

Run on real examples before release and checked against a written rubric. Not trained or gated. How we test.

PlannedCheck an invoice before you send or pay it

Next to be trained and gated. Its output can be scored automatically. No dates.

Limits

  • One skill, in two versions. The Opus version has a held-out test; the Qwen version does not yet. We do not know whether the result holds for other models.
  • The model started at 100% there, so the gains are about cost. The paper reports larger accuracy gains where models start lower.
  • Our checks are automatic, so this only works for jobs a program can score. Many skills cannot be scored that way.
  • The Setup section of the Agent Browse skill was added by hand, after problems in our test setup. It did not go through the gate.

Try it yourself

The starter is in the open repo. It needs only Python 3 and either the claude command or any local or hosted model with an OpenAI-style endpoint. It has the tasks, the scoring, the reflection step, the gate and the example run above, with a step-by-step tutorial.

git clone https://github.com/proskillpacks/skillopt-agent-browse
cd skillopt-agent-browse/starter
python3 run.py --skill none --split sel --tag baseline --model haiku

Open the starter The full research repo