Do the Skills Earn Their Keep?
The skills my agents rely on are knowledge artifacts — so I measure whether they earn their keep: coverage, reliability signals, and skill-to-outcome quality. Evidence over vibes.
I — Why measure the skills at all
Everything the agents know lives in skills — small, addressable knowledge files an agent loads on demand: a Sitecore API contract, a hydration pattern, a naming trap I only learned by hitting it. Writing them is easy. Knowing which ones actually pull their weight is not.
So I built a measurement pass. It reads the raw transcript telemetry and the run manifests and asks one blunt question of every skill in the catalogue: has this earned its keep, or is it dark weight? Not to grade anyone — to keep the knowledge base honest, so it sharpens by evidence instead of accumulating by habit.
in the catalogue
loaded (39 active)
never invoked
shipped quality
II — Four lenses, not one number
A single "score" would lie. The value is in reading four lenses side by side — a skill can be popular and unproven, or quiet and load-bearing.
III — What it surfaces
I will not dump the full table here — the point is the shape, not a leaderboard. A few of the things one run surfaced:
- All fourteen Pages-Router skills are dark. That is not rot — it confirms the build strategy is App-Router-first. The lens told me the dark set was a decision, not a gap.
- A handful of skills get re-read most sessions. The theme-hydration and resolver-pattern skills are loaded again and again within single sessions — a flag to add a TL;DR at the top so the answer is found on the first read.
- The cross-cutting skills are the highest-leverage ones. A dozen skills are loaded by three or more different agent types. Investing in their clarity pays back across the whole org, not one lane.
A skill can be popular and unproven at the same time. One number hides that; four lenses show it.
IV — What it deliberately is not
This is a lens, not a scoreboard. It is directional, not statistical — the sample is small and the correlations are read as hints, not verdicts. It does not rank people, and it does not reward throughput: a skill loaded a hundred times that never rode a shipped product is unvalidated, not proven. The honest labels are the whole point.
Dark is a question, not a verdict.
A never-loaded skill is either an unreached domain or a discoverability bug. The measurement does not delete it — it asks which, and lets me prune, rename, or wait on purpose.
Re-reads point at the docs, not the agent.
If agents keep returning to a skill mid-session, the skill is the thing to fix — a Quick Reference at the top usually turns three loads into one.
Proven beats popular.
Usage counts are cheap. The signal that matters is whether products that used a skill actually shipped well — which is why usage and outcome are separate columns, never merged.
Measure against my satisfaction, not just green tests.
The last lens correlates skills with an operator rating. A run can be all-green and still not be what I wanted — so the human verdict is a first-class metric, added on purpose.
V — Where it goes
The measurement is young and gets sharper as more products ship — every shipped run adds an outcome signal, and the operator-rating lens is still filling in. The direction is the same one the whole framework runs on: a loop that keeps updating itself. The skills are how the agents get smarter; this is how I check that the getting-smarter is real.