Tracing AI Agent Skill Usage with SkillTrace

A hands-on walkthrough for observing which skills your coding agent actually used

hideya

13 min read

·

Jul 13, 2026

--

Listen

Share

SkillTrace is a free observability tool for AI agent skill usage.
This article provides a step-by-step guide to using the software.

Press enter or click to view image in full size

Press enter or click to view image in full size

Why “AI Agent Skills” are so hard to debug

AI coding agents are starting to use reusable “skills”: small packages of instructions, references, and sometimes scripts that tell the agent how to perform a specific kind of task.

A skill might tell an agent how to review a pull request, fix a TypeScript error, inspect a database migration, or follow a project-specific debugging checklist.

This is powerful.

But it also creates a new debugging problem:

When an agent can choose from multiple skills, how do we know which ones it actually used?

Did it read the skill file?
Did it follow the skill?
Did it skip part of it?
Did it use a reference file?
Did it later believe that the skill influenced the run?

These questions are surprisingly hard to answer.

Tool calls are relatively easy to observe because they cross a clear boundary. A model calls an outside tool, passes arguments, receives a result, and the surrounding system can log that exchange.

Skills are different.

A skill may be read as a natural-language file, absorbed into the LLM context, and then influence the agent’s decisions without producing a clean external event that says, “this skill was used here.”

That is the problem SkillTrace is trying to explore.

What is SkillTrace?

SkillTrace is a pre-alpha observability tool for AI agent skill usage. It compares four evidence streams from the same run:

• Passive file-access traces
• Semantic declarations via dedicated MCP tools
• Structured post-run reflection
• Agent execution logs

The goal is not to prove exactly what the model “really thought.” The goal is more practical:

Put multiple imperfect signals side by side so you can debug skill usage.

This article walks through SkillTrace using a tiny toy demo first, then explains how to apply the same workflow to your own repository.

What SkillTrace captures

SkillTrace compares four kinds of evidence.

1. Passive traces

Passive traces are observed file-access events.

For example:

• SKILL.md was read
• A file referenced by skill was read

This is useful because it is independent from the agent.

But passive traces have an important limitation: reading a file does not prove that the agent actually used it.

A skill may be read, considered, ignored, partially followed, or overridden by another instruction.

2. Semantic traces

Semantic traces are explicit MCP tool calls made by the agent.

SkillTrace provides a local MCP server. When instrumentation is active, the agent is instructed to report events such as:

• skill start
• reference read
• skill finish

This gives you structured declarations like “I am starting to use this skill” or “I read this reference file.”

But this is still a self-report. The model may skip the declaration, misattribute something, or fail to follow the instrumentation.

3. Reflection

At the end of a run, SkillTrace asks the agent to reflect on what influenced the work.

A reflection may include:

• which skills it believes it used
• which skill files it read
• which reference files influenced the work
• which project files it believes mattered
• which steps it followed
• which uncertainties remained
• what skill changes it recommends

Reflection is not ground truth.

Its value comes from comparison.

If passive probing saw a reference file but reflection omitted it, that is interesting.
If the model declared a skill but passive probing did not see the corresponding file access, that is interesting.
If everything aligns, that is useful too.

SkillTrace is built around this comparison.

4. Agent execution logs

SkillTrace reads privacy-filtered local records maintained by Codex CLI, Claude Code, or Gemini CLI, including skill reads, tool operations, and verification outcomes.

Best-effort execution-log collection can be very informative and annotate a supported run in any mode. However, it never becomes an expected source as the logs are agent-owned, version-unstable records that were not designed as definitive skill-use evidence.

Press enter or click to view image in full size

Requirements

SkillTrace currently targets command-line agent workflows.

You need:

• Node.js 22+
• npm
• one of Codex CLI, Claude Code, or Gemini CLI
• macOS or Linux

Platform notes:

• On macOS, SkillTrace uses fs_usage for passive file-access probing and may ask for your admin password.
• On Linux, SkillTrace uses inotifywait, so you may need to install inotify-tools.

SkillTrace currently supports two instruction profiles (skill file directory formats): AGENTS.md with .agents/skills, or CLAUDE.md with .claude/skills.

SkillTrace is pre-alpha developer tooling. Expect rough edges, platform-specific behavior, and occasional missing traces.

In particular, there is a tendency to forget to make a self-report when using smaller models.

Install SkillTrace

Install the package from npmjs.com globally:

npm install -g skilltrace

Check the installed version:

skilltrace --version

Start the local daemon:

skilltrace daemon start

On macOS, this may ask for your admin password so SkillTrace can run fs_usage for passive skill-file access probing.

Open the UI:

http://localhost:7555

For a Linux container or VM where you want to open the UI from the host machine, bind the daemon to all interfaces:

HOST=0.0.0.0 skilltrace daemon start

Only do this in a trusted local network or isolated development environment.

Register the MCP server

SkillTrace uses MCP tools to record semantic skill-usage events.

The easiest setup path is:

skilltrace mcp install

This registers SkillTrace with every supported agent CLI it can find on your PATH.

Check the registration:

skilltrace mcp status

You can also target one client:

skilltrace mcp install --agent codex
skilltrace mcp install --agent claude
skilltrace mcp install --agent gemini

After this, run a quick diagnostic check:

skilltrace diagnostics

Use the verbose version if something looks wrong:

skilltrace diagnostics --verbose

First, try it on a toy skill

Before tracing your own repository, it is useful to run SkillTrace against a tiny demo project, that is included in this project’s repo.

This gives you a clean run where the expected skill usage is easy to inspect.
The procedure is as follows:

git clone https://github.com/hideya/skilltrace.git
cd skilltrace
mkdir -p tmp
cp -RP examples/type-fix-demo tmp/type-fix-demo
cd tmp/type-fix-demo
npm install

# If the daemon is not already running:
skilltrace daemon start
skilltrace mcp install
skilltrace diagnostics

skilltrace run --note "demo type-fix run" -- \
  codex "Fix the TypeScript error using the available skill"
# skilltrace run -- claude "Fix the TypeScript error using the available skill"
# skilltrace run -- gemini "Fix the TypeScript error using the available skill"

The normal SkillTrace workflow is one command. It starts the trace, runs the agent specified via the command line with the terminal attached, and stops the trace automatically when the agent exits.

When the AI ​​agent asks something like, “Allow to call the skilltrace MCP server tool?”, chose “Allow.”

When using smaller LLMs, calls to MCP tools may occasionally be skipped. If this issue occurs, please try using a larger model.

Then open: http://localhost:7555

You should see a new run in the SkillTrace UI.

A smaller LLM sometimes forgets the instructions to invoke the MCP tools. Try a larger model if you encounter this issue.

To retry the toy demo from a clean copy:

cd ../..
rm -rf tmp/type-fix-demo
cp -RP examples/type-fix-demo tmp/type-fix-demo
cd tmp/type-fix-demo
npm install

This toy demo is intentionally small. The point is not the TypeScript fix itself. The point is to see whether the agent read the skill, declared skill usage, used reference files, and reflected on the work afterward.

What to look for in the UI

The main page should look like the following. If you have made only one run, you’ll see one run item on the page.

Press enter or click to view image in full size

This shows grouped trace runs, status, trace mode, result, model/client context, and mode comparison.

Open a run detail page by clicking one of the run items:

Press enter or click to view image in full size

The run detail page is where the interesting parts appear.

You will see:

• a timeline of captured events
• run context
• Git snapshot information if available
• captured instruction contents
• consistency table
• reflection

The most important part is the consistency table.

It compares passive, semantic, and reflection evidence and asks whether there is consistent evidence of skill usage.

Press enter or click to view image in full size

For example:

• Did passive tracing observe SKILL.md access?
• Did the agent declare skill usage through MCP?
• Did the final reflection say the skill influenced the run?
• Did the reference files line up across evidence streams?

Passive-only runs are labeled as Captured rather than Pass, because there is no second evidence stream to compare.
SkillTrace does not treat file access alone as proof of skill usage. It treats it as one evidence stream.

Press enter or click to view image in full size

Try different trace modes

Instrumentation can affect agent behavior.

If you ask an agent to explicitly think about skill usage and report semantic events through MCP, the agent may become more careful, more verbose, or behave differently.

SkillTrace provides three trace modes so you can compare this effect.

skilltrace run --mode full -- <agent> ...
skilltrace run --mode passive_reflection -- <agent> ...
skilltrace run --mode passive_only -- <agent> ...

full

Captures:

• passive file access
• live semantic MCP declarations
• final reflection

This is the most informative mode, but also the most intrusive.

Use it when you are developing or debugging a skill and want maximum visibility.

passive_reflection

Captures:

• passive file access
• final reflection

It does not ask the agent to emit live skill lifecycle declarations.

This should interfere less with the agent’s normal task flow.

passive_only

Captures:

• passive file access as the only verdict-bearing consistency source
• best-effort agent execution logs

There is no instruction injection.

This should minimally interfere with the agent, though passive probing may still have platform-specific overhead or blind spots.

Passive-only mode can show that files were accessed, but not whether they were actually used.

Compare modes

After you have successful runs for the same target repo in different trace modes, the runs page can compare them.

A typical workflow is:

1. Start with full mode to debug whether the agent reads and declares the expected skill usage.
2. Try passive_reflection to reduce live semantic reporting.
3. Try passive_only to observe file access with minimal intervention.

Compare Modes checks whether the same skill and reference files appear across those runs.

Press enter or click to view image in full size

This helps answer a practical question:

Does the agent still appear to use the same skill when tracing becomes less intrusive?

That is important because observability itself can change behavior.

SkillTrace does not hide this problem. It gives you a way to inspect it.

Using SkillTrace on your own repo

Once you have tried the toy demo, the normal workflow is the same.

From a target repo:

cd <repo>
skilltrace run -- codex "fix the TypeScript errors in this repo"

The same wrapper works with the other supported command-line agents:

skilltrace run -- claude "review this change"
skilltrace run -- gemini "update the tests"

Add a note if you want the run list to show what you were trying:

skilltrace run --note “trying to simplify AGENTS.md” -- codex "..."

Target repo requirements

SkillTrace currently supports two instruction profiles (skill file directory formats).

agents

For Codex and Gemini CLI user’s repositories using:

AGENTS.md
.aents/skills/

claude_code

For Claude Code user’s repositories using:

CLAUDE.md
.claude/skills/

SkillTrace expects each skill root to use the common one-directory-per-skill layout:

<repo>/.agents/skills
  └── <skill-name>
        ├── SKILL.md
        └── <reference-dirs>
              └── <reference-files>

By default, skilltrace run auto-detects the supported profile.

If a repo has more than one instruction surface, or if you want to be explicit, use:

skilltrace run --instruction-profile agents -- <agent> ...

or:

skilltrace run --instruction-profile claude-code -- claude ...

SkillTrace injects a temporary tracing-policy instruction into the selected instruction file, writes .skilltrace/instrumentation.md, and creates .skilltrace.json when needed.

When the specified agent exits, SkillTrace removes the temporary instruction and generated files when they are unchanged.

Only one trace session can be active at a time.

Git provenance

When repeatedly modifying Skill files and verifying their behavior, you may want to know the state of the Skill files actually used during a run.

To facilitate this, when the target repo is inside a Git worktree, SkillTrace records a lightweight snapshot of the skill files at run start.

It captures:

• HEAD commit and branch
• broad changed-file status
• bounded diffs for instruction-relevant files
• bounded plain-text contents for changed instruction-relevant files

This helps compare successful and failed runs against the skill and instruction state they used.

This is especially useful when you are iterating on AGENTS.md, CLAUDE.md, .agents/skills/, or .claude/skills/.

If a run behaved differently from a previous run, the Git snapshot helps answer:

Was the skill or instruction surface different?

In the run detail page, changed instruction files are highlighted in the Run snapshot panel. You can click a file to inspect the captured plain-text contents used by that run (lines with uncommitted changes are highlighted).

A light look under the hood

SkillTrace is built around a simple idea:

Do not rely on one signal. Compare multiple imperfect signals.
At a high level, it has these parts.

Press enter or click to view image in full size

Local daemon

The daemon hosts the local server and web UI.
It also manages the active trace session and receives captured events.

Passive file-access probe

On macOS, SkillTrace uses fs_usage.
On Linux, it uses inotifywait.
This probe watches for skill and reference file access during a run.

MCP semantic logger

SkillTrace exposes a local MCP server.
When the agent sees the injected tracing instructions, it can call SkillTrace MCP tools to report semantic events such as skill start, reference read, and skill finish.

Temporary instruction injection

During skilltrace start, SkillTrace injects a temporary tracing policy into the selected instruction file.
This tells the agent how to report skill usage.
During skilltrace stop, SkillTrace removes the temporary instruction.

Post-run reflection

At the end of a run, SkillTrace can ask the agent to summarize which skills, references, files, steps, and uncertainties influenced the work.
This reflection is useful, but it is still self-report.

Trace store and UI

SkillTrace stores the run data locally and presents it in the UI.
The run detail page compares passive traces, semantic declarations, and reflection.
The UI is designed to make mismatches visible.

What SkillTrace is not

SkillTrace is not a replacement for general agent observability platforms.

General observability tools trace things like:

• model calls
• tool calls
• spans
• latency
• cost
• production behavior

SkillTrace focuses on a narrower question:

How do we know whether a natural-language skill was activated, declared, and reflected as influential in a specific agent run?

Also, SkillTrace is not a postmortem generator or automatic skill-improvement system.

For now, it focuses on the layer before that: making skill usage observable enough that we can debug it.

Known limitations

SkillTrace is pre-alpha.

Some important limitations:

• Codex CLI, Claude Code, and Gemini CLI are the first supported command-line workflows.
• Codex App support is not yet available.
• Passive file-access probing is platform-dependent.
• macOS passive probing may require admin privileges.
• Linux passive probing depends on inotifywait.
• Semantic traces and reflections depend on agent cooperation.
• Reflection is not ground truth; it may omit, misattribute, or overstate influence.
• Instrumentation may change model behavior, especially in full mode.
• Passive-only mode can show that files were accessed, but not whether they were actually used.
• Execution-log collection currently supports Codex CLI rollout files, Claude Code project-session files, and Gemini CLI chat-session files.
• Agent execution logs use local, client-owned, version-unstable formats. Missing, ambiguous, or changing logs are nonfatal and may yield no events.
• Passive SKILL.md access may be startup discovery rather than task-specific use; SkillTrace treats entrypoint-only scans as neutral discovered rows.
• SkillTrace currently focuses on observability, not automatic postmortem generation or skill improvement.

These limitations are not incidental. They are part of the problem space.

Skill usage is not easy to observe because skills often dissolve into the LLM context.

SkillTrace is an experiment in making that process more inspectable.

Privacy and data

SkillTrace itself is locally executed and no remote connection is established (except Google Fonts loading by the UI).

However, it may capture sensitive development context.

Depending on trace mode and repository state, captured data may include:

• skill files and reference files
• injected instrumentation instructions
• Git metadata
• changed-file status
• bounded diffs for instruction-relevant files
• bounded plain-text contents for changed instruction-relevant files
• agent-declared summaries, uncertainties, and file attribution
• MCP semantic logging events
• normalized execution-log facts such as skill paths, operation target paths, verification categories and outcomes, agent session/client/model identifiers, and collection status
• SkillTrace version and local runtime metadata such as OS platform, CPU architecture, Node.js version, and probe backend

Do not run SkillTrace on sensitive repositories unless you understand what is being recorded. Review captured runs before sharing logs, screenshots, or run exports.

Local SkillTrace data is stored under:

~/.skilltrace

Why this matters

The long-term idea behind SkillTrace is this:

The unit of human knowledge accumulation may be shifting from documents to executable work units enriched with execution evidence and failure histories.

Agent skills should not become trusted reusable knowledge merely by being shared.

To become trustworthy executable units of collective intelligence, skills need evidence:

• how they were activated
• how they were used
• where they failed
• how those failures informed improvement

SkillTrace is only a small first step towards it.

Hoping this tool meets your needs

AI agent skills are still early.

But they are becoming an important way to package procedural knowledge for agents.

If agents can choose from multiple skills, we need to understand how those skills are activated, declared, reflected, and sometimes misused.

SkillTrace is a small local tool for exploring that problem.

It compares passive file-access traces, MCP semantic declarations, and structured post-run reflection so developers can inspect what happened during an agent run.

Try the toy demo first.

Then try it on your own skill workflow.

I hope this tool meets your needs :-)