← Back to blog

Introducing Skills Trace: Runtime Observability for AI Agent Skills

Skills Trace Team
launchobservabilityai-agents

The Problem: Skills Disappear at Runtime

Modern AI agents rely on structured capabilities — tool definitions, prompt templates, capability modules — that we call skills. During development, these skills are well-defined and version-controlled. But once an agent is deployed, those skills are serialized into LLM request payloads and become invisible to existing observability tools.

You can monitor model latency, track token usage, and log prompt inputs and outputs. But you cannot answer the most fundamental question: which skills are my agents actually using in production?

The Solution: Gateway-Level Skill Detection

Skills Trace operates as middleware inside your AI gateway. It inspects inbound LLM request payloads — without modifying them — and detects which skills are present using a multi-strategy fingerprint matching pipeline.

When a skill is detected, Skills Trace emits a structured telemetry event containing the skill identity, detection confidence, and metadata. The prompt itself is never logged — only the skill identity.

How It Works: 6-Strategy Detection Pipeline

Skills Trace uses a cascading detection pipeline that tries up to six strategies, short-circuiting on the first match:

  1. Watermark Detection (confidence: 1.0) — Looks for embedded watermark tokens injected by skills-check during development.
  2. Frontmatter Hash (confidence: 0.9) — Matches YAML frontmatter hashes from skill definitions.
  3. Content Hash (confidence: 0.7) — Full-content SHA-256 hash matching against the fingerprint registry.
  4. Prefix Hash (confidence: 0.5) — Matches the first N characters of skill content for partial matches.
  5. Tool Schema (confidence: 0.6) — Detects skills by matching tool/function schema structures.
  6. Function Signature (confidence: 0.4) — Matches function name and parameter patterns.

The pipeline achieves sub-20ms p99 latency for the full scan and sub-5ms for watermark-only detection.

Gateway Support: Drop-In Middleware

Skills Trace ships with adapters for the most popular AI gateway patterns:

  • @skills-trace/express — Express.js middleware for custom proxy servers
  • @skills-trace/vercel — Vercel AI SDK language model middleware
  • @skills-trace/cloudflare — Cloudflare Workers request handler
  • skills-trace (PyPI) — Python callback adapter for LiteLLM proxies

Each adapter integrates in a single line of configuration. The core detection engine is shared across all adapters.

Enterprise Features

Skills Trace is built for production deployments at scale:

  • Multi-tenant routing — Route telemetry by tenant, team, or environment with TenantRouter.
  • Configurable sampling — Random or deterministic sampling modes to control telemetry volume.
  • Prometheus metrics — Export detection counts, latencies, and cache hit rates via MetricsCollector.
  • OpenTelemetry — OTLP span emission for integration with existing tracing infrastructure.
  • Registry signature verification — Ed25519-signed fingerprint registries for supply chain security.
  • Multiple emitters — Fan out telemetry to JSONL, HTTP, SQLite, PostgreSQL, or any combination.

What's Next

This initial release lays the foundation for runtime skill observability. On our roadmap:

  • Additional gateway adapters (AWS Bedrock, Azure AI Gateway)
  • Smarter detection with ML-assisted fingerprint matching
  • Cost attribution per skill across models and providers
  • Skill drift detection and lifecycle alerts
  • Integration with skills-check for closed-loop skill management

Get Started

Skills Trace core is open source. Install the detection engine and your gateway adapter of choice to start seeing which skills your agents are using in production.

← Back to blog