Turn your agent's production failures into regression tests: a hands-on guide to Tracely

Tracely (jwuthri-cloud, MIT, ~1,150 stars) is trace-native CI/CD for AI agents: it grades every production trace as it lands, freezes the failing runs into hermetic regression cases, and blocks the pull request that would ship the failure again — replayed for $0, with no API keys.

• What: Tracely — trace-native CI/CD for AI agents (jwuthri-cloud/Tracely-ai), MIT license, by Julien Wuthrich.
• Traction: ~1,150 GitHub stars and 146 forks since Oct 11, 2026; Python, self-hostable in one click.
• The loop: production trace → failure detection → regression test → CI gate → alert — tests are promoted from real failing traces, never hand-authored datasets.
• Stack: FastAPI backend, Celery/Redis workers, ClickHouse traces, Postgres/pgvector registry, MinIO blobs, Next.js 15 UI — one docker compose file; web on :3001, API on :8000.
• Replay cost: $0 in CI — recorded tool/LLM fixtures, deterministic and offline; tracely gate exits non-zero, posts a commit status, and upserts a PR comment.
• Demo seed: docker compose --profile demo up -d --build --wait opens a populated workspace (traces, clusters, cases, gates) with the default ingest key tracely_dev_key.

Every eval tool for AI agents asks you to hand-author a dataset: sit down, invent questions, write ideal answers, keep them current as the product changes. That dataset is a guess about what might break. Meanwhile production already handed you the real thing — a trace of the exact run that failed, with the exact input, the exact tool calls, the exact model responses. Tracely (jwuthri-cloud/Tracely-ai, MIT, ~1,150 stars, created Oct 11, 2026) starts from that trace: it grades every agent run as it lands, clusters the failures into issues, freezes the bad runs into hermetic replayable cases, and blocks the pull request that would ship them again. In this guide you will boot the full stack in Docker, instrument an agent, turn one of its failures into a regression test, and wire the CI gate that refuses the bad PR.

What you'll need

• Docker + Docker Compose — the entire product (API, worker, UI, ClickHouse, Postgres, Redis, MinIO) ships as one compose file.
• Python 3.10+ for the tracelyai-sdk (the CI gate CLI ships in the same package).
• An OpenAI / Anthropic / Google / Mistral / LangChain key for your own agent — optional for the demo, which ships pre-populated with traces.
• A few GB of disk for the container images, and about 15 minutes.

Step 1 — Boot the whole stack in demo mode

The fastest way in is the demo profile: it starts ClickHouse, Postgres, Redis and MinIO, runs every migration, seeds the default project and ingest key (tracely_dev_key), then populates traces, clusters, cases and gates — so the app opens with the product's real screens instead of an empty shell.

git clone https://github.com/jwuthri-cloud/Tracely-ai && cd Tracely
docker compose --profile demo up -d --build --wait
open http://localhost:3001

The web UI lands on :3001, the backend API on :8000 (remap with TRACELY_WEB_PORT / TRACELY_BACKEND_PORT). The default auth mode is dev (open, no login); self-hosted teams switch to local (email/password) via the AUTH_MODE variable. When you're done, docker compose down stops it (add -v to wipe data).

Step 2 — Instrument your agent with the SDK

Install the SDK with the extra for your provider — [openai], [anthropic], [langchain] or [all] — initialize once at startup, then wrap a run. Your normal provider calls are captured automatically, with no span code:

pip install "tracelyai-sdk[openai]"

import tracely_sdk as tracely

tracely.init(
    endpoint="http://localhost:8000",   # your Tracely API
    api_key="tracely_dev_key",          # an ingest key
    service_name="support-agent",
    env="prod",                         # prod | staging | ci | dev — the gating axis
    instrument="auto",                  # auto-detect openai / anthropic / google / mistral / langchain
)

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
    )

That produces a GENERATION span with model, messages, tokens, latency, tool calls and cost. For your own logic there are @observe and the manual agent / tool / llm / retriever / guardrail context managers. Non-Python services can skip the SDK entirely: any OTLP/HTTP exporter works if you point it at POST {endpoint}/v1/traces with Authorization: Bearer tracely_dev_key. One gotcha from the project's own skill: a missing conversation id turns one thread into orphan rows, and no flush() loses a script's last spans.

Step 3 — Let the judges grade your traces

Open the trace explorer. Agent semantics (agent.id, conversation.id, turn, step) are first-class indexed columns, so runs group into conversation threads instead of a flat span soup, and the waterfall shows agent → thinking → skill → generation → hand-off with the failing span in red and its I/O beside it.

Evaluators are columns on the trace table, not a separate tab: LLM-as-judge at conversation, run or span level, plus structural checks that need no model at all. Scores stream in live over SSE as judges finish, so you watch a run get graded in place. Failures then cluster — structurally and semantically — so 31 broken runs become one issue with a count, not 31 rows to read, and each cluster can suggest the evaluator that would have caught it. There is also a calibration screen where you label judge verdicts against human review, so you catch an over-flagging judge before you let it gate a release.

Step 4 — Freeze a failing trace into a regression case

This is the heart of the product. One click promotes a failing trace into a hermetic case: recorded input, tool and LLM outputs bundled as fixtures, with a fail-to-pass contract attached — the case must fail on the old code and pass on the fix, or the promotion isn't trusted. The recorded run is the test: no hand-authored dataset, no guess about what might break. Multi-turn behaviour gets scenarios instead: a scripted conversation, or an adversarial goal a red-team model improvises against (goal achieved = FAIL — the scenario is inverted by design).

Step 5 — Make the CI gate block the PR

The promoted suite replays in CI against recorded fixtures: deterministic, offline, no API keys and no model spend. Three wiring options, depending on how your CI can reach your agent. The hermetic one — re-running your agent on each case's recorded input with recorded outputs as fixtures — looks like this:

- uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master
  with:
    api: https://tracely.your-co.dev
    key: ${{ secrets.TRACELY_KEY }}

- run: pip install tracelyai-sdk
- run: tracely replay planner --entrypoint my_pkg.agent:run    # a Python agent
  # …or any language:  tracely replay planner --cmd "node run.js"  (reads $TRACELY_INPUT)
  env:
    TRACELY_API: https://tracely.your-co.dev
    TRACELY_KEY: ${{ secrets.TRACELY_KEY }}

tracely gate exits non-zero, posts a commit status, and upserts a PR comment — the PR that reintroduces the failure is blocked. Hermetic replay needs your agent to route tool/model calls through the SDK's call_tool / call_llm seam; add --live to make real calls instead (real calls, real cost). If you'd rather Tracely drove your agent itself, register your agent's HTTP endpoint and author scenarios — nothing to install, import or shim — and the same Action gates any-language services. All you ever hand CI is your ingest key: it identifies your workspace.

Step 6 — Drive Tracely from your coding agent

Every backend serves an MCP endpoint at /mcp, so a coding agent reads your traces and writes your evaluators with no glue code:

claude mcp add --transport http tracely http://localhost:8000/mcp   --header "Authorization: Bearer tracely_dev_key"

Then MCP gives your agent your data; the Tracely skill gives it the know-how — how to instrument, what to evaluate, how to wire the gate — so "add Tracely to this agent" becomes one sentence:

npx skills add https://github.com/jwuthri-cloud/Tracely-ai --skill tracely

It works with Claude Code, Cursor, Copilot, Antigravity and anything else the skills CLI supports. Alerts get the same treatment: a rule is a when (a gate fails, a live conversation breaks on a judge, a failure mode appears that nobody has seen before, a rate crosses a line) plus a drawn flow — Slack, email, a webhook with your own headers, an LLM step, a Python expression — and the project ships an in-app assistant that draws the flow for you from a sentence.

What you built

A closed observability→regression→CI loop for an AI agent: the full stack running locally in Docker, your agent instrumented with three extra lines, its traces graded by judges and clustered into issues, one real production-style failure frozen into a hermetic case, and a PR gate that replays that case for $0 and refuses the regression — plus an MCP endpoint and a skill so a coding agent can operate the whole workspace on your behalf.

Honest limitations

• It is nine days old. Created Oct 11, 2026, it is moving fast — expect breaking changes in the SDK, the skill and the gate Action while the API settles.
• The demo stack is heavy. ClickHouse, Postgres, Redis and MinIO just to try an idea; there is no hosted trial beyond the one-click Railway deploy, and judges degrade without an LLM key (server-wide OPENROUTER_API_KEY or a per-workspace key) rather than telling you loudly.
• Hermetic replay has a seam requirement. Your agent must route tool/model calls through the SDK's call_tool / call_llm for fixtures to work; otherwise you replay --live and pay for the model calls.
• Judges are still judges. LLM-as-judge verdicts can over-flag; use the calibration screen before you let one block releases, and test every alert flow before trusting it to page you.