Aug 2026 · LLM telemetry, observability, Prometheus, OpenTelemetry
In the previous post I built the entire pipeline by hand: five agents, five storage formats, one adapter per agent, a dedup wrapper, a Pushgateway, Grafana. At home nothing emitted metrics, so I built everything.
At work it happened the other way around. The telemetry already existed. Claude Code ships native OpenTelemetry metrics (cost, tokens, model, per user) straight to the staging Pushgateway.
The org spends real money on AI coding tools and almost nobody can see the invoice. The Anthropic Admin Console is reachable by a handful of people (CTO and a few others).
A question nobody could answer: how much do we spend, per model, per team, and is it worth it?
The original plan was a parser for the JSONL files in
~/.claude/projects. My own rule: metadata only, never prompt content.
Smoke test on staging: a manual push under the job
claude_code_eve_smoketest to the preview Pushgateway. Instead of my
homegrown metrics, the NATIVE Claude Code ones appeared
(otel_scope com.anthropic.claude_code, v2.1.235). The curl that revealed
everything:
curl -s http://pushgateway-preview.<company>.com/metrics \
| grep -A2 'job="claude_code_eve_smoketest"' | grep cost_usage_total | head -5
claude_code_cost_usage_total{instance="<work-machine>",job="claude_code_eve_smoketest",model="claude-haiku-4-5-20251001",otel_scope_name="com.anthropic.claude_code",otel_scope_version="2.1.235",query_source="auxiliary",session_id="9699d657-...",user_email="eve@<company>.com"} 0.001251
Rich labels out of the box: user_email, user_id,
model, query_source (main vs auxiliary: the agent overhead
costs almost as much as the main answer), effort,
session_id, terminal_type, organization_id.
The emitter already exists and it is Anthropic's: zero instrumentation, zero code to maintain. The work became configuring the export, a dashboard, and onboarding.
claude_code_cost_usage_total is a COUNTER. Counters reset: new process,
restart, re-emission.
The naive panel query was sum(claude_code_cost_usage_total). When the
counter resets, sum() loses everything the reset ate: cost under-reported by 90%
or more. The personal smoke test caught a REAL defect before rollout. That is
exactly what a staging pilot is for.
# WRONG: sum() on a resetting counter under-reports cost by 90%+ sum(claude_code_cost_usage_total) # RIGHT: increase() handles counter resets sum(increase(claude_code_cost_usage_total[$__range]))
Lesson: the correct query depends on the metric type, not on what sounds right.
pushgateway-preview.<company>.com: test the full pipeline without
touching prod. The dashboard "Claude Code - Usage & Cost" on staging Grafana came
alive with real smoke test data. I will not post a screenshot of my work Grafana,
but the dashboard already does its job: usage split by project and model.