Token spend

Zimmer records every Anthropic API call its agents make, and prices them at current list rates on the Costs page. This is Zimmer’s own accounting. It is a different thing from Inference, which reads Anthropic’s rate-limit headers to answer “how much headroom is left in the window”. Costs answers “what did we spend it on”. Neither substitutes for the other, and they can disagree without either being wrong.

The tables

Table	Holds	Keyed on
`session_token_usages`	Inference an agent session did — the bulk of spend	`request_id`
`adhoc_token_usages`	Inference Zimmer’s own code made outside any session	`request_id`

A third table, token_usage_features, splits each session request across the context-management features it was carrying. It is an estimate rather than a measurement — see Context features.

The split is not cosmetic. Ad hoc calls — auto-generated session titles, push-notification summaries, the CLI status probe — have no session to hang off, so they would be invisible in the session table by construction. They are also the population most likely to contain a bug that bills quietly forever, which is a good reason to give them a page of their own rather than folding them into a total.

`request_id`, not `uuid`

This is the one detail worth internalising before trusting any number here.

A single API call is written to the transcript JSONL as several assistant lines — a thinking block and a text block are separate lines — and every one of those lines repeats the same usage object. On top of that, resuming a session replays its whole prior history into a new file, so the same call recurs across files too.

Summing per line, or per line uuid, therefore counts one API response once per content block. Measured against this deployment’s own corpus that over-counts tokens by 79%. The API’s requestId is one call, and a unique index on it is what makes the count correct and the ingestion idempotent.

Two consequences follow from the unique index:

Re-scanning a file is free, so the recurring sweep and a full backfill can run at once.
A line with no requestId is skipped rather than stored under a substitute key. Inventing a key is what produced the 79% error in the first place.

Lines whose model is <synthetic> are skipped too: those are locally-generated error and interrupt notices that never reached the API.

Volumes are stored; prices are not

The tables hold token counts. Dollars are computed on read by TokenPricing.

Rates change, models get added, and the useful questions are comparative — what would this workload have cost on Sonnet, what does last month look like at today’s rates. Baking dollars into the rows would foreclose all of that and leave a column that silently ages. Nothing needs a backfill when a price changes; the page just reports differently.

The same reasoning is why the REST API returns its rate table alongside every figure: a dollar amount without the rates that produced it is not reproducible.

Cache tokens are three columns, not one

cache_creation_tokens is split into cache_creation_5m_tokens and cache_creation_1h_tokens, because the two bill differently:

Bucket	Rate, relative to that model’s base input
Cache read	0.1×
Cache write, 5-minute TTL	1.25×
Cache write, 1-hour TTL	2×

A single collapsed cache_tokens column cannot be priced at all. The distinction is not academic: on this deployment cache writes are the largest single line item — several times the cost of everything the models actually produced — and 95% of them are on the 1-hour TTL. A page that showed one cache number would hide both facts.

Older transcript lines predate the cache_creation sub-object. Those are charged at the 1-hour rate, the conservative reading and the right one here.

How usage gets in

Ingestion is per runtime, because where a runtime records what it spent is a property of the runtime. RuntimeRegistry::Bundle#usage_ingestor_class is the slot that says which:

Runtime	Ingestor	Reads
`claude_code`	`TokenUsageIngestionService`	`~/.claude/projects/<sanitized-working-directory>/*.jsonl`
`pi`	`PiTokenUsageIngestionService`	`sessions.transcript`, for `agent_runtime = 'pi'`
`codex`	`CodexTokenUsageIngestionService`	`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` and `.jsonl.zst`

session_token_usages.agent_runtime records which, and GET /api/v1/costs/records both returns it and filters on it — a consumer reconciling against one provider’s bill needs to be able to subtract the others’.

Two of the three read something other than a plain per-session file, and each is what forced the slot to exist:

Pi is the only runtime whose conversation is not in its home — PiRuntimeAdapter points --session-dir at <clone>/.pi/sessions, so the transcript is reaped with the clone. Reading it off disk would lose a Pi session’s whole spend the moment it was archived, so the ingestor reads the durable copy TranscriptPollerService keeps in sessions.transcript instead. See Pi’s token usage below.
Codex writes into a date-partitioned tree four levels deep and Zstandard-compresses a rollout once it finishes, so half the corpus is not a text file at all — and its token events carry no per-call identifier and no model. See Codex’s token usage below.

Two jobs drive ingestion, and both run themselves — there is no step here that needs a shell on the production box:

TokenUsageIngestionJob, on a 10-minute cron, running every ingestor the registry names and scanning only transcripts touched in the last two hours. The lookback overlaps the interval generously so a missed run, a deploy, or a late-written transcript closes itself on the next pass. One ingestor failing is logged at error — which pages — and does not cost the others their sweep.
TokenUsageBackfillJob, on a 5-minute cron, sweeping the whole corpus once. It works against a token_usage_backfills row — one row per sweep — in two-minute slices, recording a cursor after each committed chunk. On the first tick after a deploy, if no sweep has ever finished, it starts one. Once one has, the job is an indexed lookup and nothing else, on that deploy and every deploy after it.

bin/rails token_usage:backfill   # same object, no budget, in the foreground — for a developer
bin/rails token_usage:summary    # what is stored, and how far back it goes

The rake task is a convenience, not the delivery mechanism. It resumes the run already in flight rather than starting a competing one.

Why a job and not a rake task

Getting history into the ledger used to mean somebody SSH’ing into production and running rake token_usage:backfill. That is an operational step this deployment is not supposed to have: deploy is the delivery mechanism for ops actions, and the Costs page was quietly wrong until a human found the time.

The property that makes an unattended sweep safe is the one the unique index already gave us: ingestion is idempotent on request_id. A re-swept directory writes nothing, so a slice that dies mid-chunk, a recurring sweep overlapping the backfill, and a full re-scan all cost time and nothing else. The cursor only advances on a committed chunk, so an interrupted run resumes rather than restarting.

TokenUsageBackfillJob runs on the maintenance queue, deliberately not pollers or default: it holds its thread for minutes, while those queues carry latency-sensitive pollers and control work respectively.

How complete is the page?

TokenUsageBackfill.coverage is the single answer, rendered three ways so the surfaces cannot disagree:

Surface	Shows
The Costs page	A panel: backfilled or not, progress while sweeping, the date the ledger starts, and a Re-scan history button
`GET /api/v1/costs`	A `ledger_coverage` object alongside every rollup
`get_costs` (MCP)	A Partial history warning while sweeping; the covered window once finished
Supervisor	`token_usage_backfills` — every sweep, its cursor, counters and last error, read-only

covers_since is the oldest call actually stored — the only defensible answer to “how far back does this go”. Before a backfill it is roughly the deploy that shipped ingestion; after one it is the oldest transcript on disk.

A re-scan can be asked for from the page’s button, POST /api/v1/costs/backfill, or the backfill_token_usage action on the action_health MCP tool. All three are the same idempotent request: they join a sweep already in flight rather than starting a second.

Ingestion deliberately does not hook into TranscriptPollerService. That service is a live-broadcast path running inside AgentSessionJob, so hanging accounting off it would put a write on a hot path — and it would still miss both sessions that finished before the feature existed and the app’s own claude -p calls, which have no session and therefore no poller.

How a Claude Code transcript is attributed

Claude Code transcripts live under ~/.claude/projects/<sanitized-working-directory>/, and that directory name is the only evidence of what produced them.

Directory	Goes to	Labelled
`-home-rails--zimmer-clones-…`	`session_token_usages`	agent root from the clone subdirectory
`-rails`	`adhoc_token_usages`	`cli_status_probe`
`…headless-inference…`	`adhoc_token_usages`	`session_title`
anything else	`adhoc_token_usages`	`unknown`

cli_status_probe is a closed source: it exists because CliStatusService checked Claude Code’s auth with claude whoami, and whoami is not a subcommand — so the CLI read the word as a prompt and answered it with a full agent turn, every two minutes on cron (#536). The check now reads ClaudeCredentialHealth in-process and makes no model call, so no new rows carry this label. The historical ones stay: spend that happened is still spend.

Linking a row back to a Session uses two strategies, because neither covers the corpus alone: the transcript filename is <session_id>.jsonl for a main transcript, and the clone directory (created per session) covers agent-*.jsonl subagent files and resumed sessions whose runtime uuid drifted. A row that matches neither is still stored — spend that happened is still spend — and shows up as unattributed rather than disappearing.

Pi’s token usage

A Pi session file is a tree of JSONL entries, and four of its entry shapes can carry a usage object: an assistant message, a tool result that did nested LLM work, a compaction, and a branch summary. All four are ingested, which is what makes Zimmer’s total agree with the one Pi shows in its own footer.

Three things differ from the Claude Code path, and each follows from the format:

The key is synthesised. Claude Code stamps every call with the API’s own requestId. Pi records an OpenRouter responseId on assistant messages but on none of the other three shapes, so rows are keyed pi:<pi session uuid>:<entry id> — globally unique, stable across re-ingestion, and stable across a fork, whose copied transcript prefix therefore writes nothing rather than double-counting its source’s spend.
The model is <provider>/<model> — openrouter/anthropic/claude-opus-4.6, the same id ModelCatalog offers and the session config stores. A compaction or branch summary carries no model of its own, so it is attributed to the model in force when it ran.
A bare cacheWrite is a 5-minute write. Claude Code’s unsplit cache-creation figure is charged at the 1-hour rate, the conservative reading of a line that predates the sub-object. Pi is the other way round: cacheWrite1h is an explicit field and Pi’s own cost math prices anything outside it at the short rate, which is what OpenRouter billed.

That last point is what makes the money right rather than merely plausible. Zimmer prices Pi rows the way it prices every other row — stored volumes at TokenPricing’s list rates — and for the Anthropic models on OpenRouter those rates are the published ones, so the figure reproduces the cost Pi recorded beside the volumes to the cent. Measured across the three production Pi sessions that existed when this shipped: $0.582929 either way.

History is covered twice over, and it needs both. A one-time post-deploy task, 20260906190000_ingest_pi_session_token_usage, sweeps every Pi session that existed when the ingestor shipped — in id order, 25 at a time, resumable from its cursor. That task is terminal by design, so TokenUsageBackfillJob also sweeps Pi’s whole corpus at the head of every run it works: without that, any gap wider than the recurring job’s two-hour lookback — a worker outage, an ingestor bug found a day later — would lose that spend permanently with no surface to ask for it back. So Re-scan history on the Costs page, POST /api/v1/costs/backfill and action_health’s backfill_token_usage all recover Pi spend, exactly as they do Claude Code’s.

Both are no-ops on a second run for the same reason every ingestion run is: rows are keyed on request_id and written with insert_all ... unique_by.

Codex’s token usage

A Codex rollout is JSONL too, but it supplies less than either sibling format does, and every decision below follows from something it does not carry.

The corpus is not where the Claude scanner looks. Rollouts live under ~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<uuid>.jsonl — four levels, date-partitioned, resolved through CodexHome so CODEX_HOME moves both the writer and the reader together. A finished rollout is Zstandard-compressed in place to .jsonl.zst, so most of the corpus is not a text file; both extensions are read, streamed a chunk at a time through the zstd-ruby the transcript source already depends on.

The delta is recorded, not the running total. Tokens arrive on an event_msg line whose payload type is token_count, and it reports them twice — total_token_usage, cumulative for the whole rollout, and last_token_usage, just this turn. One row per API call is what the table means, so the turn’s delta is what lands. That is also the more complete of the two readings: on production rollout 019ffff8-… the cumulative counter freezes across a context compaction while the delta still reports the 21,983 tokens the summarization spent, so the running total ends 21,983 short of the sum of its own deltas.

The key is synthesised from the timestamp, because a token_count event carries no identifier of any kind — no request id, no turn id, no event id. Rows are keyed codex:<rollout uuid>:<event timestamp>. The obvious alternative, the event’s ordinal position in the file, was rejected: it is not stable under a dropped line, so one unparseable record in the middle of a rollout would shift every later ordinal and re-ingest the rest of the session as new spend. Two events sharing a millisecond are numbered in file order (…#1), which an append-only file makes deterministic.

The model is tracked forward. It is not on the token event; it is on turn_context (and on the thread_settings_applied event), emitted at the head of a turn. Both are followed as the model in force, so a rollout whose model changes mid-session attributes each turn to the model that served it. A token event arriving before either has been seen is skipped and counted in the run’s skipped figure — model is NOT NULL, and inventing one would put a wrong rate on real volume.

Attribution has two strategies, like the Claude path. The rollout uuid is Codex’s own thread id, which Zimmer captures into sessions.session_id; failing that, the cwd Codex stamps on session_meta is the session’s clone path, and the same clone_path LIKE lookup resolves it. A rollout matching neither is still ingested with a null session — spend that happened is still spend — and shows as unattributed rather than disappearing.

History is covered the same two ways Pi’s is: the one-time post-deploy task 20260907090000_ingest_codex_session_token_usage pages the rollout tree in sorted path order, 25 files at a time, resumable from its cursor — and TokenUsageBackfillJob re-sweeps the whole corpus at the head of every run it works, so Re-scan history recovers Codex spend too.

Picking a window

The Costs page carries both a set of one-click horizons — 24 hours, 7 days, 30 days, 90 days, 1 year — and an explicit from/to calendar range. Both resolve through CostWindow, which is also what every drilldown link on the page carries, so clicking into a breakdown never silently changes the window under you.

The calendar fields are plain <input type="date"> elements. That is the control a phone renders as its own native calendar, and it needs no JavaScript. A reversed range is swapped rather than rejected, a half-filled one still resolves, and a span longer than a year is clamped to the most recent 365 days rather than handed to Postgres as a full-table scan.

The same window is expressible over MCP and REST: get_costs takes days, or from/to as YYYY-MM-DD; GET /api/v1/costs takes days, or from/to as ISO-8601 instants.

Drilling in

The daily-spend chart and the breakdown tables both reveal what is behind a figure.

Each bar in the daily chart is a button. Hovering it on a pointer device, or tapping it on a phone, fills the readout strip below the chart with that day’s session/ad-hoc split, tokens, calls, and biggest agent roots. The readout sits in normal flow rather than floating over the bar, which is what makes it legible at a 375px viewport.
Each row in a breakdown table is a <details> element. It opens on click, on tap, on Enter, and with JavaScript disabled; hover_details_controller.js adds hover-to-open on pointer devices only, and a row you opened deliberately stays open when the pointer leaves.

Every figure a drilldown shows is precomputed in the same cached snapshot as the page — the per-day and per-root detail come from one extra grouped query each, not one query per bar.

Narrowing the whole page

Opening a row says what that row is made of within the current page. Narrowing makes the whole page about it: the daily series, the model split, the token decomposition, the feature estimate and the experiment cohorts all recompute for one agent root, or one session.

Each By agent root row carries an Only this link, and each Most expensive sessions row does too. They are ?agent_root= and ?session_id= on the same page — the two arguments get_costs takes and GET /api/v1/costs/records filters by. The folded other (N) row is several roots at once, so it gets no link: past CostAnalytics::TOP_N a root is reachable by typing ?agent_root= or by asking get_costs, not by clicking. Beware the router’s split identity while narrowing — see the limitation.
A narrowed page says so in a banner and offers the way back. CostScope (app/services/cost_scope.rb) is the object behind it, the parallel of CostWindow: the window and the scope round-trip together, so changing the horizon keeps the root and clicking into a root keeps the horizon. The re-scan button carries both as well.
session_id wins when both are given, which is what get_costs does with the same pair. A session_id that is not digits is ignored rather than read as session 0.
The scope is part of the snapshot’s cache key. Without that a narrowed page would read the fleet’s cached bundle back under the root’s heading.

Two things a narrowed page deliberately does not show the same way:

Ad hoc spend. Those are Zimmer’s own calls, made outside any session, so they carry no agent root: a root-scoped page excludes them rather than showing the fleet’s. A session-scoped page keeps the ad hoc calls made about that session — its generated title, its push summaries — which is the column GET /api/v1/costs/records?kind=adhoc&session_id= filters on.
Burn rates. They are current fleet rates over a fixed sample of recent sessions, not a rollup of anything on the page, so they are not rendered under a heading that says one agent root.
The experimental-setting cohorts. The cohorts themselves narrow, but the corpus they are counted against does not — ExperimentAnalytics reads the deployment’s whole tagging history to say how many sessions carry each label. A one-session A/B comparison is not a comparison anyway, so the panel is a fleet-page control.

Per-session cost

Each dashboard card and each session detail page carries the session’s own total, in muted gray. It is a secondary signal, not a headline: it sits beside the session id and the timestamp rather than anywhere prominent.

The figure comes from SessionTokenUsage.cost_by_session, one grouped query per rendered collection, warmed by preload_session_costs in each partial that renders a list of cards. A per-card lookup would be a per-card round trip on a page that renders hundreds of them.

A session with no stored usage renders nothing rather than $0.00 — usually it just means its transcript has not been swept yet, and a zero would assert it was free.

Burn rate by harness + model

The Costs page answers “what did we spend”; the scheduler needs “what does a minute of this cost”. BurnRateRecomputeJob derives the second from the first every 20 minutes and writes it to harness_model_burn_rates, one row per (harness, model).

The rate is dollars per minute of elapsed session time: for each of the last HarnessModelBurnRate::SAMPLE_SESSIONS (25) sessions of that combination, the cost of its calls over the span from its first to its last, summed across the sample before dividing. Summing before dividing is deliberate — averaging per-session rates would give a two-call session the same weight as a two-hour one. A session with a single call is floored at one minute rather than dividing by zero.

harness is the agent root, the same dimension the by-root table above breaks spend down by, because that is what predicts a session’s spend shape: a router turn and a merge-gate turn move very different money on the same model.

Only Claude Code rows are sampled — SessionTokenUsage.quota_bearing. This is the one place where “what did we spend” and “what draws down the quota window” come apart, and both readers of the distinction are easy to miss because neither says Claude in its name. These rates exist for the spot gate, which prices the running Claude fleet against an Anthropic window; a Pi row is an OpenRouter invoice and a Codex row draws on a ChatGPT plan, and neither has a claim on that window. Their own rates could never be looked up anyway — a Pi row’s model reads openrouter/anthropic/claude-opus-4.6 and a Codex row’s gpt-5.6-terra, where a Claude session’s config reads opus — but both would still land in the cost-weighted fleet average that prices a combination nobody has sampled, and a Codex row would drag it toward zero because its model is unpriced. QuotaCapacityCalibrator carries the same filter for the same reason, and there it matters more: it divides spend by Anthropic’s reported utilization, so a dollar Anthropic never counted inflates the capacity estimate in proportion. The Costs page, the REST index and get_costs are deliberately not filtered — they are asked what Zimmer spent, and the answer is every runtime.

The prices are TokenPricing’s, applied through the same cost_sum_sql the rest of this page uses. There is deliberately no second pricing path — a rate priced differently from the page would make “this session burns $0.40 a minute” and “this root spent $6,116 this week” two numbers nobody could reconcile.

The spot gate multiplies these by its re-check interval to project what admitting one more session will spend before anyone looks again. A combination that has never been sampled is priced at the cost-weighted fleet average rather than at nothing, so an unknown harness cannot look free; a combination with no spend in the last 30 days has its row dropped rather than left to inform the scheduler forever.

Context features

token_usage_features answers a question the usage tables cannot: not what did this call cost, but what was it carrying. The injected goal block, the operator’s session notes, skill bodies, MCP responses, tool output, extended thinking — each is a decision someone made, each bills again on every turn it stays in the context, and none of them is individually visible in a bill. (The session hierarchy and the human-message record have their own lines too, and are the worked example of a feature that stopped being injected — see When a feature moves rather than disappears.)

These numbers are estimates, and the page says so

The API reports one usage total per request with no per-content-block decomposition. Nothing it returns says how many input tokens were the goal text versus a skill definition versus an MCP tool result. So a per-feature figure cannot be measured — it can only be estimated from what the transcript records, and the estimate is built so it cannot mislead:

ContextFeatureAttributor walks each transcript in order, measuring the characters each feature contributes.
Characters convert to tokens at a fixed ratio — 3.7, which is measured rather than guessed. Between two consecutive requests the fixed prompt prefix cancels, so the change in billed tokens over the change in transcript characters reads the ratio directly; over 2,782 such pairs on this deployment the median is 3.57 and the mean 3.93. Re-measure with rake token_usage:calibrate_chars_per_token.
Every share is divided by max(estimated, actual), so the parts can never exceed the request’s real totals. When the estimate overshoots, all shares scale down together rather than any one being trusted.
Whatever is left is carried as an explicit unattributed line, never spread across the features.

That residual is large, and its size is the finding rather than a defect: on this deployment about 56% of tokens are unattributed. It is the fixed prompt prefix — the harness system prompt and the tool schemas of every MCP server attached to the session — plus per-request web-search charges. None of it appears in a transcript. Because it is a per-request constant, the lever that shrinks it is attaching fewer tools to a session, not writing shorter prompts.

Marginal, not cumulative — and why dollars ≠ tokens

Content added at turn 3 is in the prompt for every turn after it, so “the goal block is 3,600 characters” does not answer “what did the goal cost”. The attributor keeps two buckets per conversation:

Bucket	What it holds	What it is billed as
`carried`	everything already in the prompt	cache read, at 0.1× base input
`pending`	everything added since the last request	cache write, at 1.25× or 2×

cache_read_tokens splits across carried; input_tokens + cache_creation_* across pending; output_tokens across the assistant’s own blocks. This is why the page reports both tokens and dollars and why they rank differently: a feature re-appended to every turn is cache-written at up to 2× on every turn, while one that lands once in the prefix is written once and read back at a tenth forever after. Dollars is the column that decides.

Adding a feature

Append one entry to ContextFeatureRegistry and re-run ingestion:

feature(
  key: "my_thing",
  label: "My thing",
  blurb: "One line the page shows under the bar.",
  owner: :zimmer,
  pattern: %r{<my-thing>[\s\S]*?</my-thing>}   # or a block: { |block| block.type == "tool_result" }
)

That is the whole procedure. Because ingestion is a re-runnable scanner over the transcripts on disk, a detector written today is backfilled over everything still retained — no instrumentation at the call site, no waiting for fresh data.

Retention bounds the backfill. Claude Code prunes ~/.claude/projects on its own schedule; on this deployment the corpus goes back about 30 days in bulk. A new detector can therefore see roughly the last month, not all history. The usage rows already ingested keep their totals — only the per-feature split is limited.

owner is what makes the table actionable. :zimmer marks the features this repository chose to inject and can therefore choose to stop injecting, shrink, or serve from a cheaper model; :harness and :work are cost you can see but not directly legislate.

When a feature moves rather than disappears

Dropping an injected block usually means the content still gets read — it just arrives another way. Provenance is the worked example: session_hierarchy and human_messages are no longer injected at all, and what a session fetches with get_session_provenance instead lands on its own provenance_tool line rather than in the shared “MCP responses” bucket. Without that third line the change would read as pure savings while the bytes had merely moved. When you stop injecting something, give what replaces it a line of its own, placed before the generic detector that would otherwise swallow it.

Keep the detector for the block you stopped injecting. Ingestion re-scans the transcripts still on disk, so a removed detector does not zero a line — it strands a month of history in the residual. session_hierarchy and human_messages still match, and trending to zero on new transcripts while provenance_tool picks up is the before/after.

And give a detector to whatever the removal exposes. A block with no detector falls through to prompt, whose owner is :work — so a block Zimmer chose to inject gets billed to the user’s task. Dropping the provenance blocks left <unavailable-mcp-servers> and <attached-files> sitting directly after the goal suffix with nothing claiming them, so both got a line of their own (unavailable_mcp_servers, attached_files), and the goal region’s lookahead was widened to stop at each of them rather than swallowing them.

What the attribution cannot see

Extended thinking is under-counted. The harness writes thinking: "" into the transcript and keeps only the cryptographic signature — across 955 thinking blocks in this deployment’s recent corpus, not one retained its text. The signature is counted; the reasoning is not, and the difference lands in the residual. No detector can fix this.
System reminders are rarely persisted, so the CLAUDE.md contents injected on the first turn usually fall into the residual too.
Server-tool charges are not split. A web search bills per request, not per token, and there is no defensible way to divide a per-request charge across the content features inside that request. It stays on the parent row and lands in the residual.
Volume. The table grows at roughly the number of features detected per request — about 8× the parent table. request_id carries a foreign key to the parent’s unique index, so deleting a usage row takes its attribution with it.

Check the reconciliation yourself at any time:

bin/rails token_usage:attribution_report DAYS=30
bin/rails token_usage:calibrate_chars_per_token

Experimental settings

An experimental setting is a global switch that changes how Zimmer drives its agents — MCP tool search is the first, Settings → Experimental is where they live, and ExperimentalSettingsRegistry is the catalogue. Every session is tagged with what each one was when it started and when it last ran, and the Costs page compares the cohorts that tagging produces.

The registry is the single edit point. One entry there gives a setting its toggle on the settings page, its write path in AppSettingsController, its per-session tag, and its section in the report. A setting cannot be togglable and untracked.

Two values, not one

session_experimental_flags stores value_at_start and value_at_end per (session, setting). Intermediate toggles are deliberately not tracked; the pair exists to answer a different question — is this session’s cohort label safe? A session whose two ends disagree ran under both values of the setting, so it is bucketed as mixed and excluded from both cohorts rather than rounded into one.

Observed, not derived

The value could be re-derived at read time from the session’s date and the date the setting shipped. That works exactly once, for the single step change a setting makes when it lands, and stops working the moment the setting is toggled back. Storing what was actually observed is what lets cohorts interleave in time — which is the difference between an A/B test and a before/after chart.

The date-derived path exists only for history that predates the table. ExperimentalFlagBackfillJob labels those sessions from landed_at in the registry entry: created_at decides the start value, the session’s last recorded API call decides the end value, and rows written that way are marked source = "backfilled". The report shows how many of each it is reading.

The job cannot reach forward past the first live observation, and that bound is load-bearing. landed_at describes exactly one step change, so it knows nothing about a later toggle. Without the cutoff, a session parked in waiting would be labelled from its creation date, then run under whatever the setting had since become, and land in mixed — the backfill would silently destroy exactly the interleaved cohort a deliberate toggle was flipped to collect. The cutoff is MIN(first_observed_at) over the observed rows, falling back to an hour ago on the one tick where nothing has been observed yet and every session really is history.

For mcp_tool_search the boundary is 2026-08-22 13:55:34 UTC (b59d9ad7, which shipped it on for everyone).

Retiring a setting

When the question a setting was asking has been answered and the losing branch is deleted, its registry entry and its AppSetting column go with it. The session_experimental_flags rows already written under that key are left alone — they honestly record what those sessions ran under — and the report simply stops rendering a section for a key the registry no longer knows. provenance_via_mcp went that way: the answer was to stop injecting provenance entirely, so there is no longer a switch to compare across, and the before/after lives on the context-features table instead.

Why the report hedges as hard as it does

The settings are global, so nothing is randomized and a cohort is “whoever ran while it was on”. For a backfilled setting the cohorts are literally before this date and after this date, which perfectly confounds the setting with everything else that changed around then — including other merges landing the same afternoon. That is stated on screen, next to the number, rather than in a footnote.

Three things follow from it:

Cost per API call is the headline, not cost per session. Per-session cost mostly measures how long sessions happened to be, which is task mix. Per-session is still shown — it is what the bill feels like — and labelled as the confounded one.
Sample sizes are always on screen, and a side thinner than 5 sessions or 50 API calls in the window prints no percentage at all. A dramatic-looking delta over four sessions reads exactly like a real one, which is the failure this refuses.
The same agent root is compared against itself in a drilldown, using only roots present on both sides. Holding the root constant removes the largest single source of task mix. It does not remove the rest.

The honest reading of a thin report is that the data cannot yet support a claim. That is a correct outcome, not a broken page.

Adding a setting

Append an entry to ExperimentalSettingsRegistry::BUILT_INS with the AppSetting boolean column that backs it, plus a migration for the column. Set landed_at only when the setting shipped as a step change for everyone; leave it nil and only sessions from then on get tagged, which is the more honest default for a setting that ships off. A Zimmer Extension flagged experimental needs nothing at all — it is picked up from Zimmer::ExtensionRegistry.experimental automatically.

Reading it back

Web: the Costs page, alongside Inference.
REST: GET /api/v1/costs for rollups, GET /api/v1/costs/records for the rows themselves, paginated and filterable by session, agent root, model, or source. See the REST API.
MCP: the get_costs tool, in the health group. Fleet-wide by default; scopeable to one agent root or one session, and windowed by days or by an explicit from/to. Every report carries the context-feature split, always labelled as an estimate, and the experimental-setting cohorts, always labelled as observational.
MCP, from inside a session: the same tool name on the self_session surface every session is given, as a composite override hard-scoped to the caller. A session can ask what it cost; it cannot ask what the fleet cost, and agent_root is refused outright. See the MCP server.

What the dollar figures are not

They are not a bill. These accounts are subscription-billed, so list price here is a comparable unit across models — the thing that lets you say Opus cost 4× what Sonnet would have on the same volumes — rather than money owed.

A model with no rate configured prices at zero and is named explicitly on the page, in the API response, and in the MCP output. That is deliberate: a wrong rate is worse than a visibly missing one, because it lands inside a total that reads as authoritative. A new model id showing up in that warning is the signal to add it to TokenPricing::RATES.