Spot and priority sessions

Quota is finite and the work is not. On a busy afternoon a merge gate, three scheduled sweeps and a label poller can be spending the same 5-hour window that the thing you actually asked for needs.

Zimmer’s answer is to classify sessions by where they came from, and to let the automated ones wait.

Class	Behavior
priority	Starts whenever it is ready. Never consulted about quota or concurrency.
spot	Starts while every quota window has non-reserved capacity for it, the fleet’s burn rate is inside that window’s pacing curve, and a session slot is free. Otherwise it waits and starts later. And stops if a window runs out of room while it is running.

A held or paused spot session is deferred, never cancelled. Nothing is lost.

Zimmer models each quota window in dollars: a calibrated estimate of what a full window is worth in Opus spend, a priority reserve carved out of it, and a non-reserved remainder that spot work is expected to consume in full before the window rolls over. The operator sets the reserve as a percentage on /inference; the dollar figure beside it is derived. A deployment sitting idle should be idle because it has genuinely spent its budget, never because the gate was being careful.

Where the class comes from

Three things can decide a session’s class. The first one that speaks wins:

Order	Source	Set it	Scope
1	The session itself — sessions.scheduling_class	Scheduling class on the new-session form, Run as spot on any Quick Router surface, scheduling_class on start_session / POST /api/v1/sessions; afterwards action_session (change_scheduling_class), PATCH /api/v1/sessions/:id, or the button on the hold banner	That one session, and anything it spawns
2	The trigger that fired it — triggers.scheduling_class	The trigger’s edit form, or action_trigger	Every session that trigger spawns from now on, plus its own already-spawned sessions still waiting
3	Its genesis — the default for where the work came from	/inference (only for the origins no trigger produces)	Every deriving session of that genesis, past and future

Most sessions never touch 1 or 2: both columns are NULL, and the class is derived. That is what keeps the defaults live rather than frozen into history.

The Quick Router’s spot opt-in

The Quick Router is the fastest way into Zimmer, and everything it creates is web_ui genesis — priority. That is right for the usual case and wrong for the one it kept forcing: a long unattended sweep you are not waiting on, typed into the same box, competing for quota with work someone is watching.

Every Quick Router surface carries a Run as spot checkbox, unchecked by default:

Surface	Where the checkbox is
The chat-bubble Quick Router panel, on every page	Under the prompt, above Submit / Submit & Open — both submit paths read it
The dashboard’s inline prompt (md: and wider)	At the right of the attach-button row
The dashboard’s full-screen prompt overlay (phones)	Above the Submit button
The mobile joystick’s Quick Router petal	Opens the chat-bubble panel, so it inherits that one

Leaving it unchecked submits no class at all: sessions.scheduling_class stays NULL and the session keeps deriving from web_ui, exactly as before — so promoting or demoting web_ui on /inference still moves these sessions, which stamping “priority” on the row would have quietly stopped. Only an explicit tick writes anything.

The choice is per submission rather than a preference. The box clears when the surface closes — a submit, the phone overlay closing, the panel being dismissed with Escape, the backdrop or the X. A half-typed prompt is kept across a close and the class deliberately is not: re-ticking a box is cheap, and a tick left over from an hour ago would silently park the next prompt behind the gate.

A spot submission lands at the top of the spot queue. Choosing spot here is a statement about quota, not about importance — a human typed this one seconds ago. Leaving it at the default precedence of 0 would file it beneath every automated spot session already ranked above 0, and behind every older session tied at 0, so SessionsController ranks it with Session.precedence_above_top_spot instead: a few points above the highest spot session currently queued. Each checkbox says so, and it is re-rankable afterwards from the Ranked view like any other spot session. An unchecked submission is untouched, including the chat bubble’s ordinary child-sits-just-above-its-parent bump.

Genesis

A session’s genesis is where its line of work ultimately came from — not who physically inserted the row. It is a column on sessions, assigned once at creation.

Genesis	Means	Default class	Class set on
web_ui	A human typed it into the Zimmer web app: the new-session form, the dashboard quick prompt, the chat bubble, or the Invoke button on a trigger.	priority	/inference
slack	A Slack trigger fired on a DM or a channel message.	priority	the trigger
github_issue	A github_issue trigger fired — the feed the issue-work gate reads.	spot	the trigger
github_label	A github_label trigger fired — the ready to merge feed the PR merge gate reads.	spot	the trigger
schedule	A cron-scheduled trigger fired.	spot	the trigger
ao_event	A session-state trigger fired because another session changed state.	spot	the trigger
api	Created over POST /api/v1/sessions or MCP start_session with no parent session, or fired by hand over POST /api/v1/triggers/:id/invoke / action_trigger’s invoke.	spot	/inference
unknown	Origin could not be established — chiefly rows created before genesis was recorded.	priority	/inference

Five of the eight kinds restate a trigger condition type, so their class lives on the trigger, not in a global per-kind setting: one noisy Slack trigger can be spot without demoting the eleven other Slack triggers that have a human waiting on the answer. The three that no trigger produces keep a per-kind setting on /inference.

Two of the defaults are policy calls worth stating plainly:

• unknown is priority on purpose. A session Zimmer cannot explain is never one it throttles. The failure mode of the classifier is “runs anyway”, not “silently held”.
• schedule and ao_event are spot because recurring automation runs again by definition, so a deferred run costs little. Set the trigger to priority if that is wrong for a particular one.

Genesis is inherited

A session created with a parent takes its parent’s genesis verbatim. That single rule is what makes the classification safe to act on:

Holding #103 would strand a request Tadas is waiting on. Holding #203 is exactly the intent. Both fall out of the same rule, with no special-casing of agent roots — which matters, because the gate roots (issue-work-gate, pr-merge-gate) live in a deployment’s own catalog, not in Zimmer.

An explicitly declared genesis outranks inheritance. The chat bubble is the case that needs it: it carries a parent so the conversation threads, but a human typed the message, so it declares web_ui rather than inheriting spot from whatever session was on screen.

Forks inherit through metadata["forked_from_session_id"], the only lineage edge a fork has.

An explicit scheduling_class is inherited the same way. A router told to run one long batch as spot spawns children that are also spot, without every spawn call having to repeat it — and without moving any other session that shares the genesis.

Where you see it

Genesis appears on every node of the Session hierarchy panel, as a genesis · class pill beside the agent-root pill — so “this whole branch is spot” is readable at a glance, and an outlier stands out. It is also on every dashboard card, in get_session, and in quick_search_sessions.

Stored only when someone chose it

sessions.scheduling_class is NULL on most sessions. When it is, Session#priority_class resolves from the stored genesis on every read, through the per-genesis override map in AppSetting#genesis_class_overrides.

That sparseness is the design. Promoting web_ui to spot reclassifies every deriving session of that genesis, including ones that already exist — which is what changing a default has to mean. A class denormalized onto every row at creation would only ever apply to sessions created after the click, and would freeze today’s defaults into all of history.

A trigger’s class is stamped rather than derived, because the selector is read once, when the trigger fires. So it needs a path of its own to reach sessions that already exist — and it has one: changing a trigger’s class also moves that trigger’s own sessions that are still waiting. That is the case the setting is usually changed for. An operator flipping a trigger to priority during a quota backlog is trying to release the work that is stuck right now; before this existed the change reached future sessions only, and the backlog sat exactly where it was with nothing to say why (#480).

Three clauses bound it, and each one is there to stop a click promoting work nobody asked to promote:

• This trigger’s sessions only — matched on metadata.trigger_id, never the genesis. Five of the eight kinds restate a condition type, so a genesis-wide sweep would drag in the work of every other trigger that shares the kind. That is promote_genesis’s blast radius, and it is why promote_genesis is the wrong lever for a trigger.
• Still waiting only. A session that has already started is past the gate this setting governs.
• Still carrying the class the trigger stamped. A session somebody moved by hand stays where they put it — a per-session choice outranks a trigger-wide one.

Clearing the selector back to “Default” moves them too, back to deriving from their genesis: the stamp follows the trigger’s, so a session spawned before the change and one spawned after it are indistinguishable. The form’s flash notice, action_trigger’s response and PATCH /api/v1/triggers/:id’s reclassified_waiting_sessions all report how many sessions actually moved between classes — a rewrite that resolves to the same class is not counted, because nothing moved.

A session the change promotes is then started, rather than left to wait out a re-check the promotion made moot. Each one goes through Sessions::StartNow — the same owner every other promotion path uses — so a session held behind a deferred AgentSessionJob scheduled up to an hour out has that job pulled forward instead of a second one being enqueued beside it. Until this, the change landed the class and nothing else, and the backlog an operator had just promoted went on waiting exactly as long as before (#423).

The release runs in TriggerPromotionReleaseJob, off the operator’s request and after the trigger’s save has committed. Both halves matter. Releasing is not a set operation the way the reclassification above is — each session needs its own queue read and its own reschedule — so a long backlog inside a PATCH would time out halfway through, leaving the operator unable to tell a failed release from a failed save. And the class has to be visible before any job is pulled forward, or the worker reads the session as still spot and holds it straight back.

Four populations are left alone, and each is re-read at release time rather than trusted from the write:

• A demotion, or a rewrite that resolves to the class the sessions already had. Nothing moved.
• A session that is no longer waiting, or whose class did not end up priority. It started, was moved by hand, or its promotion rolled back in between.
• A session carrying a spot-ceiling pause or an auth-outage park. Each has its own resume owner, and a pause_into_spot_queue park is a per-session choice a trigger-wide one must not override — the same rule that leaves a hand-moved session’s class where somebody put it.
• A session in a frozen category, which is opted out of every bulk start.

One session that cannot be started does not abandon the rest of the backlog.

To move a session this does not reach — one that has started, or one from a different trigger — move that session: the Make this session priority button on its hold banner, the Scheduling class selector on its detail page, action_session with change_scheduling_class, or PATCH /api/v1/sessions/:id.

The gate

With gating on, a spot session takes a turn — its first or its next — while all three of these hold, for both windows.

Check	What it means	Reason when it fails
The cap	What the fleet plus this session is projected to spend over the next ten minutes keeps total spend inside the window’s non-reserved budget. This is what protects the priority reserve, and it is absolute.	at_utilization_limit
The pace	The fleet’s burn rate, including this session’s, is inside remaining spot budget / time left in the window — the rate that reaches 100% of the budget exactly as the window rolls over.	at_utilization_limit
A free slot	Fewer sessions are running than Max sessions at once. Skipped only for a session that is already running when the gate runs — it is counted in the fleet itself.	fleet_at_cap

Why a curve rather than a target

The gate used to compare pooled utilization against a flat target percentage: under it, spot work ran flat out; at it, everything stopped. Three problems followed.

A percentage says nothing about how much capacity is left — “76% of a 65% target” cannot be compared to “keep $200 back for priority work”, so a reserve could not be expressed at all. A hard target wastes capacity: work burned through the allowance early and then idled for hours. And it bursts and then idles, when what is wanted is some work happening at every hour of the day.

The pacing curve is self-correcting in both directions, because its numerator is what is left and its denominator is the time left to spend it in. Spend below the curve and the sustainable rate rises, releasing more work; run ahead of it and the rate falls, holding work back until the window catches up. No cliff at either end.

There is always room for one session

A session is not infinitely divisible. If the sustainable rate were below what a single session burns, the pace check alone would admit nothing and leave the whole budget unspent — the opposite of what the model is for. So when no spot work is in flight, the pace check is waived and only the cap applies: one session runs, gets ahead of the curve, and the next admission waits for the curve to catch up. A duty cycle rather than an outage.

Two things do not count as spot work in flight, and both of them used to.

Priority sessions don’t. Priority work running is work happening and it does spend against the same window — but it is not the work this waiver lets through, and the spot budget goes unspent whether or not a router is running. A live deployment always has some priority session on a worker, so keying the waiver on the whole fleet turned it off permanently: production spent four days at 0–1 sessions with 33 spot sessions held and the waiver never fired (#693).

The session being admitted doesn’t. By the time the gate reads the fleet, the asking session is already inside AgentSessionJob#perform — its agents job carries a performed_at while its own row still reads waiting, so it is reported as a turn awaiting a worker. Counting it made “is any spot work in flight?” answer yes on a completely empty deployment, every time, which put the waiver out of reach at the door regardless of the question above.

Priority spend is not thereby ignored: every running session’s burn is in the rate both checks are tested against, and the reserve is never waived. A fleet of priority work that has genuinely eaten the spot budget still holds the session — on the cap, which is the check that owns that question.

Before the first calibration

QuotaCapacityCalibrationJob needs a window with measurable spend and utilization before it can price one. Until then a window has no dollar figure, and the model reasons in fractions of the window instead: every quantity means the same thing on a 0–1 scale, the cap and the curve work identically, and only the burn-rate projection is unavailable. Every surface says which mode a window is in — an estimate is never presented as a measurement, and “not calibrated” is never rendered as $0.00.

The same checks decide for a session that is already running, which is what makes the budget a ceiling rather than only a starting line — see The budget is a ceiling.

The concurrency limit

Max sessions at once (default 10) is what bounds how fast the quota can be spent, and its semantics are deliberately asymmetric:

• Priority sessions are never held by it. A priority session starts whenever it is ready, even with every slot taken.
• Priority sessions still count toward it. The number counted is every Claude Code session a worker is running, whatever its class. (Codex and Pi sessions spend nothing against a Claude account, so they do not take a slot.)
• So ten running priority sessions leave zero spot slots — priority work is meant to crowd spot work out, and that is the intent rather than a side effect.

It is checked when a session starts, and for a spot session that is the whole of it: lowering the limit under a running fleet holds the next spot start and never interrupts spot work already underway on the strength of the number alone.

A priority start is the exception, and it is the third bullet made true of work already in flight. Until preemption a priority session starting into a full fleet simply ran one over the ceiling, so “ten running priority sessions leave zero spot slots” was a claim about sessions that had not begun yet — the spot work already running kept every slot it had. Now the priority session takes a slot off a running spot session instead, and the fleet converges back to the operator’s number rather than growing past it.

What the ceiling counts, and what it does not

A turn is handed to a session — by a fired wake trigger, a follow-up, a poller, or the end-of-turn handoff to a queued message — well before a worker starts executing it. Between the two sits the agents GoodJob queue, which is only GOOD_JOB_AGENTS_THREADS (default 12) deep, and on a busy deployment that gap runs to minutes.

Since #1040 that queue reads waiting rather than running — see running means a worker thread has the turn — so the status column and the ceiling now agree about the common case. The ceiling still computes its population rather than counting the column, because neither status is a clean answer on its own:

Population	Counts?	Why
A turn a worker is executing (running, with a live job)	Yes	It is the fleet doing work, and it is the only population that is.
A turn queued for a worker (waiting, with a ready agents job), or a running row between jobs	No	It has taken nothing yet. What it is waiting for is a worker, so counting it makes the ceiling a limit on how much work is waiting rather than on how much is running.
A turn whose worker is still making its clone and spawning the CLI (waiting, with a job that has a performed_at)	No	No agent process exists yet and no quota is being spent. This is exactly how a first start behaved before #1040, so no denominator moved.
A dormant waiting row — a spot hold, a ceiling pause, a quota park, a session asleep on its own wake	No	It has no ready job in the lane at all, which is how it is excluded. A held session’s re-check job is parked on a future scheduled_at, and its owner is the spot ladder rather than the worker pool.
A running row asleep on its own future wake with no AgentSessionJob at all — none running, none queued	No	Nothing will happen to that session until its wake fires, so it is not even waiting for a worker.

The two uncounted rows are still told apart, and both halves of the sleeper rule are load-bearing — the second is the one that is easy to get wrong. “Asleep” alone is not enough, and neither is “a start path would refuse it”: AgentSessionJob’s pause guard is conjoined with session.waiting?, so it does not fire for a running row — a queued job would run the session and take a worker while the ceiling had stopped counting it. So the test is that no AgentSessionJob exists for the session at all, asked through PendingAgentTurns, which reads the job rows rather than sessions.running_job_id (that column is written from inside perform, so a session whose job is still queued has a blank one). “And no worker is on it” falls out of that, and it is what keeps a busy session counted: arming a wake mid-turn is the ordinary orchestrator pattern — a router calls wake_me_up_later and then keeps working — so “has a wake armed” alone would stop counting a session at the moment it is busiest.

A row reaches running-while-asleep when its turn ends with something else already in flight for it — a queued message the handoff path picks up, or a recovery job CleanupOrphanedSessionsJob enqueued — and that something then finishes without pausing it.

Both /inference cards print what the count leaves out — “17 more waiting for one of the 8 worker slots · not counted” — because a queue that is invisible is the whole explanation for why nothing of yours is moving. Waiting for one rather than queued, deliberately: that bucket is every row with a turn coming that no worker has started, which is turns in the agents lane plus rows between jobs (the handoff window, a first spawn not yet enqueued, and the orphans CleanupOrphanedSessionsJob repairs). Reading the count with the queue folded in is exactly how #957 was reported. RunningTurns is the one place the distinction is made; both ceilings read through it.

The consequence for tuning: the agents pool is a hard bound on both ceilings. The count is turns a worker is running and the pool runs GOOD_JOB_AGENTS_THREADS (default 12) of them, so a ceiling above that can never be reached — the spot gate would never report fleet_at_cap, and top-up would always see the fleet as having room, while work keeps queueing behind the same twelve workers. Nothing clamps the setting: the number you type is yours, and growing the pool is a deploy away. Both /inference cards and get_spot_policy say so when your number is above the pool, and print the effective ceiling — min(configured, GOOD_JOB_AGENTS_THREADS) — beside it. Both shipped defaults — spot_max_concurrent_sessions 10 and the top-up ceiling 3 — now sit under the default pool of 12, so an un-retuned deployment gets ceilings that bind. A deployment that raised its ceilings to work around the old pool of 8 should bring them back under 12, or it keeps the unreachable-ceiling behaviour for no reason.

What actually bounds the pool: memory, and not the way you would guess

GOOD_JOB_AGENTS_THREADS is sized by what the sessions cgroup pool can hold, not by the database, because each of its threads runs a whole agent session. The history is worth keeping, because the intuition it corrects is a common one.

Measured over the 24 hours to 2026-09-05T14:16Z, at 8 threads and before #981’s fix, against the worker’s 10 GiB memory.max, strongest evidence first:

Metric	Value	Weight
memory.stat anon peak	9.07 GiB of 10 GiB	Decisive — anonymous memory is unreclaimable, so this is what decides whether N sessions fit
memory.events oom_kill	≥ 5	Decisive — real kills. Per-container maxima, and the counter resets when a deploy recreates the container, so the true total is higher
memory.current peak	10,737,324,032 B — 94 KB under the cap	Corroborating — includes reclaimable page cache (file peaked at 7.28 GB), and a cgroup doing heavy file I/O fills toward its limit with cache as a matter of course
memory.events max (forced reclaim)	133,791	Corroborating — cache-driven reclaim fires this too

The failure that produces is #981: eight in-budget sessions — no runaway, largest process 943 MB — summed over the container cap and the kernel OOM-killed the GoodJob worker itself, taking every in-flight AgentSessionJob with it. Per-session cgroups do not help here and are not meant to: cgroup v2 is hierarchical, so zimmer.sessions/sessions/session-<id> sits inside the container’s cgroup and charges the same 10 GiB. ZIMMER_SESSION_MEMORY_MAX_MB bounds one runaway; ZIMMER_SESSIONS_MEMORY_MAX_MB bounds the sum, on a pool the Rails worker is not inside — so the pile-up now kills a session rather than the worker that runs all of them.

That changes the victim, not the demand — and the victim is the part that mattered. Because the pool’s memory.max is absolute, admitting more sessions no longer endangers the worker through session memory; it spends pool headroom instead. That is what let the thread count go 8 → 12: overshoot costs one session, which GoodJob retries, where it once cost every session plus the worker.

One path stays outside the pool, and the qualifier above is doing real work. bin/docker-entrypoint runs the cgroup delegation after starting the inner dockerd, deliberately — so the daemon and the .agent-containers dev stacks it manages are charged to the container cgroup, alongside the worker. GOOD_JOB_AGENTS_THREADS is the only thing bounding that path, which is the residual risk in raising it. See nested Docker for the accounting.

The counter-intuitive corollary: do not raise the pool to match the threads. The pool is sized from what must survive a pile-up, not from how many sessions are admitted. Two tenants live outside it and both have to fit in the residual — the Rails worker (~1.6 GiB measured) and the inner dockerd with the dev stacks it runs (~1.5 GiB), ~3.1 GiB together. At 6144 the residual is 4096 MB and covers that. At 7168 it would be 3072 MB — under the measured need — so the container cap would fire first, and that OOM selects across the whole container and takes the worker. The pool has to fire first, so 6144 stayed put when the threads moved.

What 12 costs is headroom for concurrent heavy work. Measured on the live worker at 12 session cgroups: pool anon 3154 MB of the 6144 cap, ~263 MB per session, leaving ~3.0 GB — about five concurrent capped test suites at ~560 MB each. The conservative figure from #981’s peak task dump (~382 MB per session) would leave ~1.6 GB, or two to three. The real tolerance is somewhere in that band and is not pinned down. The connection budget is not what binds first, and is not roomy either: 12 threads derive 91 required backends against the 97 a db-s-2vcpu-4gb cluster serves, and 15 would derive exactly 97 — the whole plan, zero margin, which is the other reason 12 rather than 15.

Its sibling: the backlog top-up ceiling

Max sessions at once says how much work may run. The Backlog top-up card directly below it on /inference says how little work means the fleet has room for more — the ceiling no_sessions_in_progress fires under, which is what hands an under-used fleet more to do. The two are read together: raising the concurrency limit without raising the top-up ceiling buys slots nothing fills them, which is exactly what left a ten-slot fleet running at two.

Its count is sessions a worker is running, on the same reading the concurrency limit uses (see What the ceiling counts): a backed-up spot queue does not suppress top-up, and sessions in waiting do not count against it, of any class. Both ceilings are therefore about work actually executing — “hold spot work above 10 on a worker, top up below 3” — but they are not the same count, so do not expect the two numbers on the page to match. The concurrency limit reads Session.running_claude_code_count: Claude Code sessions only, frozen categories included. The top-up ceiling reads FleetIdleMonitor.running_sessions: every runtime, frozen categories excluded. A fleet running Codex or Pi work shows up in the second and not the first. Both go through RunningTurns, so they agree about what a running row means and differ only on runtime and frozen categories. The full rules live under no_sessions_in_progress.

The card’s Under its ceiling since is the moment the fleet crossed below that ceiling, not the last time a session started — sessions coming and going underneath the ceiling do not move it. On a fleet that stays under its ceiling, top-up’s cadence is therefore the cooldown alone; see Why the cadence is the cooldown alone.

Every change to both ceilings is recorded

The gate switch, both reserves, the concurrency limit, the preemption switch and the three top-up thresholds move from five surfaces — the spot gate, backlog top-up and genesis forms on /inference, the action_spot_policy MCP tool, and a Rails console — and a change to any of them is invisible from the outside. A cap that quietly goes back down does not fail anything: spot work is deferred, never cancelled, so the only symptom is a fleet running slower than it is paid for, which looks exactly like a quiet day.

So every persisted change to the operator-set scheduling policy writes one line, naming the surface that made it and every value that moved:

[FleetPolicy] changed via web:/inference spot gate form: spot_max_concurrent_sessions 12 -> 8
[FleetPolicy] changed via mcp:action_spot_policy set_top_up session #15373: fleet_idle_max_sessions 3 -> 12

An MCP write carries the calling session as well, because one API key is shared by the whole fleet and names a caller but not a session.

The covered columns are AppSetting::FLEET_POLICY_ATTRIBUTES: the gate switch, both reserves, the concurrency limit, the preemption switch, the three top-up thresholds and the genesis class overrides. The state the pollers write on their own sweep — fleet_idle_since, fleet_idle_event_fired_at, quota_pool_available — is deliberately outside it, because a running commentary from the pollers would bury the handful of lines that matter.

Two things worth knowing about the line:

• It is WARN, and that is deliberate. The OTel exporter ships WARN and above, so an INFO record reaches container stdout and nothing else — and there is no shell on the production box to read stdout with. See Observability. WARN does not page, so a legitimate operator change records itself without waking anyone.
• It records; it does not alert. Nothing counts these lines, and no health check compares the live policy against an intended one. A revert is reconstructible from one VictoriaLogs query rather than from the fleet’s agent transcripts, but it is not announced.

update_column and update_all skip the callback, as they skip every callback. Nothing in Zimmer writes these columns that way — the pollers use them only for their own state columns — but a console session reaching for either would move the policy without a record.

Read across the whole pool, not one account

Utilization is the pool average — every Claude Code account’s latest reading, averaged. It is the same number /inference prints as Avg 5-Hour Utilization (servable accounts) and Avg 7-Day Utilization in its Account Pool section, computed once in ClaudeAccountPool and read by both, so the page’s headline figure and the gate’s decision cannot disagree.

Deciding on a single account meant one account at its cap stopped the whole fleet while the rest of the pool sat idle. Rotation moves work off a refused account onto the accounts that still have headroom, so the quota a deployment can actually spend is the pool’s, not whichever account happens to be serving this minute.

Every account counts, whatever its status — active, quota_exceeded and needs_reauth alike. An account in needs_reauth is one Zimmer cannot serve from right now, not one whose quota is spent: its windows keep draining while it waits for a human, and its headroom is real again the moment they log back in. Leaving it out would shrink the denominator to the serving accounts and make the average jump every time an account fell out of the pool or came back.

The 5-hour figure is narrower than the weekly one. It averages only the accounts whose 7-day window still has room, because those are the accounts a turn could actually land on. An account whose week is spent is left out of it — and says so on the weekly figure, where it reads 100% honestly and is what holds work while the week is spent.

It used to be counted at 100% in the 5-hour figure instead of being left out, on the reasoning that its 5-hour headroom cannot be served. That is true of the account and false of the number: the pacing curve reads this figure as 5-hour capacity consumed, so the substitution became a floor of weekly-spent ÷ accounts-read on a curve that restarts at zero every five hours — and nothing the fleet did could bring it down, because it was not about the 5-hour window at all. On 2026-09-10 that was 2 accounts of 7, all seven reading 0.0% on their 5-hour counters: a pooled 28.57% against a curve at 25.15%, holding 33 spot sessions on a fleet that had been idle for four days (#693). When every account’s week is spent there is nothing servable to average, and the figure falls back to the whole pool — which then reads 100%, correctly, without a substitution.

An account with no reading at all contributes nothing and is left out of the denominator too — the decision says how many of the pool’s accounts it averaged. When nothing has a readable window the gate falls open on no_snapshot.

When the pool comes back

The Account Pool section leads with the answer to “when does work get unblocked?”, as a clock ticking down by the second to a wall-clock time. It comes off the same ClaudeAccountPool measure as the figures below it.

An account can serve a request when both of its windows have room, so the moment it comes back is the later of the two resets it is actually waiting on — a window that already has room contributes nothing, because that room is there now. The pool’s moment is the earliest of those across its accounts.

That is the whole of the rule, and it is worth being concrete about why it is not two separate answers. An account sitting on an empty 5-hour window with its week spent comes back the moment its week does, whatever its 5-hour window is doing; an account over its 5-hour cap with plenty of week left comes back when the 5-hour window rolls. Measuring the two windows separately — a “next usable 5-hour reset” over the accounts with weekly allowance left, and a 7-day reset over the rest — cannot express the first of those, so a pool whose soonest relief was a weekly reset twenty minutes out would advertise a 5-hour rollover hours later.

The banner has three states, and the empty ones say which emptiness they are:

• A countdown, when every account with a reading is out of capacity and at least one of them can say when it returns. It names the wall-clock moment beside the clock and how many accounts are out.
• “Work is not blocked”, when an account has room on both windows right now. There is nothing to count down to, and a clock ticking toward the next rollover would read as a wait that is not one.
• “Nothing here says when work resumes”, when everything with a reading is out and none of them recorded a reset time. A zeroed clock would read as “any moment now”.

The tick is driven in the browser from an absolute ISO-8601 instant in the markup (unblock_countdown_controller.js), not from a duration the server rendered — /inference is a page people leave open, and “in 22m” is right for one second and wrong for every second after it. The server renders the same clock string from the same instant, so the first paint is already correct and the page still tells the truth if JavaScript never runs. When the deadline passes while the page is open the clock stops at now and says the reading is stale rather than counting into a negative wait.

Under the weekly average, Next 7-day reset still describes its own window: it is measured only over accounts whose week is spent, because those are the ones a weekly rollover returns to service. It is the detail behind the headline, not the headline — an account whose week returns at noon but whose 5-hour window is spent until 2pm is not servable at noon. When no account is weekly-blocked the note says that rather than naming a rollover on an account that was never blocked. It reports the soonest reset recorded among the spent accounts and counts them separately, because a spent window does not always carry a reset timestamp — when none of them does, the note is the count alone.

Nothing here counts a reset time that has already passed. A past timestamp describes a window that has already rolled over, which is the same rule the counters follow, so it is not something the pool is waiting for.

get_spot_policy reports the same two answers, from the same measurement rather than from a second one of its own: the pool’s moment as Account pool capacity beside the decision, and the 7-day rollover as Next 7-day reset under the weekly window. Where there is a time to give, it is given as a countdown and a UTC wall clock — the countdown is what a session deciding between sleeping on a wake and escalating acts on, and the absolute time is what survives being quoted into a later message. Where there is not, the tool says which of the two absences it is, in the same words the banner uses. SpotGateService::PoolCapacity carries the values off ClaudeAccountPool::Measure; the three banner states above are the three sentences the tool prints, so the page and the tool cannot answer this differently.

The gate stops reading the pool as soon as it can answer without it — gating turned off short-circuits before it ever measures — so get_spot_policy falls back to ClaudeAccountPool.measure when the decision carries no capacity of its own. The page has no such short-circuit, and with gating off the tool would otherwise go silent about a pool the page was still reporting on.

That answer is deliberately separate from the ceilings below. Account pool capacity is Claude’s quota — when an account can serve a request at all. The ceilings are Zimmer’s own gate. They come apart in both directions: a pool can be out of capacity while the gate sits open, and the gate can hold spot work on a pacing curve while every account has room.

The times are rendered as UTC on the server and rewritten to the reader’s own clock in the browser (the local-time Stimulus controller), which names the zone it converted to; the UTC reading stays on hover, and stays on screen if JavaScript never runs.

Targets and the concurrency limit are set together on the Claude Code tab of /inference, on the same page as the windows they are measured against, and all three are settable over MCP with action_spot_policy (set_gating).

Fail-open

Every uncertain condition allows the session, and the reason is named so the UI can say which:

Reason	Meaning
gating_disabled	The toggle is off.
no_snapshot	No Claude Code quota reading to decide on.
unavailable	The gate could not be evaluated at all.
within_limits	Both windows have room and are inside their curves, with a slot free.
at_utilization_limit	Held. A window’s non-reserved budget is spent, or the fleet is running ahead of that window’s pacing curve.
fleet_at_cap	Held. Every session slot is taken — by spot work, priority work, or both.

A monitoring gap must not become an outage of all automated work.

Three ceilings, two reason strings

at_utilization_limit covers two ceilings that behave differently, and the difference matters: only one of them stops work that is already running.

Decision#ceiling	What it means	Does it pause running spot sessions?	What lifts it
fleet_cap	Every session slot is taken. No window is involved.	No	A running session finishes
spot_budget	A window’s non-reserved budget is spent.	Yes	The window rolls over
pacing_curve	The budget has room, but the fleet is spending it faster than the window can carry.	No	The fleet’s burn falls

The two window ceilings share one reason string because at_utilization_limit is persisted on sessions — it is written into spot_pause_reason and read back by the banner on every session carrying it, so splitting the string would make those banners unreadable. Decision#ceiling derives the distinction instead, and /inference and get_spot_policy both report it.

SpotHoldExplanation turns it into the two sentences those surfaces render: why it’s held, and held until.

”Held until” is a condition, not a clock

Only spot_budget has a time in it. The other two are conditions, and Zimmer states them as conditions rather than reaching for the nearest plausible number.

The pacing curve is the case worth spelling out. The sustainable rate is the budget left divided by the time left, so while the fleet burns faster than that rate the numerator falls faster than the denominator and the rate keeps dropping. Waiting widens the gap. What lifts a pacing hold is the fleet’s burn falling, which means running sessions ending — and nothing in the model predicts when that happens.

So the surfaces say: “When the fleet’s burn falls to or below $X/min”, where $X is the sustainable rate less what the next session is itself projected to spend — the gate tests the sum of the two, and Window#within_pace? is <=. The budget and a free slot still have to hold, and the sentence says so. The window’s rollover is offered after it, labelled as an upper bound on the wait rather than a forecast of it — the rollover refills the budget, so the hold cannot outlast it.

When one session on its own is priced above the whole sustainable rate, there is no fleet burn low enough to admit it and the copy says so: nothing fits beside the work already in flight, and once no spot work is in flight the idle-fleet waiver admits one session so the deployment runs in a duty cycle.

A hold can involve both window ceilings at once — one window’s budget spent while the other is only ahead of its curve. Decision#ceiling reports spot_budget then, because that is the stricter of the two, and Decision#budget_spent_windows is what the copy names: saying “the 5-hour and weekly windows’ budget is spent” would be false of the second, and bounding the wait on the weekly rollover would over-claim a window that clears as soon as the fleet slows.

What “hold” does

A held session stays in waiting — the status Zimmer already uses for “created, not started” — and AgentSessionJob re-enqueues itself after ten minutes plus a little jitter. GoodJob persists the delayed job in Postgres, so the retry survives a worker restart or a deploy. When a slot frees, or the curve catches up, the same job starts the session normally.

A re-check job that a worker shutdown catches mid-pickup is re-enqueued verbatim rather than treated as an interrupted session — see waiting is two different situations, and neither is a recovery. The chain has no redundancy along it: each re-check is what forges the next one. So when a link is lost, a sweep re-forges it — see A hold that loses its re-check.

Archiving a held session ends it deliberately. The delayed job is left in the queue — nothing cancels it — but AgentSessionJob refuses an archived session before it reaches the gate, so that job fires once, logs why it stopped, and schedules nothing further. The hold record is left on the session as it stands: it is the history of why the session sat in the queue until it was trashed, and an archived session shows no hold banner anyway.

The jitter matters at a backlog: without it, sessions held in the same minute re-check in the same minute forever, every one of them reading the same fleet size before any of them has started.

Consecutive holds back off

Jitter spreads a held population out. It does not make it smaller — and the size is what matters to the queue. A flat ten-minute interval means N held sessions put a fixed N / 10 min of AgentSessionJob work onto the agents queue for as long as the hold lasts, an arrival rate that cannot fall when the deployment is struggling. That is what it is for: on 2026-08-20, ~80 quota-held sessions re-checking every ~11 minutes held a standing ~8 jobs/min against sixteen agents threads, eleven of them occupied for hours by live sessions. When a host-latency episode pushed each re-check into the tens of seconds, arrivals outran service and the GoodJob ready queue grew without draining until SystemHealthMonitorJob paged.

So each consecutive hold doubles the interval — 10m, 20m, then on until the ceiling clamps it — and the ceiling depends on why the session is held, because the two reasons clear on very different timescales:

Hold reason	Ceiling	Why
at_utilization_limit	1 hour	A pool window comes back down over hours. Re-checking more often than this cannot learn anything new, and this is the reason that produces the long-lived holds.
fleet_at_cap	30 minutes	A slot frees whenever any running session ends, which is unpredictable and often soon.

So a utilization ladder runs 10m, 20m, 40m, 60m, 60m…, and a fleet-cap one 10m, 20m, 30m, 30m….

Jitter is added after the ceiling, so a population pinned at the ceiling still spreads out. The ladder resets in two situations, and both are the caller saying so rather than anything inferred here:

• The session gets through. spot_hold_count is one of the spot_hold_* metadata keys cleared on start, so the next outage begins again at ten minutes rather than resuming where the last one left off.
• A person asks for this session directly. Restart, action_session’s restart and POST /api/v1/sessions/:id/restart run one implementation — Sessions::RestartFromScratch — so the keys they except from the metadata they carry forward are one list, Session::RESTART_FROM_SCRATCH_KEYS. They have to except them: those paths re-enter the gate looking exactly like a scheduled re-check — no prompt, no resume flag — so without it they would read as another consecutive hold and push the ladder up, making someone who asked for the session now wait longer than if they had left it alone.

Restarting resets the ladder; it does not bypass the gate. A session the gate still refuses is held again, back at ten minutes. The lever that starts it now is Make this session priority.

The cost is real and is not hidden: a session can now sleep longer than it strictly had to, up to its ceiling, if the condition clears early. That is bounded, visible as spot_hold_retry_at on the session’s detail page, and a human who wants it now can make the one session priority — which is what the hold banner already says.

Refusing instead would mean the gate silently deletes work: a github_issue trigger that fires once during a busy afternoon would never run at all.

A hold that loses its re-check

The ladder above is a chain of single delayed jobs, and until 2026-08-31 it had no redundancy anywhere along it. A hold is a promise — “re-checking at 02:43:15” — kept by exactly one job, and each re-check is what enqueues the next one. Lose one link and the session waits forever, in a state indistinguishable at a glance from a session merely queued.

That is not hypothetical. Session 7507 was held for the 145th time at 02:12:30Z; the hold record committed, and 23 seconds later the worker was gone. GoodJob re-picked the row and raised InterruptError, and the hold’s own log line never reached the database — it was still in the in-memory LogBuffer, which is how we know the execution died between the metadata write and the flush, taking the un-enqueued re-check with it. At 02:43:15Z nothing happened, and nothing ever would have. Eleven hours later the session page was still showing a human 5 of 5 session slots taken while the live gate said within_limits at 1 of 5.

Two changes make that recoverable rather than terminal.

The durable record is the session, not the job. spot_hold_retry_at on the session is what the ladder rests on, and SpotHoldSweepJob reconciles it against reality every five minutes. A hold more than ten minutes past its own re-check time, with no AgentSessionJob still queued against that session, is a broken ladder: the sweep advances the stamp under a row lock and enqueues the turn again. The advanced stamp is its own idempotency key, so the next pass leaves the session alone. It re-arms at most ten a pass, spread over three minutes, so a recovered backlog walks back onto the ladder rather than hitting the gate at once.

A re-arm is a re-check, not an admission: it puts the same turn back through the gate, which decides again. If the gate is open the session runs; if it is still closed the session is held again, with a fresh re-check behind it.

The refused prompt is recorded on the session too. “Deferred, never dropped” is the promise the hold makes, and while the only copy of the prompt was the delayed job’s argument list, a lost job broke it. A deferred resume now writes spot_hold_prompt alongside the rest of the hold record, so the sweep replays the real turn. A hold recorded before that — every session stranded today — has no prompt to replay and comes back on a recovery nudge instead, which is said out loud in its log.

A re-armed first turn carries its attachments. The lost job carried this turn’s images and files as arguments, and AgentSessionJob reads them from nowhere else, so a bare re-arm brought the turn back with its prompt and without the screenshot the prompt referred to. A re-armed start now reads them off the durable volume through the same Sessions::FirstTurnAttachments the other first-turn doors use, and the log line names what the turn is carrying rather than only asserting that it was carried (#789).

The volume is the right place to look only when the session has never run, which is why that read is gated on a blank session_id — the same predicate Sessions::StartNow uses. The gate holds a session before AgentSessionJob stamps one, so a genuine first start reads blank, as does a restart from scratch. A start hold on a session that has already run is a different thing: an OAuth resume enqueues a promptless new-session job, which this gate can hold, and everything on that session’s volume belongs to turns it has already had.

A re-armed resume carries its attachments too, from a different place. A deferred resume writes spot_hold_images and spot_hold_files beside spot_hold_prompt — the descriptors hold! was handed, not the bytes, which are already durable on the volume — and the sweep replays them from there (#890). The three keys are written and dropped together, so a hold that carries no attachment clears an earlier hold’s rather than leaving it beside a prompt it never belonged to.

It must be the record and not the volume, and that is the whole reason the two branches differ. On a resume the volume holds every attachment the session has ever received, including ones earlier turns already consumed, so “everything on disk” would put an old screenshot on a new turn. Replaying the wrong attachments is worse than replaying none: both are silent, and only one of them is also wrong.

A resume whose prompt was lost as well comes back on the recovery nudge with no attachments. The nudge is a different turn — Zimmer’s own sentence about a stalled ladder — and an attachment belongs to the prompt that referred to it.

Both stores hand a descriptor back with string keys while the CLI adapters index it with symbols (image[:path]), so the round trip goes through Sessions::AttachmentDescriptors, which owns that conversion in both directions. A descriptor that skipped it would enqueue a turn that looks like it is carrying a screenshot and reads as carrying none — the same silent failure wearing the other face. hold! normalizes once at the door and builds every carrier from the result, because it hands one turn’s attachments to as many as three of them — the delayed job, the hold record, and the enqueued_messages row a second refused turn is parked in — and two copies of one turn built from different values is how they come to disagree.

The replay also checks the bytes are still there. A hold record outlives the files it names: the sweep re-arms records nobody has touched for hours, while the durable-storage cleanup reaps a session’s attachment tree on its own schedule. Handing the adapter a path that is gone is not a missing screenshot — load_image_as_base64 reads it, the Errno::ENOENT surfaces as a failed spawn, and the session is stamped spawn_failed. A repair path must not be able to do more damage than the thing it repairs, so a descriptor whose file has vanished is dropped and the turn comes back short an attachment rather than not at all. The start branch gets that for free, by reading what is on disk in the first place.

An interrupt no longer mistakes a hold for a stranded session. A held session is dormant on purpose, exactly like a ceiling pause or an auth-outage park, and AgentSessionJob’s interrupt recovery now recognises it as such. It used to read only the pause record (spot_pause_reason), so a held session — spot_hold_reason, a different population with a different resume owner — fell through to the recovery path and was stamped paused_by: "recovery" on top of its hold. Session 7507 was: twelve auto-continue attempts against a clone deleted days earlier, then abandoned at 02:54:32Z, leaving it in waiting holding a re-check that had already been lost.

A hold a promotion has overtaken

The gate holds spot sessions. A hold record on a session that is no longer spot is a fossil, and until 2026-09-06 nothing cleared one: hold_if_needed returned early on a priority session without touching the record it found, so a promoted session kept its spot_hold_* keys through its whole run and every surface that reads them read them as live.

That produced a record which contradicted itself. Session 6934 simultaneously said Scheduling class: priority (set on this session) and Held by the spot gate (at_utilization_limit), with Holds so far: 52, over a frozen gate sentence ending “Priority sessions are unaffected” — and its banner offered the Make this session priority button to a session that already was (#423). Both halves cannot be true, and the class is the one that is.

Two changes, in the two places the record can be wrong:

• The gate clears what it will not act on. A session that reaches hold_if_needed and is not spot has its hold record dropped, the same way a session the gate lets through does. This is the last door a promoted session comes through, not the first — every promotion path releases the hold itself — so reaching it with a record means something moved the class without going through one: a demote-then-promote, a write straight to the row, a genesis-wide policy flip.
• The surfaces that replay a record say when its class has overtaken it. The session page’s hold banner and get_session both render one shared sentence (SpotSessionHold::PROMOTED_SENTENCE), so they cannot drift; the banner also stops describing a priority session as a spot one and withholds the Make this session priority button, and get_session stops telling an agent to promote a session that already is.

SpotSessionHold.held? is deliberately not narrowed to spot sessions. That predicate is what the stalled-ladder sweep keys its population on, and a promoted session whose re-check was lost still needs the sweep to put a turn back on it. Narrowing it would hide exactly that session from the one thing that can rescue it.

A hold lasts as long as the number does

There is no escape hatch and no deadline: while a window is ahead of its curve or out of budget, spot work waits. That is the intent — but the curve is what makes the wait short. A window ahead of pace is back inside it as soon as the clock moves far enough, which is minutes rather than hours, and a window whose budget is genuinely spent waits for the rollover. Both are visible on /inference the whole time, in dollars. Promoting one session to priority is the lever for a single piece of work that cannot wait.

Every turn is gated, not just the first

The gate used to read “is this a first start?”, and everything else — a fired wake_me_up_later backstop, a queued follow-up, a Slack or GitHub poller message, a heartbeat nudge, a restart — went straight through. All of those arrive at AgentSessionJob carrying a prompt, and every one of them begins a new turn that spends fresh quota. On 2026-08-22 a spot session woke on its own backstop trigger and ran a full turn while this gate was reporting at_utilization_limit at 87% of a 65% target, force-pausing 22 running spot sessions and holding 141 more at the starting line.

So the admission check covers every turn. While a window has no room for spot work, the only way a spot-designated session runs is to be promoted to priority first. Two things still pass through, because neither spends anything:

Passes through	Why
clone_only	Sets up a clone and configures MCP. No agent is spawned.
resume_monitoring	Re-attaches to a process that is already running. Holding it would orphan that process, not save a token.

One hold reason is skipped for a session that is already running when the gate runs: fleet_at_cap. Its deliverer has flipped it to running, so it is counted in the running fleet itself — refusing it for a full fleet would refuse it on the strength of its own slot, and would refuse every session the ceiling sweep resumes (those are flipped to running before their jobs run). The utilization reading has no such problem: the pool’s windows are measured independently of this session.

That exemption is keyed on the session’s status, not on “this turn carries a prompt”. A turn the gate has already deferred once is sitting in waiting and holds no slot, so its re-check is an admission like any other and the concurrency limit applies to it in full.

A deferred turn is not a lost turn. The prompt that woke the session, and any images or files attached to it, are re-enqueued verbatim on the same backed-off re-check as a first-start hold — GoodJob persists that delayed job in Postgres, so it survives a worker restart or a deploy. The session goes back to dormant waiting (not needs_input: nobody has to do anything about it), and nothing announces it as needing a human — no push notification, and no session_needs_input event that would wake a parent watching this session about a turn that never ran.

Two details keep that true when something delivers to the session again during a hold, which can last the best part of an hour — a second child waking its orchestrator, say:

• The gate takes custody of the prompt, so pending_follow_up_prompt is dropped. That marker means “no job has picked this up yet” and AgentSessionJob prefers it over its own argument; left in place, the second delivery’s marker would overwrite the first, and the first deferred job would deliver the second prompt and discard its own.
• A second refused turn is queued, not given a second job. Two jobs racing one session means the concurrency guard drops whichever loses, so the later prompt goes into enqueued_messages — the durable queue Zimmer already drains at a session’s next turn boundary — and is delivered after the turn ahead of it. The hold record is left alone in that case: the re-check it names is the one that will actually fire, and a queued prompt is not another rung on the backoff ladder.

The session detail page shows a Held for quota headroom banner — Next turn held for quota headroom when it was a resume — naming the reason, the next check time, and how to run it now. get_session reports the same through spot_hold_reason / spot_hold_retry_at / spot_hold_count, and says explicitly that the queued prompt is still coming.

A status-summary fork is refused, never queued

A status-summary fork is Zimmer’s own seconds-long bookkeeping: it exists to run one turn, write a blurb about another session, and be harvested. It is not the operator’s work, and every list an operator reads already drops it. The scheduler treats it the same way, in two places.

The gate is asked before the fork is made. SessionStatusSummaryGenerator already checks the login pool before forking — if there is no account to run a fork on, it writes the blurb with one pool-independent claude -p completion instead. It asks the spot gate the same question, for the same reason: “the fleet is full” and “the pool is empty” both mean the fork cannot deliver, and neither is a reason to stand a session up. So while the gate is refusing, no summary fork is created at all — the blurb still arrives, from the path that spends no slot.

That check is deliberately stricter than the hold’s own reading. A dispatched fork is running by the time it reaches the gate, so the fleet_at_cap exemption above applied to it and the concurrency limit never did: every summary fork started as one more Claude Code process on top of a fleet already at its cap. Asking before the fork exists is what puts it back under the cap.

A fork that reaches the gate anyway is thrown away, not held. The two checks race — the fleet can fill between them — so SpotSessionHold refuses a summary fork’s turn like any other and then disposes of it: the fork is archived, its claim on the summary record is released with the gate’s own sentence, and a headless retry is enqueued. It is not an exemption. The fork does not run, so nothing here lets a session past the fleet cap or past burn pacing; what changes is that a refused fork ends instead of joining the queue.

That branch sits above the fleet_at_cap exemption described earlier, and deliberately. The exemption spares an already-running session the concurrency limit because refusing it would refuse it on the strength of its own slot — and because the refusal would strand it. Neither half holds for a summary fork: it is running only as bookkeeping, since its deliverer flipped it and no process has been spawned, and its refusal is a disposal rather than a deferral, so there is nothing to strand. Below the exemption, a fork refused for a full fleet would have been waved through to run as one more process on top of it — which is the escape the check before the fork exists is closing.

Holding one was the wrong disposal. A ten-minute deferral, then twenty, then an hour, parks a hidden session row and its working directory for far longer than the turn it was deferring would have taken — and its claim on the summary record ages out after PENDING_TIMEOUT, at which point the next generation forks again for the same source while the first is still queued. Nothing abandoned a fork that was dispatched and never ran, so that pile-up had no bound on it (#712).

The budget is a ceiling

Admission alone made the budget a floor under when new spot work stops, not a ceiling on what spot work spends. A session admitted with room to spare goes on running, and a fleet of them carries the window straight past the line: on 2026-08-20 the spot gate card read “Holding spot sessions: 5-hour window at 89% of its 80% target” while twelve sessions ran and three accounts sat in quota_exceeded. The gate had stopped admitting and then watched the work already in flight climb toward 100%.

So SpotCeilingSweepJob re-evaluates the same decision every five minutes and applies it to running sessions:

The gate says	What happens to running spot sessions
at_utilization_limit, ceiling spot_budget	Every running spot session is paused.
at_utilization_limit, ceiling pacing_curve	Nothing. The pace is an admission device: killing a running turn to enforce a curve spends a lost tool call protecting nothing, since the same money is spent either way, just later. It is also what keeps the idle-fleet waiver coherent — otherwise the sweep would pause the session the waiver had just admitted, and the two would flap.
fleet_at_cap	Nothing. A running session already holds its slot; pausing it would free that slot only for another spot session the same cap would hold. A priority session arriving at a full fleet is the one case that does take a slot — see A priority session takes a slot, which is a different trigger on a different path.
anything else	Sessions dormant in the queue are resumed, highest precedence first (oldest pause first within a tie).

SpotSessionPause reads Decision#stops_running_work?, not held?, which is what draws that first line. A fleet merely ahead of the curve is throttled at the door and never interrupted.

Priority sessions are never paused, on any reading. Nor are Codex or Pi sessions (they spend nothing against a Claude window — every pause and preemption path filters on agent_runtime = 'claude_code') or status-summary forks (Zimmer’s own seconds-long bookkeeping).

What a pause does

The session’s CLI process is terminated and the session goes dormant in waiting — the same shape a wake_me_up_later sleep leaves behind, reached the same way (pending_sleep is set, and the pause callback carries it needs_input → waiting).

waiting, not needs_input, is deliberate: a session in needs_input lands on the homepage action queue, which is for work a human must act on. A quota pause is not — and ten of them at once would bury the sessions that genuinely need a person.

Pausing interrupts a turn. What the agent had already written to disk stays written; the tool call in flight is lost, along with any reasoning not yet flushed to the transcript. That cost is paid once per pause, and it is recorded rather than silent:

Where	What it says
metadata	spot_pause_at, spot_pause_reason, spot_pause_detail (the gate’s own sentence), spot_pause_count, and paused_by: spot_quota
The session log	A warning line naming the window, what it ran out of, and what brings the session back
The session page	A Paused mid-run for quota headroom banner, with the same Make this session priority button the hold banner carries
MCP	get_session reports the pause, why, and when it resumes

paused_by: spot_quota is what keeps the recovery sweeps out of it: it is neither user (which stops auto-continues) nor recovery (which would have DeploymentRecoveryJob resume the session straight back into the window that stopped it). A Refresh all on the dashboard skips these sessions for the same reason.

What brings it back

The next sweep that finds the gate open resumes them, with the standard system-recovery nudge telling the agent to pick up where it left off. Two things shape which and how many:

• A resume margin. Resumption decides against a reserve WIDENED by SpotGateService::RESUME_MARGIN_PCT (5 points of the window): with a 20% reserve, a paused session resumes only once the fleet is inside a 25% reserve. Holding a session that never started costs nothing, so admission uses the plain reserve; resuming one that was interrupted mid-turn costs a lost tool call, so it waits for real headroom rather than resuming the instant the fleet dips under the curve and pushing it straight back over. The margin is applied as extra reserve, so the money it protects is the same money the reserve protects — and the reserve the page shows is still the one the operator set.
• A batch of five per sweep, and never more than the free slots under Max sessions at once. A window that has just come back down is at its most fragile — every session resumed starts spending again immediately — so the fleet walks back up over successive sweeps rather than restoring all at once. Sessions past the batch keep their place at the front of the next one.
• A batch of one when the pacing curve was waived rather than passed. A fleet with every spot session paused and only priority work running is exactly the state the idle-fleet waiver fires in, and the waiver’s contract is a duty cycle rather than a burst: one session goes back, which puts spot work in flight, so the next sweep five minutes later is paced normally. Only the pace is narrowed — the cap still holds the money back with the resume margin on top of the reserve, and the fleet cap still holds the slots.

A session someone promotes to priority while it sleeps is resumed by the next sweep whatever the windows say, because priority work is never gated on quota.

One thing outranks even that: a session with a wake-up still ahead of it is left alone. See A pause outranks precedence below.

A priority session takes a slot

The budget ceiling above stops running spot work when a quota window runs out of money. The concurrency ceiling now does the same thing for slots: when a priority session is about to start and every slot is taken, one running spot session yields its slot instead of the fleet running one wider than Max sessions at once.

SpotPreemption owns this. It is evaluated at the same choke point the gate already uses — SpotSessionHold.hold_if_needed, which every quota-spending turn passes through — so the moment “a priority session is about to take a slot” is knowable there and nowhere else. The priority session is never held either way; what preemption decides is whether the fleet grows or a slot changes hands.

Who yields

The spot queue’s own order, read backwards. The resume sweep takes the highest precedence first, so the session that yields is the lowest — the queue decides who runs and who stops, or it is not the queue. Precedence ties (the default is 0 for everything nobody has ranked) break on:

1. Fewest prior preemptions, from a durable per-session ledger, so the cost is spread rather than charged to whichever row sorts first every time.
2. Least human involvement — HumanMessage rows on the session itself, the same reading human intervention uses. A session someone is in the middle of a conversation with is the last to lose its turn.
3. Newest, which has least in flight to lose.

Exactly one session yields per outstanding slot. A marked session keeps its turn — and therefore keeps counting toward the cap — for the whole grace window, so the cap comparison subtracts the marks already standing. Without that subtraction, the fact that the gate is a choke point on every turn rather than every session would bite hard: a priority session taking follow-up turns every couple of minutes would take a fresh victim for the same slot on each one until the spot fleet drained.

The pause is graceful, and often free

A preemption does not kill the victim’s process. It writes the pause record and pending_sleep while the session runs, and the session’s own turn end carries it needs_input → waiting through the same execute_pending_sleep a deliberate park uses. No tool call is lost and no unflushed reasoning.

SpotCeilingSweepJob resolves the mark on its five-minute pass:

State on the next pass	What happens
The session reached needs_input still marked	It is slept into the queue. Its turn already ended; the only thing that did not happen is the transition, and execute_pending_sleep alerts rather than raising when it fails. Without this net the session sits in the operator’s action queue holding a record no sweep acts on.
The mark is older than STALE_MARK_AGE (1 hour)	It is released, whatever the fleet says. A mark only survives past its grace while the fleet keeps reading at cap, so an hour-old one means its provenance broke rather than that the turn is long.
The fleet is back under its cap	The mark is released. The preemption was never needed, the session keeps its turn, and nothing was lost at all — including the ledger, which is un-charged, so a released mark costs neither a recorded preemption nor a cooldown.
Still at or over its cap, and the mark is older than SpotPreemption::GRACE (10 minutes)	The turn is halted where it stands, exactly as a budget pause halts one. This is the only path that costs anything, and it is paid only by turns that would otherwise make the ceiling mean nothing.

A mark inside its grace is left alone, and a mark with no readable timestamp is neither escalated nor released — nothing is done on the strength of a record that cannot be read.

A session that has already asked to sleep at the end of its turn — it armed a wake_me_up_later mid-turn, say — is never marked in the first place. Its slot is on its way back without anyone taking it, and marking it would overwrite a sleep intent this ceiling does not own.

It joins the queue that already exists

The victim lands in SpotSessionPause’s queue carrying spot_pause_reason: preempted_by_priority — the same record, the same dormancy, the same sweep, the same precedence order. That is deliberate and it is the most important property here: a dormant spot session must have exactly one resume owner, and giving preemption its own park would have given it its own sweep, which is #617 rather than a feature. Its resume condition was already exactly right — a free slot inside the budget — and the slot the priority session frees when it finishes is precisely that.

Thrash is bounded by a cooldown. A session preempted within SpotPreemption::COOLDOWN (30 minutes) is not eligible again, read off a ledger a resume deliberately does not clear. Without it a session resumed into a free slot is the lowest-ranked running spot session again a minute later and yields again immediately — pause, resume, pause, with a lost turn each cycle and no work in between. A session already carrying any pause record is never a candidate either; its slot is on its way back already.

When nothing is eligible, nothing happens and the fleet stays one over its cap, which is what it did before preemption existed. The same is true of a slot already promised: with a mark outstanding the cap reads as satisfied, so a second priority start in the same window takes nobody. Every error path lands there too: preemption is called from the gate whose whole promise is that it only defers, so it never raises and never holds anything.

Where you see it

Surface	What it says
The session page	A Preempted by a priority session banner, naming the slot it gave up and what it is waiting for — a free slot and a Claude account under both quota targets, because it waits in the same queue as the budget ceiling’s sleepers and answers to the same resume decision. A spent window keeps it asleep even once a slot frees.
The session log	The mark, and then either the release or the halt
/inference	Spot sessions preempted by priority work, its own figure beside the paused and held ones
get_session	The same lines, plus which session took the slot and how often this session has been preempted
get_spot_policy	Priority preemption: on/off, and the same count

Turning it off

spot_preemption_enabled on AppSetting, on by default. It is a separate switch from spot_gating_enabled so an operator can stop the one part of the policy that interrupts work already underway without turning the gate off and letting the whole fleet run unpaced. With it off, a priority session starting into a full fleet runs one over the limit — the behaviour that predates this.

Both surfaces can set it: the Let priority work take a slot checkbox on /inference, and action_spot_policy with "action": "set_gating", "preemption_enabled": false.

Joining the queue on purpose

The same dormancy is reachable deliberately, and it is the answer to “this session should wait, and no time I could name is the right one”. pause_into_spot_queue on action_session sleeps the session and hands it to this sweep with no wake-up trigger and no wall-clock time at all.

It is the same record and the same resume path as a ceiling pause, with three differences, because nothing interrupted this session:

• spot_pause_reason is user_spot_queue, so the banner, get_session and the session log all say it was parked deliberately rather than describing a turn it never lost. It is also left out of the “spot sessions asleep in the queue” count on /inference, which is about what the ceiling cost.
• A session that resolves to priority is set to spot, since the sweep resumes a non-spot sleeper on its very next pass. Make this session priority on the banner reverses it, and that next sweep resumes the session — which is the intended way back out.
• The panel’s Resume with box still applies: with no trigger to hang the prompt on, it rides on the session and is delivered when the sweep reaches it, in place of the recovery nudge.

Its place in line is whatever precedence the session already carries — parking it does not re-rank it. Any unfired wake-up armed from the same control is cancelled, because picking the queue after picking a time means “not then, this instead”.

The sweep runs every five minutes, but what bounds how fast the ceiling reacts is the reading, not the sweep: utilization comes from quota snapshots, which land when ClaudeUsageSamplerJob samples (the serving account every 15 minutes, a spare within 75), when an account rotates, and when someone opens /inference.

Precedence: ranking the spot queue

scheduling_class answers “does this session wait for quota headroom”. It says nothing about which of the waiting ones goes first — and with a permanently long spot queue that is the question that actually decides what gets done.

precedence is that ordering. Every session carries one; only spot sessions are ordered by it.

• Higher is handled sooner, on an absolute scale. 100000 comes before 50, and 50 comes before 0. It is not a 1..N rank and nothing renumbers it — values are sparse on purpose, so there is always room to slot work between two existing entries. Ties break on created_at, oldest first.
• It lives on every session, priority ones included. A priority session demoted to spot has to land somewhere sensible, and a spot session promoted to priority has to keep its place for when it is demoted back. A column populated for half the rows would lose that on every round trip.
• A spawn lands just above its parent. start_session with no precedence puts the new session one point above the session named in parent_session_id, so the child that finishes its parent’s job runs before unrelated work queued beneath it and a tree of work stays contiguous. Name a value only when you mean to move the work relative to everything else in the queue. A fork inherits the same way, through forked_from_session_id — with one exception: a status-summary fork takes the default 0 rather than the bump. It is not continuing the source’s work, it is writing a blurb about it, and inheriting the bump ranked Zimmer’s own bookkeeping one place above the very session it was summarizing.
• A trigger can predefine one, alongside the class it already carries, so a feed is ranked once rather than one spawned session at a time. A trigger takes an absolute number only — the value is stamped on every session it ever spawns, and a standing instruction to jump the whole queue is not a rank.
• “Put this first” is a placement, not a number. Where a surface offers it, a symbolic top_of_spot stands in for the integer and the server resolves it against the live queue as part of the same write. The Ranked view’s demote button, the Quick Router, the two MCP session tools and the REST API’s create and update all offer it; a trigger takes an integer only — see Placing something at the head of the queue.

Placing something at the head of the queue

“Work on this before the rest of the spot queue” is the one placement common enough to be worth naming, and it is the one nobody should be computing by hand. Session.precedence_above_top_spot answers it: a few points (SLOT_GAP) above the highest-ranked non-archived spot session there is right now. Reading the live maximum rather than a stored high-water mark is the whole point — a queue whose top has since been archived does not keep inflating the scale.

Session.precedence_for_place("top_of_spot") is the single door onto it, and every surface that offers the placement comes through it:

Surface	How it asks
Ranked view, Demote to spot	PATCH /sessions/:id/update_scheduling_class with place=top_of_spot
Quick Router, Run as spot	resolved server-side on submit; the checkbox copy says so
MCP start_session, action_session (change_precedence / change_scheduling_class)	place: "top_of_spot"
REST API POST /api/v1/sessions, PATCH /api/v1/sessions/:id	"place": "top_of_spot"

Two surfaces that rank a session do not offer it, and that is deliberate. The session detail page’s scheduling-class control posts only the class — its Precedence field beside it is the integer, and a demotion there keeps whatever rank the session carried. A trigger takes an integer only: the value is stamped on every session it ever spawns, so a placement resolved once at edit time would be a stale number by the first spawn, and one resolved at every fire would be a standing queue-jump rather than a rank.

On MCP and over REST the argument is mutually exclusive with precedence — they are two answers to the same question, and passing both is an error rather than a silent winner. Passing neither leaves the ordinary behaviour untouched: a spawn still lands one point above its parent, and a demotion still keeps whatever rank the session already carried.

Where the surface is placing a session that already exists, the resolution both excludes that session from the measure and floors the result at the rank it already holds. The first stops “put this at the head” from walking the top row SLOT_GAP higher on every call; the second stops it overshooting the other way, since a session on top at 1000 with a runner-up at 10 would otherwise be rewritten down to 15 — still the head of the spot queue, but now beneath a priority session carrying 500 that would outrank it on a later demotion. Together they make re-placing a session that is already first a no-op.

Why a symbolic form rather than a documented recipe

The MCP tools used to tell an agent to read the current top with quick_search_sessions and pass a value a few points above it. That is two round trips, and it is racy: another session can land above the computed value between the read and the write, leaving the session that was meant to go first sitting second. It also hands the agent a number it has to re-derive — including the archived exclusion — instead of asking the server the question it already knows how to answer.

The symbolic form narrows that window to a single request; it does not lock the queue. The resolution is a MAX read and then a separate UPDATE, so two callers placing at the top in the same instant read the same maximum and write the same value, and ranked breaks the tie on created_at — the older session goes first. That is the honest claim, and it is the one the tool descriptions make.

A human message jumps the queue

A named human speaking to a spot session moves it to the head of the spot queue and gets its next turn moving, and sends the least human-involved queued session to the bottom in exchange.

”A human intervened” is a fact, not a guess

It means a HumanMessage row on this session. Nothing else.

Zimmer already draws that line at the input boundary rather than from the text: HumanMessageCapture writes a record only when the authenticated actor was established — Tadas typing into the web UI, or a Slack user id that resolves through the seeded roster. An agent’s follow_up over MCP, a router-composed spawn prompt, a fired wake_me_up_later, a heartbeat nudge, a polled GitHub comment and a system-recovery resume all arrive as the same kind of user turn and record nothing. So the trigger has no heuristic in it, and the mistake that would have promoted half the fleet — treating “a prompt arrived” as a person — is not reachable.

Here, not the hierarchy. A human talking to a router is not intervening in the twelve sessions under it; that is exactly the here / elsewhere distinction get_session_provenance already draws.

Creating a session is not intervening in one. The new-session form and the Quick Router’s prompt box capture a HumanMessage too, and they already decide placement for themselves — the Quick Router’s Run as spot checkbox is where top_of_spot came from. Those two entry points are named and skipped, so a fresh human-typed session is not re-placed and does not demote anybody.

The rank alone is not enough

The promotion goes through Sessions::StartNow, the same door the Ranked view’s Start entry and a promote-to-priority use. Landing the rank and stopping there is #423: the class change applied, and the session then sat exactly as long as before because its deferred re-check was still up to an hour out.

Moving a turn is not passing the gate, and this does not pretend otherwise. The session stays spot, so a full fleet or a spent budget holds it again — what it buys is the front of the queue and the next slot, which is what “get going immediately” can honestly mean for spot work.

The placement uses the instance form of top_of_spot, so a session already at the head keeps its number rather than walking up SLOT_GAP on every message.

The exchange, and why it refuses more often than it fires

Without a counterweight every intervention adds SLOT_GAP to the top of the queue forever and the numbers inflate away from anything an operator set by hand. So one session goes to the bottom — chosen by fewest human messages, then oldest last human message, then fewest prior demotions, then highest precedence, then newest. Highest, not lowest: among equally-uninvolved sessions the one to send down is the one currently jumping the queue on nothing, and it is the only one whose demotion changes the order at all — picking the lowest instead makes the exchange a permanent no-op after a single round, because that session is already at the bottom and stays selected forever while the top goes on inflating.

Three rules each end in demoting nobody, which is a perfectly good outcome:

• Nobody less involved. The promoted session has just been spoken to, so a candidate has to be strictly below its count. A queue in which everything has had as much human attention as this one has nothing less wanted in it.
• The starvation exemption. A session created longer ago than Sessions::HumanInterventionPromotion::STARVATION_EXEMPTION (24 hours) is never demoted, whatever its involvement. This is the anti-starvation rule stated as a rule rather than left to luck: “least human involvement” is a property that never changes on a session nobody talks to, so without it the same row would be demoted forever. An unattended session sinks at most until it is a day old; from then on every promotion goes past it rather than over it. created_at rather than time-in-queue, deliberately: queue time resets every time a session runs, so a session that keeps almost-running would never accrue any, and being wrong in the “exempt too early” direction is the right direction for a rule about starvation.
• Already at the bottom. A session whose rank is already the lowest is left alone rather than walked down another SLOT_GAP for no change in the order.

Candidates are spot sessions dormant in waiting — the queue itself. A running session is not in it, and a priority session is not ordered by precedence at all.

The demoted session is not stopped, not cancelled and not held. It keeps its turn, its park record and its resume owner; the only thing that changed is an integer. It runs when the queue above it drains — and one message from a human puts it straight back on top.

Where it runs

HumanMessage’s after_create_commit enqueues HumanInterventionPromotionJob on the default queue. Out of process on purpose: the work is a queue read, a placement write, a StartNow that reaches into GoodJob’s own tables and a second write on another session, and none of that belongs in front of the request that was delivering the human’s message. A failure in the re-ranking is logged and swallowed — the message itself has already been delivered.

A pause outranks precedence

Precedence answers which waiting session goes first. It does not answer whether a session may start at all, and one thing overrides it unconditionally: a wall-clock pause.

A session that an agent slept with wake_me_up_later sits in waiting and keeps whatever precedence it had. Nothing in the columns distinguishes it from a session merely queued behind the gate — both are waiting spot sessions with a number. So the rule is enforced on the start, not on the ordering:

A session paused until a time it has not reached does not start, whatever its precedence and whatever its scheduling class. It stays in the queue, at its rank, and the selector takes the next candidate.

A pause is a floor, not a promotion

“Not before this time” is the whole of what a pause says. It does not say “and then run regardless of the queue” — the spot queue stays the scheduler for spot work. When the wake comes due it delivers a prompt like any other turn, and that turn answers to the spot gate: a spot session whose window has no room for it is held and stays dormant in waiting, to be started by the queue in precedence order, while a priority session goes straight through because priority work is never gated on quota.

So the two mechanisms compose in one direction only. A pause can keep a session out of a queue slot it would otherwise have taken; it can never take one the queue was not going to give.

Five places could otherwise have started it early, and each declines:

Who	What it does instead
AgentSessionJob (a first start — a spot-hold re-check, a fleet slot opening)	Stands down and logs why, without re-arming the re-check timer. The armed wake is the next event in the session’s life.
SpotSessionPause (the ceiling sweep, every 5 minutes)	Skips it before the promotion branch, so promoting a paused session to priority does not start it either. Counted as held.
AuthOutageParkService (the un-park sweep, every 15 minutes)	Skips it. This one matters twice over: its resume goes through resume!, whose cancel_pending_one_time_wake_triggers callback would have destroyed the pause without a trace.
action_session restart — the start path in the awaken-waiting-sessions skill	Refuses with an error naming the pause and telling the caller to take the next candidate.
POST /api/v1/sessions/:id/restart — the same door for a script or an integration	Refuses the same way, with the same sentence.
A fleet-maintenance agent working the ranked queue	quick_search_sessions marks the row **Paused:** yes with the wake time, so the agent skips it deliberately — and the two rows above refuse regardless of what the agent decides.

The last row is the reason the guards are code rather than only prompt text. The fleet-wake selector is an agent reading a skill, which is a judgement; the refusals are a guarantee that holds for any caller — the skill, a script, a future integration.

Both restart doors refuse at the surface rather than deeper down, because each resumes the session before it enqueues anything and resume’s cancel_pending_one_time_wake_triggers callback consumes the pause on the way past. A guard further in would arrive after the pause was already gone.

Two paths deliberately still consume the pause, because both mean a caller is taking this session over rather than a selector working a list: the web UI’s Restart button, and a follow_up addressed to the session directly.

The pause is a deferral, not a cancellation, and its expiry is what makes that true. Past its moment the wake is no longer “ahead of” the session, every guard above stops applying, and the session is an ordinary queue candidate again — reachable both by its own wake firing and by the spot sweep, whichever gets there first. Sleeping means the scheduler has yet to reach it, read through the same TriggerCondition#schedule_due? the firing path uses, so an overdue wake describes a stuck session rather than a resting one.

Only a wall-clock pause blocks a start. Session#awaiting_scheduled_wake? — the broader reading a refresh uses — also counts a session-scoped ao_event watcher, and that one has no time component at all: if the watched session fails or is archived it is “still ahead” forever. Declining to nudge on that is free; declining to start on it would put a session permanently beyond every automated path on the strength of one dead watcher. So the start guards read Session#paused_until_scheduled_time?, which counts only unfired one-time schedules.

The Ranked view

/?view=ranked — a fourth dashboard view beside Categories, Last Touched and Created, and the only one that is a management screen rather than a reading one. Priority sessions stack above the queue, the spot queue is listed under them highest-precedence first, and both halves are editable in place:

Do this	And	Which means
Type a number in a row’s precedence field and press Enter	the row moves to its new position immediately, with no page load	PATCH /sessions/:id/update_precedence
Drag a row between two others	it takes the midpoint of the two values it was dropped between	PATCH /sessions/:id/reorder_precedence, which is handed the two neighbours and derives the value
Drag between two adjacent values, where no midpoint exists	the neighbours are nudged one apart each, and the dropped row takes the middle of the gap that opens	the same request — 21 and 20 become 22 and 19
Open a row’s ⋮ menu and press Demote to spot on a priority row	it lands SLOT_GAP (5) above the current top of the spot queue	PATCH /sessions/:id/update_scheduling_class with place=top_of_spot
Open a row’s ⋮ menu and press Promote to priority on a spot row	it moves up to the priority section, keeping its rank for a later demotion, and starts	the same endpoint, plus Sessions::StartNow
Open a row’s ⋮ menu and press Start now	the session’s queued turn is taken now instead of when the scheduler gets to it	POST /sessions/:id/start_now
Open a row’s ⋮ menu and press Trash	the session is archived and the row leaves the queue	POST /sessions/:id/archive — the same action every other Trash affordance posts to, speed bump and Undo toast included

Every write is optimistic: the row moves first and the server’s answer corrects the numbers behind it, including any neighbour that was nudged. A write that fails rolls the row back and reloads, because the server’s order is the only one that counts.

The nudge is what keeps an integer column usable without a renumbering pass — one extra write per drag, and no global compaction ever. A nudge can push a neighbour onto the value of the row beyond it; that is left alone deliberately, since equal precedence is legal and cascading the nudge upward would turn one drag into an unbounded write. The next drag into that spot separates them.

The Ranked view opens on waiting, running, needs_input and failed rather than the dashboard’s usual needs_input-only default: its whole subject is work that has not started. An explicitly chosen filter still wins, as everywhere else.

Each row carries one visible control — a ⋮ overflow menu holding promote/demote, Start now, a link to the session and Trash. The row itself is the reading surface: a drag handle, the rank, the status and the title. On a phone the row is two lines (title above, rank and status below), because a single line at 375px leaves the title about eighty pixels.

The title opens the session in the dashboard’s right-side drawer rather than navigating away — the same drawer a card’s View button opens, on the same session-drawer#open action. It stays a real <a href>, so middle-click, ⌘/Ctrl-click and “Open in new tab” still open the session in a new tab: only a plain, unmodified left click is intercepted, and below the sm breakpoint the drawer declines entirely and a phone gets the full session page.

Starting a queued session now

Sessions::StartNow is the operation behind Start now, and the one every promotion to priority performs on the rows it just promoted — the Ranked view’s Promote, the hold banner’s button, action_session’s change_scheduling_class, PATCH /api/v1/sessions/:id, and a trigger selector change. It exists because the hold banner’s “Make this one session priority to start it now” was not true: promoting removed the reason a session was held and changed nothing about when it would next be asked, so a session somebody had just decided was urgent went on waiting out a re-check up to an hour away.

A waiting session is dormant in one of three shapes, and each has its own door back in:

Shape	What starts it
Held at the starting line by SpotSessionHold	its already-queued delayed AgentSessionJob is rescheduled to now, through GoodJob’s own reschedule_job
…and it is still spot	the gate is asked again now rather than in forty minutes — it is not bypassed. A window still over its target holds the session again, and the message says so rather than claiming a start it cannot promise. Promotion is what removes the gate, which is what the hold banner has always pointed at
Paused mid-run by the ceiling, or parked from pause_into_spot_queue	SpotSessionPause’s own resume — the same locked re-check the sweep performs, restoring the prompt left with the park
Neither, and it has never run	a fresh AgentSessionJob — the one branch that builds a turn out of nothing rather than moving one that exists
Neither: it has run before and has nothing scheduled	nothing, from a promote. It is stranded rather than queued, so Start now sends it the same continue nudge Refresh does

The first row is the one worth internalizing. A held session is not a session with nothing scheduled: the gate takes custody of the refused turn and rides it — prompt, images and files — on a delayed job. Enqueuing a fresh job alongside it would leave two jobs for one session, and only the first is protected. AgentSessionJob’s concurrency guard stands a job down while running_job_id points at a live job, but the deferred one fires whenever it likes, and a session that has already finished its turn and gone back to waiting has no live job to stand it down. That second turn would run for real, re-delivering a prompt that was already delivered. Pulling the queued job forward keeps it at one job, one turn, prompt intact.

That last-but-one row is the only one that has to rebuild the turn’s attachments too. AgentSessionJob receives images and files exclusively as job arguments, so a job built from scratch starts with none — which is how a session whose prompt was “here is the screenshot, fix this” came to be started with the prompt and without the screenshot. Sessions::StartNow re-reads them from the durable volume, where they sit keyed by session id, and says in the session’s log what the turn is carrying. It leaves out anything a queued follow-up already owns: both live in the same per-session directory, and a screenshot attached to a message somebody queued for later belongs to that message, not to the turn before it. The other rows must not re-read storage — their job already carries its own copy. The reader itself is Sessions::FirstTurnAttachments, shared with the other door that builds a first turn from nothing: Restart from scratch, in the web UI, the REST API and MCP alike.

A session asleep on a wake-up it has not reached is refused rather than started. The wake is that session’s next event and it carries its own prompt; starting underneath it would race the two. AgentSessionJob refuses such a start on its own — this only says so before the click.

So is a session whose queue could not be read. An unreadable queue looks exactly like an empty one, and a held session that has never run would then take the “enqueue its first turn” branch on the strength of it — producing the second turn the whole design exists to avoid. The hold record is left alone on every path that does not actually start something, so a session that turns out to have nothing queued keeps the banner explaining why it is dormant.

The queue stays live

Rows subscribe to the sessions_ranked Turbo Stream. It is a separate stream from the card grid’s sessions_index_individual because a ranked row is not a card and is not keyed on dom_id — which is exactly why the queue’s statuses used to go stale until the page was reloaded.

Two kinds of message travel on it.

A status change replaces one element per row: the status pill (_ranked_row_status, written by Session#broadcast_ranked_row). That bound is the design, not a shortcut. The row around the pill holds two pieces of state the server does not own — a precedence the user may be halfway through typing, and the row’s position while SortableJS is dragging it — so replacing the whole row would let a background status change destroy an interaction in progress. A running session’s pill carries a spinner, so the queue reads at a glance as “these are the ones actually moving”; it arrives and leaves with the pill, because the pill is the thing being replaced.

A membership change sends an envelope (_ranked_delivery, written by Session#broadcast_ranked_membership on create, on a status change and on a scheduling-class change). The server cannot decide whether a row belongs on your screen, because one stream serves every open page and each has its own filters — the operator watching live work and the one who ticked “Archived” to go through the trash are on the same channel, and a session going archived means “leave” to the first and “stay, relabelled” to the second. So the envelope carries the session’s filterable facts plus the row already rendered inside an inert <template>, and ranked-queue#deliveryTargetConnected decides:

The page’s filters say	What happens
the status is excluded now	the row leaves
the scheduling class no longer matches its section	the row moves sections
it is admitted and not on the page	the row is inserted, in precedence order
anything else	the envelope is discarded

The row travels in a <template> so its ids are never in the document until the page accepts it — an envelope for a session you filter out cannot leave a stray ranked_row_<id> behind, and one for a row already on screen cannot duplicate it. The envelope is consumed and removed either way.

Neither kind is sent for a status-summary fork. Those are Zimmer’s own bookkeeping sessions (see the Status summary), and every server-rendered session list already drops them with Session.excluding_status_summary_forks — so a stream that carried them would put rows on the queue that a reload then took away. The exclusion lives inside Session#broadcast_ranked_membership and #broadcast_ranked_row rather than on the callbacks that call them: a fork’s status changes reach both methods through broadcast_status_change, which calls them directly, so a guard on the callback registration covered only the create and scheduling-class paths.

Three rules keep that safe, and each costs a little freshness:

• Deliveries are held during a drag and flushed on drop, so nothing is inserted or removed under a moving pointer.
• A row holding focus or a half-typed value is never moved or removed — the same rule the reconnect backfill applies. A delivery never writes a precedence input belonging to a row it is not moving, so an uncommitted number cannot be clobbered at all.
• A section already at its 200-row cap takes no insert, because the server truncates at the same number and says on the page that it has.

What it still deliberately does not do:

• No re-sorting. Precedence, not status, decides the order. A precedence someone changes in another tab does not move rows here — that would move a row out from under a pointer for a number nobody on this page typed.
• No inserting into a narrowed page. A search query, an agent-root filter or a genesis filter cannot be evaluated client-side for a session the page has never rendered, so live_insert is off whenever one is in force and the page declines rather than guessing. Removal stays sound under all three: a row on screen already matched them, and neither a status nor a class change alters that.

The section header counts and the “nothing here” placeholders are recounted by a MutationObserver in ranked_queue_controller.js, so they stay true whether the row arrived or left via a broadcast, a promote or a demote.

None of the remaining staleness is permanent. Both lists are data-live-region="sync", so a page whose socket died — every reopen of the installed iOS PWA — is reconciled against a fresh render on reconnect, which recovers the order, the grouping and any row the stream never delivered. The reconciler skips a row holding focus or a dirty field, so it cannot eat a precedence someone is halfway through typing either. See the reopen backfill.

Precedence decides who gets the headroom back

Two sweeps hand out recovered capacity, and both read precedence:

• The fleet wake starts quota-parked spot sessions, in precedence order — see When the pool runs dry.
• SpotSessionPause puts back the spot sessions the ceiling paused mid-run when a window comes back down, highest precedence first, oldest pause within a tie. Its budget is bounded by the free slots and MAX_RESUMES_PER_SWEEP, and that budget is usually smaller than the population it holds — so the order is what decides which work resumes, which is the same question the ranked queue answers.

The two populations are different (auth_outage_reason parks versus paused_by: "spot_quota") and neither can start the other’s sessions.

Within the outage-parked population there is a further boundary, and it runs between Zimmer and the fleet wake rather than between two Zimmer sweeps: AuthOutageWakeAuthority says whether a given park is the ranked fleet wake’s to start (spot) or Zimmer’s own 15-minute sweep’s to resume (priority). Zimmer’s sweep starts only what it owns, and asks for a fleet wake on behalf of what it does not — see When the pool runs dry.

The paused_by: "spot_quota" population has no such split. A session a priority session preempted is in it, not in a population of its own: same record, same sweep, same precedence order. It is reported under its own number because its cause is different, not because anything else about it is.

Three parks, one headline

A session can carry all three park records at once, because none of them clears the others: the gate writes spot_hold_* when it refuses a turn, the ceiling writes spot_pause_* when it interrupts one, and AuthOutageParkService writes auth_outage_* when the login pool has nothing usable. The session page and get_session both have to answer one question anyway — why is this waiting? — and the three answers point at three different resume owners, so naming the wrong one sends the reader to a sweep that is not coming.

SessionWaitingReason is the single ranking both surfaces read:

Rule	Why
Newest wins. Each mechanism stamps when it was recorded; the last one to fire is what put the session where it is now	Session 7503 carried a ceiling pause recorded fifteen seconds before its auth-outage park, and read back the pause
A mechanism with no usable timestamp ranks below every mechanism that has one	”No stamp” is not “the beginning of time”
An overdue hold cannot be the current reason while another mechanism is present	SpotSessionHold.held_sessions excludes a session that also carries a pause or a park, and #rearm! refuses one — so the sweep that repairs a stalled ladder will never touch it, and recheck_sentence’s promise would be false. An overdue hold that is the only mechanism keeps the headline, because there the sweep really is its owner

Losing mechanisms are named rather than dropped. get_session prints one Also on the record, and not why it is waiting now line each, with the reason it lost. The session page draws one box per mechanism rather than a list, so its spot banner renders whichever of the hold and the pause ranks higher — and when an auth-outage park has superseded that, it says so in prose and drops the re-check sentence, whose promise of a sweep does not hold for a session carrying a park. Before this, get_session concatenated the hold and pause lines with no precedence between them and never rendered the outage park at all, so a session parked on an empty pool read back a two-day-old spot hold and a promise of a sweep that skipped it (#642).

Quick filters

Three one-click filter states sit above the search box, because they are the ones worth reaching without touching five checkboxes: All (every status, both classes), All Unarchived (every status but the trash), and All Priority Unarchived. Each is a complete filter state rather than a control that combines with the others, and each persists exactly as pressing Apply filters does.

MCP parity

Capability	Web UI	MCP
Read a session’s genesis and class	Hierarchy panel, dashboard card	get_session
See whether priority preemption is on, and what it has stopped	/inference spot gate card	get_spot_policy
Turn priority preemption on or off	/inference spot gate form	action_spot_policy (set_gating, preemption_enabled)
See that a session was preempted, and which session took its slot	Session page banner	get_session
Filter by class or genesis	Dashboard segmented control	quick_search_sessions (priority_class, genesis)
Read the windows, the concurrency limit, and the current decision	Spot gate card on the Claude Code tab of /inference	get_spot_policy
Read each window’s estimated capacity, dollars remaining, dollars reserved, and spot budget left	Account Pool section and spot gate card on /inference	get_spot_policy
Read the fleet’s burn rate and the sustainable rate the curve allows	Spot gate card on /inference	get_spot_policy
Read the $/min of each harness + model combination	Burn rate table on /costs	get_costs
Read which of the three ceilings is holding spot work, and what lifts it	Spot gate card on /inference	get_spot_policy
Read how many spot sessions are asleep in the spot queue	Spot gate card on /inference	get_spot_policy
Read when the account pool regains capacity, and the soonest 7-day rollover behind it	Account Pool section on /inference	get_spot_policy
Read why one session was paused mid-run, and what resumes it	Banner on the session page	get_session
Read which of a hold, a pause and an auth-outage park is why a session is waiting	Ranked banners on the session page	get_session
Toggle gating, set the two priority reserves, set the max sessions at once	/inference	action_spot_policy (set_gating)
Read the backlog top-up ceiling, stretch and cooldown, and where the fleet sits against them	Backlog top-up card on /inference	get_spot_policy
Set the backlog top-up ceiling, stretch and cooldown	Backlog top-up card on /inference	action_spot_policy (set_top_up)
One-click promote a genesis (non-trigger kinds only)	/inference	action_spot_policy (promote_genesis / demote_genesis)
Reset all genesis classes	/inference	action_spot_policy (reset_genesis_classes)
Set a trigger’s class	Trigger edit form	action_trigger (scheduling_class)
Read a trigger’s class	Trigger page, /triggers badge	search_triggers, get_spot_policy
Choose a class when spawning	Scheduling class on the new-session form; Run as spot on every Quick Router surface	start_session (scheduling_class)
Change one session’s class	Scheduling class on the session detail page, or Make this session priority on the hold banner	action_session (change_scheduling_class)
Start a queued session now, without waiting out its re-check	Start now in the Ranked view’s ⋮ menu; promoting a waiting row does it too	action_session (start_now, or as a side effect of change_scheduling_class to priority)
Park a session in the spot queue with no wake-up time	— (MCP only; the web UI has no control that sleeps a session)	action_session (pause_into_spot_queue)
Sleep a session until a chosen wall-clock time	— (MCP only)	wake_me_up_later
Stop a running session’s turn while parking it	— (MCP only)	action_session (pause_into_spot_queue with halt: true; the default lets the turn finish, and self_session does not offer it)
Rank a session in the spot queue	Precedence on the session detail page; the Ranked view’s inline field, drag handle and ⋮ menu	action_session (change_precedence, or precedence alongside change_scheduling_class)
Put a session at the head of the spot queue	Demote to spot in the Ranked view’s ⋮ menu; Run as spot on a Quick Router submission	action_session and start_session (place: "top_of_spot")
Choose a rank when spawning	Precedence on the new-session form	start_session (precedence)
Predefine the rank a trigger’s sessions get	Precedence on the trigger edit form	action_trigger (precedence)
Read a session’s rank	Ranked view, session detail page	get_session, quick_search_sessions
Read the spot queue in the order it will be worked	Ranked view	quick_search_sessions (status: "waiting", priority_class: "spot", order: "precedence")

The page and the tool render the same decision — SpotGateService.evaluate, of which there is exactly one — so the card’s badge and the tool’s answer cannot disagree.

Both MCP tools are in the health group, not sessions: they are about the deployment’s quota posture rather than about one session, and a self_session connection has no business rewriting the global policy from inside a session it is being throttled by.