The session lifecycle

Every session is in exactly one of five states. The transitions between them are enforced by AASM in app/models/concerns/session_state_machine.rb, and each one carries side effects that are as important as the state change itself.

The states

State	DB value	Meaning
waiting	1	Not executing. Either a turn has been handed over and is queued for one of the agents lane’s worker threads, or the session is dormant — a spot hold, a ceiling pause, a quota park, or asleep on its own wake. The initial state.
running	0	One of the agents lane’s worker threads is executing a turn: an agent process is alive and a monitoring job owns it.
needs_input	2	The agent’s turn ended, or it’s blocked on an elicitation. This is your to-do list: agents archive themselves on completion, and a session waiting on its own PR archives when that PR merges, so what stays here is meant to need you. See goals.
failed	4	Terminal error. Resumable.
archived	3	In the trash. Restorable until the clone is reaped.

The integer values are load-bearing (they’re the existing ActiveRecord enum). A sixth state, corrupted (5), was removed — sessions now go to failed instead.

running means a worker thread has the turn

Zimmer executes agent turns on the GoodJob agents lane, and that lane is ConnectionBudget.good_job_queue_threads[:agents] threads deep — 12 on the Tadasant production deployment. Everything above that number is a durable row waiting for a thread.

resume used to land in running, stamped by whoever handed the session a turn. The agents queue sits between that hand-over and any worker picking the job up, so running routinely held a large population of turns nothing was executing: the dashboard would read “20 running” while /inference correctly said the pool runs 8 turns at once, and both were right about different things (#957, #1040).

So the split is now in the status column:

The turn is…	The session reads
queued in the agents lane, waiting for a thread	waiting
held by a worker that is making the clone and spawning the CLI	waiting
executing — an agent process is alive and monitored	running

Two consequences are worth carrying around.

waiting has a fourth meaning. SessionWaitingReason used to rank three dormancy mechanisms — a spot start-hold, a spot ceiling pause, an auth-outage quota park — each with a named resume owner. “Queued for a worker” is a fourth, and its owner is GoodJob’s own poller rather than a Zimmer sweep. It is read off the job row, never off a metadata marker, so it cannot outlive a worker that died: a session whose turn is gone stops claiming one and falls to StrandedSleepRescue like any other stranded sleeper. The session page draws a banner for it and get_session prints a line.

running? is no longer the “a turn is already in flight” guard. Every caller that used it to decide “queue this prompt rather than starting a second turn” — the web, REST and MCP follow-up routes, Trigger#follow_up_session!, EnqueuedMessageDrainJob, Sessions::MessageParent — asks Sessions::LiveTurn.underway? instead, which reads the agents job rows and fails closed. So does Session#claim_system_recovery_turn!, which is what stops a recovery sweep enqueuing a second turn against a session that already has one queued.

What did not change is any ceiling’s denominator. RunningTurns#on_a_worker counted running rows whose job had a performed_at, and it still does — the pre-spawn window is reported as awaiting a worker, exactly as a first start was before the change. What moved is which status the uncounted turns wear. SpotGateService’s projected burn still prices the queue too, because a queued turn will spend as soon as a thread frees up; it now reads the job rows to find it.

The full machine

The old docs/SESSION_STATE_MACHINE.md was missing five of these

It documented start, sleep, pause, resume, fail, and archive. It did not document block_on_elicitation, unblock_from_elicitation, or the three unarchive_to_* events. It also claimed you cannot archive a running session — you can; archive transitions from waiting, running, needs_input, and failed (the UI exposes this as force-archiving a stuck session).

The events, and what they actually do

start — waiting → running

Guarded on git_root being present. Resets the elapsed-time counter, records the session’s experimental-setting flags, and logs.

This is the only way into running, and it is fired from inside AgentSessionJob#perform — on the worker thread, once the turn’s process has been spawned. Everything that merely hands a turn to a session fires resume, which lands in waiting. See running means a worker thread has the turn.

Experimental-setting flags

start and resume — and no other transition — call record_experimental_setting_flags, which writes one row per experimental setting into session_experimental_flags. The first call fixes the session’s start-of-life value; every later one moves its end-of-life value. So a setting toggled between two turns shows up as a disagreement between the two rather than silently landing in one cohort.

Those two are the transitions at which an agent process is about to spawn, and a setting like MCP tool search takes effect in the spawn environment — so observing there records what the session actually ran with. The terminal transitions look like the natural home for the end-of-life value and are the wrong one: archive and fail fire at bookkeeping moments that can land arbitrarily long after the session last ran. HealthMonitorService#archive_old_sessions archives everything untouched for seven days in a loop, so recording there would re-stamp each old session’s end value with today’s setting, flip it to mixed, and quietly drain the control cohort of the comparison the labels exist to support.

It is bookkeeping and behaves like it: the write is a single upsert, it swallows its own errors, and the caller rescues again around it, so a cohort label can never be the reason a session fails to start. Both of those rescues make the same exception the side-effect reporter does — they do not swallow on a transaction Postgres has already aborted, because a transition that is going to roll back either way is not one a swallow can save. See Experimental settings for what the labels are for.

AgentSessionJob adds a second guard ahead of the transition, and it is the one that makes a pause mean something: a session with a one-time wake-up still ahead of it does not get a first start. waiting is both “queued to spawn” and “deliberately asleep”, and precedence says nothing about which — so without this, every automated starter reads a paused session as a runnable one. The job stands down, logs why on the session, and does not re-enqueue itself; the armed wake is the next event in that session’s life.

Only a first start is refused. A wake firing on time arrives as a follow-up prompt, resume_monitoring re-attaches to a live process, and clone_only spends no turn — none of those is an early start, and refusing them would strand the wake this guard exists to protect. See A pause outranks precedence for the other three callers that decline, and why the guard lives here rather than in a prompt.

Entering running is reported to the fleet-idle monitor

An after_commit on the status column — not a hook on start and resume — calls FleetIdleMonitor.record_session_started! whenever a commit lands a session in running. The fact no_sessions_in_progress needs is “a session entered running”, and every path that produces it has to count: both AASM events, an elicitation unblocking, a session created directly in running. Missing one would let an idle stretch run straight through a moment the fleet was full, because a fleet that fills up and empties again inside one cron tick is invisible to the sweep that samples for it.

It does not cover update_column / update_all, which skip callbacks — no caller writes status that way, and the sweep takes its own reading on its next tick regardless, so this is the fast path rather than the only one.

The hook reports the start; the monitor decides what it means. It asks the ceiling before clearing anything, so a session starting on a fleet that stays under its ceiling leaves the idle clock exactly where it was. fleet_idle_since is the moment the fleet crossed below its ceiling, and only crossing back over it ends the stretch — a clear on any start at all is what used to make the column mean “when a session last started” and the /inference card claim a stretch far shorter than the real one. What re-arms the event between fires is the cooldown, not the fleet. See Why the cadence is the cooldown alone.

After the commit, and best-effort: a transition is never slowed or rolled back by this bookkeeping.

pause — running → needs_input

Fired when the agent’s turn ends (the process exits normally). This is the workhorse transition, and it does nine things beyond changing status:

1. warn_if_pr_goal_captured_no_url — if the session’s goal mentions a pull request and custom_metadata["github_pull_request_urls"] is still empty, write one warning log to the timeline. GithubPrUrlHook only records a PR it can see the session open, and an empty list is otherwise indistinguishable from “no PR to record” — see transcript hooks. Never raises, and once per session, not once per event: fail and archive call it too, and the dedup is on the warning log itself, so a session that pauses, warns, and later archives says it once. Skipped on a recovery pause, for the reason steps 4 and 5 are skipped on one — see which pauses announce themselves.

2. cleanup_running_job — clears running_job_id.

3. bump_needs_input_transition_counter — one increment of custom_metadata["needs_input_count"], handed to both of the next two steps as their debounce marker. Bumped here rather than inside either consumer, because two bumps would make each one’s marker stale to the other and suppress both. It has its own rescue for the same reason every other side effect does: a raise here must not wedge the transition.

4. fire_settled_needs_input_ao_event(marker) — wakes anything watching this session, if the session is still at rest when the settle window closes. See a turn boundary is not a rest. Skipped entirely for a recovery pause; see which pauses announce themselves.

5. enqueue_debounced_needs_input_push_notification(marker) — see below. Skipped for a recovery pause, for the same reason. Steps 4 and 5 are the pause’s announcement and travel together.

6. enqueue_session_inference_if_needed — LLM-generates a title and category if still pending. Skipped when a SessionTitleJob for this session is already queued and not yet claimed: it reads the transcript when it runs, so a second one behind it would only find the work done. The job Session schedules two minutes after creation counts, so a session that comes to rest inside those two minutes is titled when that job fires rather than at the pause. Without a transcript the title is deterministic, from the human’s own prompt rather than the composed one the runtime received.

7. enqueue_status_summary_refresh — the only automatic trigger for the Status summary. The generator still refuses when the session has not moved since the last one, so a transition that added no transcript costs nothing. Skipped the same way when a SessionStatusSummaryJob for this session is queued and not yet claimed — see one generation at a time.

   Both checks are PendingSessionJob, a read on the job table. A job already running does not count: it took its snapshot when it started, and the fresh enqueue is answered cheaply by its claim. The checks exist because a session sleeping and waking on a short self-wake pauses once per wake, and without them each pause stacks one more of each job behind the ones already waiting.

8. execute_pending_sleep — if a wake-up was scheduled while the session was running, the sleep was deferred to here; now it fires.

9. drain_enqueued_messages_after_pause — if the session is coming to rest with a message still queued for it, schedule the delivery. See below. Runs last, and after execute_pending_sleep, so it reads the state the sleep left behind. It does not stop at a sleep: waiting is a resting state a queued message is owed delivery in too, and which populations of waiting genuinely cannot take one is EnqueuedMessageDrainJob’s call to make under its own re-read.

Steps 3–7 are skipped for one kind of session: a status-summary fork, which is Zimmer’s own throwaway (marked in metadata by SessionStatusSummaryGenerator::FORK_MARKER). Its pause means its one turn is finished, so it is harvested and archived instead of notifying anyone or landing in the action queue.

A turn boundary is not a rest

pause runs at the end of every turn, and plenty of those turns end with the session on its way somewhere else rather than waiting for anyone. Steps 8 and 9 above are the two routine cases, and both are decided inside this same callback: a session holding a pending_sleep goes straight back to waiting, and a session with a queued message is resumed by EnqueuedMessageDrainJob seconds later.

That made session_needs_input a much noisier signal than its name suggests. Router session #9964 was woken four times in 25 minutes by child #9966 — a completely healthy session that had opened a PR and was cycling through the open-pr skill’s bounded self-wake, crossing needs_input for microseconds on each pass back to waiting. Every one of those wakes cost the router a full agent turn and four trigger writes, because a fired one-time wake destroys its siblings and they all had to be re-registered.

So the event is now settled rather than emitted on the edge. The job is enqueued with a wait: of SessionStateMachine::NEEDS_INPUT_SETTLE_WINDOW (30 seconds) and the marker from step 3; when it runs, it is dropped unless Session#resting_in_needs_input? still holds — the session is still in needs_input — and unless the marker still matches, which is how a later transition supersedes an earlier one’s event.

fail and archive emit through the unchanged fire_ao_event_triggers, immediately: a terminal state cannot flap, and delaying the event that ends a wait would be pure latency.

resting_in_needs_input? asks about status and nothing else, which is deliberate and is the part worth reading the reasoning for — it is in Triggers. The short version: every richer test considered can only still be true when the window closes if the thing that was going to move the session has failed, and those sessions are precisely the ones a watcher must hear about. Nothing re-emits this event, so suppressing them would lose the wake rather than delay it.

block_on_elicitation emits through the same settled path, and bumps the same counter. That is a change: it never bumped before. It matters only in the narrow case where a pause and an elicitation land inside the 60-second push debounce, where the elicitation now invalidates the in-flight needs_input push — which is the right outcome, since that path already sends its own elicitation_pending push and the transition’s comment says avoiding a double-notify is the point.

The debounce is worth understanding. Sessions sometimes flap running → needs_input → running between turns, and without debouncing every flap would push a notification. So the push job is enqueued with a 60-second delay (NEEDS_INPUT_DEBOUNCE) carrying a monotonic marker from custom_metadata["needs_input_count"]. If the session churns during the window, the marker won’t match and the deferred job no-ops.

Which pauses announce themselves

Steps 4 and 5 are the pause’s announcement: the settled session_needs_input wake that reaches every watcher, and the push notification that reaches the human. running → needs_input is one transition covering several unrelated situations, and only some of them mean “a person is now needed here”.

Pause	Announced?	Why
The agent’s turn ended	yes	The session is at rest waiting on a human. This is what the transition is for, and it is the hot path.
A human hit Pause (paused_by: "user"), an interrupt, an API/MCP/web pause	yes	Also at rest, and a watcher wants to know the session stopped.
Zimmer recovering its own interrupted process (paused_by: "recovery")	no	The session is not waiting on anybody. It is on its way back to running under a continuation that owes it a restart.
Zimmer giving up on a session that never wrote a line (unstarted_turn_restart_abandoned)	yes	Writes no recovery marker, because no sweep can restart it. See A session that never wrote a line.
A status-summary fork	no	Zimmer’s own bookkeeping, harvested rather than queued. Skips steps 3–7, not just the announcement.
Parked on an auth or quota outage (AuthOutageParkService)	yes	Deliberately writes no recovery marker, precisely so it is not swept back into an exhausted pool. A parked session is a real stop and is announced as one.

The settle window above does not cover the recovery case, and the arithmetic is why the carve-out exists. The boundaries the window suppresses leave needs_input within microseconds (a self-wake) or ten seconds (a queued-message drain). A recovery pause does not: the fastest thing that moves the session is RecoveryContinuationJob’s 30-second delay, and behind that CleanupOrphanedSessionsJob’s five-minute cron, so at settle time it is still sitting in needs_input and resting_in_needs_input? — status and nothing else, by design — says yes. It would fire.

So the marker is read at the source instead. Every recovery path writes metadata["paused_by"] = "recovery" immediately before pause! — AgentSessionJob’s dead-process branch and its GoodJob::InterruptError handler, and SessionRecoveryService#transition_to_needs_input — and SessionStateMachine#recovery_pause? reads it in the pause callback. The state transition still happens. The session really is in needs_input, really is in the homepage action queue, and the timeline still says so. Only the two outward signals are withheld.

That matters because a fired one-time wake destroys its siblings. The pattern wake_me_up_when_session_changes_state documents for watching a child is a wake set plus a wake_me_up_later deadline, so a single spurious session_needs_input costs the watcher the whole set — including the backstop that was supposed to catch a genuinely hung child.

The pause transition is not the only door into that wake. Trigger#fire_ao_event_immediately_if_state_matches exists so that a watcher registered after the transition already happened does not sleep forever: it row-locks the watched session at trigger-creation time and fires at once if it is already in the target state. A status-only test there would deliver the same spurious wake a few seconds later, through the other door — and a router that spawns a child and immediately watches it is the common shape, with a freshly-spawned session whose process died being exactly the case this carve-out is about. So the immediate-fire path asks announcement_deferred_to_recovery_sweep? too, and leaves the watcher armed rather than firing. The armed watcher is reached by whichever announcement eventually happens: both enqueue the same AoEventTriggerJob, and that job re-queries the enabled conditions when it runs, so a condition created mid-pause is picked up rather than missed.

The suppression is a deferral, not a deletion, and that is what makes it safe. RecoveryContinuationJob (asked for by the parking code itself, on a 30-second delay), CleanupOrphanedSessionsJob (every five minutes) and DeploymentRecoveryJob (once at boot) all select on paused_by = 'recovery' and auto-continue what they find. SessionContinuation bounds that at MAX_CONTINUE_ATTEMPTS — roughly an hour — and when it gives up it drops the marker, writes an error-level “will not be retried again” line, and makes the announcement the pause skipped, via Session#announce_deferred_needs_input!. So a recovery-paused session that is never continued still wakes its watchers and still pushes, exactly once, at the moment it stopped being Zimmer’s problem and became a human’s.

Which is why the carve-out asks whether a sweep is actually coming, not merely whether the marker is set. A session parked in a frozen category is excluded from every query in both sweeps (Session.not_in_frozen_category), so there is no deferral to make — nothing continues it, and SessionContinuation never runs to announce it later either. That pause is announced at the time, like any other stop. AgentSessionJob’s recovery-pause writers do not check the category, because they run inside the session’s own job rather than in a bulk recovery flow; SessionRecoveryService bails on a frozen category before it ever pauses.

Two edges the deferred announcement deliberately does not cover. A session abandoned in failed already fired session_failed and an unconditional failure push when it failed. A session bounced to waiting by execute_pending_sleep is dormant, and telling a watcher it “needs input” would be a claim about a state it is not in — the settled event would drop it anyway.

The missing-PR warning is deferred on the same test, and lands with the same announcement. Step 1 of the pause is a backstop, and its budget is one warning per session — so the pause that spends it has to be a pause the session actually came to rest in. A recovery pause is not: nothing about the session’s pull-request work is settled, it is on its way back to running, and it is the one pause nobody is told about. Session 5679 spent its whole budget on a deploy interrupt six minutes in, ran for two more days, opened a PR through a route GithubPrUrlHook did not recognise, and came to rest with nothing recorded and nothing said (#558). So pause skips step 1 whenever it skips steps 4 and 5, and a frozen-category recovery pause writes it at the time, exactly as it announces at the time.

The make-good is SessionContinuation’s give-up branch, one line above the deferred announcement — and unlike the announcement it is not gated on resting_in_needs_input?. The warning is a factual timeline note rather than a claim that a human is needed, so it is due in whatever state the session was abandoned in, and the state that needs it most is the one the guard excludes: a recovery pause carrying pending_sleep is bounced straight on to waiting by execute_pending_sleep with nothing armed to resume it. fail and archive stay unconditional behind that, so the warning is deferred rather than lost: a session that is never resumed says it when it is failed or trashed.

A session does not idle on its own queue

needs_input, waiting and archived are the states in which a session stops draining its queue, and the same invariant covers all three: a session must not come to rest with a message still queued for it. What differs is what can be done about it. Archiving ends every delivery path, so the honest answers there are to refuse the transition and to record the discard — below. The two resting states end nothing. An idle session is precisely the condition a queued message is waiting for, so the answer there is to deliver it and keep running.

Both resting states count, not just needs_input. That was the gap behind #566. The three create surfaces all promise the caller delivery “when the session becomes idle”, and a session resting in waiting — asleep on an open-pr self-wake, slept by action_session sleep, resting after a park — already is. Nothing was scheduled to come back for those rows at all: a Github::PrStatusEvaluator merged-PR notice was queued onto session 6377 at 22:56Z, and was still pending when Sessions::ArchiveGuard named it five hours and nine minutes later. Session#idle_for_queued_delivery? is the one predicate both schedulers read, and it is needs_input? || waiting?.

Most of that already happened before this transition. AgentSessionJob drains the queue at each of its four turn-end paths and does it before calling pause!, so the session hands the turn over while still running and never flaps through needs_input — no spurious session_needs_input wake, no push notification. That is still the hot path. What that ordering cannot cover is:

• The race. The drain reads the queue, finds it empty, and then pause! commits. A message enqueued in that window lands pending on a session that is already idle.
• Every pause that isn’t a turn ending. The MCP pause action, POST /api/v1/sessions/:id/pause, the web pause button, Sessions::InterruptService and SessionRecoveryService all call pause! directly with no drain of their own.
• A message queued onto a session that is already idle. None of the three create surfaces — the web queue form, POST /api/v1/sessions/:id/enqueued_messages, MCP manage_enqueued_messages — checks the session’s state, and all three answer that the message goes out when the session becomes idle. An EnqueuedMessage after_create_commit hook covers this one.

Nothing else would have picked any of them up. HeartbeatSweepJob is the only sweep that wakes an idle session, and it deliberately skips one holding a pending message — on the assumption, previously untrue, that something else was about to deliver it.

Both entry points schedule an EnqueuedMessageDrainJob rather than draining inline. AASM runs after callbacks inside the transition’s own transaction, and delivering means resuming the session — a second AASM event nested in the first, plus an AgentSessionJob enqueue — from whatever thread called pause!, including a web request. The job runs once the transition is committed, after a 10-second delay that lets the callers which pause-then-immediately-deliver (Sessions::InterruptService, SessionContinuation’s auto-continue) finish first.

The job itself opens no transaction and takes no advisory lock, which is load-bearing rather than an omission. EnqueuedMessageProcessorService#process_next_message opens its own transaction and rescues everything inside it, and a Rails transaction block joins an open one instead of nesting under a savepoint — so wrapping the call would turn the service’s rescue from “roll the claim back and return false” into “swallow the error and let the outer transaction commit whatever got written”, which can mean a message claimed and destroyed with no AgentSessionJob behind it. Unwrapped, the service’s transaction is the outermost one and a mid-delivery failure rolls the message back to pending. Concurrency is already the service’s job: the claim is a FOR UPDATE SKIP LOCKED on the row plus a lock! on the session, and AgentSessionJob’s end-of-turn drain calls it bare for the same reason.

The job refuses to deliver in seven states, because in each the session genuinely cannot take a message and delivering would make things worse:

State	Why not
Blocked on an MCP elicitation	The agent process is still alive. Resuming spawns a second process against one clone and orphans the round-trip.
Parked by AuthOutageParkService (auth_outage_reason)	A fresh turn hits the same quota or auth wall, burns the message, and parks again. AgentSessionJob’s own drain reads the same marker.
paused_by: "mcp_retry"	A retry carrying the original prompt is already scheduled; delivering now races it into the same failing MCP server.
waiting with a running_job_id	waiting is not only a resting state — it is also where a session sits for the whole of its first start, from the moment AgentSessionJob claims running_job_id through the clone, the session-id write and the spawn. Delivering there buys nothing and costs a round trip: the processor takes its resume! branch, destroys the row, and the replacement job is refused by the concurrency guard as a duplicate of the live first-start job — which now parks the prompt straight back in the queue rather than losing it, but only after churning its position and origin. A session genuinely at rest carries no job id — pause clears it, and so do SpotSessionHold#return_to_queue! and AuthOutageParkService.resume_parked!.
waiting with no runtime session id (session_id.blank?)	A follow-up prompt into a session with none is reclassified by AgentSessionJob as a fresh start, which runs session.prompt — and the way a message reaches that turn is by being merged into the prompt column, permanently. A queued message is a turn with an origin and a position, and spending it there loses both. It goes out on that first turn’s end-of-turn drain instead, where it runs as itself. This asks about the session id alone, not never_ran?: the two differ by transcript.blank?, and the population that matters most is a session that has run and whose stale runtime id was later released.
waiting and held at the spot gate (SpotSessionHold.held?)	SpotSessionHold refuses the turn at the door and re-queues it as a fresh row, so a drain would churn the message’s position and origin on every pass. The gate’s own re-check is already the thing that will run it.
waiting and paused in the spot queue (SpotSessionPause.paused?)	Same gate, resumed by SpotCeilingSweepJob rather than by a re-check.

The last four are the waiting-only ones, and one candidate is deliberately not among them: a session dormant on its own armed wake-up (paused_until_scheduled_time?). That is the case the whole invariant is about — the open-pr skill has a session sleep on a bounded self-wake so it does not occupy the human’s action queue while its PR is rated, and the merged-PR notice is queued rather than sent precisely because that session is waiting. The notice is what the session is asleep waiting for, so delivering it early is the point rather than a cost. EnqueuedMessage#stale? is what keeps that honest: a notice whose subject moved on while it sat in the queue is retired rather than delivered, so a session is only ever woken early for a message that still says something.

Delivering early does not spend the wake. The drain resumes the session through Session#resume_for_follow_up!, which takes the preserving branch of cancel_pending_one_time_wake_triggers: the message is delivered, the session takes its turn, and its own wake-ups — the schedule and any session-scoped ao_event watcher — are still armed when that turn comes to rest. See A follow-up does not cancel a wake below for why, and for the paths that still consume.

The dead-process case. A drain does not need the old agent process — it enqueues a fresh AgentSessionJob that resumes the runtime session from session_id and the working directory, exactly as a human follow-up does. So a session whose process died still takes its message. What it cannot survive is a missing clone or session_id: there AgentSessionJob fails the session with a failure_reason, and the message stays pending on a failed session, where the recovery paths (SessionContinuation#continue_with_queued_user_message) already prefer a queued user message over the automated recovery prompt when they auto-continue it.

No spin loop. Each successful delivery destroys a row, so the queue strictly shrinks. Failure is what needs bounding: three attempts, 30 seconds apart, counted in metadata["enqueued_drain_attempts"] and cleared by resume so each idle spell gets its own budget. After that the job stops and raises an alert, deduped per session. It deliberately does not retire the messages to undelivered the way an archive does: it could not deliver them, but the next turn anybody gives the session drains the queue normally. Retiring here would destroy a still-deliverable message in order to record that this job could not deliver it.

One consequence worth naming: a user-initiated pause on a session with a queued message resumes within seconds. That is the invariant working as stated rather than an oversight — the queued message is input the user was told would be delivered, and pausing the current turn is not the same as withdrawing it. To stop it, delete the queued message.

A queued message outranks an injected recovery nudge

The drain triggers above cover a message that arrives while the session is at rest. The opposite ordering is a separate window, and it is the other half of #566: the message was already queued, and something then resumed the session with an injected AutomatedPrompts::SYSTEM_RECOVERY nudge. Neither drain trigger can fire for that — a mid-turn kill and resume never produces a transition into a resting state, and the after_create_commit hook ran and correctly declined hours earlier.

Every automated resume in Zimmer ends the same way — AuthOutageParkService.resume_parked!, AgentSessionJob#auto_continue_after_interrupt, SessionRecoveryService#auto_restart_session, HealthMonitorService#retry_failed_sessions, SessionContinuation, the restart surfaces — all of them call AgentSessionJob.enqueue_with_prompt(session.id, AutomatedPrompts::SYSTEM_RECOVERY). So the decision lives at that one choke point, in AgentSessionJob: if the prompt this turn carries is a system-recovery nudge and the session has a pending message, the message takes the turn and the nudge is dropped.

EnqueuedMessageProcessorService performs the handoff and does it under the locking it already owns — the row is claimed FOR UPDATE SKIP LOCKED, the session stays running (the pre-pause handoff path, so there is no running → needs_input → running flap), running_job_id is released and a fresh AgentSessionJob is enqueued carrying the message’s content and attachments. The arriving job stands down.

The session has to be running for that to hold, and the check requires it. Releasing running_job_id is something only the handoff branch does, and that branch is selected by the session already being running. From needs_input or waiting the processor takes its resume! branch instead, which leaves running_job_id pointing at the job that is standing down — so the fresh AgentSessionJob would be refused by #perform’s concurrency guard as a duplicate, and the message it carried would go back to the tail of the queue having lost its place in it, with no turn behind it either way. Every caller that injects a nudge resumes the session before enqueueing the job, so the guard costs nothing on the paths it is for. It also settles the session’s armed one-time wakes: a running session’s may_resume? is false, so no resume runs and cancel_pending_one_time_wake_triggers never fires — the wakes a system-recovery resume deliberately preserves survive the handoff exactly as they survive the nudge. The per-turn markers pending_follow_up_prompt / pending_follow_up_sent_at are dropped before the handoff, because the follow-up arm reads pending_follow_up_prompt || follow_up_prompt — left standing, the job the handoff enqueues would find the nudge sitting there and deliver that in place of the message that just won the turn.

Scoped to the nudge on purpose. A human follow-up, a trigger prompt, a poller message and a restart-with-the-initial-prompt are each somebody saying something specific and newer, and the queue drains behind them at the end of the turn as always. The nudge is the one prompt with nothing to add over the queue — “you may have been interrupted, carry on” — so delivering the queued message instead is carrying on, with the reason attached. SessionContinuation had preferred the queue over the nudge on the deployment-recovery path alone since before this; the choke point generalises that one path’s decision to every path that shares its prompt.

Why it mattered: session 7681 was killed and auto-continued four times in ninety minutes on 2026-08-23, and a message queued at 13:48Z was still pending at position 1 at 15:19Z, having lost to the nudge on every cycle, while broadcast_message_count moved 136 → 284 → 533. A session being restarted every few minutes could never drain its queue at all.

And a follow-up the session is still holding outranks it too

The queue is not the only place an undelivered prompt lives. metadata["pending_follow_up_prompt"] is the other, and it means the same thing one step earlier: a prompt was accepted for a job that has not picked it up yet. A follow-up sent to an idle session goes there, not into the queue — the session is resumed and a job is enqueued to carry the prompt straight to the agent.

That is the window #1023 reported. The sender was told the follow-up was sent; a recovery path resumed the target first, carrying the nudge; and no turn for the prompt ever appeared in the target’s transcript. It was silent from both ends — the sender had a success result and an empty manage_enqueued_messages, and the recipient got a message indistinguishable from an ordinary interruption. Three hand-offs vanished that way in one afternoon and stalled a PR for four hours.

Session#recovery_turn_prompt is where the precedence now lives, and all four automated resumes read it rather than reaching for the constant: AgentSessionJob#auto_continue_after_interrupt, both sweeps through SessionContinuation#continue_recovered_session, and SessionRecoveryService#auto_restart_session. If the session is holding a follow-up, the recovery turn carries that; otherwise it carries the nudge. The session’s own timeline says which, so “why did this session get a nudge instead of my message” is answerable from the session page rather than from a raw transcript read.

Two properties keep it exactly-once in both directions:

• The marker is not cleared by the read. The follow-up arm of AgentSessionJob#perform consumes it — moving it to active_follow_up_prompt just before it spawns — so the prompt survives an enqueue that then fails. STALE_RETRY_METADATA_KEYS, which every recovery claim clears, deliberately does not list it: an undelivered prompt is not retry state.
• Whoever takes custody drops it. Sessions::RequeueSkippedPrompt releases the marker when it moves that same prompt into the durable queue, and SpotSessionHold removes it when the gate takes the turn. Two live copies of one prompt is two turns, which is the same defect wearing the other mask — the one #1009 landed guards for.
• And it never displaces a message that outranks it. EnqueuedMessageProcessorService claims a queued message and destroys its row inside the transaction that enqueues the job carrying it — so from that moment the message is a job argument, and a marker left standing would be delivered instead of it. The processor moves the held prompt to the tail of the queue and releases the marker, so the claimed message takes this turn and the held prompt takes the next one. Identical text, a copy already queued, and a nudge each collapse to just releasing it.

The producer side had to be fixed for any of it to hold. Session#deliver_follow_up! stamped the marker, but the two routes that deliver a follow-up straight into an idle session — Mcp::Tools::ActionSession#direct_follow_up and POST /api/v1/sessions/:id/follow_up — did not, so for them the accepted prompt existed only as the argument of the job they enqueued. A worker shutdown discards that job, and with it the only copy. Both now stamp it, immediately after the resume, so a reader who sees the marker is guaranteed to also see running.

The interrupt path is the same loss reached from the other side, and AgentSessionJob’s #preserve_interrupted_prompt closes it: when handle_interrupt_error throws a job away and recovery-pauses the session, the prompt that job was carrying is handed back to the row first, so the resume that follows delivers it. It refuses in five cases — no prompt, a nudge, a prompt already handed to the runtime (active_follow_up_prompt contains it, so re-stamping would replay a message the agent already acted on), an earlier undelivered prompt already in the slot, and a copy already in the queue — and every refusal is written to the session’s timeline with the prompt text, because a drop that is written down is not this bug.

resume — waiting | needs_input | failed → waiting

Handing a turn over, not starting one. resume is what every delivery path fires — a fired wake, a web or API follow-up, a poller’s comment, a recovery sweep, an un-park — and it leaves the session in waiting, queued for one of the agents lane’s worker threads. The start fired by AgentSessionJob#perform is what makes it running. waiting → waiting is a real transition rather than a no-op: the side effects below are the whole point of firing it.

It is still refused from running, and that refusal is load-bearing — may_resume? is half of how Session#claim_system_recovery_turn! tells “nobody is driving this session” from “somebody already is”. The other half is now the job rows, because a queued turn no longer shows up in the status column. See running means a worker thread has the turn.

Unguarded, deliberately. The preconditions for resuming — a clone on disk, a runtime session id to resume into, a live process to reattach to — are established or validated by AgentSessionJob, which recovers what it can and fails the session with a specific failure_reason when it cannot. The state machine does not re-check them, so a resume you ask for is a resume the job gets to attempt. Clears a pile of stale state: MCP failure flags, the paused_by marker, the blocked_on_elicitation and lost_elicitation markers, any pending_sleep, and the enqueued_drain_attempts counter (so each idle spell gets its own retry budget). It also decides what happens to the one-time wake-ups this session was waiting on, which is a decision with four answers rather than one.

A follow-up does not cancel a wake

cancel_pending_one_time_wake_triggers runs on every resume and asks one question: does this resume replace the wait the session was on, or add to it? The kind of resume is carried on three transient flags, never persisted, each set by the caller that knows the answer.

Kind of resume	Set by	What happens to the wakes
A takeover — restart, restart-from-scratch, a resume of a failed session	nothing; the default	Consumed. The wait is replaced, so the wakes are moot. Each condition gets last_triggered_at stamped, which closes it permanently and lets CleanupStaleTriggersJob collect the row as a Trigger#dead_one_time_wake?.
A system-recovery nudge — a deploy restart, an orphan sweep, a hung-process reap	Session#resume_for_system_recovery!	Preserved, and the session is put back to sleep afterwards when a still-fireable one-time schedule backstops that re-sleep. The session never chose to wake.
A wake fire — the session’s own wake_me_up_later or state-change watcher firing	Trigger#follow_up_session!	Held across the woken turn and retired when that turn comes to rest. Consuming them at fire time, before the woken turn has re-armed anything, is the no-trigger window of #569.
A follow-up — a router’s follow_up, a human’s message, a queued message draining, a Slack or GitHub trigger	Session#resume_for_follow_up!	Preserved, with nothing marked and nothing re-slept. The session takes the turn it was handed and its own wake fires afterwards, from needs_input, exactly as it intended.

The last row is #898. Consuming there was silent in both directions: the sender was never told it had just become the only thing that could wake the session, and the session — which had usually already written “my self-wake fires at 08:06” into its own transcript — answered, came to rest in needs_input, and sat there indefinitely. Session 13403 spent the morning of 2026-09-04 like that, holding a nearly-finished PR, until an unrelated third session happened to nudge it.

Preserving is deliberately the eager side of that trade, and the cost is worth stating honestly. The wake carries the prompt the session wrote for itself, so a follow-up that redirected the session is followed by a wake pointing back at the abandoned work — and a session running open-pr or wait-for-ci re-arms its own wake when it wakes, so this can be several turns rather than one. All of it is in the transcript, all of it is the session deciding with full context, and all of it is bounded by the same self-wake budget the skill already keeps. A consumed wake costs an indefinite strand nobody can see at all.

Two things keep it honest:

• A wake that can no longer fire is still consumed — a session-scoped ao_event whose watched session is archived or gone. Preserving one would leave an enabled, unfired row that reads as an armed wake on /triggers and is collected by nothing. A one-time schedule is always preserved, including one already overdue, because an overdue schedule is a wake the scheduler has not reached yet rather than a wake that is lost.
• A wake that outlives its session is already somebody’s job. CleanupStaleTriggersJob destroys one-time reuse triggers whose target session is archived, and it excludes resuscitate_archived triggers because those are an explicit opt-in to waking an archived session. Nothing in the preserve branch second-guesses that: a wake_me_up_later wake never sets resuscitate_archived, so firing at an archived target skips silently and the sweep collects the row.

The sender is told, too. action_session follow_up adds an Its own wake-up line to its result, and POST /api/v1/sessions/:id/follow_up returns a pending_wake object, so a router redirecting a sleeping session can see that the session still wakes itself and does not schedule a duplicate.

Four reset policies, declared together

Every path that hands a session back to a job first drops the per-turn metadata the last turn left behind — retry counters, failure reasons, dormancy markers. Four sets are in use, and they are declared next to each other on Session so the differences read in one place (#508).

Constant	What it is	Who uses it
STALE_RETRY_METADATA_KEYS	the default	every ordinary resume and restart
RESTART_FROM_SCRATCH_KEYS	the default plus SETUP_ARTIFACT_KEYS and the spot-hold ladder	Sessions::RestartFromScratch
PRE_PROMPT_RESTART_KEYS	the default plus runtime_started	restarting a session that failed before its initial prompt
RECOVERY_CONTINUE_KEYS	the default minus paused_by	SessionContinuation, when it delivers a queued message

RESTART_FROM_SCRATCH_KEYS takes the setup artifacts because the attempt it replaces failed partway through and may have left half a clone behind, and the spot-hold ladder because a person asking for this session by name is not the scheduled re-check the gate would otherwise read it as. PRE_PROMPT_RESTART_KEYS drops runtime_started so the replacement spawns with --session-id rather than --resume; --resume against a conversation that was never written raises No conversation found.

RECOVERY_CONTINUE_KEYS is the one where a wrong key set loses a session rather than leaving a stale counter. paused_by is the marker both recovery sweeps select on, so SessionContinuation clears everything else before it hands the turn to EnqueuedMessageProcessorService and drops paused_by only once delivery has succeeded. A refused delivery then falls through to the automated recovery prompt with the session still detectable. Clearing it up front puts the session outside every later recovery pass.

The API and auth error-scan positions — api_error_last_checked_line and auth_error_last_checked_line — are in none of the four. They record which transcript errors have already been handled, and clearing one makes the scanner re-read old entries, which is how a stale quota error misclassifies a new transient rate limit. context_length_last_checked_line is the deliberate exception: it is on the default set, so every reset clears it.

Restart from scratch is one operation behind three doors

The session page’s Restart button, POST /api/v1/sessions/:id/restart and MCP action_session’s restart all reach Sessions::RestartFromScratch when there is no conversation to prompt into — setup never completed, or the session never ran. It owns the whole sequence: the git_root guard, RESTART_FROM_SCRATCH_KEYS, the transaction and its database retry, the two log rows, the resume, the enqueue of a fresh first turn carrying its attachments, and the claim of the new job’s id. Each surface keeps only its own authorization and its own way of rendering the answer, so a dropped Postgres connection during a restart is retried identically through every door.

Refusing a session asleep on a wake-up it has not reached stays at the surface, because the doors genuinely disagree: MCP and the REST API refuse (an agent working a ranked queue must not start a session that asked to be left alone), and the Restart button does not (a person clicking it on one session is taking that session over).

Which sessions each door restarts

The entry conditions are the other thing each surface keeps, and for a long time the web door was strictly the narrowest: it re-derived its own failed?, so a session stranded in needs_input was restartable by an agent and by a script and by a human not at all — no button on the session page, the card or the phone sheet, and a refusal behind it if there had been one (#830).

Door	Entry condition	Accepts
MCP action_session	may_resume?, then refuse_if_paused!	failed, needs_input, waiting — minus any session asleep on an armed wake
POST /api/v1/sessions/:id/restart	may_resume?, then the same pause refusal	the same three, minus the same
Web Restart	Session#restartable_by_hand?	failed, needs_input — minus one blocked on an elicitation

#restartable_by_hand? is may_resume? && !waiting? && !blocked_on_elicitation? — built from the predicate the other two gate on rather than from a second list of statuses, so the set of statuses the web door admits is a subset of theirs by construction. It subtracts two things.

waiting, because every shape that session takes is a reason to: its turn is already queued for a worker (a restart would enqueue a second), or it is asleep on a wake-up that resume would consume on the way past, or it is dormant in the spot queue and a restart puts it back on the window that parked it. A waiting session that is genuinely stalled has its own control — Refresh, which sends the continue nudge under #continue_nudge_on_refresh? and is rendered on every card.

A needs_input session blocked on an elicitation, which is the one shape of needs_input that is not stranded either: its agent process is alive and blocked on a synchronous MCP round-trip whose answer form is rendered on the same page as the button. Restarting would clear the blocked_on_elicitation marker with the elicitation still active, drop the session out of the action queue while a human answer is still the only thing that finishes it, and disarm #clear_stale_elicitation_block!, which repairs from that marker.

What it does not subtract, and this is the one place the doors still genuinely differ: a session asleep on an armed one-time wake. #paused_until_scheduled_time? is status-agnostic, so a needs_input session can hold one — a follow-up preserves one-time schedules rather than consuming them — and MCP and the REST API refuse exactly that. The web door does not, and that is the same interactive-door distinction as above, older than this predicate: a person clicking Restart on one session is taking that session over, where an agent working a ranked queue must not start a session that asked to be left alone. So the web door is a subset by status, not a subset of what those two will actually accept.

The button is not the same promise in both states it is offered from, so it does not read the same:

• from failed it fires on the click, as it always has;
• from needs_input it asks first, because Restart there is a takeover — it resumes a live session and enqueues a turn nobody asked for. Which turn is SessionsController#restart_with_continue_prompt’s decision and the dialog says which of its three answers applies: the whole setup pipeline again, from scratch, with the original prompt (the never-ran case that produced the issue); that same original prompt into a freshly spawned runtime, for a turn that died before its prompt was ever delivered; or an automated SYSTEM_RECOVERY continue prompt into the existing conversation. SessionsHelper#restart_confirmation branches in that method’s order for that reason.

The three render sites (_session_header_actions, _session_card, the mobile joystick’s sheet) all ask the same model predicate and take their dialog copy from the same helper, so what is rendered cannot drift from what the controller will accept. They do not all render it the same way: the card’s footer row has a width budget that #607 spent, so a card offers failed the prominent button and needs_input — which on the default board is every card — a row in its overflow menu instead. SessionsHelper#restart_placement owns that split.

mcp_servers_status is reset on resume, not deleted

clear_stale_mcp_failure_metadata drops four keys outright — should_fail_session, mcp_connection_checked, mcp_failed_servers, mcp_failure_reason — because each is a verdict the previous run reached, and a resume that inherited should_fail_session would fail the new run before its servers had any chance to connect.

custom_metadata["mcp_servers_status"] is treated differently: every entry is reset to {"status": "pending"} for the servers in Session#all_mcp_servers, rather than the key being deleted. Its entries do all have to go — a connected recorded by the process that just exited says nothing about the one about to start — but deleting the key is what left it missing entirely on sessions that plainly had MCP servers (#465). It came back only once the next run got far enough for TranscriptPollerService to reach McpStatusPersisting, and a run that died before its transcript appeared never got there. The REST API and the get_session MCP tool hand custom_metadata back verbatim, so an absent key reads as “this session has no MCP servers” — and, in the triage that reported the defect, as “the servers never came up”. pending says what is actually true at a resume, and the detector upgrades each entry as its evidence arrives.

The reset spans the union of all_mcp_servers and the names already in the hash. all_mcp_servers alone would hand the key’s survival to a catalog read that fails soft — plugin_mcp_servers returns [] when the AIR catalog cannot be resolved — so a blip at resume time would empty the reset for a plugin-only session and delete the key, which is the defect itself.

An unchanged reset writes nothing, so an ordinary resume of an already-pending session issues no extra UPDATE. AgentSessionJob then applies a related but weaker operation immediately before the spawn: Session#seed_mcp_servers_status_floor!, which adds pending only where no entry exists and so leaves a carried-over status alone. It runs there because that is the first point at which air prepare has run and all_mcp_servers therefore includes whatever AIR auto-injected. The MCP server OAuth page has the detector-side half of the same floor, under “A server that fails before it connects is listed as pending, not omitted”.

A live execution is not an interruption

“Interrupted” is one column. GoodJob::Job#perform raises InterruptError whenever it picks a row that already has a performed_at — it never asks whether anything is still executing that row, because in the case it was written for (a worker that died) there is nothing left to ask.

A row becomes re-pickable the moment its advisory lock goes away, and with GoodJob’s default :advisory strategy that lock is a session-scoped lock on one pooled Postgres connection, held for the whole execution. For every other job in Zimmer that window is milliseconds. For an agent session it is the length of an agent’s turn — minutes to hours, and the worker’s own comment in config/environments/production.rb already says so. Lose that one connection anywhere in that window and the lock is released while the agent is still working; the dequeue scope is finished_at IS NULL and does not exclude rows that have a performed_at, so the poller picks the live row straight back up.

That is a phantom re-pick, and it was the dominant source of recovery nudges. Over three hours of production logs on 2026-08-29, ~38 interrupt events across ~19 sessions produced ~36 delivered nudges and not one stand-down by any existing guard. Every fire’s embedded Interrupted after starting perform at '<time>' was 30 seconds to 8 minutes older than the fire itself; in every one the previously spawned CLI was still running and had never been terminated, and a second CLI was spawned two to seven seconds later — two agents against one clone. One session took 16 nudges in 71 minutes in a self-feeding loop, each fire naming the job the previous fire’s nudge had just spawned. The fires arrived in fleet-wide clusters (three different sessions inside the same second, seven such moments in three hours), which is one worker losing many locks at once rather than per-session flakiness — the connection-layer saturation of #329 is the mechanism that fits.

So the handler’s first guard asks whether this job is still executing, here, right now. AgentSessionJob::LIVE_EXECUTIONS is a process-local set of job_ids currently inside #perform. A re-picked execution never registers in it: GoodJob’s InterruptErrors around_perform is declared on ApplicationJob, and a superclass’s around callback wraps its subclasses’, so it raises before the inner one can add anything. A hit therefore means a different thread in this same process is inside #perform for this same job — which is only true when the lock was lost out from under a live execution. A worker that really died takes its set with it, so the genuine case never sees a false hit.

Standing down there means standing down completely: no running_job_id cleared, no paused_by written, no pause, no nudge. Each would be an act against a session whose agent is mid-turn.

Suppressing the handler’s own nudge is not enough on its own, because the re-pick leaves the row lying about itself. GoodJob stamps it with an error at re-pick time and a finished_at when the raise is rescued, and CleanupOrphanedSessionsJob#orphaned_running_session? and DeploymentRecoveryJob#orphaned_running_session? both return true on either of those before they reach any liveness question. So a phantom re-pick the handler correctly ignored would still be swept within five minutes, running the identical cascade under a different log line. Both sweeps therefore ask AgentSessionJob.executing? first, and GoodJob’s cron runs inside the worker, so the set they read is the one the live execution registered in.

The set is deliberately not durable. A re-pick landing in a second worker process finds it empty and falls through to the recovery path, which is the safe direction; Zimmer’s worker role is one container running one good_job process, so in practice every re-pick and every sweep lands where the entry is. That gap is recorded in limitations.

A finished turn is not an interruption

The other way a row is re-picked when nothing was interrupted is after the turn already ended. AgentSessionJob#handle_interrupt_error used to treat every re-pick as “the deploy killed us mid-turn” and resume the session with AutomatedPrompts::SYSTEM_RECOVERY.

That is right when the session was still running. It is wrong when the turn had already finished. A normal completion transitions running → needs_input and, in the same callback chain, clears running_job_id — so the ownership check that would otherwise catch a stale re-pick sees nothing to defer to, and the recovery path resumes a session that was correctly waiting for its human. Over one representative production week that race accounted for 44% of interrupt events and 39% of all recovery nudges sent.

So the handler stands down when the session is already at rest. The test is deliberately positive — it names the states that are genuinely done being driven, and everything else falls through and recovers as before. metadata["paused_by"] carries most of that:

paused_by	Reached needs_input by	On InterruptError
absent	the agent finishing its turn	stand down — nothing was interrupted
"user"	somebody pausing it by hand	stand down — the pause was deliberate
"recovery"	an earlier recovery pass parking it — or degrade_mcp_servers! resuming a session on the servers that did connect	recover, as before
"mcp_retry"	schedule_mcp_retry parking it for a delayed retry	recover, as before

A negative test (“anything but recovery”) would have been wrong: mcp_retry’s only route back to running is its delayed retry job, and both recovery sweeps match paused_by = 'recovery' exactly — so standing down on it would strand the session where nothing looks.

paused_by is not the whole story either. blocked_on_elicitation reaches needs_input from running carrying no paused_by at all, and keeps its running_job_id, because the agent process is still alive mid-turn waiting on an approval. That is not a session at rest, so it is excluded from the stand-down and still recovers.

A session still running when the interrupt lands is unaffected and recovers exactly as before.

waiting is three different situations, and none of them is a recovery

The handler used to push every waiting session through waiting → running → needs_input so a sweep could pick it up. waiting is reached three unrelated ways, and that was wrong for each.

No runtime session id yet. perform did not reach the point where it issues one, so nothing durable exists and there is nothing to resume. The repair is to run the job again, so the handler re-enqueues this job’s own arguments verbatim and leaves the session in waiting.

Pausing it instead is what stranded spot-held sessions. A spot hold lives entirely in waiting, and the only thing that ever schedules the next re-check is the re-check job itself — so severing that chain meant the session could never start again. One issue-work-gate session was held correctly 120 times over 22 hours, then had one re-check job re-picked and spent the next 20 hours in needs_input with an empty transcript on a human’s action queue.

The replay is bounded by MAX_INTERRUPTED_START_REQUEUES. Exhausting it fails the session with a failure_reason naming the cause, which is loud and visible — the one thing a queued session must never do is disappear quietly.

Asleep on purpose. wake_me_up_later runs a session running → needs_input → waiting inside a single pause callback, via execute_pending_sleep. An interrupt landing in that window dragged the sleeper back awake and spent a turn on a recovery nudge, cancelling the wake it had just scheduled. Nothing was interrupted there either — the turn finished — so the handler stands down and lets the wake fire.

The discriminator is Session#awaiting_scheduled_wake?, not a metadata flag, because a session that slept successfully carries none: execute_pending_sleep clears pending_sleep once sleep! succeeds, and writes no paused_by. The armed-wake query is the only signal there is.

Parked on an outage. AuthOutageParkService parks a session in waiting when every account in the pool is out of quota, and it arms nothing at all — its own sweep over metadata->>'auth_outage_reason' is what wakes it. So it matched neither of the signals above and fell through to the recovery path, which stamps paused_by: "recovery" over the park. The auto-continue then declines (the session is waiting, not needs_input) — but the marker is the part that does the damage: DeploymentRecoveryJob and CleanupOrphanedSessionsJob both match [:needs_input, :waiting] with paused_by = 'recovery', so within a sweep or two they resume a session into the outage that parked it. Quota depletion is budget pacing, not a failure signal; a park is a wait, and this was how the wait got ended early. AuthOutageParkService.parked? is the discriminator, the row-level sibling of SpotSessionPause.paused?.

Anything else in waiting that has run. Chiefly the window between the session id being issued and start! firing, which spans the clone, the AIR prepare and the spawn — seconds to minutes, and a deploy is exactly what lands in it. That session is stranded rather than resting, and unlike the first case it has a session id and a clone, so it takes the ordinary recovery path. pause is running → needs_input, so it stays in waiting carrying paused_by: "recovery" — which is swept, because both continuation queries match [:needs_input, :waiting] on that marker and resume accepts waiting.

A session that never wrote a line is restarted, not parked

AgentSessionJob’s resume_monitoring path has exactly one plan: re-attach to the pid recorded in metadata["process_pid"]. When that pid is dead there is nothing to monitor, and the question that decides what to do next is whether there is a conversation to come back to.

metadata["runtime_started"] does not answer it. That flag is written the moment Zimmer records a spawned pid, before the runtime has produced a line, so a process killed in its first seconds leaves it true over a conversation that was never persisted. RuntimeConversationPresence answers it properly, and asks both transcript stores — Zimmer’s polled copy and the runtime’s own file — so a merely lagging poller can never be enough to conclude that nothing was written.

The runtime wrote…	What happens
a conversation	Park with paused_by: "recovery", as before. The session is mid-turn; a resume picks up where it left off.
nothing	Sessions::RestartUnstartedTurn replays the session’s own prompt into a fresh spawn. Nothing was consumed and no partial work exists, so the stored prompt is exactly what should run.
nothing, RetryBudget::EMPTY_TURN.max times	Come to rest in needs_input with failure_reason: "unstarted_turn_not_recoverable", metadata["unstarted_turn_restart_abandoned"] naming the reason, and no recovery marker — no sweep can do anything a third restart would not. The pause announces itself.

A third vantage point reaches the same two rows: a resume whose clone has gone missing from disk, which asks the identical question and hands the no-conversation answer to the same service. See a clone that vanished is rebuilt, not fatal. Its give-up is the row above, unchanged — needs_input, not failed, which is also the only one of the two states that accepts the follow-up that would rebuild the clone.

This is the same judgement ProcessLifecycleManager#handle_empty_turn makes when a process exits under a live monitor, arriving from the other direction: there the turn ended in front of us, here it ended while nobody was watching. They share both the budget and the empty_turn_recovery_count key deliberately — it is one event seen from several vantage points, and a session that has already burned its restarts in-process does not get a second allowance because the next failure happened to be a worker interruption. It is one RetryBudget object (RetryBudget::EMPTY_TURN), so it is also one reset: a stable stretch hands the restarts back to whichever vantage point needs them next, rather than either one being a lifetime cap.

The restart takes a new runtime session id (or, for a runtime that mints its own, drops the stored one) and turns runtime_started off, so the replacement spawn builds --session-id rather than --resume. Re-asserting the old id would hit the #519 trap: a transcript holding only the runtime’s own bookkeeping is simultaneously too present to create against (“already in use”) and too empty to resume (“no conversation found”).

The behaviour this replaced: on 2026-09-02 a worker interruption caught production sessions 12265 and 12267 at once. 12265 had made tool calls and resumed harmlessly. 12267 had produced nothing — zero assistant turns, zero tool calls — and sat in the homepage action queue with a completely empty transcript, indistinguishable from a session asking a question, for nine and a half minutes, until an unrelated orphan sweep reached it.

The recovery pause asks for its own continuation

A recovery pause is a promise that a sweep will continue the session, and with only CleanupOrphanedSessionsJob’s five-minute cron to keep it the wait is whatever is left of that cron’s period. The two mechanisms can also cancel out: the nudge AgentSessionJob enqueues after a worker interruption is dropped when a recovery job is already queued (“Skipping job - session already has a running job”), and when that recovery job then fails to adopt its dead pid it re-enqueues nothing at all — leaving the session with no pending work of any kind.

So the dead-process branch asks for the continuation itself. RecoveryContinuationJob is scheduled with a 30-second delay and delegates to the very same SessionContinuation the sweeps use, so there is one implementation of “continue a recovery-paused session” and one attempt budget. Every guard the sweeps apply is re-asked of the row at delivery time — still paused_by: "recovery", still needs_input or waiting, not in a frozen category — because all three can change inside the delay window, and Session#claim_system_recovery_turn! re-reads the row FOR UPDATE so a cron tick landing at the same moment cannot produce two turns. The cron stays the backstop rather than the mechanism.

This covers one of the six writers of paused_by: "recovery", deliberately. It is scheduled by AgentSessionJob’s dead-process branch — the one the nine-and-a-half-minute stall came through. SessionRecoveryService#transition_to_needs_input, AgentSessionJob’s InterruptError and MCP-retry parks, and the two sweeps’ own re-parks still rely on the cron.

Auto-continue gives up

SessionContinuation#continue_recovered_session needs a session_id and a clone on disk. For a session paused before it ever started, neither will ever exist, and no amount of waiting produces them. Both sweeps select on metadata["paused_by"] = "recovery", so an unbounded retry meant the 5-minute orphan cron re-read the same session forever — 500+ identical “auto-continue skipped” log lines for one session that never ran.

After MAX_CONTINUE_ATTEMPTS failures the sweep gives up: it drops the paused_by marker (which is what both sweeps select on, so the session stops being swept), records metadata["recovery_continue_abandoned"] with the reason, and logs it once at error. The session comes to rest wherever it already was — needs_input or waiting on the ordinary path, still failed when the caller was the InterruptError-failed branch — for a human to restart, which is the honest state for one Zimmer cannot restart itself.

With one exception, and it is the permanent case the budget is not sized for: a session that never ran. See below.

A session that never ran comes to rest in waiting, not in the action queue

needs_input is the homepage’s action list — the set of sessions a person is expected to decide something about. A session that was interrupted while its first job was starting, or that the spot gate held before it was ever dispatched, has nothing to ask. waiting is the state that already models “waiting for compute”, and it is the only one anything re-dispatches from: every start path reads waiting, so a never-dispatched session parked in needs_input is both a slot in somebody’s queue and a dead end where the work simply never happens.

On 2026-08-22 a sweep of the spot queue found 42 spot-class sessions in needs_input, exactly one of them there for a sanctioned reason. Two populations accounted for most of the rest: ~28 interrupted in one window on 2026-08-20 (interrupted_start_requeue_count, a job_started_at, and a transcript holding only the spawn prompt), and sessions held at at_utilization_limit and then abandoned by a recovery pass with recovery_continue_abandoned: "no session_id found, working directory not found or invalid".

Sessions::ReturnToQueue is the rest-state selection, called at the two points where Zimmer stops trying to continue a session it paused for its own recovery and would otherwise leave it wherever the pause put it:

• SessionContinuation#abandon_or_retry_continue — the sweeps’ give-up above.
• AgentSessionJob#auto_continue_after_interrupt — the immediate continuation after an interrupt, which declines when there is no runtime session to resume. Without this the session waits out twelve doomed sweep attempts (about an hour) before reaching the give-up, sitting in the action queue for all of it.

It moves a session only when every one of these holds, and each condition is doing work:

Condition	Why
Resting in needs_input	sleep is needs_input → waiting; anywhere else there is nothing to correct
No runtime session_id	The predicate every other recovery path uses for “has never run”
RuntimeConversationPresence says no conversation	A runtime that mints its own id (Codex) can have written a whole turn while Zimmer’s column is blank; re-queueing that session would run its prompt twice
A prompt to run	The same carve-out StalledSessionStart makes — a prompt-less session is waiting on a human, and no sweep would start it
Not in a frozen category	A parked bucket every bulk flow leaves alone
No wake of its own armed	StalledSessionStart partitions a session with a pending one-time wake out of its batch, and that disqualifier is not a metadata marker, so the return could not drop it — a session moved with one armed would be read by neither owner
unstarted_requeue_count under MAX_RETURNS	See the bound below

The move drops paused_by with it. That marker is what both recovery sweeps select on and one of StalledSessionStart’s dormant markers, so leaving it behind would hand the session straight back to the sweeps that just gave up on it while hiding it from the sweep that can actually start it.

Who picks the session up again. waiting is only the right answer because something reads it, and two owners cover every shape this moves:

The returned session	Picked up by
ordinary — no dormant marker	StalledSessionStart (StalledStartSweepJob, every 5 minutes) restarts a waiting session with no runtime session id, a prompt and no queued turn
held by the spot gate	SpotHoldSweepJob (every 5 minutes) re-arms the re-check ladder. SpotSessionHold.held? requires waiting, so a held session parked in needs_input is invisible to it — which is exactly how the second population stopped being re-checked

“Picked up” is not always “restarted”, and the difference is deliberate. A session older than StalledSessionStart::MAX_STALL_AGE (1 day) is failed by that sweep rather than started, with a failure_reason naming the wait — its first turn is stale by definition, and a day-old one is a gamble a human should take deliberately from a failed row they can see. So a returned session that has been sitting for days lands in failed, not in running. That is still the point of the move: failed is a visible row with a reason on it, and needs_input was a slot in the action queue that nothing was ever going to act on. The spot-held row is unaffected either way — spot_hold_reason keeps those sessions out of StalledSessionStart’s population entirely, and SpotHoldSweepJob puts them back on the gate’s ladder rather than aging them out.

The bound. Sessions::ReturnToQueue::MAX_RETURNS caps how many times one session may be sent back, for the same reason MAX_INTERRUPTED_START_REQUEUES caps the replay: a session that keeps reaching a give-up, being returned, being re-dispatched and reaching the give-up again would loop for as long as the deployment stays broken. Past the budget the session rests in needs_input after all, with unstarted_requeue_exhausted recording why, and a human is the right next step. The counter is deliberately not in Session::STALE_RETRY_METADATA_KEYS: a resume clears that set, so a session that runs for one second and dies again would get its whole budget back on every cycle.

A session that has a conversation behind it is untouched by all of this. Its needs_input may be a real question, and the give-up still rests it there and still makes the announcement the recovery pause deferred.

The budget is sized for the other thing the validation reports. A missing working directory can be transient — a volume not yet mounted after a boot, a clone being restored — so the count is set to roughly an hour against the 5-minute cron, long enough for a blip to clear and still bounded. The writes go through merge_metadata! rather than update!: a read-modify-write would run Session’s validations, including the globally coupled agent-root and catalog-skill checks, and a RecordInvalid swallowed by the caller’s per-session rescue would stop the counter advancing — restoring the unbounded loop for exactly the sessions least likely to be recoverable.

The nudge names the path that sent it

A dozen paths enqueue the same SYSTEM_RECOVERY constant — a deploy, an orphan sweep, a SIGTERM retry, an API-error retry, a quota park lifting, a manual restart. Sending one undifferentiated string means neither the agent nor the human reading over its shoulder can tell which fired, and “this should only happen on a deploy” is the reasonable conclusion it invites — in that same week, fewer than a third of these nudges were within ten minutes of a deploy.

AutomatedPrompts.system_recovery(reason:) appends one line naming the path, after the standing instructions, so the prompt’s meaning is unchanged for an agent that ignores it. AutomatedPrompts.system_recovery? is the matching predicate — compare with it rather than == against the constant. Both branches of AgentSessionJob#resume_for_recovery_prompt preserve the session’s wake-ups, so a misread no longer loses them; what it loses is the conditional re-sleep, which only the recovery branch performs. A recovered session that should have gone back to sleep comes to rest in needs_input instead.

Three producers name themselves today, and between them they account for the large majority of nudges by volume: the InterruptError auto-continue, SessionContinuation (which covers both the orphan sweep and deployment recovery, via continuation_source), and AuthOutageParkService resuming a session whose login pool refilled. The rest — the SIGTERM retry, the API-error retry, the auth-recovery resume, the health monitor, the manual restarts from the web UI, the REST API and the MCP tool, and the ProcessLifecycleManager continuations — still send the bare constant. system_recovery(reason: nil) returns it unchanged, so converting one is a one-line change; the gap is unfinished work, not a designed-in default.

A system-recovery resume keeps the wake-ups

That cancellation is right for a takeover and wrong for a recovery. When Zimmer restarts a session’s process after an interruption — a deployment restart, an orphaned process, a hung process reaped, a health-monitor retry — the session did not choose to wake, so the wake-ups it was sleeping on are still exactly what it is waiting for. Consuming them there is how an orchestrator gets stranded: it comes back with nothing armed, ends its turn, and sits in needs_input indefinitely with children it was supposed to be watching.

Recovery paths therefore call Session#resume_for_system_recovery! rather than resume!. It sets the transient system_recovery_resume flag for the duration of the transition, and cancel_pending_one_time_wake_triggers takes a preserve branch instead:

• The pending conditions are left armed — nothing is consumed.
• If one of them is a one-time schedule (a wake_me_up_later deadline backstop), pending_sleep is set so the session returns to waiting once the recovery turn ends. A wall-clock wake fires regardless of what else happens, so sleeping on it cannot strand the session.
• If the only pending wakes are session-scoped ao_event watchers, the session comes to rest in needs_input with those watchers still armed. A watched session may have reached the state being watched for during the outage, and that transition is not replayed — so sleeping would trade a long stall for an indefinite one. Trigger#follow_up_session! delivers to a needs_input session just as well, and the operator can see it in the meantime.

That re-sleep is conditional at execution time as well as at resume time. It is recorded as pending_sleep plus a pending_sleep_requires_wake marker, and execute_pending_sleep drops it rather than sleeping if nothing is armed by the time the turn ends. A backstop whose wall time elapsed during the outage is due the moment recovery resumes the session, so it can fire mid-turn, destroy its siblings and hand off to a new turn without ever pausing — and sleeping on that stale intent would leave the session in waiting with nothing armed and no paused_by, invisible to both recovery sweeps. A deliberate sleep (POST /api/v1/sessions/:id/sleep on a running session) carries no marker and is always executed, because arming nothing is exactly what it means.

A wake-fire resume holds the wake-ups until the turn it starts is over

The other exception, and it is about when rather than whether. When a wake fires and resumes its requester, the rest of the wake group really is moot — but only once the woken turn has run. It used to be voided at fire time, in two places at once: this callback consumed every pending one-time wake aimed at the session, and the firing job then destroyed the sibling trigger rows. Re-arming is the woken turn’s job and happens at the end of that turn, so between the fire and a successful re-arm the session held nothing at all — and a turn interrupted anywhere in that window (a deploy restart, a killed process, a transient failure) left it asleep with no wake, indistinguishable from a session sleeping correctly (#569).

Trigger#follow_up_session! therefore sets a transient wake_fire_resume flag around its delivery, and cancel_pending_one_time_wake_triggers takes a hold branch: the conditions are left unfired and the trigger rows are stamped wake_held_at. A held wake is still enabled and can still fire. What the stamp records is that this turn owes it a retirement.

Only a trigger that is nothing but one-shot wakes is held, because holding hands the whole row to the retirement. A trigger mixing an unfired one-shot with a recurring schedule or a Slack condition does other work, so its one-shot is consumed the old way and the row is left alone.

retire_held_wake_triggers pays that debt, from the pause and archive callbacks. Two details carry the whole design:

• It runs on pause unless the pause is one Zimmer entered on the turn’s behalf — Session#turn_stood_down_before_it_ran?. Three markers answer it, and all three mean the prompt was never delivered: paused_by = "recovery" (a process Zimmer restarted under a live turn), failure_reason = "undelivered_turn" (a turn that raised during setup — ParkUndeliveredTurn states explicitly that it writes no paused_by), and auth_outage_reason (a turn stood down because the account pool was empty). A woken turn that never ran has not had its chance to re-arm, so retiring there would re-open the window at the one moment it has to stay shut.
• It only takes rows carrying wake_held_at. Wakes the woken turn armed for itself carry no mark, so the pause that retires the old group leaves the new one alone. failed triggers are exempt as everywhere else — they are the record of a wake that tried and could not, and the user’s to clear; re-arming one clears its hold mark, so the re-arm is not destroyed by the next pause.
• It is deliberately not called from fail. A failed session is not reliably a finished one — CleanupOrphanedSessionsJob recovers failed sessions carrying GoodJob::InterruptError or the recovery marker, which is an interrupted turn wearing a different status.

Retiring is as load-bearing as holding, and both halves are tested. A wake that outlives its wait and fires into a later, unrelated one fails silently in exactly the same way the bug does.

One recovery path deliberately does not preserve: when continue_recovered_session finds a queued user message, it delivers that instead of the recovery prompt, via a normal resume!. A waiting user message means someone has taken the session over, which is the case the cancelling semantics are for.

The automatic recovery paths do not call resume_for_system_recovery! directly — not the two sweeps, not the hung-process auto-restart, not the stranded-sleep rescue, not the failed-session retry, not the auto-continue after a job interruption. They go through Session#claim_system_recovery_turn!, which takes the row lock, refuses a session the trash swallowed since the caller read it — or one whose work has been handed to a replacement that is carrying it (#801) — and calls the preserving resume only on the way to :claimed. See Spawning and monitoring for all of that guard and for the resumers outside this family that lock by hand instead, #554 for what it costs when it is missing, and #753 for the tail of the family.

The invariant this restores: a session that was in waiting with wake-ups registered does not silently end up in needs_input with none.

sleep — needs_input → waiting

The “wake me up later” path. The session goes dormant and a one-time schedule trigger will resume it. If the wake-up is scheduled while the session is running, metadata["pending_sleep"] = true is set and the actual transition happens on the next pause.

A park that passes "halt": true does not wait for that next pause — it stops the turn itself (Sessions::HaltRunningTurn terminates the process and pauses the session), so the session is dormant in waiting before the tool call returns. The deferred sleep remains the fallback for a halt that cannot land. See Parking a session that is still running.

Agents reach this through the wake_me_up_later MCP tool, which goes through Sessions::ScheduleWakeUp — a time in the past, or inside the 30-second grace window, would fire-and-drop in the scheduler and leave the session asleep forever, so the service refuses it. Only needs_input, running and waiting sessions can be scheduled; from failed or archived the auto-sleep silently no-ops and the trigger would point at a session nothing can wake.

A wake is only a wake while it can still fire, and Session#awaiting_scheduled_wake? is where that is decided — for the refresh nudge, for the start guards, and for the repair sweeps. A one-time schedule stops counting once its moment passes unfired; a session-scoped ao_event watcher stops counting once the session it watches is archived or deleted, because the firing path keys on transitions into the watched state and an archived session makes no more of them. A watched session that merely failed still counts, deliberately: it can be restarted, and it can still be archived. See A wake is only armed while it can still fire.

A wake in flight is not a wake that was lost, and both halves of that predicate allow for it — SessionStateMachine::SCHEDULE_FIRE_SETTLE, ten minutes. “Due” and “fired” are never the same instant: ScheduleTriggerJob is a one-minute cron, so a one-time schedule reads due, enabled and unfired for as long as it takes the next tick to reach it, and a session_archived watcher on a session that has just archived is waiting on an AoEventTriggerJob that was enqueued after that transaction committed and has not run yet — which is exactly why the archive callback spares that watcher while destroying its siblings. Inside that window both count as armed; the siblings it spares nothing for — a session_needs_input watcher on the same archived session — read unfireable straight away, as they always did. Outside it, they do not: the scheduler has had every chance and the wake is not coming. Without the window, a wake scheduled for a round five-minute boundary raced StrandedSleepSweepJob’s own tick and lost — session 13229’s wake came due at 11:20:00, the sweep announced to #alerts that it could never fire at 11:20:08, and it fired at 11:20:18 into a session the rescue had already resumed, losing the prompt it carried.

A human’s levers on a sleeping session are narrower than they look, and worth stating exactly. Start now (the Ranked view’s ⋮) resumes a session parked in the spot queue, which arms nothing — but it refuses one asleep on a wall-clock wake, because Sessions::StartNow treats an armed wake as outranking the queue. For that session a human has two routes and they do different things. A follow-up from its session page is delivered and the session takes a turn, but it leaves the wake armed — #preserve_pending_wakes_across_follow_up keeps one-time schedules, so the message adds to the wait rather than ending it. Genuinely taking the session over means cancelling the wake at /triggers, where it is listed as Wake session #<id> at <time>. The Restart button is a third thing: it is not offered for a sleeping session, because sleeping means waiting and the button is offered only for failed and needs_input — but a needs_input session can hold an armed wake too, and there Restart consumes it, deliberately, because a person clicking it is taking the session over.

The spot queue — the same sleep with no wake-up. action_session’s pause_into_spot_queue sleeps the session and hands it to the spot scheduler instead of arming anything: Sessions::PauseIntoSpotQueue writes the same dormancy record a mid-run ceiling pause writes (SpotSessionPause), so the sweep that already resumes spot work picks this session up on the next pass where a Claude Code account is under both quota targets and a session slot is free — highest precedence first. Two consequences worth knowing before calling it:

• It makes the session spot, if it was not already, because the sweep resumes a non-spot sleeper on its very next pass. That is reversible and it is the way back out: Make this session priority on the banner promotes it, and the next sweep resumes it.
• It replaces any wake-up the session had already armed — parking with no wake-up means “not then, this instead”, and a leftover wake would pull the session straight back out of the queue.

Its queue position is whatever precedence the session is already carrying; the Ranked view is where that is changed.

A sleep with a wall-clock wake armed outranks every automated reason to start the session early — its precedence, its scheduling class, a recovered quota pool, a freed fleet slot. The places that would otherwise have started it are listed in A pause outranks precedence, along with the limit of what a pause claims: it is a floor under when the session may run, not a promotion past the spot queue when the moment arrives. A spot-queue park is the deliberate exception to the whole thing — it arms nothing, so the spot sweep resuming it is the point.

Zimmer also uses this path on its own behalf: AuthOutageParkService parks a session here when the login pool runs dry, which is what keeps a quota-blocked session out of the heartbeat sweep’s reach. See Agent harness auth.

A session can be in both states at once — parked on the pool and asleep on a wake somebody chose. The pool recovering answers the first and says nothing about the second, so AuthOutageParkService.wake_parked_sessions! skips it. That skip is load-bearing: its resume goes through resume!, and cancel_pending_one_time_wake_triggers would have consumed the pause on the way past, leaving no record that the session was ever paused.

The same service also guards the exit paths, not just the moment the wall is hit. A session whose turn was never delivered — active_follow_up_prompt still set, because AgentSessionJob removes it only on clean completion — stopping while its runtime’s pool has no available account is the outage, not a finished turn. AuthOutageParkService.park_undelivered_turn! parks it here rather than letting the loop’s “the process is gone” fallbacks pause it onto somebody’s action queue. See The park has to survive the paths that do not know about it.

Every sleep records why it happened

sleep is the funnel every running → waiting passes through — a running session is marked pending_sleep, paused to needs_input, and slept from the pause callback — so it is where Sessions::StopRecord writes the answer to why did this session stop. Three metadata keys, on every sleep:

Key	What it holds
stopped_reason	One of auth_outage_park, spot_hold, spot_pause, scheduled_wake, deliberate_sleep, system_recovery_resleep, halted_turn, unstarted_requeue, user_pause, unattributed
stopped_detail	A sentence a human can read
stopped_at	When the session went dormant, UTC

All three are cleared by start and resume, so they only ever describe a session that is dormant now. SpotSessionHold.return_to_queue! writes its own copy, because refusing a turn at the gate is the one running → waiting that does not go through the state machine.

unattributed is a finding, not a blank. It is what the record says when nothing on the row explains the stop, and it also writes a warning to the session’s own timeline and renders in get_session as “went dormant with no attributable cause”. This closes #608, where three priority sessions each started cleanly — fresh job_started_at, new pid, every MCP server connected — and were back in waiting sixty to seventy-five seconds later with no exit_status, no auth_outage_reason and no auth_outage_parked_at. A wake could not be verified from the wake side, a scheduler could not account for a start budget that had silently un-done itself, and the row looked exactly like one that was never woken.

Provenance rides with the intent. A running session is marked pending_sleep now and slept at the end of its turn, so the mark and the mechanism’s own record used to be two separate writes. Every writer now stamps pending_sleep_reason in the same statement as the flag — Sessions::StopRecord.pending_sleep(reason) — and the flag and its stamp are cleared together. So even a mechanism whose follow-up write is lost cannot produce a stop that nothing explains.

AuthOutageParkService was the mechanism that could, and it is fixed at the source too: the outage keys and the sleep intent are now one merge_metadata!. Under the old ordering the session was marked pending_sleep first and the outage recorded several statements later, with a session-log insert in between and one rescue around the lot — so a log write that raised left the sleep intent behind with nothing beside it, AgentSessionJob read the exit as an ordinary completed turn (no exit_status), and the pause carried the session to waiting in silence.

block_on_elicitation / unblock_from_elicitation — running ⇄ needs_input

This pair exists because an elicitation is not a turn ending. An MCP server made a synchronous request and is blocked waiting for the human; the agent process is still alive. So block_on_elicitation surfaces the session as needs_input (to get it on your homepage and into the notification path) but does not call cleanup_running_job — killing the process would break the round-trip.

A metadata marker (blocked_on_elicitation) distinguishes this from a real pause, and guards the flip back.

Stranded elicitation blocks are a real failure mode

If the reactive unblock is missed — a swallowed AASM::InvalidTransition from a state race, or the MCP server crashing mid-round-trip — the marker is left set with nothing to clear it, and the session sits in needs_input showing a phantom “blocked on elicitation”.

CleanupExpiredElicitationsJob sweeps for this every 5 minutes and calls clear_stale_elicitation_block!, which strips the marker and leaves the session in needs_input; flipping it to running would create a phantom running session with no monitoring job. A minutes-old stranded block has no live round-trip to resume into.

Leaving it there and saying nothing is its own lie, though — a session parked in needs_input with no banner reads as merely idle. So a stranded needs_input session gets a lost_elicitation marker naming what happened, which the session page renders as a banner. See Elicitation.

fail — waiting | running | needs_input → failed

Cleans up the running job, fires session_failed triggers, enqueues a push notification that bypasses the per-session opt-in, reports the failure when a trigger created the session, and — like pause — enqueues a Status summary refresh. The reasoning behind the unconditional push: by the time fail! fires, retries are already exhausted, so this is a final non-self-resolving event. A silent status flip would be worse than an unwanted push.

A status-summary fork that fails is harvested (recording the failure on the source session’s summary) instead of notifying, exactly as on pause.

A trigger’s session that fails takes the work item with it

A trigger fire is a one-shot event, and the session it creates is the only thing carrying it. The fire is spent the moment that session exists — GithubTriggerPollerJob records the item key in seen_items, SlackTriggerPollerJob advances its cursor, AoEventTriggerJob consumes the wake — and each of them does that deliberately, because the alternative is dispatching the same event twice (#704).

So when that session reaches failed, the work item is dropped and nothing points at it any more. failed is terminal: no message is delivered to it, no wake reaches it, and the merge-gate contract reads it as unreachable. Nothing said so either. On 2026-08-23 the PR ready to merge → merge gate trigger fired correctly on a ready to merge label, its session died 86 seconds later at first start, and the PR sat with no gate on it for eleven hours — the trigger’s history showed the fire, the PR showed a clean label and no comment, and no alert fired anywhere (#632).

OrphanedTriggerFire closes that. When a failing session’s genesis is one whose fire is genuinely consumed — github_label, github_issue, slack or ao_event — and it carries a trigger_id, fail enqueues OrphanedTriggerFireJob, which:

• writes an ERROR line on the session’s own timeline saying the fire is spent and its subject has nobody on it, and
• raises an alert naming the trigger, the session, why it died, and the GitHub PR or issue if one can be found in the prompt the fire carried.

“Why it died” is sent in two places, and the split is security-relevant rather than cosmetic. Session#failure_summary — a closed case over enumerated failure_reason values that never interpolates runtime output — goes in the report’s details: prose, assembled here. The raw exit_status and exception_message go through AlertSnippet into a field of their own, for redaction, clamping and UTF-8 coercion. Both are needed: the summary renders process_failed as “Process failed”, while the sentence that identifies the 7844 failure — “Runtime session id … is already in use” — lives only in exit_status. And the raw half has to be redacted, because AirPrepareError embeds air prepare’s full stderr and air prepare is the step that resolves .mcp.json’s ${VAR} credential substitutions, so that text can plausibly carry a secret value. This is the first path by which either field leaves the box, and a secret that has been sent to an alert channel cannot be un-sent. UnclassifiedFailureReporter makes the same call for the same reason.

metadata.trigger_id alone would be too wide, because a trigger creates sessions on paths where a retry is coming or a human is already watching, and telling either of those that “no retry is coming” would be false. A recurring schedule re-fires on its next interval and a system_event is re-armed when nothing handles it; a manual Invoke (Triggers::ManualFire, from the web UI, REST or MCP action_trigger) is somebody pressing a button and watching the session they just made; a burst-notice session is not a work item at all. Session#genesis is stamped at creation and is exactly that distinction, so the population is keyed on it, plus the burst_notice marker. One gap is knowingly left — see Limitations.

Three things it deliberately does not do:

• It does not re-dispatch. The population includes the merge gate — the one mechanism authorized to merge without human sign-off, and the one Zimmer already has a scar from double-dispatching. The only after-the-fact guard against redoing work already done is “did the runtime write a conversation”, which RuntimeConversationPresence answers off a clone the reaper may already have removed and which fails toward “a conversation exists” — so an automatic retry would decline precisely in the long-delay case that hurt. Surfacing is what was missing; every recovery in that timeline was a human noticing.
• It does not touch an ordinary failure. The predicate is asked inside the transition, so a session with no trigger behind it — nearly all of them — costs one hash lookup and enqueues nothing.
• It does not collapse two drops into one message. Unlike UnclassifiedFailureReporter, where N sessions hitting one unknown failure mode are one fact, N orphaned fires are N distinct work items each needing its own re-dispatch. The dedup key is per session, and the bound on noise is instead that each session reports at most once: orphaned_trigger_fire_reported_at is checked on entry, and that key is in Session::STALE_RETRY_METADATA_KEYS, so a session genuinely restarted and failed again reports again. The stamp goes down after the report rather than before — stamping first would turn a Slack outage, or a deploy landing in the window, into a permanent silent drop, which is this bug reintroduced inside its own fix. The cost of that order is that a retried job writes a second identical timeline line; the Slack side is covered by the per-session dedup key.

The start failure itself is a separate matter and has its own owner — #519, fixed by ProcessLifecycleManager#handle_session_id_conflict minting a fresh runtime session id. This is about what happens to the trigger’s work when a session dies for any reason at all.

A turn that never started parks instead of failing

failed is where a fault comes to rest, and that is the right home for a turn that broke while the agent was working. It is the wrong home for one that broke before the agent existed, because failed is not in the needs_input action queue the homepage presents as the user’s to-do list, and nothing sweeps it — CleanupOrphanedSessionsJob and DeploymentRecoveryJob restart a failed session only when it died on GoodJob::InterruptError. A follow-up carrying a live user request that dies in setup is therefore dropped in silence: production session 3949 lost one for two days (#439).

So Sessions::ParkUndeliveredTurn intercepts AgentSessionJob’s catch-all rescue and comes to rest in needs_input instead, on five conditions:

1. No agent process was spawned by this job — asked of ProcessLifecycleManager#current_pid and the job’s own local pid. A turn that died after its process existed is a runtime fault with a transcript to read, and still fails.
2. No further attempt is queued. AgentSessionJob declares retry_on for Timeout::Error, Errno::ECONNRESET and Errno::ETIMEDOUT, and the catch-all re-raises — so a turn dying on one of those is not over, and another attempt will run this same prompt. Parking it would announce an ending in the action queue while a retry was still queued, and a human acting on that announcement would race the retry into delivering the prompt twice. Those three keep the failed path they have always had, which is a real gap and is written up in Limitations.
3. The turn was carrying a prompt. An undelivered prompt is what makes this a person’s problem. A promptless turn has nothing anybody is waiting on.
4. The session is running or waiting. pause transitions from running only, so a turn that died during setup — which since #1040 leaves the session waiting rather than running — takes the start! it had earned first, and then parks. Without that the park is refused and the caller fails the session into exactly the state this class exists to keep it out of.
5. Not a status-summary fork, which must never take a slot in the action queue.

The park writes failure_reason: "undelivered_turn", the exception class and message, and the prompt — under undelivered_prompt, plus a copy on the session’s own timeline, which is what a human actually reads. Nothing re-delivers that prompt automatically, and the UI copy says so.

It is deliberately not put back in pending_follow_up_prompt, which is the obvious-looking place and is a trap: #perform’s follow-up arm reads pending_follow_up_prompt || follow_up_prompt, so a value left there wins over the next turn’s real prompt. Three delivery paths enqueue a prompt without stamping the marker — the REST follow_up endpoint, MCP action_session’s direct follow-up, and EnqueuedMessageProcessorService — so all three would send the dead turn’s prompt in place of the one a human just wrote. The third is reachable from this park’s own pause!, which drains the queued-message backlog.

A failure before the first agent turn is retried, not failed

The park above owns the turn that was carrying somebody’s prompt. The turn that was carrying nothing — a session’s first one, still being set up — has a different right answer, because nobody is waiting on it and nothing has been done yet: run it again.

On 2026-09-02 an OverlayFS EXDEV came out of the AIR CLI’s install swap on the production host — Invalid cross-device link @ rb_file_s_rename - (/opt/air-cli/node_modules, /opt/air-cli/.retired/node_modules) — and did so for 45 minutes. AirPrepareService.ensure_air_installed! rescues SystemCallError and re-raises it as AirPrepareError; that class is in neither of AgentSessionJob’s two prepare-specific rescues, so it reached the catch-all, which stamped failure_reason: "exception" and called fail!. 14+ sessions went terminal in that window, every one of them killed before it had taken a single turn. Three were the only live shepherd for an open PR; one was the session spawned to fix the underlying bug. Nothing retries failed and Sessions::StartNow refuses it, so they sat there for about seven hours until a human-run sweep restarted them (#785).

AirPrepareService’s own 5s/10s/20s ladder could not have caught it and was never going to: that ladder wraps the air prepare subprocess, and this was a Ruby-side raise in the install that runs before it.

So AgentSessionJob#retry_bootstrap_failure runs first in the same catch-all, and the gate is when the failure happened, not what it was. Session#before_first_agent_turn? is the whole classification — metadata["runtime_started"] blank and the transcript blank, both, the same caution never_ran? takes. A turn that raised before an agent ever spoke destroyed nothing: no conversation, no half-applied edit, no pushed branch. There is no allowlist of retryable exception classes here on purpose, because an allowlist only ever knows about the outages that already happened, and the next fault will be a shape nobody enumerated either.

Note what is deliberately not one of the signals: session_id. AgentSessionJob stamps the runtime session id right after the clone and before air prepare, so every failure this predicate exists to catch already has one — which is why Session#never_ran?, whose first clause is session_id.blank?, is the wrong question here.

Five conditions, and the session is left waiting with its configuration intact so the ordinary start path picks it up again:

1. No agent process was spawned by this job — the same question the park asks, of the same two pids.
2. The turn carried no prompt. That is the park’s case and must stay there — this retry clears the runtime session id, which routes the replacement down #perform’s fresh-start reclassification, and that arm merges the follow-up text into session.prompt when the session already has one, rewriting the column. A human’s message would be delivered joined to the original prompt rather than as the turn they sent. The two paths are disjoint by this line.
3. The session is waiting — what a turn still in setup reads as since #1040. This is also what keeps a clone_only turn out (created needs_input) and a resume_monitoring one (it takes start! before any of its setup can raise).
4. Before the first agent turn, as above.
5. Budget left.

Note what is not a condition, where the park has one. The park declines while a retry_on attempt is queued, because parking announces an ending in the action queue and a human acting on that announcement would race the retry into delivering one prompt twice. This path announces nothing — and because taking it suppresses the catch-all’s raise, no retry_on attempt is ever scheduled behind it, so exactly one replacement exists either way. Declining here would leave the three classes most likely to be a transient outage (Timeout::Error, Errno::ECONNRESET, Errno::ETIMEDOUT) going terminal, which is the failure this exists to end — and would let what raised decide after all. There is no status-summary-fork carve-out either: a fork must never take a slot in the action queue, and staying waiting takes none.

The ladder is 30s / 2m / 5m / 15m / 30m plus up to 30s of jitter — about 52 minutes in total, chosen to outlast the 45-minute window that produced the outage, with the jitter there so a fleet-wide fault does not re-land in lockstep the way it arrived. It is bounded because the opposite error is exactly as silent: a genuinely broken configuration retrying forever would never reach anybody either.

Each retry discards what the failed attempt built — Session::SETUP_ARTIFACT_KEYS, the runtime session id, and the clone tree itself. That is not tidiness: #perform reuses an existing clone when it finds one, and that arm does not run air prepare at all, because it assumes the clone was prepared by whatever made it. For a bootstrap failure that assumption is false, so a retry which kept the clone would skip the very step that failed. The tree is deleted through AtomicCloneRemoval rather than left for a reaper, because the youngest bar any clone reaper applies is OrphanCloneFilesystemCleanupJob::PRESSURE_AGE_THRESHOLD at two hours and the whole ladder fits inside it — five abandoned clones per failing session, none of them reclaimable for an hour, is how a fleet-wide bootstrap fault fills the volume, and a full volume is itself a bootstrap failure. That is the one feedback loop the retry budget does not bound.

The give-up path keeps the clone and clears only the runtime session id. setup_complete? is session_id.present? && clone_root.present?, so nulling the id alone already leaves the terminal session needs_restart_from_scratch?; dropping clone_path as well would make two things false at once, since the teardown line tells the human the clone is preserved for debugging and DeferredCloneCleanupJob finds a failed session’s clone through clone_root.

What is kept on both paths is everything that is the session — prompt, agent root, MCP servers, catalog selections, goal, attachments — and the replacement job carries the original job’s own arguments, so the retry is the same turn rather than an approximation of it.

The retry is quiet in one specific sense, and it is worth being precise about which. The catch-all’s own Error in agent execution line and backtrace still land on the session’s timeline on every attempt, followed by a warning explaining the re-queue — nothing about the fault is hidden from anybody reading the session. What the retry does not take is the raise e, and that is the paging path: Sentry’s ActiveJob integration and the terminal ERROR the zimmer_backend_log_errors Grafana rule reads. A fault about to fix itself should not page five times. Giving up is where the volume goes. When the budget is spent the session takes the ordinary loud path — an ERROR line, fail!, and the re-raise into Sentry and the terminal ActiveJob ERROR the zimmer_backend_log_errors Grafana rule reads — under failure_reason: "bootstrap_retries_exhausted" rather than the generic "exception", so the failure that outlived five automatic attempts does not read like the first one. That reason is a member of Session::PRE_PROMPT_FAILURE_REASONS; combined with the discarded setup artifacts it leaves the session needs_restart_from_scratch?, so the Restart button on the page a human lands on re-runs the whole pipeline instead of trying to --resume a conversation that was never written.

It writes no paused_by: "recovery" marker either: that marker promises a sweep will continue the session, and a boot that is deterministically broken would just fail again on every one of those attempts and end in the same silence. Without it, announcement_deferred_to_recovery_sweep? is false and the pause callback makes the announcement itself — the settled session_needs_input wake fan-out and the debounced push. That push is routed through build_failure_body rather than the usual needs_input LLM summary, which would otherwise describe the previous, unrelated turn.

resume drops everything the park stamped, scoped to a row whose failure_reason is the park’s own. Without that, the marker outlives the turn that earned it and the next ordinary pause renders a failure on a session that has just worked.

The session’s ERROR lines, its stamped exception and the re-raise into Sentry and the terminal ActiveJob ERROR (which is what the zimmer_backend_log_errors rule reads) are all unchanged. What does change, besides the resting state: the session page’s failure block is gated on Session#shows_failure_details? rather than on failed?, and HealthMonitorService’s failure_reason_distribution queries status: :failed, so a parked boot failure does not appear in it — see Limitations.

The sibling case is Sessions::RestartUnstartedTurn, which parks the same way when a process is gone and wrote nothing; this one is for a process that never existed.

It also runs warn_if_pr_goal_captured_no_url, last in the callback so nothing above it can be skipped. A session that dies mid-turn never reaches pause, so a PR it opened and never named would be recorded nowhere at all (#313).

When recovery restarts a session into silence

Zimmer’s own recovery is a loop, and until #988 it had no exit. CleanupOrphanedSessionsJob calls a running session hung once its last_timeline_entry_at is older than CleanupOrphanedSessionsJob::INACTIVITY_THRESHOLD (15 minutes); SessionRecoveryService terminates the recorded pid and restarts the session; and the restart’s own resume calls reset_elapsed_time_counter, which sets last_timeline_entry_at to now. Fifteen minutes later the sweep reaches an identical verdict about an identical session.

That is correct while the restarts work. What nothing bounded was the case where they do not work at all. Four production sessions wedged this way on 2026-09-05 — 14313 (92 minutes), 14474 (36), 14391 (3h10m) and 14501 (2h45m) — across two agent roots and two clones. In each one the recorded process_pid named no process, the transcript file was frozen, needs_input_count climbed as recovery re-queued job after job, and process_identity was byte-identical across every relaunch, which is proof no restart ever reached a spawn: a new process on the same host cannot reproduce an old one’s started_at_ticks. The status read running the whole time, so no session_failed fired, no push went out, no parent’s wake trigger tripped, and the sessions held slots indefinitely. The person whose queued work never landed was the one who eventually noticed.

Sessions::SilentRecoveryGuard is the exit. It runs at the two places recovery gives a session another turn — SessionRecoveryService#auto_restart_session and SessionContinuation#continue_recovered_session, the latter shared by all three sweeps — and it spends one RetryBudget::SILENT_RECOVERY attempt per restart that started a turn and wrote nothing. At the maximum it fails the session with failure_reason: "recovery_produced_no_output" and drops paused_by, which is what stops both sweeps selecting it.

The verdict takes two facts, and needing both is what keeps live sessions safe:

Fact	Read from	Why not one without the other
the turn actually started	metadata["job_started_at"] advanced	AgentSessionJob stamps it only after every stand-down guard has passed, so a spot hold, a quota hold and a superseded turn all leave it unchanged — and are therefore invisible to this budget by construction. Quota depletion is budget pacing, never a failure signal.
the turn wrote nothing	Session#transcript_line_count unchanged	Written by TranscriptPollerService alone, so unlike last_timeline_entry_at no state transition can move it. One tool call, one assistant message, and the whole budget is handed back.

An auth-outage park is checked explicitly on top of those, because the job that parks has already stamped job_started_at.

A spent counter never fails a session on its own. Reaching the maximum is not the terminal condition; a fresh silent restart while the maximum is already spent is. A session that starts producing output again — because a human restarted it, because a later sweep worked, for any reason — is read as recovered on the next pass and starts from zero.

What it deliberately does not bound: a restart that does spawn a process which then produces nothing is the empty turn, and RetryBudget::EMPTY_TURN has bounded that from both vantage points since #727. The two are different failures with different evidence, so they do not share a counter.

archive — any state → archived

Sets archived_at, retires any messages still queued for the session, dismisses notifications, fires session_archived triggers, cleans up triggers watching this session, sets a trash expiry, and — last, for the same reason as on fail — runs warn_if_pr_goal_captured_no_url. A session trashed straight from needs_input is one nobody is coming back to, so this is where a PR opened and never recorded gets said out loud.

Neither this nor fail is literally terminal — resume runs from failed and the three unarchive_to_* events from archived — so the warning keeps pause’s point-in-time honesty: it says what was true when it was written (“no PR URL yet”) and, because the dedup is once-per-session, it is not retracted if the session is later revived and does open one.

Status-summary forks are carved out inside the hook rather than at the call site. The generator strips the goal a fork inherits, but only in prepare_fork — a fork abandoned before that point still carries the source’s “open a PR”, so the goal check alone would not have covered it.

So are sessions that never got an agent turn, and this is the carve-out archive and fail need most: both transition directly from waiting. A session created with a PR-flavored goal and trashed before it started, or bulk-archived by HealthMonitorService’s seven-day sweep, never had a turn in which to open a PR — so there is no miss to report, and saying it anyway is noise in exactly the place the warning is meant to be a signal (#356). Session#before_first_agent_turn? decides it, and it errs toward speaking: a session carrying runtime_started in any form — including the false that Zimmer writes when it finds a killed process wrote no conversation — or a transcript in any form, is one where a runtime was spawned, and it is still warned about.

The archive line names who did it

Every other transition has one obvious cause. archive has six unrelated callers — the web UI, the REST API, the MCP API, HealthMonitorService’s stale-session sweep, status-summary fork cleanup, and a session archiving itself — and they all used to leave the same five words in the timeline. So “why did my session get archived?” could not be answered from the session page: a human clicking Trash and another agent archiving the session out from under its own unfinished work looked identical, and telling them apart meant reading a different session’s raw transcript off disk.

Callers set session.archive_actor immediately before archive!, and the line names it:

[State Machine] Session moved to trash by session #5225 via the MCP API
[State Machine] Session moved to trash by a user in the web UI
[State Machine] Session moved to trash by Zimmer's stale-session sweep (untouched for 7 days)

The actor is transient, never persisted, and never inferred. An archive from a console or a test says by an unrecorded caller rather than claiming an actor it does not have.

On the MCP API the actor is self-declared, through the same acting_session_id argument follow_up uses and for the same reason: the API key is shared by the whole fleet, so nothing about the request identifies the caller. An archive that declares nothing is logged as by an undeclared MCP API caller — which still answers the question that matters, because a human archiving a session does it from the web UI. The injected self-session server names itself separately (via the self-session MCP server), since its tool group narrows the actions a session may take, not the session_id it may aim them at.

Archiving retires the message queue

Archiving ends every path by which a queued message could still be delivered. EnqueuedMessageProcessorService claims pending rows only, and for a live session the only thing that calls it is AgentSessionJob’s end-of-turn drain — which an archived session never reaches, because the monitoring loop sees archived? and terminates the process instead of pausing.

Those rows used to stay pending anyway, and that is what made a dropped message silent. Every reader of the queue — the panel on the session page, GET /api/v1/sessions/:id/enqueued_messages, the MCP manage_enqueued_messages list — treats pending as still going to be sent. So did the sender: follow_up without force_immediate auto-queues onto a running session and answers “Message queued (session is running). It will be sent when the agent completes its current task”, which is a promise rather than a receipt.

The window is not exotic. Goals routinely tell an agent to archive itself once its work is done, so a session that is running when you queue a message may legitimately never become idle again: it goes running → archived and the queue never drains. Any two messages arriving inside one agent turn can hit it. Production session 6073 did — a user’s second Slack message queued behind their first, the session archived at the end of that same turn, and the message sat pending in a queue nobody would ever drain.

So archive moves them to undelivered, a fourth, terminal status alongside pending/processing/sent. An archive is the reason a row lands there but not the only one — the delivery-time staleness check retires a poller notice whose state had already moved on, on a session that is very much alive (see Background jobs). What the two have in common is that nothing is left worth delivering, which is what the status records. Three things follow from that:

• The queue can no longer misreport itself. undelivered is not pending, so the claim query cannot take it — including after an unarchive_to_*, where a weeks-old message would otherwise arrive as if it had just been sent.

• The archive line names what was lost, next to the unresolved-PR clause and for the same reason:

  [State Machine] Session moved to trash by session #5225 via the MCP API — 1 queued message was
  never delivered and is now marked undelivered: "What do you mean I pulled yellow onion?..."

• An alert fires, deduped per session, unless the caller forced past the archive guard, or the archive stranded nothing but a notice Zimmer had addressed to the session itself. Unlike the unresolved-PR clause an unforced strand is an anomaly: a message was accepted and never delivered, and the only reason to find that out from a user noticing is that nothing else said it.

The row itself is kept, not destroyed — its content is the thing the sender was promised delivery of — and the session page lists it, marked as never delivered. That listing sits outside the follow-up form rather than inside the live queue panel, because the form is hidden once a session is archived, which is the state these rows exist in.

Archiving over a queued message is refused, and force is the way through

Retiring the messages records the loss; it does not undo it, and recording a discard is not the same as intending one. So an archive over a non-empty queue is refused, on every surface — the MCP archive and bulk_archive actions, POST /api/v1/sessions/:id/archive, and the web UI’s Trash button:

Cannot archive session 6073: 1 queued message has not been delivered. Archiving discards it.

  1. What do you mean I pulled yellow onion? I didn't do that, add it back

Do not archive. If you are this session, end your turn instead — the queued message is delivered
as your next turn, and archiving after that succeeds because the queue is empty. If you are
archiving another session, leave it alone and let it consume what you sent it.

Only if you have read the message above and are deliberately discarding it: re-call with
"force": true. That is not the recommended path — the message was accepted from someone who was
told it would be delivered, and forcing throws it away.

The refusal leads with what not to do, because for the caller that meets it most — a session archiving itself at the end of a turn — not archiving is free and correct. An agent self-archives because it believes its work is finished, and a message that landed mid-turn is exactly the evidence that the belief is stale. It ends its turn instead, the pause drains the queue, the message arrives as the next turn, and the archive succeeds after that because the queue is empty.

force is the deliberate override, off by default, for a caller that has read the message and is choosing to discard it. It is named last in the error and hedged in its own schema description, because it exists for the exception rather than the rule. Forcing does not make the discard invisible: the messages are still retired to undelivered, the archive line still names them, and the discard is recorded on the log plane — so what was thrown away stays readable afterwards. What it does not do is page.

Archiving someone else’s running turn is refused too

A queued message is not the only thing an archive can throw away. Archiving a session that has an agent turn in flight terminates that process mid-turn — AgentSessionJob’s monitoring loop reloads the session, sees archived?, SIGTERMs the runtime and lets the clone go — so whatever the turn had not committed is lost.

#400 is the incident. A status-summary fork, resumed into a copy of another session’s pre-archive conversation, decided the work it was reading was finished and called action_session archive with that session’s id. The target had been handed a new prompt 45 seconds earlier and was working on it. Its transcript ends [Request interrupted by user]; its clone was deleted. From outside, the caller saw running with a growing message count and then archived — nothing distinguished the work completing from the work being destroyed, so polling the status could not catch it.

So the MCP archive and bulk_archive actions ask Sessions::LiveTurn as well:

Cannot archive session 3567: an agent turn is in flight on it right now. Archiving terminates that
process mid-turn and deletes its clone, so whatever it has not committed is lost — and nothing
afterwards distinguishes that from a session that finished.

You are not that session, so you cannot know what it is in the middle of. If you are trying to
redirect it, send it a "follow_up" instead (with "force_immediate": true to get ahead of the turn
it is in) — it keeps the conversation and the work. If you want it to stop, "pause" it and archive
once it has come to rest.

Only if you have decided to destroy the running turn: re-call with "force": true.

Four things bound it, and each matters:

• A session archiving itself is never refused. That is the normal ending: its own turn is the one in flight, and it is the only caller that knows whether that turn is finished. Refusing it would break the completion signal the whole lifecycle policy rests on. Identity comes from the connection rather than the request body — RuntimeConfigPostProcessor stamps session_id onto the URL of the Zimmer server it injects into a session’s own runtime config, and Mcp::Context carries it, so nothing a caller sends can forge it. SelfSessionActionSession#enforce_self_report! reads the same field for message_parent, with the opposite polarity for an unidentified caller: it lets one through, and this guard refuses one. A connection that names no session is not the session either, and force is one call away while the destroyed turn is not.
• Only a turn a live worker is executing counts — an unfinished AgentSessionJob row that JobLiveness calls :running. A turn still queued in the agents lane is cancelled by an archive rather than destroyed, which is what the caller asked for; and a row left behind by a SIGKILLed worker is a corpse, so a stuck session stays archivable by the sweeps that exist to clear it.
• force still works, and the loss stops being silent. A forced archive over a live turn writes a warning on the target’s own timeline — “Archived by … while an agent turn was in flight — the running process was terminated mid-turn and its uncommitted work discarded. This was not the session finishing.” — and the answer says the same thing back to the caller. The session cannot report this itself; it is the thing being terminated.
• A fork’s connection names the fork. ForkSessionService used to prepare a fork’s runtime config for the source session, which stamped the source’s session_id onto the fork’s Zimmer MCP entry — and RuntimeConfigPostProcessor#with_session_id leaves a URL that already carries one alone, so re-preparing for the fork on its first turn did not correct it. A fork therefore held a connection that identified it as its source, and would have read as that source archiving itself. The config is now prepared for the fork.

POST /api/v1/sessions/:id/archive and its bulk twin ask the same question, and they reach the unidentified-caller branch every time: the REST API key belongs to the whole fleet, so nothing about the request says who is calling. An agent that reaches for curl gets the same refusal the MCP tool would give it, and the same force override.

The web UI’s Trash button is deliberately exempt. It is the interactive door — a person is looking at the session page with the running turn rendered in front of them — which is the same distinction the Restart button draws when it consumes a pause that the non-interactive doors refuse to.

The system-initiated archives (HealthMonitorService’s stale sweep, the status-summary fork cleanup, SessionStatusSummaryHarvestJob) do not consult it, for the same reason they do not consult Sessions::ArchiveGuard: a refusal none of them could reconsider would be a fleet-wide stuck state with no human in the loop to clear it.

A forced archive records, it does not page

The alert exists for a message that was discarded without anyone reading it. force is the caller asserting that it did read it: Sessions::ArchiveGuard refuses first, prints every pending message in the refusal, and the refusal says in as many words that force is only for a caller that has read them. Paging a human about a discard a caller has already been talked through is paging them about Zimmer working.

It is an assertion, not a proof, and the gap is worth stating. Every surface skips the guard outright when force is set, so a caller that sends it on its first call is never refused and never shown anything — and on a batch the one flag covers every session in it, including ones whose queues the caller has not seen. Nothing at strand time can tell that caller from one that answered a refusal. What makes this the right trade anyway is that the alternative was worse in practice, and that the forced branch is not silent: force is off by default, named last in the refusal and hedged in its own schema description, and every forced discard still leaves the row, the archive line, and the ledger entry below.

So the branch is on the archive, not on the message:

The archive	What happens
Forced past Sessions::ArchiveGuard	Rows retire, the archive line names them, and a [StrandedQueue] forced=true line goes to the log plane at WARN. No page.
Unforced — every system-initiated archive	Rows retire, the archive line names them, and it pages for every message somebody was waiting on.

Production made the case on 2026-08-29. A spot-queue cleanup Tadas had asked for worked a list of eleven sessions that had refused archive over undelivered queued messages, read each queue as the refusal instructs, and forced. Seven pages landed in #alerts in three seconds, and because every alert in that channel spawns a router session, one authorized cleanup burned seven of them. The messages were stale recovery nudges and a router’s own backstop wake — nothing anybody was waiting on, and nothing the origin vocabulary could distinguish from a message somebody was.

Three things this is not:

• Not a hole in the guard. Sessions::ArchiveGuard still refuses, and that refusal is the load-bearing part: it is what puts the message in front of a caller that has not seen it. Only the page is dropped, and only once the caller has answered the refusal.
• Not silence. The row still retires to undelivered, the archive line still names it, and SessionStateMachine#record_strand_ledger writes a line the log plane can be queried on — session, actor, count, and each row’s id and origin. That is the fleet-wide question the per-session archive line cannot answer: what has been force-discarded, and by whom? It is logged at WARN on purpose; Zimmer backend logging errors (excludes staging) pages on ERROR and FATAL, and writing it at ERROR would move the page rather than remove it.
• Not “automated messages don’t page”. The origin column records who wrote each message — caller for anything queued on someone’s behalf, and automated_pr_merged, automated_merge_conflict and automated_recovery_nudge for the notices Zimmer addresses to a session on its own behalf — and it is emitted on the REST payload and in the MCP list, so a retired queue can be explained from outside the database. Exactly one of those origins is exempt on the unforced path, and automated_pr_merged is deliberately not it. A system sweep that strands a PR-merged notice still pages, and that matters: a fork wrongly credited with its source’s PR gets the merge notice queued onto it and is then archived by the harvest job, and this alert is how that bug was found.

A strand nobody was waiting on records, it does not page

force answers “did the caller read this?”. It does not answer the other question the alert needs, which is “was anybody waiting on it at all?” — and there is one message in Zimmer’s queue where the answer is no.

On 2026-08-31 at 17:14:34Z, #alerts paged over session 8810. Everything in the chain was working:

• 8810 was a machine-created status-summary fork of #7340 — a throwaway whose only job is to write a status blurb for its source. No human ever spoke to it; Zimmer’s own record of human-authored messages reports zero anywhere in its hierarchy.
• It was spot-held, and its re-check stopped firing. SpotSessionHold’s sweep re-armed it with Zimmer’s own recovery nudge — “you may have been interrupted; continue if you were mid-task, otherwise keep waiting. No human sent it.” The gate refused that turn too, so the nudge went into the durable queue rather than onto a job.
• The fork then failed to boot at all (Runtime session id … is already in use) and the status-summary cleanup archived it. That cleanup does not consult Sessions::ArchiveGuard and does not set force — correctly, on both counts — so the strand took the loud branch.

The page said “whoever queued them was told they would be sent”. Nobody had queued it. Zimmer had written it, to a session that no longer existed, and then paged a human about throwing it away — burning a router session for a message with no author and no reader.

So the unforced branch subtracts one thing, on the message rather than on the archive:

The retired queue	What happens
Anything somebody was waiting on	Rows retire, the archive line names them, and it pages — unchanged
Only notices Zimmer addressed to this session	Rows retire, the archive line names them, and a [StrandedQueue] forced=false line goes to the log plane at WARN. No page.
A mix of the two	It pages about the caller’s messages only, the body names how many nudges it left out, and the suppressed rows still get their forced=false ledger line

EnqueuedMessage::SELF_ADDRESSED_ORIGINS is that list and it has one entry, automated_recovery_nudge. The bar for a second is that the message must carry nothing still true once the session is archived. The recovery nudge clears it: its whole content is a question — were you interrupted? — put to a session that is now terminal, so it has no answer, no author waiting on one, and no reader left to discover the loss from.

AutomatedPrompts::HEARTBEAT is the near-miss, and it is left out on purpose. It reaches the same queue by the same route and would meet the criterion — but every origin added here permanently narrows the one alert that reports a message being thrown away, so the bar for widening it is a firing this exemption would have prevented, not an argument that it would fit. The heartbeat goes on the list when it pages, and not before.

automated_pr_merged deliberately does not clear it, and that contrast is the design. A merge is a fact about the world that outlives the archive. An unforced strand of that notice is how the mis-credited-PR bug behind #555 was found — a status-summary fork that inherited its source’s PR, had the merge notice queued onto it, and was archived by the harvest job. Keyed on “automated” this exemption would have silenced its own smoke detector; keyed on this one origin it cannot. automated_merge_conflict is out for the same reason: an unresolved conflict is still unresolved afterwards, and nothing else reports it.

Where the stamp comes from. SpotSessionHold#queue_behind_scheduled_turn is the only place a refused turn’s prompt becomes a durable queue row, and it is a funnel: a trigger fire, a human’s follow-up and Zimmer’s own nudge all reach it as the same opaque string. It stamps automated_recovery_nudge when AutomatedPrompts.system_recovery? recognises the prompt — the same predicate AgentSessionJob already keys resume_for_system_recovery! off — and caller otherwise, caller being the default and the wider bucket. Reading the body is confined to that one write; every reader afterwards asks the column. Getting it wrong is bounded in both directions and silent in neither: a nudge mis-stamped caller pages exactly as it did before, and a caller’s message could only be mis-stamped by being byte-identical to the nudge template.

Three things this is not, mirroring the forced branch:

• Not a hole in the guard. Sessions::ArchiveGuard is untouched and still refuses an archive over a queue holding a recovery nudge. The refusal is what puts an unread message in front of a caller; only the page is dropped, and only for a message with no reader.
• Not silence. The row still retires to undelivered, the archive line still names it, and record_strand_ledger writes the forced=false ledger entry — so “nothing was lost” is a claim somebody can audit rather than an absence.
• Not a fix for why the nudge was stranded. A status-summary fork that cannot boot is a real defect and is tracked separately. This changes who finds out about it: the fleet’s log plane rather than a 5pm page about a message nobody wrote.

One bulk archive is one page

Each stranded queue is reported on its own, naming its own session: a sweep that strands queues on N sessions has discarded N distinct messages, and one alert standing in for all of them is the summary that hides the other N−1. That is right for the record and wrong for the delivery, which is what produced seven separate pages — and seven router sessions — from one call.

The delivery is fixed a layer out rather than in this process. Every one of those reports carries the same message, “Queued messages stranded by an archive”, with the session id in its context — so GlitchTip groups them into one issue and Grafana groups them by alertname over a five-minute interval. N records, one page, and no per-session detail lost on the way: the complete record is the ERROR record for each session, the archive line on the session itself, and the undelivered rows.

The sweep is the path this matters most on, because it is the only one of the three that can strand unforced across many sessions at once — the two caller-facing bulk paths only strand when the caller forced, and a forced strand no longer pages at all. On those two the batch is defensive: it keeps the one-page-per-call property true of whatever per-session alert is added next. The web UI’s bulk action is not wrapped because it never strands: it skips any session with a queue and reports the count.

Every state that can archive is covered, needs_input and failed included. Nothing drains their queues, which makes the discard there certain rather than merely likely — and a refusal without an override would leave those sessions permanently un-archivable, which is why force and the full coverage are one design rather than two. The same reasoning puts force on the narrowed self_session schema: a session archiving itself is the caller that meets the refusal most, and would otherwise be the one caller with no way past it.

The surfaces differ only in how force is expressed, not in whether the discard is refused:

Surface	Refusal	Override
MCP archive	ToolError naming the messages	"force": true
MCP bulk_archive	per-session error, batch continues	"force": true, applied to the whole batch
POST /api/v1/sessions/:id/archive	422 with the same text	force: true
POST /api/v1/sessions/bulk_archive	per-session entry in errors, batch continues	force: true, whole batch
Web Trash	flash naming what would be lost	an Archive anyway button that re-posts with force
Web bulk archive	skipped, and the count is reported	none — archive those individually to confirm

The web bulk action deliberately has no force: a bulk selection is not a claim to have read each session’s queue, and there is no per-session confirmation to hang an override on. Every Trash affordance — the session header, the session card, the mobile joystick — posts to the same controller action, so the two-step covers all of them without touching any of them.

System-initiated archives do not consult the guard at all. HealthMonitorService’s stale sweep, status-summary fork cleanup, and SessionStatusSummaryHarvestJob archive unconditionally. That asymmetry is deliberate: a refusal those callers could hit would be a fleet-wide stuck state with no human in the loop to clear it, and none of them is a caller that could reconsider and let the message land. Their discards are still recorded, because the retirement runs on the transition itself rather than at the call site.

…and what the archive cost

Archiving is what takes a session out of Github::PrPollPass’s scope — with_github_prs excludes archived sessions — so it also ends any chance of the merge message the PR goals in config/goals.json promise the session: “the pull-request poller sends this session a message when the PR merges, and THAT MESSAGE IS YOUR SIGNAL TO ARCHIVE”. A session archived before the poller’s next pass never receives it — including the reading that message carries of what the merge fired, which for a PR whose merge deploys is the half of the work that happens after the merge.

When the session still has tracked PRs that Zimmer never saw reach a terminal state (merged or closed), the archive line says so and names them:

[State Machine] Session moved to trash by session #5225 via the MCP API — 1 tracked pull request
had not reached a terminal state, so no merge notification will be delivered for it:
https://github.com/tadasant/strad/pull/152

This rides on the archive line rather than raising a warning of its own, deliberately. A merge gate archives the producing session within seconds of merging — well inside the poller’s 30-second cadence — so an unresolved PR at archive time is the common case rather than an anomaly worth alerting on. What it is worth is one sentence in the place someone asking where their session went is already looking.

The clone is not deleted immediately. DeferredCloneCleanupJob runs after a short undo window and then either deletes the clone (if it’s clean) or preserves unpushed artifacts for TRASH_RETENTION_PERIOD, which is 4.days. EmptyTrashJob deletes them once that expires.

That job is the only thing that deletes an archived session’s clone. AgentSessionJob kills the agent process when it sees the archive and leaves the clone where it is, because a second deleter beats the preservation by about ten seconds and there is nothing left to preserve from a tree that is already being unlinked. #653 is what happens when that ordering is not respected, and it costs two things: the session’s uncommitted work is destroyed rather than saved, and the preservation dies on ENOENT against a tree the other deleter has already renamed out from under it (see AtomicCloneRemoval), which pages. Nothing leaks by waiting: StaleCloneCleanupJob (archived with no trash deadline, one hour) and EmptyTrashJob (at the deadline) are the backstops if the deferred job never runs.

Preservation is not instant on a large tree — a git bundle create, a git add -A and a git diff --binary over the whole working tree — so the job re-reads the session’s status immediately before deleting, and stands down if it is no longer archived. An unarchive inside that window puts a session back to work on this clone, and that is not something the undo window could undo.

A clone that has vanished by the time preservation starts is not a failed preservation. There is nothing to preserve and nothing to keep, so CloneArtifactService logs it at .info — the same call check_dirty_state makes for the same race — and the job clears the trash deadline rather than holding a session in the trash for four days over a clone that does not exist. “Vanished” is decided by looking at the disk, not by reading the exception: Errno::ENOENT also comes back from a git that is not on PATH, and that is a real failure on a clone that is still there. A clone still on disk that cannot be read stays .error and is held for the full window, because it is then the only copy of the session’s work.

A git that had to be killed for exceeding its deadline is held for the same reason, reached from the other direction. Every git command in this path runs under a wall-clock watchdog (CloneArtifactService::GIT_TIMEOUT_SECONDS, 120s) so that a wedged one cannot hold a thread of the two-thread maintenance lane forever — see the sweeps’ section. When it fires, the dirty check reports dirty rather than clean: clean is what authorizes the delete, and a timeout means the question was never answered. The job preserves instead, and a preservation that times out too holds the clone for the full window.

If artifact extraction fails outright, the clone is kept — it is then the only copy of that session’s unpushed work — and the job writes the same 4.days deadline on the session and a warning log the session’s owner can see. Unarchive inside that window finds the clone on disk and restores it as it stands; EmptyTrashJob reaps it, its Docker resources and its artifacts at the deadline. The deadline is anchored to archived_at here rather than inherited from whatever is already on the row: a retry that reaches this branch after an earlier run cleared trash_after would otherwise leave the clone to StaleCloneCleanupJob, which reaps it — unpreserved, and without tearing Docker down — an hour after archive. The cost of the hold is a limitation.

Preservation does not take the working tree entirely on faith. A tree that is 50 or more deleted tracked files and almost nothing else is not uncommitted work — it is a clone that an interrupted recursive delete mangled — so CloneArtifactService drops those deletions from working_tree.patch (keeping the additions and modifications), logs at .warn, and records dropped_deletions in the artifact metadata.

The deletions are filtered out of the diff already in memory, not by re-running git with --diff-filter=d. The guard fires precisely when a concurrent recursive delete is gutting that clone, so a second chdir into it raced the delete: the directory was gone 11 ms after the warning and the whole preservation died on ENOENT (#425). Nothing re-enters a clone the guard has just proven is being deleted.

.warn rather than .error because this refusal is self-healing — the corruption is dropped, the real work still travels in the bundle and the filtered patch, and the deleted files come back from HEAD — and routing it through StructuredLogger#error paged on every clone the guard successfully defused. DeferredCloneCleanupJob stamps the drop on the session as mangled_clone_dropped_deletions, and MangledCloneReportJob sums a day of them into one line, so the frequency stays countable without paging.

Unarchive applies the same test before it replays a patch, because patches captured before the guard existed are still on disk. There it refuses the patch whole rather than filtering it: git can re-take a filtered diff from a live clone, but nothing can cheaply split a patch file, so a legacy patch’s additions are declined along with its deletions. And after restoring artifacts, UnarchiveSessionService checks the clone it produced: if the agent root’s subdirectory is gone, or the tree is now overwhelmingly deletions, it puts the clone back to the commit it was checked out at before the restore (which unwinds a fast-forwarded bundle too, so damage carried by commits is reverted rather than reset onto) and deletes what the restore added. A clone missing its root subdirectory fails air prepare with ENOENT and takes the session down with it, so a pristine clone is the better of the two losses.

A restore never shortens the conversation

Unarchive re-materializes session.transcript at the runtime’s resume path, because normally the stored copy is the durable record and the file is gone or stale. Sometimes that direction is backwards. The runtime’s transcript lives under ~/.claude/projects, keyed by the sanitized working directory and the runtime session id rather than inside the clone, so it outlives the clone being deleted — and it can hold turns the stored copy never caught.

A session that archives itself writes its closing turn to disk and is killed moments later, so anything the last poll missed exists only in that file. Writing the stored copy over it destroyed the only remaining record of the agent’s answer, and left the resumed agent with no memory of what it had just said — session 13908 woke up and had to be asked to repeat itself.

So the restore compares first. When the file on disk holds more lines than the stored copy, it is kept as it is and session.transcript is healed from it instead. The guard only ever grows the record: a file that is shorter or equal is the ordinary stale case and is still overwritten, and a failure to read it falls back to writing the stored copy rather than failing the unarchive. Once the file is judged longer, though, that verdict stands even if copying it into the record then fails — falling back to the write at that point would destroy the turns the guard exists to save. It is the same rule TranscriptPollerService applies in the other direction, where a shorter file on disk means the clone was recreated and must not be allowed to erase history.

The read goes through the runtime’s transcript source rather than a bare file read, because that is where decompression and secret redaction live and session.transcript is served to the UI, the REST and MCP APIs, and the transcript archive. The adopted turns are not re-broadcast into a timeline that has just rendered them, and nothing special is needed to get that: every unarchive clears broadcast_message_count as one of the stale retry keys, and the poller rebuilds it from the stored transcript — the healed one — on its next pass.

The clone unarchive rebuilds is not required to have the path the session row remembers. A session freezes its agent root’s subdirectory at creation time, so renaming that root’s directory in the catalog leaves every pre-existing session naming a path main no longer has — and the refusal is permanent, which stranded both halves of the population: archived sessions that could not be unarchived and live ones that could not resume once the reaper took their clone (#921). UnarchiveSessionService and AgentSessionJob now hand GitCloneService the root’s current path as well, resolved from the catalog by metadata["agent_root_key"], and the clone lands on it when the stored path is absent and the current one is there. The row is corrected in the same breath, before the damage check above reads session.subdirectory — see the router root’s two names. A root the catalog no longer carries — and a root that moved its tree to the repo root, which is a limitation — still fails the unarchive, and it fails as a warning rather than an exception-tracker event: nobody on call can act on a missing directory that the person clicking Unarchive is already reading in the flash message.

In both cases the artifacts stay on disk and the session log records where — unarchive clears trash_after, so nothing else will ever reap them, and the path is the only way back to whatever real work the patch held. See issue #411 and the limitation this does not close.

The session’s scratch directory and prompt attachments are on the trash schedule too, not the undo-window one: nothing can rebuild them from a remote the way a clone is rebuilt, so they are kept for the whole time archive is undoable and reaped by EmptyTrashJob. That is why a clean-clone session can still hold a trash_after — it is the deadline for whatever is still retained, which is not always the clone. See how long scratch lasts.

Restoring a session that never ran

Everything above assumes the session has a conversation to come back to. Some do not. A session created by a trigger, held at the starting line by the spot gate for a whole quota window, and then paused or archived has no runtime session_id and no transcript — its agent process never launched. Session#never_ran? is that pair, and both halves have to be blank.

Restore and restart used to refuse those sessions outright with “Session has no session_id” (#557). That precondition is about resuming a Claude Code conversation, and applying it to a session that never had one made Trash irreversible for exactly the class of session where starting over is cheapest and most obviously correct — its prompt, agent root, skills, plugins and lineage are all intact and none of its work has been done.

A never-run session now takes a separate, much shorter path on both doors:

Action	What it does for a session that never ran
Restore (web Restore, POST /api/v1/sessions/:id/unarchive, MCP unarchive)	No clone, no transcript write, no air prepare. UnarchiveSessionService drops the half-written setup artifacts of the aborted spawn and returns the row to needs_input.
Restart (web Restart, POST /api/v1/sessions/:id/restart, MCP restart)	Takes the existing restart-from-scratch path — clear the setup artifacts and the spot-hold keys, null the session_id, and enqueue the full setup pipeline with the session’s original prompt and its first-turn attachments.

Restore lands the session in needs_input, which is a state the web Restart button is rendered for — so the fresh start is one click from the page a person is already looking at, rather than something only an agent or a script could ask for (#830).

Restore does not clone, on purpose: the fresh start clones for itself, and a clone built here would be one the fresh start never uses and a reaper has to sweep. Whatever starts the session next lands in AgentSessionJob, which already reclassifies a follow-up prompt to a session with no session_id as a fresh start.

The dangerous mistake is the inverse one, so the branch is deliberately narrow. A session holding a transcript with no session_id — what ProcessLifecycleManager#release_stale_runtime_session_id! leaves behind on a runtime that mints its own conversation id — has hours of work in it. Restarting that fresh would silently discard the conversation, so it is not never-run: it stays on the resume path, and the failure to restore it stays loud on every surface. Every other restore failure — a clone that would not rebuild, a DB error, a row that cannot leave its state — is likewise still a failure, not a quiet reroute into a fresh start.

Trigger#resuscitatable_session? is the same predicate, used for a different decision: a recurring trigger will not reuse a never-run session even though the restore would now succeed, because a follow-up into a session with no session_id runs that session’s own prompt with this fire’s prompt merged into it — a fire folded into somebody else’s prompt is not a fire. The trigger spawns instead. See Triggers.

The undo window is one number

Archiving from the web UI leaves a toast with an Undo button on it. That button is honored for SessionStateMachine::UNDO_ARCHIVE_WINDOW — 30 seconds — after the archive, and it is the only place that number is written down. Three things read it and none of them restates it:

Reader	What it does with it
ApplicationHelper#flash_duration_ms	How long the toast stays on screen, so the button is gone the moment it stops working
SessionsController#undo_archive	Refuses a click that arrives after the window, with “The undo window has expired. Use the restore feature instead.”
DeferredCloneCleanupJob::CLEANUP_DELAY	Waits the window out, plus five seconds, before reaping the clone — undo puts the session back to work on that clone without rebuilding it

That is not tidiness. The two numbers were literals in two files and drifted: the toast lived 30 seconds while the controller accepted 5, so Undo was already refusing for 25 of the 30 seconds it was on screen (#1036). A test that pinned the controller’s window to a number would not have caught it, so the regression test reads the duration off the rendered toast and asks the controller to honor a click at the last instant that toast is still up.

Undo is deliberately generous rather than exact, because it is not a destructive path: it restores a session out of the trash, which is what the ordinary Restore action does with no time limit at all. The window scopes the toast affordance; TRASH_RETENTION_PERIOD and EmptyTrashJob are what actually bound the trash.

Side effects are swallowed by design — but no longer silently

Almost every callback in the state machine is wrapped in a bare rescue (preserve_debug_info is the exception — it only logs). That part is deliberate and unchanged: a broken notification service should not be able to wedge a session in running, so a failed side effect never aborts the transition. The consequence is still real — cleanup can not happen while the state advances anyway.

What changed is that swallowing no longer means silence. All 22 callbacks route their rescue through one helper:

rescue => e
  report_swallowed_side_effect(__method__, e, alert: true)
end

alert: true logs and reports the exception to GlitchTip through ErrorReporter (the same seam the trigger pollers and SystemHealthMonitorJob use). alert: false logs only — which is still an ERROR record, and still pages; the difference is the structured event and its backtrace. The split is by consequence, not by severity of the exception:

	Callbacks	Why
Alerts	set_archived_at, cleanup_running_job, clear_stale_mcp_failure_metadata, clear_auth_recovery_budget, execute_pending_sleep, cancel_pending_one_time_wake_triggers, retire_held_wake_triggers, clear_pending_sleep, set_blocked_on_elicitation_marker, cleanup_watched_session_ao_event_triggers, fire_ao_event_triggers, clear_trash_expiry	The failure leaves persistent state inconsistent and nothing reconciles it — a resumed session that re-fails on a stale MCP flag, an armed wake-up that fires into live work, a restored session still queued for deletion.
Logs only	reset_elapsed_time_counter, log_state_change, clear_blocked_on_elicitation_marker, clear_paused_by_metadata, mark_notifications_stale, dismiss_notifications, enqueue_failure_push_notification, enqueue_debounced_needs_input_push_notification, enqueue_session_inference_if_needed, set_trash_expiry, enqueue_deferred_cleanup	Cosmetic, best-effort by construction, or already covered by a reconciling sweep — CleanupExpiredElicitationsJob for a stranded elicitation marker, StaleCloneCleanupJob for a nil trash_after, EmptyTrashJob for a missed cleanup enqueue. A second alert path to an event that already self-heals is just noise.

The report identifies the callback, and the session rides in its context. A sick database hits the same callback for every session in flight, and GlitchTip groups by exception and stack, so the wave collapses into one issue rather than one per session — while each event still says which session it was.

Note this covers reporting only, with one exception: a failure that aborted the transaction is not swallowed. Swallowing exists so a transition can finish with one side effect missing, and once Postgres has aborted the transaction the transition cannot finish at all — every later statement in it raises PG::InFailedSqlTransaction and lands here in turn, and the transition’s own requires_new: transaction cannot commit either (an outermost one turns its COMMIT into a rollback; a savepoint fails on RELEASE). So the state change is already lost. Swallowing there only buries the cause under its consequences and pages once per callback. DatabaseTransactionState.aborted_by? asks libpq directly — PQTRANS_INERROR on the pool the error came from — and report_swallowed_side_effect logs the abort and re-raises, so the first failure ends the transition and the caller sees the statement that actually failed.

That is #924. On 2026-09-04 a worker connection held a cached plan across an app_settings migration; the SELECT behind record_experimental_setting_flags failed, a silent rescue in AppSetting.current returned a default, and the three callbacks after it each reported an InFailedSqlTransaction of their own. #alerts got four ERROR records naming only consequences and no record at all of the statement that failed.

The blast radius is wider than that one path, deliberately. Any callback whose failure aborts the transaction now ends the transition — a check constraint on logs.create! in log_state_change fails pause! outright, where before it cost one timeline line. That is the honest outcome rather than a new one: the transition was never going to persist. It is narrow in the way that matters, though — a failure a requires_new: savepoint absorbs (Session#assign_slug, GateDecisions::Record) leaves the connection healthy, aborted_by? says so, and the swallow contract applies unchanged. Everything else about that contract is untouched: an ordinary failed side effect is still swallowed, and the transition still completes.

Note

A failed one-time scheduled wake is still destroyed silently — that is a separate path from these callbacks and is tracked in Limitations.

Who else moves sessions around

The state machine is not the only actor:

• HeartbeatSweepJob (every 30s) re-nudges needs_input sessions with heartbeat_enabled by injecting a heartbeat prompt and resuming them. It skips sessions blocked on an elicitation or with pending enqueued messages — resuming those would spawn a second process.
• CleanupOrphanedSessionsJob (every 5 min) catches sessions marked running whose process is gone — and, since #988, stops catching the same one forever. See When recovery restarts a session into silence.
• StalledStartSweepJob (every 5 min) catches the opposite end: a session that never started. A first turn rides on exactly one AgentSessionJob, and — unlike a spot hold, a ceiling pause, an auth park or a recovery pause — a session that has never run carries no marker saying so, so a lost start job leaves a plain waiting row that looks exactly like a session created a second ago. StalledSessionStart re-enqueues it, and fails it after three attempts rather than re-queuing forever. See Background jobs.
• StrandedSleepSweepJob (every 5 min) covers the other half of the same hole: a session that did run, went to sleep on a wake, and lost the wake. waiting is one word for two states — a session resting on a wake it will get, and a session resting on a wake it will not — and nothing distinguished them, because a legitimate sleep carries no marker either. The predicate is no fireable wake, not no wake: an ao_event watcher whose watched session is already archived is an enabled row that will never fire, so a sweep keyed on the absence of trigger rows would walk straight past it — and, in the other direction, a wake that has come due within SCHEDULE_FIRE_SETTLE is one the scheduler has not reached yet rather than one that was lost. StrandedSleepRescue resumes the session with a SYSTEM_RECOVERY nudge through the same claim_system_recovery_turn! door orphan cleanup uses, and stops after three rescues rather than resuming forever. Production session 6412 sat in waiting for 38.7 hours before this existed (#855). See Background jobs.
• ZombieReaperJob (every 5 min) reaps dead child processes that nothing is waiting on. It deliberately leaves alone any pid a live waiter has claimed — see Background jobs. Reaping a pid the monitoring loop was waiting on costs the loop the exit status it would have routed through handle_exit; the loop’s signal-0 fallback answers that case with handle_unreaped_exit, which runs every recovery that reads evidence rather than a status — see Spawning.
• SessionRecoveryService handles hung and interrupted sessions on a best-effort basis; SigtermRetryService covers gracefully SIGTERM’d sessions (deploys) with a bounded retry ladder (RetryBudget::SIGTERM, 3 attempts); an abnormal signal death (SIGKILL from an OOM kill, SIGSEGV, …) is instead resumed by ProcessLifecycleManager#handle_signal_death (RetryBudget::SIGNAL_DEATH, 3 attempts) — see Retry budgets.

How a process actually gets terminated

Pausing, archiving, following up and recovery all end in the same place: ProcessTerminationService#terminate, given the session’s process_pid. It walks a ladder and stops at the first rung that works.

1. SIGTERM to the process group (-pid). Agent children are spawned with pgroup: true, so this reaches the leader and every grandchild it started — MCP servers, node, gh.
2. SIGTERM to the leader alone, if the group could not be signalled (no such group, or not ours).
3. SIGKILL, group first and then the leader, if it is still alive after its SIGTERM grace.
4. A group SIGKILL sweep, once the leader is confirmed dead.

Two details are load-bearing.

Liveness is answered by reaping, not by signal 0. Process.kill(0, pid) succeeds for a child of ours that has already exited — an unreaped child holds its pid as a zombie until someone waits on it. So the service asks waitpid(pid, WNOHANG) instead: a status back means the child had exited and is now collected, nil means it is genuinely still running, and ECHILD means it is not our child, where signal 0 is the right answer and is used as the fallback. Once a pid is reaped it is never probed again — it belongs to the OS and can be recycled.

Before this, every liveness check answered “still running” for a child that died on the first SIGTERM: termination burned ~15–25s of sleep in a GoodJob thread, sent two redundant SIGTERMs and two SIGKILLs, and then reported :error for a kill that had worked.

The group SIGKILL sweep is deliberate. Grandchildren are reparented to init when the leader dies; nothing else cleans them up. They received the group SIGTERM in step 1, so the sweep gives the group one more second to drain, then SIGKILLs it — and skips silently when the group is already empty, which is the common case. A group we may not signal, or a sweep that fails, never downgrades the result: the leader is dead either way, and that is what the caller acted on.

What the dashboard shows you

The dashboard opens on the sessions that are waiting for you and nothing else: needs_input only, out of the box. That is the point of the queue. Everything else — the run you started five minutes ago, the failures you already read, the trash — is one tick away in the Filters section in the sidebar.

Filters is a single form covering four things: the search box, the status multi-select, the scheduling class narrowing, and an Advanced disclosure holding the agent-root, genesis and transcript-contents controls. One form, one Apply, and the whole filter state travels together.

Status is a multi-select over the five states, and selecting none means every status. There is no separate trash toggle: archived is simply one of the five, so trash visibility is answered by the same control as everything else rather than by a second toggle arguing with it.

You want	Tick
The queue (the default)	Needs input
Live work	Waiting + Running
The trash	Archived
Everything	nothing

Two behaviours are worth knowing:

• A selection persists. Applying the form writes the statuses and the scheduling class to a sessions_filters cookie, so the choice survives a reload and a bare visit to / — the same shape of preference as the view-mode cookie. Reset filters deletes it and returns you to needs_input only.
• The ticked boxes describe the result set in every view, search included. Searching does not quietly widen the status filter. That means a search from the default view returns needs_input matches only — to search the trash, tick Archived; to search everything, tick nothing. The status summary sits directly above the search box so the narrowing is visible rather than surprising.

The scheduling class is a filter rather than a search: it narrows whichever view you are in and leaves the category grid in place. A free-text query, an agent root, or a genesis is a search, and replaces the grid with a flat result list.

The equivalent for an agent is quick_search_sessions, whose status argument takes one status or an array of them.

Card order is yours to set, and it stays set

Inside a section, cards are ordered by where you dragged them — sessions.sort_order ascending. A card nobody has placed sits where it arrived, which is newest-first: every session Zimmer creates is stamped as it is created. Grab a card by the grip bar at the top of it and drop it where you want it, in its own section or in another one. The drop POSTs the section’s order to POST /sessions/reorder, naming the card you moved, so it survives a reload, a trip into a session and back, and paging the section.

How that is stored (SessionCardOrder):

• A drag moves one card, next to its neighbour. The server puts the moved card immediately above the card that is now below it (or immediately below the card above it, if you dropped it last) and moves nothing else. It never reads a card’s index in the posted list as its position: sections paginate at 50 independently, and the browser’s copy of a page can hold cards the server would render elsewhere — a new session a broadcast prepended onto page 2, say. Anchoring on the neighbour means a drag on page 2 cannot renumber page 1, and a card you did not touch stays put — including one the status filter is hiding, and a favorited card rendered up in Starred.
• An arrival goes on top. A new session, a card the auto-categorizer or set_category moves, and the cards of a deleted category all take one below their new section’s lowest sort_order, so they appear at the top the way a new session always has, and nobody else’s row is rewritten to make room. Rows nothing has ever placed tie at the column default 0 and fall back to created_at DESC. Note what “on top” means once you have arranged a section by hand: a new session outranks the card you dragged to the top, because the alternative — inserting it by created_at into an order you chose — has no defined answer.
• A drag rewrites what moved, not the section. The section’s existing values are handed back out along the new order, so only the cards whose rank changed are written — dragging a card from 40th to 1st writes 41 rows, not the thousands of archived sessions sharing its section. The exception is a section whose values tie — one nobody has reordered, or one where two sessions arrived in the same instant and read the same minimum — which is renumbered once on its next drag. Drags that touch the same section, including both ends of a cross-section drag, are serialized with an advisory lock; creating a session takes no lock, which is why that tie can happen at all.
• The order is global, not yours. Zimmer is a single circle of trust with no User model, so there is no principal to hang a per-viewer order on — the position lives on the session row, the same way position lives on the category. Everyone sees the board you arranged.
• Starred cards are not drag-orderable, and lose nothing by it. A favorited card floats out of its section into the pinned group, which sits outside the drag-and-drop controller and carries no grip bar — its category placement is invisible while it is starred. Starring does not touch sort_order, so unstarring puts the card back where it was.

A cross-section drag is one write, not two: the moved card’s category change and its placement land in the same transaction. Right-clicking a card’s grip bar opens the same move as a menu, which puts the card at the top of the page of that section you have open.

The equivalents for an agent are manage_categories’ reorder_sessions action over MCP and POST /api/v1/sessions/reorder over REST.

Manual refresh

The dashboard’s refresh controls are the human counterpart to those background actors. There are four of them, and they all end up in SessionsController:

Control	Action	Scope
The per-card icon next to a session’s status badge	#refresh	that one session
”Refresh all” in the header	#refresh_all	every non-archived session outside a frozen category
The icon in a category section header	#refresh_category	that category’s non-archived sessions
The icon in the Starred group header	#refresh_starred	every non-archived favorited session

#refresh_starred deliberately does not skip frozen categories the way #refresh_all does. The Starred group renders every favorited session regardless of its category, so the button acts on exactly the cards sitting under it — starring is a per-session opt-in that outranks the category’s parked flag.

The per-card icon is hidden for a running session. A running session is already streaming into its card; there is nothing a refresh would tell you that the card does not.

How the dashboard answers

Every mutating dashboard control — the four refresh buttons plus Trash, Undo, Restore, bulk trash, Pause and the category moves — responds to a Turbo request with a Turbo Stream, not a redirect. The stream replaces one element: #flash, the layout’s single toast container (app/views/shared/_flash.html.erb). The cards themselves are not in the response, because they re-render on their own over the sessions_index_individual and session_<id>_status broadcast channels. The redirect existed only to carry the message.

Two actions stream a card as well, because a broadcast cannot reach one:

• Trash removes the card it just archived, and streams the “Session moved to trash.” toast carrying the Undo button. The toast needs the #flash target to land in — before that id existed, the card vanished with no way to undo it.
• Undo prepends the card back into the grid it belongs to (#sessions_grid for Uncategorized, #category_grid_<id> for a category). Its own restore broadcast is a replace, and there is nothing left in the DOM to replace.

#archive, #unarchive and #pause stream one more thing: the session page’s own status badge and header actions. Session#broadcast_status_change already pushes both over the cable, but a broadcast is fire-and-forget and this whole change exists because a cable can be silently dead. The direct reply to the user’s click cannot be lost, so a status-changing action carries its chrome rather than trusting the socket. Both targets are absent when the click came from a dashboard card, where those streams are a no-op.

That chrome is what makes trashing from the session page itself work: it leaves you on the page, with the toast and its Undo, and the button you just clicked turns into Restore. Only the dashboard card’s Trash takes something out from under you, and all it takes is the card.

A stream only happens if the client asks for one, and that is a client-side property, not a controller one. The card’s Trash link carries data-turbo-method, so Turbo builds the request and sends Accept: text/vnd.turbo-stream.html. The detail header’s Trash button — the one the session drawer puts in front of you — builds its own form in archive_countdown_controller instead, and has to submit it with requestSubmit(). A native form.submit() fires no submit event, so Turbo never sees the submission: the browser POSTs for real, #archive answers on its format.html branch, and the redirect reloads the dashboard from the top, discarding the scroll offset the drawer was opened over.

Every one of these actions keeps its format.html branch, which still redirects with a real flash. That is what a non-Turbo client — and most of the controller test suite — gets.

The session detail page is on the same footing: it carries a cable-reconnect Stimulus controller that watches each <turbo-cable-stream-source> for the connected attribute turbo-rails sets and clears, and re-subscribes any source still dark after a grace window (backing off up to 30s). It replaced a <meta http-equiv="refresh"> that fired on a five-second timing window whether or not the cable had actually dropped — and, being a top-level navigation, dismissed the session drawer along with the user’s place in it. Session#recently_recovered? now only decides whether to show the recovery banner.

It does not replace stream-visibility-recovery — the two answer different failures. A socket that dies while the page stays visible has missed nothing, so cable-reconnect re-subscribes that source in place. A page that was hidden (backgrounded PWA, bfcache restore) and came back with a dead socket has also missed content, and re-subscribing would not bring it back, so stream-visibility-recovery handles that case in two steps: reopen the ActionCable consumer, which re-subscribes every subscription on the connection and restores live updates without rendering anything, then backfill the gap.

The drawer loads its own URL

The dashboard’s right-hand session drawer is a real Turbo Frame navigation, not an innerHTML injection — that is what makes the detail view’s turbo_stream_from subscriptions and Stimulus controllers connect, so a transcript streams live inside the drawer and the follow-up and archive controls work.

What it navigates to is a separate path: /sessions/:id/drawer renders the same detail body wrapped in <turbo-frame id="session_detail"> with no application layout, and /sessions/:id renders the full page with no frame. Which body you get is decided by the URL and nothing else.

That separation is load-bearing, and it replaced an arrangement where one URL served both bodies and a Turbo-Frame request header chose between them. Every cache between the view and the reader keys on the URL, so a shared URL meant any of them could hand the drawer’s frame request the frameless body — and Turbo would render its Content missing placeholder in place of the session. The one that actually did it is not an HTTP cache and no response header can reach it: Turbo 8’s link prefetching keeps a hover-prefetched request keyed only by URL, and splices it into any later GET fetch for that URL — a frame’s own src load included — discarding the Turbo-Frame header that decided the body. It happens in browser memory and never touches the network, which is why Vary: Turbo-Frame could not help and why disabling prefetch on one link never did either: the key is the URL, so any other link to the same session seeded the entry just as well.

Four rules follow, and all of them are pinned by tests:

• Nothing links to a drawer path. Turbo’s prefetch cache is only ever seeded by hovering an <a>, so as long as no anchor points at /sessions/:id/drawer, no entry for it can exist. A drawer trigger’s href stays the full session page — so middle-click, ⌘-click and the no-JS path all do the obvious thing — and it hands the drawer the frame’s URL in data-session-drawer-url instead. SessionsHelper#session_drawer_link_data builds that pair.
• Nothing inside the drawer navigates the frame. A plain same-origin link in there navigates the frame, and every page it could land on — /triggers, /costs, the dashboard, the session’s own full page — has no session_detail frame, so the drawer shows Content missing again by a different door. Every link the drawer renders carries data-turbo-frame="_top", including the ones that answer with a Turbo Stream and so never navigate anything: a rule with case-by-case exceptions is one nobody can check. The session-hierarchy links to other sessions carry the drawer url as well, so clicking one swaps the drawer in place; on the full page, where no drawer exists, they fall back to an ordinary navigation.
• The log-level filter re-fetches the frame’s own address. Changing the log level is a server round trip — the server filters timeline items before paginating them — so the detail body has to be re-rendered with a filter param. log_level_filter_controller picks which address to re-fetch off the enclosing frame’s src: a frame carrying one was lazy-loaded from that address into some other document, which is the drawer, so the frame’s src is the session’s address; no frame, or a frame without a src, was rendered as part of this document, so the document’s own address is the body’s — that covers the full session page and /sessions/:id/drawer opened directly. Using window.location unconditionally is what broke the drawer, where it is the dashboard: opening the drawer with a non-default level saved in localStorage navigated the dashboard to /?filter=<level>, a param that means nothing there, and dismissed the drawer along with the reader’s place (#666). What the frame navigates to is /sessions/:id/drawer?filter=<level> — a drawer url, so the rule above still holds, and no anchor points at it so the prefetch cache still cannot be seeded for it. A frame navigation has no browser loading UI behind it, so the drawer dims the frame for the round trip off Turbo’s own busy attribute rather than leaving the previous filter’s content looking live.
• The re-fetch carries the open Transcript with it. The round trip re-renders the whole detail body, and the Transcript disclosure is collapsed on every ordinary load — thousands of rows is not what a returning reader wants first. But changing the log level is a request about the transcript, made by someone who has it open and is reading it, so re-rendering it shut takes the only thing they were looking at off the screen: the drawer stays open and the body underneath is filtered correctly, and yet the change reads as having done nothing. So the filter’s address carries transcript=open when, and only when, the disclosure is open at the moment the level changes; SessionsController#load_session_detail turns that into @transcript_open and _detail renders <details open>. Only the filter round trip sets the param — no link, form or redirect does, and #session_redirect_target drops it along with filter — so a reader who never opened the transcript is never handed one open. The panel comes back open with layout, which is what session-scroll’s transcriptCollapsed guard needs in order to run at all, so the reader lands on the newest items rather than at the top of thousands of rows. On the full page, where the round trip is a document navigation, the param stays in the address bar exactly as filter does, and a reload restores the same view.
• A redirect the frame follows lands on the drawer url. A frame follows a 302 with its own Turbo-Frame header still attached, so #follow_up’s “queued instead” branch and #refresh — the two that redirect rather than answering with a Turbo Stream — pick their target through SessionsController#session_redirect_target. That reads the frame header to choose a destination, which is not the content negotiation that caused the bug: each URL still has exactly one body.

test/contracts/session_drawer_frame_url_test.rb pins the arrangement, and the controller tests assert the invariants directly: each URL returns the same body whatever the Turbo-Frame header says, no link on the dashboard points at a drawer path, and — asserted over the whole rendered drawer body, so a link added later is caught without anyone remembering the rule — every same-origin link inside the drawer escapes to _top. test/system/session_drawer_log_filter_test.rb drives the log-level filter in both contexts and asserts the dashboard’s URL is untouched when the drawer re-filters, and that an open transcript survives the round trip while a closed one stays closed; the controller tests pin the transcript param itself, including that only the exact value open opens the disclosure. Opening /sessions/:id/drawer by hand is not a supported way to read a session — it answers with a bare frame and no chrome — it is the drawer’s own address.

The reopen backfill

iOS suspends a backgrounded standalone PWA. The process stops and the WebSocket dies with it, so on a real reopen the socket is always dead and the dead-socket branch runs every single time. That branch used to be a full replacing Turbo.visit, which is exactly why the app appeared to reload whenever the user switched back to it. Checking consumer.connection.isOpen() first did not fix that — it only made the case that never happens (socket survived the hide) free.

The branch now fetches the page the server would render and reconciles it into the live document, region by region, without navigating. app/javascript/lib/live_region_backfill.js does the reconcile; the regions declare themselves in the markup with data-live-region, and the values mirror what BroadcastService does to them:

Value	Broadcasts do	Backfill does	Examples
append	append children	append children the page lacks, by id	the timeline
replace	replace the element	replace it when the server’s copy differs	status badge, header actions, metadata, provenance, enqueued messages, the composer
sync	add, replace and remove children	reconcile children by id, in the server’s order	the dashboard’s session grids, elicitation banners

The strategy has to match what the broadcasts actually do, and getting it wrong is silent in one direction: elicitation banners are appended when raised and removed when answered, so marking them append would leave a dead approval prompt on screen after a reopen.

A deferred panel is fetched from its own URL. The detail screen’s transcript and provenance panels are <turbo-frame loading="lazy"> stubs, so the page the backfill just fetched has them unloaded — a skeleton with no ids in it, which backfillLiveRegions correctly finds nothing to reconcile against. Their content is at /sessions/:id/transcript_panel and /sessions/:id/provenance_panel, and the recovery fetches those too, for the frames the reader had actually opened. A frame still unloaded has nothing on screen to bring up to date, so it costs nothing.

They are reconciled, not reloaded, because reloading the frame would throw away the older pages infinite scroll had pulled in and the reader’s place among them. The transcript’s batch carries data-live-append-into="session_<id>_timeline" rather than that id itself — the page already renders the broadcast target under that name, just after the frame — and the recovery renames it before reconciling. That is what puts a recovered row in the same container, and therefore the same order, as one that arrived over the socket. It then sweeps data-live-transient placeholders inside the frame that the panel response no longer renders: the region reconcile only sweeps transients that are children of the region, and the transcript’s empty state sits beside the append target rather than inside it, so it would otherwise stay above the rows the backfill just recovered. A panel that will not come back is skipped rather than escalated to a full reload: by that point the socket is open again, so the next broadcast reaches it anyway.

Three rules keep the backfill from taking something away from the reader:

• A region in use is never swapped. Anything containing the focused element, or a field holding a value the server did not render — a half-typed follow-up, a staged attachment — is skipped.
• An append region never removes. Older pages pulled in by infinite scroll are not in the server’s tail render, and are left where they are. The one exception is a child marked data-live-transient (the empty-state placeholder), which a broadcast would have removed too.
• A sync region showing a different page is skipped. The dashboard’s category sections page inside their own <turbo-frame> without changing window.location, so re-fetching that URL returns page 1 — and syncing it would throw away the page the reader had paged to. Each grid records its page in data-live-page, and a mismatch means hands off.

Appending by id needs rows that have ids, and timeline rows are not records — a row is a Log, an MCP log, or one of the several OpenTranscripts events a transcript line fans out into. SessionsHelper#timeline_item_dom_id derives one from what the row is, and the derivation is constrained by having to agree across both render paths. In particular it excludes transcript_index: BroadcastService normalizes without one, so including it would give every live-streamed row a different id from its own re-render — and the backfill would then append a second copy of everything that had arrived over the socket. test/helpers/sessions_helper_test.rb asserts the two paths agree, and test/system/pwa_reopen_recovery_test.rb asserts a live-arrived row is not duplicated by a reopen.

Three exits are worth knowing about. A socket that reports itself open is left alone entirely and the reopen costs nothing. A socket still mid-handshake is left to finish rather than torn down and restarted. And a backfill that cannot run to completion — the fetch failed or timed out (10s), the URL now redirects because you were signed out, or the reconcile itself threw part-way — falls back to a replacing visit, because a page left half-recovered and quiet is worse than one that lost its place.

isOpen() is a readyState read, which is its limit — see Limitations.

A Turbo 8 morph was tried for this and rejected. Morphing reconciles the whole live DOM against the server’s HTML, and Stimulus controllers here keep state in value attributes the server does not render — log_level_filter_controller writes its own level-value in connect(). Idiomorph removes any attribute missing from the server’s markup, so that state reverts to its default and the controller then acts on it: in testing, a morph reverted the log filter to minimal and hid every item in the timeline. A morph also closes a <details> the reader opened, for the same reason. The region-scoped backfill avoids both by touching only what broadcasts touch.

What a refresh does depends on the session’s state:

• failed → restart it (#resume_failed_session, or restart_with_continue_prompt in bulk).
• needs_input → in bulk, continue it, unless the user paused it by hand (paused_by: "user").
• waiting → send the automated continue nudge (AutomatedPrompts::SYSTEM_RECOVERY), the same prompt every recovery path uses. See below.
• running, and any waiting session excluded below → re-read the transcript from disk.

Refreshing a waiting session nudges it

A waiting session has no live process, so re-reading its transcript tells the user nothing. What they want from “refresh” is for the session to get moving again, so Session#continue_nudge_on_refresh? routes it through restart_with_continue_prompt instead.

Two kinds of waiting session are excluded, because for them a nudge would be wrong rather than merely useless:

• Never started (session_id blank). There is no conversation to continue — the session is still queued and the spawn pipeline owns it.
• Deliberately asleep (Session#awaiting_scheduled_wake?). A one-time wake-up targeting this session that is still ahead of it — from wake_me_up_later or wake_me_up_when_session_changes_state — means the agent chose to sleep until a specific moment. Nudging it would fire the work early, and worse: restart_with_continue_prompt resumes the session, and resume’s cancel_pending_one_time_wake_triggers callback consumes the pending condition, so a premature refresh would delete the wake-up it was waiting on. If the trigger table can’t be read, the session is treated as asleep, so a database error never wakes a sleeper.

Both excluded kinds fall through to the plain transcript refresh. The rule is identical for the single-session icon and for all three bulk controls, so “refresh all starred” over a mix of statuses does the right per-session thing.

An overdue wake-up is a stall, not sleep

“Still ahead of it” is doing real work in that second bullet, and it is deliberately narrower than what cancel_pending_one_time_wake_triggers consumes on resume.

A one-time schedule whose moment has already passed without firing — a stopped scheduler, a GoodJob backlog, a crashed trigger job — describes a session that is stuck, which is exactly the session refresh exists to rescue. Treating it as “asleep” would make refresh a no-op on the one case that most needs it. So sleeping means the scheduler has yet to reach it, read through the same TriggerCondition#schedule_due? the firing path uses. A session-scoped ao_event wake has no time component — it fires whenever the watched session transitions — so an unfired one is always still ahead.

The cancel-on-resume path keeps the broader reading, and should: once a session is resumed by hand, an overdue wake-up firing later would land on an already-active session.

Bulk refresh asks this about a whole set of candidates at once via Session.ids_awaiting_scheduled_wake, which answers in one query rather than one per waiting session.