operations/production

An identity is a stable owner_id (one per replica, stable across restarts) plus an incarnation_id the process bumps each boot. Reentry requires the same owner id and the same incarnation; a new incarnation must claim or reclaim through the fenced lease path rather than clearing the old owner. What differs between deployments is the liveness metadata that identity carries.

// A stable owner id per replica plus a per-boot incarnation. `local_process`
// attaches this host's kernel boot id and pid, so a same-host peer can prove
// a crashed holder dead and reclaim its lease before the TTL. On a non-Linux
// host, or across a machine reboot (the boot id changes), it degrades to
// opaque, TTL-only reclaim.
let owner = LeaseOwnerIdentity::local_process(
    std::env::var("WORKER_ID").unwrap_or_else(|_| "worker-1".to_string()),
    std::env::var("AGENT_SERVICE_INCARNATION").unwrap_or_else(|_| boot_incarnation()),
    std::env::var("HOSTNAME").unwrap_or_else(|_| "host-1".to_string()),
);

// Cross-host / opaque holders (the common distributed case) get TTL-only
// reclaim; build them with `LeaseOwnerIdentity::opaque(owner_id, incarnation)`.
let session = core
    .session(chat_id)
    .session_execution_owner(owner)
    .open()
    .await?;

The default is a 30 second TTL with a 10 second renew interval. The constructor enforces ttl >= 3 * renew_interval, so a healthy owner can miss two consecutive renewals — a scheduler stall, a transient store error — before a peer may treat the lease as expired. A shorter TTL reclaims a crashed owner's work sooner but widens the window in which a slow-but-alive owner is falsely taken over; a longer TTL does the reverse. Choosing that number is the trade, and it is now a host decision instead of a constant baked into lash.

// One timing decision governs every durable single-writer lane lash claims:
// session-execution leases, effect replay, queued-work and turn-input
// claims, and process leases. `new` enforces `ttl >= 3 * renew_interval`, so
// a live owner can miss two renewals before a peer may treat the lease as
// expired. A shorter TTL reclaims a crashed owner's work sooner; a longer
// one tolerates slower renewal and lowers false-takeover risk.
let lease_timings = LeaseTimings::new(
    Duration::from_secs(15), // ttl
    Duration::from_secs(5),  // renew_interval
)
.expect("ttl >= 3 * renew_interval");

let core = LashCore::rlm_builder(factory)
    .provider(provider)
    .model(
        lash::ModelSpec::from_token_limits("anthropic/claude-sonnet-4.6", None, 200_000, None)
            .expect("valid model metadata"),
    )
    .store_factory(store_factory)
    .effect_host(Arc::new(InlineEffectHost::default()))
    .lease_timings(lease_timings) // omit to keep the 30s ttl / 10s renew default
    .build()?;

// lash ships no drain orchestrator (ADR-0014): each step is an explicit,
// host-owned lever. The order below and every deadline are host policy.

// 1. Stop admitting new turns. A host-layer decision — flip a readiness
//    flag, drain the load balancer. lash cannot see your ingress.

// 2. Finish or cancel in-flight turns. A live turn shares the session and
//    makes park/close fail with `SessionStillInUse` until it ends.
for session in &idle_sessions {
    session.cancel_running_turns();
}

// 3. Park resumable sessions (flush dirty state through a fresh-lease commit,
//    release the lease, keep a cheap handle) or `close()` ephemeral ones.
//    Both consume the session and need exclusive ownership.
for session in idle_sessions {
    let parked = session.park().await?;
    // Cache `parked` keyed by `parked.session_id()` and rebuild it later
    // with `LashCore::resume(parked)`; drop it instead to fully close.
    let _ = parked.session_id();
}

// 4. If you stopped an external queued-work or turn-input driver mid-claim,
//    hand its claims back so peers take the work now instead of waiting out
//    the claim TTL — `session.abandon_queued_work_claim(&claim)` and
//    `session.abandon_turn_input_claim(&claim)` — and resolve outstanding
//    durable waits as `Cancelled` with `session.revoke_durable_waits()`.

// 5. Release provider transports. The default `close()` is a no-op; the
//    Codex provider sends WebSocket Close frames on its cached sessions.
let _ = provider.close().await;

// 6. Flush the trace sink (fsync for JSONL). OTel span-export durability is
//    the host's duty: `force_flush()`/`shutdown()` your own TracerProvider.
core.flush_trace_sink()?;

// 7. Exit. Any lease this process still holds now expires on its TTL.
Ok(())

Steps 1 and 2 are pure policy: lash cannot see your ingress or your grace budget, so stopping admission and deciding whether to await or cancel in-flight turns stays with the host. Steps 3 through 6 are the levers — park/close, the claim and wait handbacks, provider close, trace flush — each an explicit call with no hidden ordering. Anything lash cannot expose as a lever without becoming an orchestrator (signal handling, drain deadlines, readiness endpoints) is host territory by design.

The agent service example wires the subset a stateless request/response host owns: an axum graceful-shutdown signal stops admission, then the process closes its retained provider handle and flushes its trace sink before exit.

A host that runs its own durable process worker has one more terminal-writing step, orthogonal to the session drain above: DurableProcessWorker::drain_owner_bound_work(). lash does not fold it into a facade drain, because only a host that constructs the worker — directly through lash::durability::DurableProcessWorker::new, or reached through RestateProcessDeployment::worker() — owns its lifecycle. Run it inside the worker's own shutdown, after stopping admission and releasing in-flight run leases. It terminalizes every non-terminal OwnerBound row this worker started (the row's first_started.owner equals the worker's lease owner) as Abandoned{OwnerDrain} under a fresh drain lease — the owner completing its own work is the ordinary graceful path, and shell.start processes carry OwnerBound. Rerunnable in-flight work (lashlang engine and subagent turns) takes the opposite contract: the worker just stops the local run task and writes no terminal, leaving the row non-terminal so the next worker re-runs it. Rows another owner started, not-yet-started OwnerBound rows (still claimable by any peer), and ExternallyOwned rows are left untouched. What happens to a host's OwnerBound work when it crashes instead of draining is the subject of background process recovery.

The pivotal distinction the sweep draws is provably dead vs. merely lease-expired. Claiming a row's lease, it separates a fenced reclaim of a holder proven dead (is_definitely_dead_for_claimant — the same-host liveness proof LocalProcess identity carries, described under worker identity) from simply acquiring a free or TTL-expired lease with no death evidence. Elapsed time alone never terminalizes anything: a slow-but-alive owner whose lease lapsed is silent, not dead, and the sweep leaves its started work exactly where it is. Only a positive death proof — or an operator's explicit authorization (below) — converts that uncertainty into a terminal fact.

The verdict each disposition yields after owner loss:

Every Abandoned write — at drain, at the sweep, or from a reconciled request — goes through the sweep's own fenced lease, so the terminal has exactly one legitimate writer and a revenant owner that reappears is rejected by its stale lease token rather than healed back to running. Abandoned is a fourth terminal state peer to Completed | Failed | Cancelled; it rides await_output and reconcile like any terminal, and (per docs/adr/0017-process-observation-is-best-effort-push-over-state-truth.md) it does not ride the best-effort event sink. On a Restate deployment the same rules hold at the engine seam: ingress skips ExternallyOwned submission, the run handler completes a re-invoked started OwnerBound row as Abandoned{Sweep} instead of re-running it, and pending abandon requests are reconciled there too.

The recipe: list the non-terminal rows, then read four raw fields off each ObservedProcess and classify.

// List the live rows (host-wide, or `list_granted_to`/`list_originated_by`
// for a session lens), then classify each from raw facts — no derived verdict.
let live = core
    .processes()
    .list(&ProcessListFilter {
        status: ProcessStatusFilter::Running,
        ..ProcessListFilter::default()
    })
    .await?;
for p in live {
    let lease_valid = p.lease_expires_at_ms.is_some_and(|ms| ms > now_ms);
    match (&p.disposition, p.first_started.is_some(), lease_valid) {
        // Lease still in the future: the owner is renewing. Not stuck.
        (_, _, true) => {}
        // Rerunnable / not-yet-started OwnerBound: the sweep re-runs or claims
        // it; a lapsed lease here is just work awaiting a peer.
        (RecoveryDisposition::Rerunnable, _, false) => {}
        (RecoveryDisposition::OwnerBound, false, false) => {}
        // Started OwnerBound, lease lapsed: silent, not dead. The sweep will
        // NOT terminalize it without death evidence. This is the stuck case
        // a host classifies from `lease_holder` + your own timeout budget.
        (RecoveryDisposition::OwnerBound, true, false) => {
            let _ = (&p.lease_holder, p.abandon_request.as_ref());
            // ...decide, per host policy, whether to authorize abandonment:
            // core.processes().request_abandon(&p.process_id, "operator", reason)
        }
        _ => {}
    }
}

core.processes().prune(cutoff_epoch_ms) physically deletes terminal rows — with their events, wakes, handle grants, and leases — older than the cutoff, returning a ProcessPruneReport of what it reclaimed. Non-terminal rows are never touched. Run it on the same maintenance cadence as the session store's vacuum / gc_unreachable reclamation (see persistence).

The agent service example wires this exactly: it captures a core.processes() handle before the core moves into app state and drives prune from a maintenance task on a fixed cadence with a retention window longer than any wait, in both the inline and Restate durability modes.

loop {
    match session.turn(TurnInput::text(text)).run().await {
        Ok(output) => {
            // A failed LLM call finishes the turn instead of erroring; read
            // the typed provider signal off the turn's issues.
            for issue in &output.result.errors {
                if issue.retryable == Some(true) {
                    // Transient provider/transport failure — safe to re-run.
                }
                if let Some(kind) = issue.provider_failure_kind {
                    let _ = kind; // Timeout, Http, Quota, Auth, Stream, ...
                }
            }
            return Ok(output);
        }
        // SessionExecutionBusy / SessionExecutionLeaseLost: another owner
        // holds or fenced the lease, so the attempt committed nothing.
        Err(err) if err.is_retryable() => continue, // back off in real code
        // Wiring/config a retry can never repair (missing facet, provider
        // unconfigured). Surface it to an operator.
        Err(err) if err.is_terminal() => return Err(err),
        // Neither typed signal: unknown. Apply your own bounded policy.
        Err(err) => return Err(err),
    }
}

// Per-turn timing, straight off the runtime clock.
let started_at = output.result.started_at(); // SystemTime the turn was claimed
let elapsed = output.result.duration(); // claim -> commit + post-persist hooks
let _ = (started_at, elapsed);

// Cumulative token usage for the session, split by source and by model.
let usage = session.usage_report();
let _ = (usage.entry_count, usage.usage);