operations/production

The contract is exact: owner_id is stable per worker or process, never per turn, while incarnation_id changes once per boot. Each runtime open then mints its own executor_id. Lease reentry requires all three values to match; two opens in one process share the host identity but remain distinct executors, so the second observes Busy and proceeds lane-less only at a persistence boundary that is already allowed to settle through the head CAS. The slack-clone example is the reference host-owner shape: constant owner id slack-clone-bot with a fresh boot incarnation. Lease ownership is platform-neutral and never inspects the holder's process table.

// Stable per worker or process, never per turn. Change only the
// incarnation when this process boots.
let owner = LeaseOwnerIdentity::opaque(
    std::env::var("WORKER_ID").unwrap_or_else(|_| "worker-1".to_string()),
    std::env::var("AGENT_SERVICE_INCARNATION").unwrap_or_else(|_| boot_incarnation()),
);

let core = builder.build(owner)?;

Mid-incident that reads smaller than it is. Once the stores are recreated, redeploying the previous image does not bring the service back. It fails at store open with a schema-version mismatch and never serves a request. That is the policy working, not a second outage. The old binary cannot read the new schema, and lash refuses the open rather than writing into rows it does not understand. So plan the bump such that forward is the only direction you need: know before the deploy whether the new version requires recreation, keep its image and configuration ready to redeploy, and treat "roll back the deployment" as unavailable from the moment the first store is recreated.

The version integers themselves live with the stores (crates/lash-sqlite-store, crates/lash-postgres-store) and move without ceremony, so do not pin a procedure to a number. Two operator-facing facts carry instead: the refusal reports the found and expected versions, and a release that changed either one says so in its notes, marked breaking (docs/PUBLISHING.md).

What this build writes is published as one table. It re-exports the owning crates' constants rather than restating them, so a version reported here and a version written to disk are the same symbol and cannot drift apart. Module artifact identity uses the semantic hash lashlang-semantic-v2; a host that cannot read an artifact must recompile and republish it.

let mut report = String::new();
for entry in durable_formats() {
    let row = format!(
        "{}: {} ({}::{}) [{:?}]",
        entry.format.name(),
        entry.version,
        entry.owning_crate,
        entry.constant,
        entry.probe
    );
    report.push_str(&row);
    report.push('\n');
}

The table describes its own limits rather than quietly narrowing, so each row carries the shape of its boundary alongside the number. Most are counters: exact-match in both directions, refused either way. One is a forward-only fence (stored bytes at that generation or older load; only a strictly newer generation is refused), and it is listed as such rather than as a counter, because an operator comparing two builds needs to know which direction is fatal. Compiled bytecode is marked identity-only: its stored bytes carry a build identity rather than a version, so a probe can recompute the identity this build would produce and compare for equality, but it cannot say which of two identities is newer. The VM ABI appears too, marked not-persisted: nothing on disk carries it, so it gates a live process rather than parked bytes. All of them stay listed, because a manifest that silently omitted the awkward ones would read as a complete account of the boundary when it is not.

for entry in durable_formats() {
    let note = match entry.probe {
        // Stored bytes carry their own version, so a probe reads it and
        // compares. What the comparison *means* is the version's own shape:
        // a counter is exact-match in both directions, a forward-only fence
        // refuses only a strictly newer generation.
        FormatProbe::Comparable => format!("version {}", entry.version),
        // Bytecode stores an identity, not a version: nothing recoverable
        // says "written by version N", so a probe can only recompute the
        // identity this build would produce and check whether it matches.
        FormatProbe::IdentityOnly => "identity only, recomputed per item".to_string(),
        // The VM ABI is never persisted, so there is nothing durable to
        // compare and the manifest reports it informationally.
        FormatProbe::NotPersisted => format!("not persisted ({})", entry.version),
        _ => "unclassified".to_string(),
    };
    writeln!(report, "{}: {note}", entry.format.name()).expect("a String write cannot fail");
}

Reading a store's recorded schema

A preflight handle is built from raw connection configuration, not from a wired store, and it exposes no method that writes. That separation is the whole point. Constructing a store is itself the side-effectful act a preflight is meant to precede: Postgres takes an exclusive advisory lock and may run creation DDL, an explicit migration, a signing-secret precondition and schema-gate telemetry, while SQLite takes the write lock and applies its schema batch. Asking through a store you already built answers a question you have already committed to.

let preflight = SqliteStorePreflight::for_session_store_root(root)
    .with_process_registry(root.join("processes.db"))
    .with_trigger_store(root.join("triggers.db"))
    .with_effect_journal(root.join("effects.db"));

let status = preflight.schema_status().await?;
match status.outcome() {
    // Every database answered, and every answer matched.
    StoreSchemaOutcome::Ready => {}
    // The refusal the open would have produced, named before the wiring.
    StoreSchemaOutcome::Refused => {
        for refusal in status.refusals() {
            eprintln!(
                "{} would refuse: expected {}, found {} at {}",
                refusal.name,
                refusal.expected,
                match &refusal.verdict {
                    StoreSchemaVerdict::Mismatch { found } => found.to_string(),
                    _ => "n/a".to_string(),
                },
                refusal.location
            );
        }
    }
    // Nothing refused on a version, but a database could not be read far
    // enough to decide. Do not boot blind: the evidence for a pass is
    // missing, not present, and PostgreSQL's structural gate refuses drift
    // that arrives here.
    StoreSchemaOutcome::Undecided => {
        for undecided in status.undecided() {
            eprintln!(
                "{} is undecided at {}: {} — investigate before starting",
                undecided.name,
                undecided.location,
                match &undecided.verdict {
                    StoreSchemaVerdict::Unreadable { reason } => reason.as_str(),
                    _ => "no reason reported",
                }
            );
        }
    }
    _ => eprintln!("unclassified schema outcome — investigate before starting"),
}

The status is a three-way answer, and a boot gate has to face all three. Ready means every declared database answered and every answer matched. Refused means a version the open would reject, named before the wiring. Undecided means a database that could not be read far enough to decide: a corrupt file, an unreachable server, structural drift Postgres reports without a version to blame. Undecided is not a pass: the evidence for a pass is missing rather than present, so a host treats it as "do not boot blind, investigate", which is why the surface returns an outcome a caller must match on rather than a boolean that would fold the undecided case into green.

The SQLite path in particular differs from an open in two ways an operator feels. It never passes SQLITE_OPEN_CREATE, so asking about a database that is not there reports it absent instead of leaving an empty one behind. And it reads PRAGMA user_version without taking a transaction, where the open path takes BEGIN IMMEDIATE first and therefore blocks on the write lock even when it is about to refuse.

What a read-only probe may still touch is stated rather than hidden. Opening a WAL database read-only can create its -shm and -wal sidecars, which are recoverable index files rather than durable content. No schema is applied, no version is stamped, no row is written, and the database file is never brought into existence by the asking. What the probe will not do is fall back to a connection that can write when the read-only open fails, because a read-write connection checkpoints a hot write-ahead log and removes it on close, rewriting the main file's bytes. A read the probe cannot perform read-only is reported undecided instead, and the remaining read path sets PRAGMA query_only so the engine, not the choice of statements, enforces the promise.

let database = verify_schema_at(path, SqliteDatabase::DurableCore).await;
match &database.verdict {
    StoreSchemaVerdict::Matches => eprintln!("ready at version {}", database.expected),
    StoreSchemaVerdict::Mismatch { found } => {
        eprintln!("refused: found {found}, expected {}", database.expected)
    }
    StoreSchemaVerdict::Unreadable { reason } => eprintln!("undecided: {reason}"),
    StoreSchemaVerdict::Absent => eprintln!("nothing provisioned yet"),
    _ => {}
}

Postgres reuses the read-only verification it has had since ADR-0052. Its handle connects lazily, so building one cannot fail because the server is briefly unreachable; an unreachable server is reported by the status call, where a host can tell it apart from a version mismatch and decide whether to wait or exit.

let preflight =
    PostgresStorePreflight::for_database_url(database_url).expect("a valid database URL");
let StoreBackend::Postgres { location } = preflight.backend() else {
    panic!("a postgres handle identifies as postgres")
};
preflight.close().await;

Reading whether stored data will open

The probe is one read-only walk over the store answering, per durable format, whether what is stored will open under this build. It reports an expected version, the versions actually found with a count each, and an undecodable bucket for items nobody could read far enough to judge. It never opens the store, never writes, and never disposes of anything: deciding what to delete is an operator's call and lash's collection story is separate.

At startup a host wants one decision, not a report. Summary mode gives it: the schema stamps plus the process-registry surfaces, bounded by parked processes and pending wakes rather than by session count, so it is cheap enough to run on every boot. A refusal comes back as a single sentence naming the format, the versions, the number of affected items and the remedy: the sentence a supervisor logs instead of restarting into the same failure forever.

let report = probe_store(handle, PreflightOptions::summary())
    .await
    .map_err(|error| format!("could not read the store: {error}"))?;
if let Some(refusal) = report.refusal_message() {
    // One line, then exit. Everything an operator needs to act is in it.
    return Err(refusal);
}
let ready = format!(
    "store preflight: {} ({} mode)",
    report.outcome.name(),
    report.mode.name()
);

Before a version bump the question is different, and so is the budget. Deep mode adds the per-session checkpoint walk that summary mode skips, reading each session's root blob to reach the checkpoint manifest and the component encodings behind it. The page size is a round-trip knob rather than a cap: the walk pages every surface to exhaustion, so a smaller page changes what the audit costs and never what it covers.

let options = PreflightOptions::deep().with_page_size(DEFAULT_PAGE_SIZE);
eprintln!("store audit: {}", options_note(&options));
let report = probe_store(handle, options).await?;

One row per enumerated format comes back in manifest order, including the formats with nothing stored, because a row that vanished when a store held none of it would be indistinguishable from one nobody checked. That distinction is carried explicitly: empty means somebody walked the surface and found nothing, not scanned means nobody walked it, and only the first of those means a drain is unnecessary. Rows also say what their answer rests on: a version read from stored bytes, a version carried by the envelope that embeds this format, or nothing at all for a format that is never persisted.

let mut lines = Vec::new();
for component in &report.components {
    let mut line = format!(
        "{}: {} (expected {}, {} probe, {})",
        component.format,
        component.verdict.name(),
        component.expected,
        component.probe,
        evidence_label(component.evidence)
    );
    write!(line, " scanned={}", component.scanned).expect("a String write cannot fail");
    for FoundVersion { version, count } in &component.found {
        write!(line, " found {version} x{count}").expect("a String write cannot fail");
    }
    if component.undecodable > 0 {
        write!(line, " undecodable={}", component.undecodable)
            .expect("a String write cannot fail");
    }
    if component.refused_without_version > 0 {
        write!(
            line,
            " identity-refused={}",
            component.refused_without_version
        )
        .expect("a String write cannot fail");
    }
    lines.push(line);
}

The counts say a bump is blocked; they do not say whose work blocks it. The same walk keeps the process and session ids as it goes, so "drain first" arrives as a list an operator can work through rather than an instruction they have to trust. Each entry names the process, its session, the status it is parked in, the format that refuses and both versions.

let mut worklist = Vec::new();
for DrainBlocker {
    process_id,
    session_id,
    status,
    format,
    expected,
    found,
    detail,
} in &report.drain
{
    let owner = match (process_id, session_id) {
        (Some(process), Some(session)) => format!("process {process} (session {session})"),
        (Some(process), None) => format!("process {process}"),
        (None, Some(session)) => format!("session {session}"),
        (None, None) => "an unattributed item".to_string(),
    };
    // Bytecode is the one format with no found version to print: its stored
    // identity says which build wrote it, never which generation, so a
    // mismatch is reportable and a distance is not.
    let found = found
        .as_ref()
        .map(|version| version.to_string())
        .unwrap_or_else(|| "a build-specific identity".to_string());
    worklist.push(format!(
        "{owner} [{}] holds {format} {found}, expected {expected}: {detail}",
        status.as_deref().unwrap_or("unknown status")
    ));
}

Every report names what it did not read. Summary mode's skipped session walk appears here, so does any surface the deployment could not reach, and so do the two enumerated formats no bounded walk enumerates. A report that quietly omitted its own blind spots would read as a complete account of the boundary while being a partial one, which is the failure a preflight exists to prevent.

report
    .not_scanned
    .iter()
    .map(|entry| format!("{}: {}", entry.what(), entry.reason()))
    .collect()

Backends supply the walk and nothing more: they return each item's logical bytes, its owner ids and a cursor, and lash owns every version comparison, so two backends cannot drift into disagreeing about the same boundary. Paging is keyset rather than offset (a cursor names the last item read), so concurrent writes cannot make a page skip or repeat rows. A surface a deployment cannot read reports itself as unscanned with a reason rather than as an empty page, because an empty page is an answer and "I could not look" is not.

if let Some(reason) = self.unreadable.get(&scan.surface) {
    return Ok(DurableScanPage {
        items: Vec::new(),
        next: None,
        coverage: ScanCoverage::NotScanned {
            reason: reason.clone(),
        },
    });
}
let all = self
    .surfaces
    .get(&scan.surface)
    .cloned()
    .unwrap_or_default();
// Keyset paging, not offset paging: a cursor names the last item read,
// so concurrent writes cannot make a page skip or repeat rows.
let start = match &scan.after {
    None => 0,
    Some(cursor) => all
        .iter()
        .position(|item| &item.cursor == cursor)
        .map(|index| index + 1)
        .unwrap_or(all.len()),
};
let items: Vec<DurableItem> = all.iter().skip(start).take(scan.limit).cloned().collect();
let next = if start + items.len() >= all.len() {
    None
} else {
    items.last().map(|item| item.cursor.clone())
};
Ok(DurableScanPage {
    items,
    next,
    coverage: ScanCoverage::Scanned,
})

1. Quiesce ingress

   Stop admitting turns, trigger sources, and queued work at the host edge, and stop the drivers that claim them. lash cannot see your ingress (ADR-0014), so this is a readiness flag or a load-balancer change you own. Steps 1 and 2 of graceful drain are the same levers, applied here to reach a quiet fleet rather than a single exiting process.

2. Drain in-flight effect work

   Let running turns and background processes finish or cancel them, then confirm the effect journal is empty before you touch a store. An invocation still mid-replay when its rows disappear is the failure mode the next step's gotcha describes. Hosts running a durable process worker also call drain_owner_bound_work(), which terminalizes the OwnerBound rows that worker started instead of leaving them for a sweep that will refuse to guess.

3. Snapshot, if that is your policy

   Optional, and entirely yours: lash ships no backup or restore, and nothing below depends on one. If you take a snapshot, take it of the whole consistency unit at once (see Backups below).

4. Bump and migrate or recreate

   Follow the release-specific schema note. For an explicit Lash-managed migration, let SchemaCheck::Enforce prove the published source shape and apply it before workers open; for a recreate cutover, start the new binary against empty durable state: a fresh database (or one with the lash_* tables dropped), a fresh effect journal, and an attachment prefix consistent with the store you just recreated. A Postgres store stamped below its artifacts (a component-50 stamp over component-51 or component-52 tables) is ledger/schema divergence, not a partial migration to continue: stop, inspect, and recreate it.

5. Verify before reopening ingress

   Prove the three durable surfaces work on the recreated state, in this order: a session opens and a turn commits; a background process registers, wakes, and runs to a terminal; a trigger fires. Only then reopen ingress. Verifying after traffic returns turns a failed bump into an incident with users in it.

Backups. lash ships no backup or restore tooling, and the procedure above never needs one. If your own prudence says snapshot before a destructive step, snapshot the consistency unit as a unit: the Postgres store, the effect journal, and the S3-compatible blob store holding attachment bytes are one unit, not three. Restoring any one of them alone yields mixed-vintage state: a journal referencing sessions that no longer exist, or effects replaying against recreated rows. Restoring is host territory and untested by lash, so treat a snapshot as forensic material and a fix-forward aid, never as a rollback plan.

Every trigger operation a host previously hand-wrote has a fenced, receipted replacement. All of them run through TriggerStore::execute_command(operation_id, command) on the trigger store the host installed, and every point mutation takes an expected_revision read from a List record or an earlier receipt.

Two migration notes carry over from the fence. Bulk enable or disable is a loop of per-key fenced commands, not one blanket statement, and each command's operation_id should be derived from the intent so a retried pass replays its receipts instead of re-evaluating against moved rows. A conflict is an ordinary outcome, not a backend failure: re-read the record and retry.

The standing policy behind the cutover, and the supported read alternative for every other lash_* table, is under private tables.

Queued-work and turn-input claims are not timed lanes. Each claim pins the session-execution-lease generation under which it was taken. Renewing the session lease preserves the generation; release or takeover makes the claim eligible for successor re-claim and hides it from lease-less live-claim views. Its completion remains valid until successor re-claim or explicit abandon, while the session-head CAS separately governs publication.

The default is a 30 second TTL with a 10 second renew interval. The constructor enforces ttl >= 3 * renew_interval, so a healthy owner can miss two consecutive renewals (a scheduler stall, a transient store error) before a peer may treat the lease as expired. A shorter TTL reclaims a crashed owner's work sooner but widens the window in which a slow-but-alive owner is falsely taken over; a longer TTL does the reverse. Choosing that number is the trade, and it is now a host decision instead of a constant baked into lash.

// One timing decision governs the three durable lease lanes:
// session execution, effect replay, and process execution. `new` enforces
// `ttl >= 3 * renew_interval`, so a live owner can miss two renewals before
// a peer may treat the lease as expired. Queued-work and turn-input claims
// pin the session-lease generation and carry no timing of their own.
let lease_timings = LeaseTimings::new(
    Duration::from_secs(15), // ttl
    Duration::from_secs(5),  // renew_interval
)
.expect("ttl >= 3 * renew_interval");

let core = LashCore::rlm_builder(lash::TurnBudget::Unbounded, factory)
    .provider(provider)
    .model(
        lash::ModelSpec::builder("anthropic/claude-sonnet-4.6")
            .context_window_tokens(200_000)
            .build()
            .expect("valid model metadata"),
    )
    .store_factory(store_factory)
    .effect_host(Arc::new(InlineEffectHost::default()))
    .attachment_store(attachment_store)
    .process_env_store(process_env_store)
    // Start bounded; tune both limits for your backend's latency envelope.
    .commit_budget(lash::CommitBudget::bounded(1024 * 1024, 512))
    .queued_work_batching(lash::QueuedWorkBatchingConfig::new(1024))
    .lease_timings(lease_timings) // omit to keep the 30s ttl / 10s renew default
    .build(session_execution_owner)?;

// lash ships no drain orchestrator (ADR-0014): each step is an explicit,
// host-owned lever. The order below and every deadline are host policy.

// 1. Stop admitting new turns. A host-layer decision: flip a readiness
//    flag, drain the load balancer. lash cannot see your ingress.

// 2. Finish or cancel in-flight turns. Exact retained turn addresses should
//    normally go through `core.turn_work_driver().request_cancel(...)`.
//    This process-local cancel-all remains a shutdown compatibility lever;
//    "shutdown" is opaque host vocabulary.
for session in &idle_sessions {
    session.cancel_running_turns_with_origin(Some("shutdown".to_string()));
}

// 3. Park resumable sessions (flush dirty state through a fresh-lease commit,
//    release the lease, keep a cheap handle) or `close()` ephemeral ones.
//    Both consume the session and need exclusive ownership.
for session in idle_sessions {
    let parked = session.park().await?;
    // Cache `parked` keyed by `parked.session_id()` and rebuild it later
    // with `LashCore::resume(parked)`; drop it instead to fully close.
    let _ = parked.session_id();
}

// 4. If you stopped an external queued-work or turn-input driver mid-claim,
//    hand its claims back for immediate reuse with
//    `session.abandon_queued_work_claim(&claim)` and
//    `session.abandon_turn_input_claim(&claim)`. Lease loss makes the claims
//    eligible for successor re-claim; only re-claim or explicit abandon
//    invalidates the old completion. Resolve outstanding durable waits as
//    `Cancelled` with `session.revoke_durable_waits()`.

// 5. Read the deployment's authoritative process status before retiring
//    an immutable deployment. This is a read, not a drain orchestrator:
//    keep the old deployment registered while `drained` is false.
let drain_status = core.drain_status(false).await?;
if drain_status.drained {
    // The host may now retire this deployment.
} else {
    // Keep it available for the remaining pinned invocations.
    let _ = drain_status.remaining_invocations;
}

// 6. Release provider transports. The default `close()` is a no-op; the
//    Codex provider sends WebSocket Close frames on its cached sessions.
let _ = provider.close().await;

// 7. Release resources owned by plugin factories. This is not a drain
//    orchestrator: intake, ordering, and deadlines remain host policy.
core.shutdown().await?;

// 8. Flush the trace sink (fsync for JSONL). OTel span-export durability is
//    the host's duty: `force_flush()`/`shutdown()` your own TracerProvider.
core.flush_trace_sink()?;

// 9. Exit. Any lease this process still holds now expires on its TTL.
Ok(())

Steps 1 and 2 are pure policy: lash cannot see your ingress or your grace budget, so stopping admission and deciding whether to await or cancel in-flight turns stays with the host. Steps 3 through 7 are the levers: park/close, the claim and wait handbacks, provider close, plugin-factory shutdown, and trace flush. Each is an explicit call with no hidden drain orchestration. Anything lash cannot expose as a lever without becoming an orchestrator (signal handling, drain deadlines, readiness endpoints) is host territory by design.

During pin-and-drain, query the Restate-side count with RestateAdminClient::open_invocations_by_deployment() and do not retire or deregister a deployment while its open_count is above zero. The result also surfaces a NULL pinned_deployment_id row for invocations not yet pinned, which the host must account for before retirement.

The agent service example wires the subset a stateless request/response host owns: an axum graceful-shutdown signal stops admission, then the process closes its retained provider handle and flushes its trace sink before exit.

A host that runs its own durable process worker has one more terminal-writing step, orthogonal to the session drain above: DurableProcessWorker::drain_owner_bound_work(). lash does not fold it into a facade drain, because only a host that constructs the worker (directly through lash::durability::DurableProcessWorker::new, or reached through RestateProcessDeployment::worker()) owns its lifecycle. Run it inside the worker's own shutdown, after stopping admission and releasing in-flight run leases. It terminalizes every non-terminal OwnerBound row this worker started (the row's first_started.owner equals the worker's lease owner) as Abandoned{OwnerDrain} under a fresh drain lease. The owner completing its own work is the ordinary graceful path, and shell.start processes carry OwnerBound. Rerunnable in-flight work (lashlang engine and subagent turns) takes the opposite contract: the worker just stops the local run task and writes no terminal, leaving the row non-terminal so the next worker re-runs it. Rows another owner started, not-yet-started OwnerBound rows (still claimable by any peer), and ExternallyOwned rows are left untouched. What happens to a host's OwnerBound work when it crashes instead of draining is the subject of background process recovery.

ProcessDrainReport.abandoned is confirmation evidence, not an attempted-work list: a process id enters it only when this pass receives Committed for its fenced Abandoned{OwnerDrain} terminal write. Every other outcome stays in deferred: Busy and Absent name ordinary claim/read results; SettledByPeer { terminal_status } and AlreadyApplied { terminal_status } name retained terminal evidence; LeaseLost { operation } means a newer owner superseded this pass; and BackendError { operation, error } names a registry failure. Retry backend-error rows after the registry is healthy; allow lease-lost rows to settle under the newer owner; do not treat any deferred row as this pass's drain write.

The sweep may acquire a free or TTL-expired process lease. Elapsed time alone never terminalizes anything: a started OwnerBound row remains non-terminal even after its holder's lease expires, because lease expiry does not prove whether its external side effects completed. Only the original owner's graceful drain or an operator's explicit authorization converts that uncertainty into a terminal fact.

The verdict each disposition yields after owner loss:

Every Abandoned write goes through a fenced lease or a validated workflow-key authority, so the terminal has exactly one legitimate writer and a revenant owner that reappears is rejected rather than healed back to running. Abandoned is a fourth terminal state peer to Completed | Failed | Cancelled; it rides await_output and reconcile like any terminal, and (per docs/adr/0017-process-observation-is-best-effort-push-over-state-truth.md) it does not ride the best-effort event sink. On a Restate deployment, ingress skips ExternallyOwned submission, the workflow-key recovery path completes a re-invoked started OwnerBound row as Abandoned{Sweep} rather than re-running it, and pending abandon requests are reconciled after lease expiry.

A process-registry failure during claim, re-read (including the execution-start read), renewal, terminal write, or lease release emits warn event process_recovery.backend_error on target lash_core::process_recovery. Its structured string fields are decision_basis=backend_error, process_id, operation, outcome=deferred, and error. This is distinct from an ordinary busy or absent result and from ProcessLeaseSuperseded: supersession emits process_recovery.lease_lost with decision_basis=lease_superseded and defers to the newer owner, while backend errors require investigation and another pass after the registry is healthy. After a transient renewal or terminal-write backend failure, recovery uses a token-fenced release: it cannot clear a successor's lease, and it makes Rerunnable work immediately retryable instead of retaining the lease TTL as implicit backoff.

The recipe: list the non-terminal rows, then read four raw fields off each ObservedProcess and classify.

// List the live rows (host-wide, or `list_observed_by`/`list_originated_by`
// for a session lens), then classify each from raw facts, no derived verdict.
let live = core
    .processes()
    .list(&ProcessListFilter {
        status: ProcessStatusFilter::Running,
        ..ProcessListFilter::default()
    })
    .await?;
for p in live {
    let lease_valid = p.lease_expires_at_ms.is_some_and(|ms| ms > now_ms);
    match (&p.disposition, p.first_started.is_some(), lease_valid) {
        // Lease still in the future: the owner is renewing. Not stuck.
        (_, _, true) => {}
        // Rerunnable / not-yet-started OwnerBound: the sweep re-runs or claims
        // it; a lapsed lease here is just work awaiting a peer.
        (RecoveryContract::Rerunnable, _, false) => {}
        (RecoveryContract::OwnerBound, false, false) => {}
        // Started OwnerBound, lease lapsed. The sweep will NOT infer a
        // terminal. This is the stuck case
        // a host classifies from `lease_holder` + your own timeout budget.
        (RecoveryContract::OwnerBound, true, false) => {
            let _ = (&p.lease_holder, p.abandon_request.as_ref());
            // ...decide, per host policy, whether to authorize abandonment:
            // core.processes().request_abandon(&p.process_id, "operator", reason)
        }
        _ => {}
    }
}

1. Read the lane. core.session_lease_diagnostics(session_id) returns a snapshot of the session's execution-lease row (owner id, boot incarnation, executor id, generation, claim time, expiry) plus a derived SessionLeaseRenewal. It never claims, renews, or releases, so it is free to run against a live session.

// Step 1: read the lane. Diagnostics only: this never claims, renews, or
// releases anything, so it is free to run against a live session. `None`
// means no durable session under this id at all.
let Some(lease) = core.session_lease_diagnostics(session_id).await? else {
    // The host's own record of an in-flight turn is what is wrong here.
    return Ok(());
};
let holder = lease.holder.as_ref().map(|holder| &holder.owner);
let generation = lease.holder.as_ref().map(|holder| holder.generation);
let _ = (holder, generation, lease.observed_at_epoch_ms);

match lease.renewal() {
    // Nobody holds the lane. A turn the host still shows as running either
    // already committed and released, or never claimed. Reconcile against
    // the session's committed head.
    SessionLeaseRenewal::Unheld => {}

    // Renewals were current: the lane is healthy, so the turn is blocked
    // inside itself. Look at the provider call, not the lease, and cancel
    // the exact turn if it has to stop.
    SessionLeaseRenewal::Current { expires_in_ms } => {
        let _ = expires_in_ms;
    }

    // Renewals stopped. The worker that swept the lane logs
    // `session_execution_lease.taken_over` naming this holder as
    // `displaced_owner_id`, so the handoff is in the log even if this holder
    // died without noticing. Do NOT kill it: it may still win the commit CAS.
    // Only `session_execution_lease.commit_cas_rejected` proves it lost.
    SessionLeaseRenewal::Lapsed { expired_for_ms } => {
        let _ = expired_for_ms;
    }
}
Ok(())

2. Read the lease timeline. The structured lease events cover the decisive lease transitions, and one covers settlement authority removed during recovery. Events emitted while holding or acquiring a lane carry session_id, generation (the lane's fencing token, which ADR 0029 calls the generation), owner_id, incarnation_id, and executor_id. Busy evidence names both claimant_executor_id and holder_executor_id; the lane-less commit advisory hashes the holder identity and claims no generation. A takeover is reported by the worker that won the lane, atomically with its claim, so it is present even when the displaced worker is dead.

Borrowed-write failure signature. A turn issue with code session_execution_lease_lost whose message originates from a plugin or lifecycle-observer write means that nested write borrowed the turn driver's held lane and its fence was refused. Treat it as loud authority loss, not a generic lifecycle_hook_failed. Its likelihood should track execution-lease renewal health and agreement between the runtime and store clocks; a rising rate without matching renewal failures usually points to clock skew or backend fence-ordering drift.

3. Read the committed outcome. The lease says who was working; the session head says what landed. Only the two together answer "did anything happen".

What each combination means:

The agent service example wires this exactly: GET /api/chats/{chat_id}/lease returns the reading plus the host-owned classification, and its tests induce each situation.

core.processes().prune(cutoff_epoch_ms, ProjectionWatermark::UpTo(cursor)) replaces eligible terminal rows with payload-free tombstones after the host projector has acknowledged them. A deployment without a projector must pass ProjectionWatermark::NoProjector explicitly. It removes events, wakes, observer edges, leases, and settled parent-end plans. The facade then reconciles exact tombstone-authorized trigger deliveries in one trigger-store transaction.

That reconciliation also removes occurrences after their committed fan-out is empty. Zero-match occurrences satisfy the same predicate at ingest. Session-owned subscriptions and receipts are reclaimed only after the ADR 0049 frontier says the owner is permanently deleted and the transaction witnesses no remaining owner deliveries. Host and platform subscription tombstones remain permanent Revive fences. ProcessPruneReport reports the coordinated delivery count; the trigger reconciliation report and structured logs carry the other counters.

A terminal process with a pending parent-end plan remains retained until that plan settles; live rows are never touched. CallerDeparted rows are detached launches whose caller vanished mid-launch, so no writer can terminalize them. They are reclaimed on the same cadence as terminals, and a retention filter may select that status directly.

Run core.processes().compact_tombstones(cutoff_epoch_ms, watermark) on the same maintenance cadence as the session store's vacuum / gc_unreachable reclamation (see persistence). Compaction refuses tombstones still guarding trigger deliveries. It remains blocked while the configured trigger store or deleted-session frontier is unhealthy. Compaction frees registry rows, not process ids. The prune deleted the process-owned session ids too, and those session tombstones are permanent. Reusing a process id after its tombstone was compacted away fails at session-store creation with StoreError::SessionDeleted naming an internal process-env:<id> the host never chose. Always mint a fresh process id.

The agent service example wires this exactly: it captures a core.processes() handle before the core moves into app state and drives prune from a maintenance task on a fixed cadence with a retention window longer than any wait, in both the inline and Restate durability modes.

loop {
    match session.turn(TurnInput::text(text)).run().await {
        Ok(output) => {
            // A failed LLM call finishes the turn instead of erroring; read
            // the typed provider signal off the turn's issues.
            for issue in &output.result.errors {
                if issue.retryable == Some(true) {
                    // Transient provider/transport failure, safe to re-run.
                }
                if let Some(kind) = issue.provider_failure_kind {
                    let _ = kind; // Timeout, Http, Quota, Auth, Stream, ...
                }
            }
            return Ok(output);
        }
        // Busy rejects before the turn starts; LeaseLost means an operation
        // observed handoff. Reload durable state before retrying: lease loss
        // alone neither proves no commit landed nor releases claims.
        Err(err) if err.is_retryable() => continue, // back off in real code
        // Wiring/config a retry can never repair (missing facet, provider
        // unconfigured). Surface it to an operator.
        Err(err) if err.is_terminal() => return Err(err),
        // Neither typed signal: unknown. Apply your own bounded policy.
        Err(err) => return Err(err),
    }
}

// Per-turn timing, straight off the runtime clock.
let started_at = output.result.started_at(); // SystemTime the turn was claimed
let elapsed = output.result.duration(); // claim -> Committed Turn + Post-Commit Delivery
let _ = (started_at, elapsed);

// Cumulative token usage for the session, split by source and by model.
let usage = session.usage_report();
let _ = (usage.entry_count, usage.usage);