durability/workflows

That shape is exactly what durable workflow runtimes (Temporal, Restate, similar) want from a library they're embedding. Lash ships a Restate adapter in lash-restate; other workflow runtimes implement the same controller trait. The workflow ledger can key each invocation, persist its result, and replay with the completed effect already in hand while Lash manages the state-machine checkpoint and response application.

Embedders who only need local durability configure InlineEffectHost plus the regular session store. Embedders who need durable in-flight effects create a handler-scoped controller, then pass it through .turn_id(...).effects(&controller).stream_to(&sink) / .run() on the facade. Lower-level runtime adapters can still build explicit ScopedEffectController values for TurnOptions::new(cancel, scope). RestateRuntimeEffectController is the first-party workflow adapter shape.

Fire-and-forget sans-IO effects such as Emit, Log, Progress, and Done remain driver and commit-pipeline responsibilities. They do not cross RuntimeEffectController because they do not ask the host to run nondeterministic work and return a response.

Every controller invocation receives a serializable RuntimeInvocation: scope identifies the session and optional turn coordinates, subject identifies the effect, caused_by carries the typed causal parent, and replay.key provides the stable replay discriminator. The live RuntimeEffectEnvelope is serializable and carries ref-only request data. Cancellation, streams, provider trace sinks, and attachment byte hydration stay in the local executor or host workflow.

The built turn replay key includes session_id, turn_id, turn_index, protocol_iteration, effect kind, and EffectId; per-tool invocations append the model tool call id. Durable tool steps derive child replay keys from the caller's invocation plus the prepared tool call id and step_id; the stable envelope hash also includes the JSON input. Scoped durable runs require a non-empty stable turn_id. Direct completions derive stable keys from the request payload, usage source, typed causal parent when present, and optional caller-provided DirectRequest::replay.

EffectId is local to one turn. Across turns, the counter resets to 1; use the full replay key, not the bare id, for globally unique effect identity.

impl<M: TurnProtocol> TurnMachine<M> {
    pub fn checkpoint(&self) -> TurnCheckpoint<M>;

    pub fn restore_from_checkpoint(
        config: TurnMachineConfig<M>,
        checkpoint: TurnCheckpoint<M>,
    ) -> Self;
}

TurnCheckpoint<M> is fully serde-compatible. Fields:

• state: MachineState<M>: the waiting variant, with the data each retains.
• pending_effects: Vec<Effect<M>>: fire-and-forget effects queued but not yet drained (Emit, Log, Progress).
• next_effect_id: u64: counter, preserves id determinism.
• messages: Vec<Message>: prompt history committed so far.
• events: semantic event log committed so far.
• protocol_iteration, protocol_run_offset: turn position.
• cumulative_usage: TokenUsage: running token totals for this turn.
• termination: TurnTerminationPolicyState: max-turns / forced-final state.
• synced_protocol_iteration: Option<usize>: last iteration the execution environment was synced.
• next_synthetic_message_id: u64, turn_causes: Vec<TurnCause>, progress_event_cursor: usize: internal bookkeeping that also round-trips through serde.

restore_from_checkpoint remains a sans-IO machine primitive for tests and protocol work. Runtime crash recovery does not persist or reload in-flight machine payloads from the store. Local/inline runs reopen the last committed session state; durable workflow integrations replay the host handler and effect history, then Lash retries the final commit.

• Mid-LLM crash → restore yields a new LlmCall with the same EffectId and identical request. The workflow's activity for that id is the same lookup and produces the same response.
• Post-LLM crash, pre-checkpoint → restore yields the pending Checkpoint effect, not another LlmCall. The model is not re-invoked.
• Mid-tool-call crash → restore yields the same ToolCalls sans-IO effect carrying every original call. The runtime invokes the effect controller separately for each PendingToolCall, keyed by call id, so hosts can short-circuit completed calls and only re-invoke pending ones.
• RLM exec crash → restore yields an ExecCode effect with the original code block and the driver's trajectory state. The exec result contains the parallel tool outputs in their structured form (success / failure / cancellation); no replayed ToolCallStarted events leak from the previous attempt.
• Plugin checkpoint crash → restore yields the Checkpoint effect again. The on_empty resume action is preserved, so a checkpoint that contributed no messages still finishes the turn the same way.

#[async_trait::async_trait]
impl RuntimeEffectController for WorkflowEffectController {
    async fn execute_effect(
        &self,
        envelope: RuntimeEffectEnvelope,
        local_executor: RuntimeEffectLocalExecutor<'_>,
    ) -> Result<RuntimeEffectOutcome, RuntimeEffectControllerError> {
        let key = envelope.invocation.replay_key().unwrap_or("runtime-effect").to_string();
        self.workflow
            .run_activity(key, envelope.invocation.clone(), || async move {
                local_executor.execute(envelope).await
            })
            .await
    }
}

The shape matches Temporal's workflow + activity split and Restate's handler + side-effect split. Each typed runtime effect can become one durable activity: the invocation carries typed scope, subject, causal parent, and replay key, and replay.key joins the activity result back to the logical turn effect. Restate immediately awaits concrete ctx.run(...) calls around the normal local executor for recorded non-sleep effects, uses Restate timers for sleeps, and keeps direct Restate primitives for workflow calls and awaited events. The store is touched only when Lash commits settled session state.

Deterministic Lash/Restate contract failures are terminal handler errors. A recorded effect whose replay hash no longer matches the current envelope, or a recorded Restate run that completed with a terminal failure, returns an explicit terminal error instead of panicking into an infinite backing-off invocation. Hosts should record a failed turn/event and clear their UI running state when the workflow reports such a terminal failure.

Restate hosts also need durable invocation identity outside the handler. RestateIngressClient submits workflow/service/object /send requests and parses the response body's invocationId; RestateAdminClient uses Admin HTTP for graceful cancellation, status lookup, and unfinished-invocation introspection. Forceful kill is reserved for test/dev cleanup after graceful cancel fails.

Process effects cross the same controller boundary for start/list/transfer/delete-session/await/cancel/signal, but durable execution of a background process is owned by the process work driver and worker, not by the turn that happened to start it. The registry stores durable state; waits live on the driver/awaiter seam. The process section below covers the durable intent record, handle grants, worker rebuild inputs, Restate workflow metadata, and event-sourced terminal state.

Successful completion uses RuntimeCommit.turn_commit. The store accepts identical retries for the same (session_id, turn_id) and rejects a changed commit hash, so a workflow replay can retry the final Lash commit without duplicating or mutating settled state.

Advanced in-process tools use ToolContext::durable_effects() to borrow this same boundary without receiving workflow-native context. run_json(...) emits DurableStep; external_event_key(...) signs a custom AwaitEventKey; await_event_json(...) waits through AwaitEvent; and emit_process_event(...) appends a domain process event only when the tool is executing inside a process.

A process is self-contained: the record carries its ProcessOriginator (host or session) as pure provenance, a captured execution-environment reference, and an optional 0..1 wake target. Session relationships are explicit, optional edges that never imply cleanup. Deleting a session revokes that session's handle grants and pending wake deliveries, never cancels a process, and the ProcessSessionDeleteReport lists the non-terminal processes left with no remaining grants (orphaned_process_ids); their lifecycle is host policy. See docs/adr/0011-self-contained-processes.md for the full model.

Records are event-sourced: terminal state and cancellation are keyed process events on the record (process.cancel_requested routes to the workflow's shared cancel handler), not mutable state fields. Tool and Lashlang process rows carry a captured execution-environment reference (a content-addressed process_execution_env artifact holding only Process Plugin Options and policy), and workers instantiate an ephemeral runtime from that immutable environment plus host-owned worker config; they never rebuild the originating session. Durable tool authority snapshots live behind plugin-owned options, not as ad hoc process-environment metadata. A Restate-backed start stores external_ref.backend == "restate" with external_ref.id == "LashProcessWorkflow/{process_id}". Wake semantics write durable wake-inbox rows only when the process has a wake target; a process without one still records wake events, but they deliver nowhere. Checkpoints drain inbox rows beside in-memory turn injections.

Waiting is deliberately not a registry capability (docs/adr/0016-process-waits-live-on-the-work-driver-seam.md). Store-backed registries expose point reads and writes only. The facade wraps inline registries with a ProcessChangeHub so local mutations wake ProcessAwaiter waiters, and the awaiter falls back to narrow point-read backoff when another process may be changing the store. External drivers can attach terminal awaits to their own durable promise; Restate attaches host awaits to LashProcessWorkflow/{process_id}/await_terminal.

Lashlang process starts validate before registration in the Lashlang runtime path: it rebuilds a candidate plugin session and Tool Catalog from the captured Process Plugin Options, constructs the candidate LashlangHostEnvironment, and checks it satisfies the target artifact's recorded Host Requirements before writing ProcessInput::Engine { kind: "lashlang", payload }. The registered ProcessEngine repeats the Host Requirements check during execution and recovery, so a recovered row cannot run against a drifted or incompatible rebuilt environment.

Turn in-flight replay is effect-host owned. A process is store-backed background work: it is fenced by a ProcessLease, resumes from registry events, and recovers through DurableProcessWorker::drive_pending_processes. Process leases carry owner id, lease token, fencing token, claim time, and expiry; stale process writers fail before committing durable state.

A turn supplies its per-invocation durable controller through ScopedEffectController. Out-of-turn starts (trigger deliveries, facade start, and recovery sweeps) write durable registry intent and execute through the worker's wired controller. A durable host never silently re-runs a process on an inline controller.

User-facing cancellation is intentionally separate from execution recovery. Host API, tool, and Lashlang cancel requests route through ProcessCancelAbility with an explicit ProcessCancelSource and reason; cancel-all lists live visible handles and cancels each through that same ability path. Internal cleanup such as cancel_unreferenced remains a registry operation because it is not a public cancel request.

The recovery sweep is disposition-driven, not a re-run-everything loop (docs/adr/0019-process-recovery-obeys-declared-disposition.md): every registration declares a required RecoveryDisposition and the sweep applies it mechanically. It lists non-terminal processes, claims each available lease, and skips work held by another live owner; then a Rerunnable row is re-run and completed exactly as before, a started OwnerBound row whose holder is provably dead is terminalized Abandoned rather than re-run (a not-yet-started one stays claimable by any peer, since a first execution is not a re-execution), and an ExternallyOwned row is never claimed or executed. Lease expiry alone never terminalizes: a started OwnerBound holder that is silent but not provably dead is left non-terminal until death evidence or a reconciled Abandon Request. Abandoned is a fourth terminal state, peer to Completed | Failed | Cancelled, carrying the writer (owner drain, sweep, or reconciled request) and the evidence that licensed it. The sweep is idempotent by persisted process_id; the Restate invocation is keyed as LashProcessWorkflow/{process_id}. The full verdict table and the crash-vs-drain paths are in running in production.

A subagent spawn is itself such a process. agents.spawn builds a SessionCreateRequest plus a TurnInput and emits one ProcessInput::SessionTurn, so a spawned child runs through the same generic process path (and therefore inherits provider re-supply, durability, and recovery) rather than a bespoke child-creation route. Recursion safety (the subagent depth ceiling and tool-hiding at that ceiling) and the submit_error terminal are carried as request config, not lost.

The rule is simple: a durable effect host running against any ephemeral store facet fails loudly, and it fails where the tier is first observable rather than silently producing a non-durable run. Core's DurabilityTier lives in lash-core; the Lashlang runtime maps its artifact-store tier into the same facade-level coherence checks. The defaulted hook lands on EffectHost / RuntimeEffectController, RuntimePersistence / SessionStoreFactory, LashlangArtifactStore, and ProcessRegistry; AttachmentStore::persistence() is reused as that trait's tier signal (Ephemeral → Inline, Durable → Durable). Durable overrides ship on RestateEffectHost / RestateRuntimeEffectController, the SQLite and Postgres stores / factories / registries, and the local-file and S3 attachment stores; every other implementation inherits Inline untouched, so inline and in-memory hosts work unchanged with no new ceremony.

The per-invocation durable controller is deliberately invisible at facade build(): a durable controller is constructed per-invocation from a host ctx, so the build-time controller is inline by construction. Build therefore validates peer-coherence among the stores it can see (a durable session store factory requires a durable attachment store and a durable artifact store, and a durable process registry requires a durable session store factory and a durable trigger store), while the effect-host-vs-store check happens later at the turn-scope and worker boundaries where the controller's tier is finally known. The rejected emitter-supplied durability stopgap was deliberately not added: the worker owns execution durability, not whoever emitted the event.

Sessions opened through the regular LashCore / LashSession path also get durability through the session store, but at a coarser grain: a turn either commits to the store atomically or fails as a whole. Run with a scoped RuntimeEffectController when you need crash recovery inside a turn: when a single LLM call is expensive enough to replay, or when an individual tool call can run for minutes and you can't afford to lose its progress.