durability/workflows

That shape is exactly what durable workflow runtimes (Temporal, Restate, similar) want from a library they're embedding. Lash ships a Restate adapter in lash-restate; other workflow runtimes implement the same controller trait. The workflow ledger can key each invocation, persist its result, and replay with the completed effect already in hand while Lash manages the state-machine checkpoint and response application.

Embedders who only need local durability configure InlineEffectHost plus the regular session store. Embedders who need durable in-flight effects create a handler-scoped controller, then pass it through .turn_id(...).effects(&controller).stream_to(&sink) / .run() on the facade. Lower-level runtime adapters can still build explicit ScopedEffectController values for TurnOptions::new(cancel, scope). RestateRuntimeEffectController is the first-party workflow adapter shape.

Fire-and-forget sans-IO effects such as Emit, Log, Progress, and Done remain driver and commit-pipeline responsibilities. They do not cross RuntimeEffectController because they do not ask the host to run nondeterministic work and return a response.

Every controller invocation receives a serializable RuntimeInvocation: scope identifies the session and optional turn coordinates, subject identifies the effect, caused_by carries the typed causal parent, and replay.key provides the stable replay discriminator. The live RuntimeEffectEnvelope is serializable and carries ref-only request data. Cancellation, streams, provider trace sinks, and attachment byte hydration stay in the local executor or host workflow.

The built turn replay key includes session_id, turn_id, turn_index, protocol_iteration, effect kind, and EffectId; per-tool invocations append the model tool call id. Durable tool steps derive child replay keys from the caller's invocation plus the prepared tool call id and step_id; the stable envelope hash also includes the JSON input. Scoped durable runs require a non-empty stable turn_id. Direct completions use a framed identity over session, optional turn, usage source, typed causal parent, and either an explicit replay key or a deterministic per-lane ordinal. Request content is deliberately excluded: unkeyed call order must be deterministic under redrive, and conditional or concurrent branches need stable explicit keys.

EffectId is local to one turn. Across turns, the counter resets to 1; use the full replay key, not the bare id, for globally unique effect identity.

impl<M: TurnProtocol> TurnMachine<M> {
    pub fn checkpoint(&self) -> TurnCheckpoint<M>;

    pub fn restore_from_checkpoint(
        config: TurnMachineConfig<M>,
        checkpoint: TurnCheckpoint<M>,
    ) -> Self;
}

TurnCheckpoint<M> is fully serde-compatible. Fields:

• state: MachineState<M>: the waiting variant, with the data each retains.
• pending_effects: Vec<Effect<M>>: fire-and-forget effects queued but not yet drained (Emit, Log, Progress).
• next_effect_id: u64: counter, preserves id determinism.
• messages: Vec<Message>: prompt history committed so far.
• events: semantic event log committed so far.
• protocol_iteration, protocol_run_offset: turn position.
• cumulative_usage: TokenUsage: running token totals for this turn.
• termination: TurnTerminationPolicyState: max-turns / forced-final state.
• synced_protocol_iteration: Option<usize>: last iteration the execution environment was synced.
• next_synthetic_message_id: u64, turn_causes: Vec<TurnCause>, progress_event_cursor: usize: internal bookkeeping that also round-trips through serde.

restore_from_checkpoint remains a sans-IO machine primitive for tests and protocol work. Runtime crash recovery does not persist or reload in-flight machine payloads from the store. Local/inline runs reopen the last committed session state; durable workflow integrations replay the host handler and effect history, then Lash retries the final commit.

• Mid-LLM crash → restore yields a new LlmCall with the same EffectId and identical request. The workflow's activity for that id is the same lookup and produces the same response.
• Post-LLM crash, pre-checkpoint → restore yields the pending Checkpoint effect, not another LlmCall. The model is not re-invoked.
• Mid-tool-call crash → restore yields the same ToolCalls sans-IO effect carrying every original call. The runtime invokes the effect controller separately for each PendingToolCall, keyed by call id, so hosts can short-circuit completed calls and only re-invoke pending ones.
• RLM exec crash → restore yields an ExecCode effect with the original code block and the driver's trajectory state. The exec result contains the parallel tool outputs in their structured form (success / failure / cancellation); no replayed ToolCallStarted events leak from the previous attempt.
• Plugin checkpoint crash → restore yields the Checkpoint effect again. The on_empty resume action is preserved, so a checkpoint that contributed no messages still finishes the turn the same way.

#[async_trait::async_trait]
impl RuntimeEffectController for WorkflowEffectController {
    async fn execute_effect(
        &self,
        envelope: RuntimeEffectEnvelope,
        local_executor: RuntimeEffectLocalExecutor<'_>,
    ) -> Result<RuntimeEffectOutcome, RuntimeEffectControllerError> {
        let key = envelope.invocation.replay_key().unwrap_or("runtime-effect").to_string();
        self.workflow
            .run_activity(key, envelope.invocation.clone(), || async move {
                local_executor.execute(envelope).await
            })
            .await
    }
}

The shape matches Temporal's workflow + activity split and Restate's handler + side-effect split. Each typed runtime effect can become one durable activity: the invocation carries typed scope, subject, causal parent, and replay key, and replay.key joins the activity result back to the logical turn effect. Restate immediately awaits concrete ctx.run(...) calls around the normal local executor for recorded non-sleep effects, uses Restate timers for sleeps, and keeps direct Restate primitives for workflow calls and awaited events. The store is touched only when Lash commits settled session state.

Deterministic Lash/Restate contract failures are terminal handler errors. A recorded effect whose replay hash no longer matches the current envelope, or a recorded Restate run that completed with a terminal failure, returns an explicit terminal error instead of panicking into an infinite backing-off invocation. Hosts should record a failed turn/event and clear their UI running state when the workflow reports such a terminal failure.

Restate hosts also need durable invocation identity outside the handler. RestateIngressClient submits workflow/service/object /send requests and parses the response body's invocationId. Foreground-turn Stop uses RestateTurnDeployment::turn_work_driver(), which addresses Lash's reserved keyed promises by session and turn id and can attach to the committed terminal evidence. RestateAdminClient invocation cancel/kill remains engine-level break-glass and must never be reported as a Lash Cancelled result.

Process effects cross the same controller boundary for start/list/transfer/delete-session/await/cancel/signal, but durable execution of a background process is owned by the process work driver and worker, not by the turn that happened to start it. The registry stores durable state; waits live on the driver/awaiter seam. The process section below covers the durable intent record, observer edges, worker rebuild inputs, Restate workflow metadata, and event-sourced terminal state.

Successful completion uses RuntimeCommit.turn_commit. Ordinary turn commits accept identical retries for the same (session_id, turn_id) and reject a changed commit hash. Session-graph appends add a versioned semantic request identity: an identity match returns the stored first result without applying any of the retry envelope, while changed content at the same version is a typed caller conflict and an encoding-version mismatch falls back to exact-hash semantics. A workflow replay can therefore retry the final Lash commit without duplicating or mutating settled state.

Tool attempts are atomic journal entries. Host code inside an attempt is opaque, including direct completions and retry delays; it does not borrow the controller to create nested durable steps. A completed attempt replays its recorded result, while an attempt retry or crash-before-record runs its in-attempt effects again. Work that needs multiple independently durable effects or waits is decomposed into process steps.

A process is self-contained: the record carries its ProcessOriginator (host or session) as pure provenance and a captured execution-environment reference. Wake subscription and session observation are indexed edge state, not lifecycle fields. Deleting a session removes its weak observer rows, clears its subscriptions, and discards its pending wake deliveries without changing process records or event logs; it never cancels a process. See docs/adr/0011-self-contained-processes.md and docs/adr/0046-process-transitions-are-events-record-is-a-fold.md.

Records are event-sourced: terminal state and cancellation are keyed process events on the record (process.cancel_requested routes to the workflow's shared cancel handler), not mutable state fields. Tool and Lashlang process rows carry a captured execution-environment reference (a content-addressed process_execution_env artifact holding only Process Plugin Options and policy), and workers instantiate an ephemeral runtime from that immutable environment plus host-owned worker config; they never rebuild the originating session. Durable tool authority snapshots live behind plugin-owned options, not as ad hoc process-environment metadata. A Restate-backed start stores external_ref.backend == "restate" with external_ref.id == "LashProcessWorkflow/{process_id}". Wake semantics write durable wake-inbox rows only when the process has a wake target; a process without one still records wake events, but they deliver nowhere. Checkpoints drain inbox rows beside in-memory turn injections.

Waiting is deliberately not a registry capability (docs/adr/0016-process-waits-live-on-the-work-driver-seam.md). Store-backed registries expose point reads and writes only. The facade wraps inline registries with a ProcessChangeHub so local mutations wake ProcessAwaiter waiters, and the awaiter falls back to narrow point-read backoff when another process may be changing the store. External drivers can attach terminal awaits to their own durable promise; Restate attaches host awaits to LashProcessWorkflow/{process_id}/await_terminal.

Lashlang process starts validate before registration in the Lashlang runtime path: it rebuilds a candidate plugin session and Tool Catalog from the captured Process Plugin Options, constructs the candidate LashlangHostEnvironment, and checks it satisfies the target artifact's recorded Host Requirements before writing ProcessInput::Engine { kind: "lashlang", payload }. The registered ProcessEngine repeats the Host Requirements check during execution and recovery, so a recovered row cannot run against a drifted or incompatible rebuilt environment.

Turn in-flight replay is effect-host owned. A process is store-backed background work: it is fenced by a ProcessLease, resumes from registry events, and recovers through DurableProcessWorker::drive_pending_processes. Process leases carry owner id, lease token, fencing token, claim time, and expiry; stale process writers fail before committing durable state.

A turn supplies its per-invocation durable controller through ScopedEffectController. Out-of-turn starts (trigger deliveries, facade start, and recovery sweeps) write durable registry intent and execute through the worker's wired controller. A durable host never silently re-runs a process on an inline controller.

User-facing cancellation is intentionally separate from execution recovery. Session tool surfaces validate observer visibility before calling the scoped ProcessService cancellation operation; cancel-all lists live visible processes and cancels each directly. There is no host-injectable cancellation policy in the runtime.

The recovery sweep is disposition-driven, not a re-run-everything loop (docs/adr/0019-process-recovery-obeys-declared-disposition.md): every registration declares a required RecoveryDisposition and the sweep applies it mechanically. It lists non-terminal processes, claims each available lease, and skips work held by another live owner; then a Rerunnable row is re-run and completed exactly as before, a started OwnerBound row remains non-terminal unless owner drain or a lapsed-lease Abandon Request authorizes terminalization (a not-yet-started one stays claimable by any peer, since a first execution is not a re-execution), and an ExternallyOwned row is never claimed or executed. Lease expiry alone never terminalizes. Abandoned is a fourth terminal state, peer to Completed | Failed | Cancelled, carrying the writer (owner drain, reconciled request, or exhausted execution engine) and the evidence that licensed it. The sweep is idempotent by persisted process_id; the Restate invocation is keyed as LashProcessWorkflow/{process_id}. The full verdict table and the crash-vs-drain paths are in running in production.

A subagent spawn is itself such a process. agents.spawn builds a SessionCreateRequest plus a TurnInput and emits one ProcessInput::SessionTurn, so a spawned child runs through the same generic process path (and therefore inherits provider re-supply, durability, and recovery) rather than a bespoke child-creation route. Recursion safety (the subagent depth ceiling and tool-hiding at that ceiling) and the submit_error terminal are carried as request config, not lost.

This is a mechanical fact, not an end-to-end guarantee. Store, registry, attachment, and artifact traits do not advertise a shared durability tier, and the runtime does not compare independently wired facets to infer deployment guarantees. Hosts establish those guarantees by choosing persistent substrates and validating them with the public conformance suites.

The inline host defaults to runtime-owned replay. SQLite, PostgreSQL, and Restate effect hosts explicitly report controller-owned replay; scoped and task controllers preserve that ownership. Restate effect hosts always require an ingress connection, and SQLite effect hosts are always file-backed, so neither concrete host has a constructor that contradicts its replay behavior.

Sessions opened through the regular LashCore / LashSession path also get durability through the session store, but at a coarser grain: a turn either commits to the store atomically or fails as a whole. Run with a scoped RuntimeEffectController when you need crash recovery inside a turn: when a single LLM call is expensive enough to replay, or when an individual tool call can run for minutes and you can't afford to lose its progress.