lash is a durable-object / actor-per-session architecture: the durable truth lives behind async store traits, and the in-process runtime holds only a rehydratable working copy. Running lash as a horizontally-scaling microservice is now a concrete deployment shape: Postgres for runtime/process/trigger state, S3-compatible attachment bytes, Restate for durable execution and failover, and session-agnostic ingress for external signals.
LashRuntime.state (RuntimeSessionState) is a rehydratable working copy, not a source of truth. It is loaded via RuntimePersistence::load_session and refreshed via refresh_session_graph_from_store; durable loads rehydrate the session's leaf-to-root history.
The other in-memory fields on LashRuntime are either ephemeral per-turn accumulators (shared_token_ledger, drained into state at commit) or per-worker coordination (managed_sessions, managed_turns, what this instance is currently running). None of it is durable truth.
Durable truth lives behind a single trait, RuntimePersistence (crates/lash-core/src/store/mod.rs), whose surface is exactly a durable-object store:
load_session, load_node: rehydrate working state.try_claim_session_execution_lease, renew_session_execution_lease, release_session_execution_lease: the durable single-writer execution lane for a session. Foreground turns, pending user input dispatch, wake delivery, and mutating session commands must hold this lease before touching session work.commit_runtime_state: operation-stamped session-head commit. The head revision CAS is the exclusive commit authority; the session execution lease only coordinates who should spend work.enqueue_queued_work, claim_ready_queued_work, renew_/abandon_/cancel_*: a durable work queue with claim_id, owner identity plus incarnation, claim_token, and claim_fencing_token. Claims pin the session-lease generation for reclaim eligibility; settlement validates claim ownership.save_/load_session_meta, vacuum, gc_unreachable.Any backend that satisfies this trait (and the ProcessRegistry / LashlangArtifactStore traits) is a valid store. The backend-agnostic conformance suite (lash_core::testing::conformance) is what guarantees a new backend behaves identically.
A request for session_id routes to any worker. The worker first claims the advisory session execution lease, then rehydrates the session from the store, runs the turn (pure lash-sansio compute + replayed effects), and commits under the head CAS plus any claim-ownership checks. Workers are identical and hold no durable state, so the fleet scales by adding replicas.
The store box is shared by the whole fleet, so it must be a networked backend every worker can reach. lash-postgres-store supplies the distributed RuntimePersistence, ProcessRegistry, TriggerStore, and LashlangArtifactStore; lash-s3-store supplies the shared attachment byte store. The local-file SQLite store remains right for embedded and single-host deployments.
Concurrent writers are normally prevented before effects: the advisory session execution lease lets one runtime owner execute mutating work for a session at a time. commit_runtime_state performs head CAS and any claim-ownership checks in the same transaction, so stale writers fail without partial mutation even after lease handoff.
Every host must pass LashCoreBuilder::build a LeaseOwnerIdentity. Keep its owner id stable per worker or process, never per turn, and change its incarnation id once per boot. A busy lane remains claimed and observable, but lane-less commits proceed to the session head CAS without displacing its holder. Choosing owner identity and lease timings for failover latency is host policy; see running in production.
You scale across sessions (shard/route by session_id), not within one (see Inherent Limits).
The seams a stateless fleet needs are already present and deliberate. The sans-IO / store / effect-replay split was built with exactly this shape in mind.
| Primitive | Role | Where |
|---|---|---|
lash-sansio pure TurnMachine | compute is deterministic (state, input) → (state, effects) | crates/lash-sansio |
RuntimePersistence | the single durable store seam; backend-swappable | store/mod.rs, conformance suite |
| session execution lease | single durable execution lane for every mutating session path | store/mod.rs, session_execution_leases |
commit_runtime_state CAS on session_head plus claim ownership | atomic session-head write and stale-writer rejection | store/mod.rs |
| queue with claim + fencing tokens | exactly-once work pickup by any lease-holding session runner | queued_work_batches |
effect replay keys + lash-restate | re-run a turn on another worker, replay effects | crates/lash-restate |
lash-postgres-store + lash-s3-store | networked runtime/process/trigger/artifact state plus content-addressed attachment bytes | crates/lash-postgres-store, crates/lash-s3-store |
rehydratable state | no durable-only memory; reload anywhere | runtime/session_ops.rs |
The networked storage and durable execution seams exist. What remains is ordinary production packaging and policy: how the fleet is routed, sized, observed, and operated.
| Choice | What it needs |
|---|---|
| Networked store | Use PostgresStorage for sessions, pending turn inputs, queued work, process registry, triggers, and Lashlang artifacts; use S3AttachmentStore for attachment bytes. The backend-agnostic conformance suites run against Postgres. |
| Turn durability across workers | Deploy turns under Restate with RestateRuntimeEffectController; a crash reruns the handler and replays effect outcomes before the final session commit. |
| Fleet work dispatch | Restate can push handlers for turn and process workflows, while shared queue and process rows retain claim/fencing semantics for recoverable work. The distributed worker E2E exercises two workers behind an h2c proxy. |
| Session-agnostic ingress | External signals must route to whichever worker holds the session, so ingress must not be bolted to one session. See Triggers below. |
| Affinity vs pure reload | A routing policy choice: session_id affinity (warm working-copy cache) vs reload-per-request (simpler, more store reads). The session execution lease supports either. |
| Operations envelope | Choose Postgres and object-store sizing, Restate retention/ingress topology, trace collection, idempotency-key policy, and replay observability for the host product. |
Some boundaries are properties of agent conversations, not of lash.
A session is a serial conversation; the session execution lease enforces one active mutating runner. You scale across many sessions, not within one; ideal for multi-tenant workloads, but a single session's throughput is domain-bounded.
Whatever networked backend is chosen must sustain lease claims, renewals, head-CAS commits, claim-ownership checks, and queue claims at fleet scale. The store choice is the scaling story.
A long turn holds or migrates a worker; a durable effect host is the answer, at the cost of replay on recovery. Restate is the first-party adapter, not the only possible workflow runtime.
Statelessness is about where durable truth lives, not the async runtime. lash-core staying on tokio is orthogonal: each worker is a tokio service; the durable truth is already 100% external to the process.
Triggers are the session-agnostic ingress that completes the stateless model. A trigger occurrence is not attached to a session: it is recorded to its own backend-swappable seam and routed to whichever worker holds the interested session/process. Only the resulting wake is session-coupled.
TriggerStore trait, parallel to RuntimePersistence. The first-party shared-worker implementation is PostgresTriggerStore; another fleet can use a different backend with the same matching and reservation semantics.source_type and source_key against registered subscriptions. Each subscription stores the registrant, captured execution environment, and optional wake target for the process it will start. Wake routing does not imply an observer edge; any initial observers are an explicit host decision. A single event fans out to 0..N subscribers, including subscriptions with no session edges. Session identity enters only through those optional edges, never through the event.occurrence_id from source_type + source key + idempotency key; delivery process id from occurrence_id + subscription_id). Delivery is exactly-once and replay-safe: any worker can run it, and replay under a durable effect host cannot double-start a trigger process.EarliestSafeBoundary: the sole point where session identity and turn-ordering apply to trigger delivery.Net for the fleet: route external signals to a runtime-level emit → match → idempotent effect-delivery → wake. Trigger delivery reuses the existing claim and replay machinery; wake processing then enters the target session through the session execution lease. The Postgres-backed implementation is exercised by the distributed Restate/Postgres/MinIO E2E, but the contract is the TriggerStore, session-store, and effect-host boundary.