Persist sessions by installing a SessionStoreFactory on the core or passing a store per session. Without one, sessions are in-process only. This page is the app-facing guide; effect replay and workflow durability live in the architecture docs.
What This Is
Session persistence stores committed runtime state for a session id: transcript graph, AgentFrame records, plugin/tool snapshots, usage deltas, pending turn inputs, queued work, metadata, and attachment references.
Persistence Scope
A session store makes a session id reopenable. It is separate from app tables and separate from in-flight workflow replay.
factory-backed
Use .store_factory(...) when sessions reopen across requests, process restarts, pending turn inputs, queued work, process wakes, or managed child sessions.
in-memory
Omit a session store for throwaway tests and local single-process demos where every session can reset on restart.
Schema Compatibility
lash rejects a persistent database unless its schema version matches exactly after any applicable explicit migration: SQLite uses PRAGMA user_version, and Postgres records component versions. Postgres additionally verifies the live catalog's structure at open, because a host that provisions the database itself writes that version stamp too.
This identity cutover also applies to external durable substrates. Before deploying, drain and recreate Restate LashDurableWaitIndex and LashDurableWaitWorkflow state. The v3 await-event key epoch is stored under the v2 wait-index namespace. Post-cutover index operations and workflows that wake cross the epoch gate and reject pre-cutover state with a recreate instruction. A fully parked pre-cutover workflow cannot run new code and v3 resolutions do not address its v2 key, so it never self-terminates; draining and purging those invocations before deployment is the only remedy.
Minimal Example
Install one session-store factory plus explicit artifact and attachment stores on the core; every session inherits them.
use std::sync::Arc;
use lash_sqlite_store::{SqliteSessionStoreFactory, Store};
let store_factory = Arc::new(SqliteSessionStoreFactory::new(data_dir.join("sessions")));
let artifact_store = Arc::new(Store::open(&data_dir.join("artifacts.db")).await?);
let factory = lash::rlm::RlmProtocolPluginFactory::new(
lash::rlm::RlmProtocolPluginConfig::builder()
.instruction_limit(lash::rlm::InstructionBound::instructions(1_000_000))
.wall_clock(lash::rlm::WallClockBound::secs(30))
.memory_limit(lash::rlm::MemoryBound::mebibytes(64))
.build(),
artifact_store.clone(),
);
let core = lash::LashCore::rlm_builder(lash::TurnBudget::Unbounded, factory)
.provider(provider)
.model(
lash::ModelSpec::builder(model.clone())
.context_window_tokens(200_000)
.build()
.expect("valid model metadata"),
)
.store_factory(store_factory)
.effect_host(Arc::new(lash::durability::InlineEffectHost::default()))
.attachment_store(Arc::new(lash::persistence::FileAttachmentStore::new(
data_dir.join("attachments"),
)))
.process_env_store(artifact_store)
// Start bounded; tune both limits for your backend's latency envelope.
.commit_budget(lash::CommitBudget::bounded(1024 * 1024, 512))
.queued_work_batching(lash::QueuedWorkBatchingConfig::new(1024))
.build(crate::example_process_owner())?;
Enumerate Sessions
Read the durable catalog through LashCore without opening or hydrating any session. The default includes live sessions and deletion tombstones; use a relation/deletion filter when a host needs a narrower roster.
Use the core-level factory for app servers. Use a per-session override only for tests or hosts that already own a concrete store.
Per-session override:
let session = core
.session(chat_id)
.store(Arc::new(my_custom_persistence))
.open()
.await?;
The examples use the first-party SQLite adapter because it is the smallest local durable setup. The runtime contract is SessionStoreFactory / RuntimePersistence; a host can provide another database-backed implementation when it preserves the same advisory session execution lease, head-CAS commits, checkpoint/blob references, pending turn-input claims, queued work, and idempotency semantics.
Without .store_factory(...) or .store(...), turns still succeed but state stays in memory. You still provide explicit in-memory effect, artifact, and attachment facets.
Distributed Backends
Use lash-postgres-store plus lash-s3-store when multiple workers must share one durable runtime state and one attachment byte store.
PostgresStorage::connect creates the exact supported schema on an empty database and rejects mismatched component versions. Runtime checkpoints, pending turn inputs, queued work, process rows, triggers, attachment manifests, and Lashlang artifacts live in Postgres. Attachment bytes live in S3-compatible object storage under content-addressed keys; MinIO uses the same implementation with an endpoint URL and path-style mode.
PostgresStoreConfig applies connection-level backstops by default: a 10 second lock_timeout and a 30 second statement_timeout. Mutating session work first claims the advisory session execution lease; session commits perform the head-revision CAS and any claim-ownership checks in one transaction. Postgres serialization failures, deadlocks, and lock-acquisition timeouts on the backstop write surface as retryable conflicts so hosts can reload and retry instead of treating ordinary write contention as an opaque backend failure.
The distributed integration harness is just restate-postgres-workers-e2e. It starts Postgres, MinIO, Restate, a deterministic OpenAI-compatible mock provider, two worker processes behind a Caddy h2c proxy, and a runner. The workers build a normal downstream-style LashCore with PostgresStorage, S3AttachmentStore, PostgresLashlangArtifactStore, PostgresTriggerStore, PostgresProcessRegistry, RestateRuntimeEffectController, RestateProcessDeployment, and JSONL trace sinks.
The scenario runs real session.turn(input).stream_to(...) turns through the public facade. It covers foreground tools, attachment creation through the configured attachment store, parent/nested/parallel Lashlang processes, durable sleep and wake paths, public trigger registration and delivery, live replay from a stored observation cursor, MinIO byte/metadata assertions, and failover where worker A exits from the real turn path after terminal finish or another durable effect and worker B completes through Restate replay. The runner asserts one provider call per journaled effect, one committed turn, no duplicate runtime rows, and no active Lash Restate invocations left in pending, ready, running, backing-off, or suspended.
Retain and Fork a Turn
Forking creates a new session head over shared durable history. It writes no transcript nodes and does not move the source session.
A live session tip is forkable through its head checkpoint at no extra retention cost. To preserve a turn after the source advances, pin it while it is still the live tip. fork_points() enumerates pinned past turns and current live tips; unpin() releases only the explicit pin.
let snapshot = session.admin().state().export().await;
let turn_node_id = snapshot
.session_graph
.leaf_node_id
.clone()
.expect("completed turn");
core.pin(&turn_node_id).await?;
// The source may now run more turns.
core.fork_at(&turn_node_id, branch_chat_id).await?;
core.unpin(&turn_node_id).await?;
The fork inherits the retained checkpoint and AgentFrame lineage. It starts with an empty usage ledger and no execution-scoped queue, lease, receipt, or wake-subscription state. Its explicit observer selector (All, None, or Only) and chosen process ids are recorded in the durable fork relation; session open idempotently reconciles an unfinished observer intent after a crash. Wake delivery remains addressed only to the subscribed source session.
History ownership is reachability, not the session id that first wrote a node. Child edges, live session heads, and explicit pins are counted references. Deleting one sibling stops reclamation at the first shared node still referenced by another head or pin.
A host-level rewind is explicit composition: pin the target, fork there under a new id, switch the product UI to that id, then delete the superseded session. The superseded host-facing id is permanently retired and cannot be reopened in that store. The agent-service example also snapshots its app-owned messages and board when the user pins a turn; Lash does not copy host tables.
Delete Sessions
Use LashCore::delete_session(..., scope) when an app-level delete should remove factory-backed runtime state for a session id.
let effect_host = core.effect_host();
let scope = effect_host.scoped(core.session_delete_scope(chat_id).await?)?;
let report = core.delete_session(chat_id, scope).await?;
if let Some(process_report) = report.process {
audit_process_cleanup(process_report)?;
}
Deletion requires a core-level SessionStoreFactory; per-session stores cannot be rediscovered from a bare id. If a ProcessRegistry is installed, Lash removes that session's weak observer edges, clears subscriptions addressed to it, discards its pending wake deliveries, and deletes its trigger subscriptions. Process records and event logs remain byte-identical; deletion never cancels a process.
A materialized host-facing session id is single-use: deletion writes a permanent tombstone, and later create, fork, queue, attachment, lease, metadata, or commit writes fail with StoreError::SessionDeleted. Deleting an id that was never materialized is a no-op. A reset must rotate to a new id. Lash-minted runtime-internal process session ids are single-use on the same terms: process pruning deletes them and writes the same tombstone, because the deleted set is also the frontier a delete reads to reclaim tombstoned history rows whose owning session is already gone.
The tombstone, await-event revocation ledger, effect journal, and Restate state form one trust domain and must be reset together. SQLite deployments must not wipe the catalog and effect databases independently. The v3 await-event promise-key cutover also requires draining and purging in-flight Restate invocations and recreating Restate wait-index/workflow state before upgrade. Operations that execute after deployment reject the old epoch, but a fully parked v2 invocation executes no new guard and cannot be reached by a v3 resolution; it remains parked until the operator purges it.
What Gets Persisted
Think in runtime concepts first. The exact SQL layout belongs to the store implementation.
graph
User inputs, assistant responses, tool calls, protocol events, prompt snapshots, and frame boundaries.
head
Current revision, current AgentFrame id, session policy, model/provider identity, and metadata for resume UIs.
snapshots
Plugin state, tool state, current RLM execution state, prompt state, and Lashlang artifacts needed for resume.
pending turn input
User-visible model input awaiting an active checkpoint or the next idle turn. Active-turn input is anchored to a live turn id and checkpoint boundary; interrupt finalization completes accepted inputs and defers only unaccepted ones to the next turn.
queued work
Durable ingress for SessionCommand mutations and non-user TurnWork such as process wakes, with leases and completed claim ids cleared by the consuming commit. Triggers are runtime-level occurrences, and timers are host-owned scheduling.
process envs
Process execution environments are captured at start as content-addressed Lashlang artifacts. Persisted process and trigger rows reference those immutable environment blobs; this alpha cutover is not backward-compatible with pre-env-ref process rows.
usage
Per-turn token deltas for uncached input, total output, cache-read input, cache-write input, and reasoning-output counts that aggregate into session.usage_report().
Private Tables
Every table lash creates is named lash_*, and all of them are private to lash for writes and reads alike.
Writes are the sharper edge. A hand-written UPDATE lands outside the revision fence and the idempotency receipt the store commits in the same transaction, so it can leave trigger, process, or session state that no retry, replay, or receipt read can reconstruct. Lash has no repair path for state changed that way, and the damage surfaces later rather than at the write.
Every question worth asking a lash_* table has a supported answer:
trigger state
TriggerCommand::List through TriggerStore::execute_command, or core.triggers().subscriptions(filter) for the projected registration view. Every mutation is another TriggerCommand: Register, Update, Enable, Disable, Delete, Revive, Prune.
session state
session.admin().state().export() for the committed snapshot, plus the RuntimePersistence contract itself when a host owns a concrete store. Session mutation stays inside turns and SessionCommands.
lease state
core.session_lease_diagnostics(session_id) for a session's execution lane (owner id, boot incarnation, executor id, generation, expiry, renewal state), and core.processes() for a process lease. Both are snapshot reads that never mutate the row; the commit compare-and-set stays the authority, so no host behavior may be gated on either. Triaging a stuck turn is the procedure they serve.
process state
core.processes() for observation, signalling, retention, and the deletion change feed. It is the whole process surface; the registry's rows are its implementation.
observability reads
The push seams, not a scan: ProcessEventSink, TraceSink, and session observation. A host projection that needs its own indexed, query-shaped tables builds them from those events and queries its own tables. See reporting channels.
Re-enabling a disabled trigger subscription is the case hosts most often reach for SQL to solve. It is a first-class fenced verb: read the live revision, then Enable at that revision under an idempotent operation id.
use lash::triggers::{TriggerCommand, TriggerCommandOutcome, TriggerSubscriptionFilter};
// Read the live revision through the command surface, not with SQL.
let TriggerCommandOutcome::List { records } = store
.execute_command(
"reenable-read",
TriggerCommand::List {
owner_scope: owner_scope.clone(),
filter: TriggerSubscriptionFilter {
subscription_key: Some(subscription_key.to_string()),
..TriggerSubscriptionFilter::default()
},
},
)
.await??
else {
unreachable!("List returns list records")
};
let Some(record) = records.first() else {
anyhow::bail!("no live subscription for `{subscription_key}`");
};
// Enable is fenced on the revision just read, and the operation id makes the
// retry idempotent.
let outcome = store
.execute_command(
&format!("reenable:{subscription_key}:{}", record.revision),
TriggerCommand::Enable {
owner_scope,
actor,
subscription_key: subscription_key.to_string(),
expected_revision: record.revision,
},
)
.await?;
match outcome {
Ok(TriggerCommandOutcome::Mutation { receipt }) => {
assert!(receipt.enabled);
}
Ok(_) => unreachable!("Enable returns one mutation receipt"),
// A concurrent writer moved the row first: re-read and retry. Never
// patch the row by hand.
Err(conflict) => retry_reenable_later(conflict)?,
}
A stale expected_revision returns a typed conflict carrying the current revision, so the correct response is re-read and retry. Hosts upgrading away from hand-written trigger SQL will find the operation-by-operation mapping under retiring host SQL.
Session Lane And CAS
At most one mutating runner holds the advisory session execution lease at a time. Stores check the expected head revision in the same transaction as the atomic commit; mismatch rolls back with no partial state.
Concurrent turns from two workers on the same session id normally serialize at the lease boundary. If a lane-less persistence boundary observes a busy holder after its work is already complete, it proceeds to the session-head CAS instead of discarding that work; a stale head returns a typed conflict with no partial write. Lease loss still vetoes a nested plugin or observer commit borrowing a held lane, because that write claims the holder's authority. Durable effect replay uses its separate authoritative lease lane and is unchanged.
Every host supplies LashCoreBuilder::build a LeaseOwnerIdentity: the owner id is stable per worker or process, never per turn, and the incarnation changes once per boot. Every runtime open mints a fresh executor id below that host identity. Reentry requires the exact triple (owner_id, incarnation_id, executor_id); a parked runtime resumed by that same open retains its executor, while a second open in the same process is a distinct Busy claimant. The core carries the host owner into every session runtime it constructs.
use lash::runtime::RuntimeErrorCode;
match session.turn(input).run().await {
Ok(turn) => persist(turn)?,
Err(lash::EmbedError::Runtime(err))
if err.code == RuntimeErrorCode::SessionExecutionLeaseLost =>
{
// The durable lane moved to another owner before commit: reopen and retry.
let session = core.session(chat_id).open().await?;
retry_or_report(err, session)?;
}
Err(lash::EmbedError::Runtime(err))
if err.code == RuntimeErrorCode::SessionExecutionLaneBusy =>
{
// A durable workflow controller's queued drain found the lane held by
// a live executor and stopped waiting: nothing was consumed, so let
// the engine's retry policy re-drive this invocation unchanged.
retry_later(err)?;
}
Err(lash::EmbedError::Runtime(err))
if err.code == RuntimeErrorCode::StoreCommitContended =>
{
// The failed commit published nothing: retry the same operation unchanged.
retry_later(err)?;
}
Err(lash::EmbedError::Runtime(err)) if err.code == RuntimeErrorCode::StoreCommitFailed => {
// The CAS backstop fired: reload and retry.
let session = core.session(chat_id).open().await?;
retry_or_report(err, session)?;
}
Err(other) => bail!(other),
}
The retry ladder distinguishes authority loss from safe redrive: session_execution_lease_lost requires reopening durable state, while session_execution_lane_busy means a durable queued drain consumed nothing and left its row pending, so the workflow engine may retry the same invocation unchanged. store_commit_contended is likewise safe to retry unchanged; an ambiguous store_commit_failed requires reload before retry.
Hosts expecting concurrent access should serialize at their layer or re-open the session after conflict and decide how to merge intent.
Garbage Collection
Run reclamation from a process, maintenance route, or CLI command after pruning branches or deleting sessions.
gc_unreachable()
Audits reachability from heads and active checkpoints and returns a GcReport. Run after pruning a branch. Correctness belongs to the transaction that severs ownership; a non-zero deletion count is a verify/repair finding that the owner-delete path left residue, not routine retention throughput.
Answers in the shared maintenance outcome contract: Ok with non-zero counters is an audit finding, Ok with zero counters is a witnessed nothing-to-do, and every stop is a MaintenanceFailure carrying the report accumulated before it — MaintenanceStop::Refused when a destructive step's precondition could not be proven, MaintenanceStop::Failed when the backend errored. No backend absorbs its own error into an empty report, so an audit that found nothing for weeks can no longer look healthy after an internal failure. A backend that implements no reclamation fails with StoreError::UnsupportedStoreOperation rather than reporting an audit it never ran.
vacuum()
Removes tombstoned graph-node rows already detached by gc and prunes terminal pending-turn-input evidence rows for the session this store handle is bound to, returns a VacuumReport. Run after gc_unreachable to reclaim file size. Every backend scopes the sweep to that one session and never reclaims catalog-wide; a handle built without a session binding refuses with StoreError::SessionNotBound (carried in a MaintenanceFailure, like every other stop) instead of widening its scope. Because of that scoping, deleting a session also reclaims its own tombstoned nodes plus any tombstoned node owned by an already-deleted session: a node can be tombstoned after its owner is gone (unpinning a deleted session's pinned leaf, or fork ancestry retired when the child session is deleted), and that residue is unreachable by any session-scoped vacuum, so the next session delete drains it.
The process registry is a separate store with its own retention lever. After the host advances the mandatory projection watermark, core.processes().prune(cutoff_epoch_ms, ProjectionWatermark::UpTo(cursor)) replaces eligible terminal rows with payload-free tombstones. It removes their events, wakes, observer edges, leases, and settled parent-end plans. The public facade then runs trigger-retention reconciliation. That transaction deletes the exact trigger deliveries guarded by process tombstones. It also deletes every occurrence whose committed fan-out is now empty, including occurrences that matched zero subscriptions at ingest.
The same transaction applies the ADR 0049 deleted-session frontier. A session-owned subscription and its mutation receipts survive while any delivery in that owner scope remains. After the owner is permanently deleted and the transaction witnesses the delivery set empty, it reclaims the subscription and receipts. Host and platform subscription tombstones remain permanent Revive name fences. Their mutation receipts remain governed by the existing prune_mutation_receipts host cutoff.
A deployment without a projector must choose ProjectionWatermark::NoProjector explicitly. The deletion change feed lets host projections drop their rows. A terminal process with a pending parent-end plan remains retained until that plan settles, and non-terminal rows are never touched. Run core.processes().compact_tombstones(cutoff_epoch_ms, watermark) on the same maintenance cadence. It refuses to compact tombstones still referenced by outstanding trigger deliveries. If a configured trigger store or deleted-session frontier is unhealthy, compaction stops and retains tombstones until it recovers.
Compaction frees registry rows, not process ids. Pruning already deleted the process-owned session ids (process-env:<id> and process-session-turn:<id>), and their session tombstones are permanent. A re-registered process id whose tombstone was compacted away therefore starts a process whose session stores cannot be created. It fails with StoreError::SessionDeleted naming an internal id the host never chose. Mint a fresh process id rather than reusing one. Keep the retention window comfortably longer than any in-flight ProcessWorkDriver::await_terminal. A late await receives the typed ProcessNoLongerRetained information outcome.
For Restate process and turn attachments, an attach-ceiling-elapsed error ends only that transport connection. The durable wait remains live, so hosts re-attach with the same process id or turn address rather than reporting terminal failure. See docs/adr/0017-process-observation-is-best-effort-push-over-state-truth.md and ADR 0021.
TriggerStore::reclaim_trigger_occurrences(cutoff_epoch_ms) remains the explicit grace-cutoff maintenance surface for hosts that want an occurrence-only pass. Terminality still comes from committed empty fan-out, never from the cutoff. The normal process-retention reconciliation does not depend on that optional pass for correctness.
End-To-End Example
The agent-service example keeps app storage and runtime storage separate: chat rows in one database, Lash sessions in the runtime store.
// One factory at boot, shared across every chat.
let store_factory = Arc::new(SqliteSessionStoreFactory::new(
data_dir.join("lash-sessions"),
));
let artifact_store =
Arc::new(lash_sqlite_store::Store::open(&data_dir.join("lash-artifacts.db")).await?);
let factory = lash::rlm::RlmProtocolPluginFactory::new(
lash::rlm::RlmProtocolPluginConfig::builder()
.instruction_limit(lash::rlm::InstructionBound::instructions(1_000_000))
.wall_clock(lash::rlm::WallClockBound::secs(30))
.memory_limit(lash::rlm::MemoryBound::mebibytes(64))
.build(),
artifact_store.clone(),
);
let core = lash::LashCore::rlm_builder(lash::TurnBudget::Unbounded, factory)
.provider(provider)
.model(
lash::ModelSpec::builder(model.clone())
.variant(lash::provider::ReasoningSelection::Effort(
model_variant.clone(),
))
.context_window_tokens(200_000)
.build()
.expect("valid model metadata")
.with_capability(adaptive_reasoning_capability()),
)
.store_factory(store_factory)
.effect_host(Arc::new(lash::durability::InlineEffectHost::default()))
.attachment_store(Arc::new(lash::persistence::FileAttachmentStore::new(
data_dir.join("attachments"),
)))
.process_env_store(artifact_store)
// Start bounded; tune both limits for your backend's latency envelope.
.commit_budget(lash::CommitBudget::bounded(1024 * 1024, 512))
.queued_work_batching(lash::QueuedWorkBatchingConfig::new(1024))
.build(crate::example_process_owner())?;
// Per request: open a session keyed by the app's chat id.
let session = core.session(chat_id).open().await?;
Full source: examples/agent-service.
Where Next
This page owns app-facing persistence. Replay and store internals are intentionally elsewhere.