Durable stepper and storage adapters for Statifier.
Documentation lives on hexdocs, including the Surviving a restart guide.
Statifier's pure interpreter contract (machine_state, event -> machine_state, effects) makes a persistence-first execution model possible: load a persisted position, step it, execute the effects, persist. Hosts running charts that span days or survive deploys should not need long-lived Session processes at all - but every host currently hand-rolls the loop, the storage guard, and the crash semantics. This package is that loop, packaged.
def deps do
[
{:statifier_persistence, "~> 0.11"},
# Optional, for the Postgres adapter:
{:ecto_sql, "~> 3.10"}
]
endA card-processing transaction: authorize it, capture it before its capture window closes, settle it. The whole run is four calls, and no process holds the chart between them.
alias Statifier.{Chart, Event, Machine, MachineState}
alias Statifier.Invoke.Types, as: InvokeTypes
alias StatifierPersistence.{Runs, Storage}
source = """
<scxml xmlns="http://www.w3.org/2005/07/scxml" version="1.0" initial="authorizing">
<state id="authorizing">
<invoke type="myapp:authorize" id="authorize"/>
<transition event="done.invoke.authorize" target="awaiting_capture"/>
</state>
<state id="awaiting_capture">
<transition event="capture.requested" target="settling"/>
</state>
<state id="settling">
<transition event="ack" target="settled"/>
</state>
<final id="settled"/>
</scxml>
"""Compile the chart once and store its bytes under its own content hash.
Nothing is keyed by a name you choose: the identity comes off the
compiled Machine, which is what makes the guard unskippable.
{:ok, machine} = Statifier.compile(source)
{:ok, chart_blob} = Chart.to_binary(machine)
{:ok, store} = Storage.new(StatifierPersistence.Storage.InMemory, [])
:ok = Storage.save_chart(store, machine, chart_blob)Every effect a step emits reaches your host through one seam - a module
implementing StatifierPersistence.Executor, or an arity-2 fun. Effects
arrive one at a time, in list order, as {tag, payload} tuples. This one
does the least a real host could do with an outbound authorization:
executor = fn
{:invoke, %Statifier.Effect.Invoke{type: "myapp:authorize"} = invoke}, ctx ->
# your own gateway call, keyed for idempotency by run and invocation
MyApp.Payments.authorize(ctx.run_id, invoke.invoke_id)
:ok
_effect, _ctx ->
:ok
end
opts = [executor: executor, invoke_types: InvokeTypes.new(types: ["myapp:authorize"])]create/4 initializes the chart, hands the resulting effects to the
executor, and persists the quiescent position under a run id you choose
- here the transaction's own key:
{:ok, run, state} = Runs.create(store, "txn_01H8", machine, opts)
#=> run.status == :active, active leaf state "authorizing"Each later event is one step/5: liveness check, guarded load, step,
effects out through the seam, persist. Between calls there is no live
process and no in-memory position - only the run record.
{:ok, run, state} =
Runs.step(
store,
"txn_01H8",
machine,
Event.external("done.invoke.authorize", invokeid: "authorize"),
opts
)
#=> run.status == :active, active leaf state "awaiting_capture"Nothing above kept state in the beam, so a deploy in the middle of the run changes nothing about how it continues. Given only the run id, fetch the record, fetch the chart bytes it names, and recompile:
{:ok, record} = Storage.fetch_run(store, "txn_01H8")
{:ok, %{chart_blob: blob}} = Storage.fetch_chart(store, record.content_hash)
{:ok, rebooted} = Chart.from_binary(blob)rebooted is compiled afresh from the stored bytes, not carried over
from before the restart, and it is what makes the stored position
readable again: Statifier interns state ids to indices at compile time,
so a position is only meaningful against the exact chart revision that
produced it. The identity guard enforces that on every load. Step a run
with a machine compiled from a changed chart and it refuses with
{:error, {:identity_mismatch, stored, supplied}} rather than silently
resuming the wrong configuration.
{:ok, run, state} =
Runs.step(store, "txn_01H8", rebooted, Event.external("capture.requested"), opts)
#=> run.status == :active, active leaf state "settling"
{:ok, run, state} = Runs.step(store, "txn_01H8", rebooted, Event.external("ack"), opts)
#=> run.status == :completed, no active leaf states:completed is reached only by the chart reaching a final state - the
lifecycle consumes the interpreter's :done itself and never hands it
to your executor. Runs.fail/4 is the one host-driven terminal
transition, and a step delivered to a terminal run comes back
{:discarded, run} rather than raising.
To read the configuration back as state ids, as the snippets' comments show it:
state
|> MachineState.active_leaf_states()
|> Enum.map(&Machine.id(state.machine, &1))
|> Enum.sort()| Module | Role |
|---|---|
StatifierPersistence.Storage |
The identity-guarded facade: charts, positions, run records. Every load is guarded; there is no unguarded path |
StatifierPersistence.Storage.Adapter |
The behaviour a backing store implements. Storage.InMemory is the reference one, Storage.Ecto the Postgres one (on another backend, minus the lock and the listings) |
StatifierPersistence.Runs |
The lifecycle: create/4, step/5, fail/4, in ADR-0004's fixed order |
StatifierPersistence.Driver |
Run-to-quiescence over Runs: performs the chart's <invoke> calls and steps each answer back in |
StatifierPersistence.Executor |
The seam every effect crosses on its way to your host |
StatifierPersistence.Serialization |
The per-run ordering strategy the fetch-to-persist tail runs inside; defaults to the adapter's own lock_run/3 |
StatifierPersistence.Testing.StorageConformance |
The conformance suite - point it at your own adapter to hold it to the same bar |
Two things the loop deliberately does not do. Effect delivery is at-least-once: a crash between step and persist re-drives the same event and re-emits the same effects with identical deterministic keys, and the loop never dedupes - idempotency on that key is yours. And a resumed run restores position, not liveness: pending timers and in-flight invocations are re-established by the host, from its own durable rows. Surviving a restart walks a demo embedder through both.
Runs steps a run once. A chart that invokes a service is not finished
when that step returns - it is waiting for an answer it cannot fetch
for itself, and every host that has embedded this package has written
the same loop on top. StatifierPersistence.Driver is that loop:
driver =
StatifierPersistence.Driver.new(store, machine,
dispatch: fn type, params, _context -> MyApp.perform(type, params) end,
effects: fn effect, _context -> MyApp.Timers.consume(effect) end,
invoke_types: Statifier.Invoke.Types.new(types: ["myapp:authorize"]),
serialization: {MyApp.RunLock, MyApp.RunLock}
)
{:ok, run, state} = StatifierPersistence.Driver.create(driver, run_id)
{:ok, run, state} = StatifierPersistence.Driver.send_event(driver, run_id, Statifier.Event.external("go"))One call is one durable step, every effect through your effects:
executor, every <invoke> through your dispatch: fun inside that same
step, and then one further durable step per answer until the chart rests.
{:ok, donedata} answers done.invoke.<id>; {:error, failure} answers
error.communication.invoke.<id> with Statifier.Session.failed_invocation/3's
own reason/attempts/detail payload, and means permanently failed
rather than "retry".
Both events are built field for field from the two doors
Statifier.Session gives a handler-backed invocation's host, origin
and origintype included, so the same chart sees the same event whether
it runs in a session or out of storage. That is asserted rather than
claimed: test/statifier_persistence/driver_session_conformance_test.exs
answers one document both ways and compares the _event each chart saw.
An answer whose invocation the chart has since cancelled is dropped, per
spec 6.4.3, and a chart whose answer re-arms its own call is bounded by
max_turns: rather than driven forever.
A durable subchart is one child per <invoke>, created inside the
parent's own step. An <invoke> that maps over a list is N of them, and N
creates cannot hold the parent's exclusion, so the children are started
afterwards - one call per child, from whatever job picks it up:
StatifierPersistence.Driver.start_child_at(driver, parent_run_id, effect, index, count,
policy: :all
)effect is the resolved Statifier.Effect.Invoke (or the whole
{:start_child, resolved, {:invoke, invoke}} instruction), index is the
child's 0-based position, and count is N. The call is idempotent on the
child's derived run id, so a re-delivered start adopts the child it
already created instead of making a second one. Scheduling those calls is
a job runner's business, not this package's.
Each child then runs as an ordinary run. When one reaches a terminal status its answer is stored on its own run record, and a settlement section under the parent's exclusion asks - through an indexed status projection, never a listing of whole records - whether all N have. Only the settlement that finds them all terminal assembles the dense, index-ordered list and answers the parent's ordinary door, once:
[
%{"index" => 0, "status" => "completed", "donedata" => %{"id" => "acct_1"}},
%{"index" => 1, "status" => "failed", "failure" => %{"reason" => "declined", ...}},
%{"index" => 2, "status" => "cancelled"}
]policy: :first_error cancels the rest as soon as one child fails: the
started siblings through the cascading cancel, and the ones whose start
job has not run yet through the child_canceller: seam, which is handed
the parent run id, the invocation id, and the indices with no run. Both
kinds read "cancelled" at their index in the same list.
A child fails on its own word, with no host in the loop, by settling in a
failure-classed final - a top-level <final> whose <donedata>
carries the reserved key statifier_persistence:run_status set to
"failed":
<final id="declined">
<donedata>
<param name="statifier_persistence:run_status" expr="'failed'"/>
<param name="reason" expr="decline_reason"/>
</donedata>
</final>That step is an ordinary successful one; the run record takes :failed
with the failure string "failed_final", and the whole <donedata> -
tag included, alongside whatever else the final carries - reaches the
parent's list verbatim. Macrostep-budget exhaustion is the other route to
:failed, and the failure string is what tells the two apart. An
unhandled error.* event is not a route: a chart that cannot continue
stays :active until its author routes the error to a final.
See StatifierPersistence.Runs for the full rule.
An adapter that cannot store a child's answer, or cannot answer the status projection, is refused at open - a child whose invocation could never be settled is not started. On the Ecto adapter both arrive with the V03 migration.
Early, under active development, and the API is not frozen before 1.0. The storage-adapter behaviour with its identity guard, the in-memory reference adapter, the run lifecycle and executor seam, per-run serialization, and the Ecto layer (configurable keys/tables, versioned migrations, and the Postgres adapter below) all exist and are conformance-tested.
Configure a persistence module on your own repo once, and migrate:
defmodule MyApp.Persistence do
use StatifierPersistence.Ecto, repo: MyApp.Repo
end
defmodule MyApp.Repo.Migrations.AddStatifierPersistence do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence)
end
One migration covers every version of the package DDL on a fresh database. If you already ran that migration when this package shipped only V01, pick the later versions up with a second ordinary migration rather than re-running the first:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceRunMetadata do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 2)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 2)
end
from: says where a call starts and version: where it ends, in both
directions, so a migration's two calls always cover the same span: up
from from: (default V01) up to version: (default the newest), down
from from: (default the newest) back to version: (default V01). A
migration that caps one end caps the other to match - see "Upgrading to
V03 before deploying 0.7.0" below for the case that bites.
then build the guarded store the rest of the package works through:
{:ok, store} =
StatifierPersistence.Storage.new(
StatifierPersistence.Storage.Ecto,
persistence: MyApp.Persistence
)
The adapter passes the same conformance suite the in-memory reference
does (StatifierPersistence.Testing.StorageConformance - point it at
your own adapter to hold it to the identical bar), stores engine
identities verbatim, and implements the optional per-run lock_run/3
as a transaction-scoped advisory-plus-row lock (ADR-0004 as amended).
In your test suite, pass sandbox: true so each test runs in its own
Ecto.Adapters.SQL.Sandbox checkout via the adapter's isolate/1.
The adapter is written against Postgres and this package's gate runs
against a real Postgres server, but only three of its callbacks are
actually Postgres SQL: lock_run/3 (advisory lock plus FOR UPDATE) and
the two metadata listings (jsonb containment). Everything else - charts,
positions, run records, the identity guard, the executor seam, resume,
and the versioned migrations, V03's Postgres-only index included - runs
on any Ecto backend. So does the input log of V05: a table, four columns
and a unique index, with no jsonb predicate, no advisory lock and no
index type beyond a unique one, and the same conformance cases on both
backends.
A host on SQLite or another backend therefore declines the lock
callback rather than getting a portable imitation of it: pass your own
serialization: {module, config} strategy, backed by an exclusion the
host already owns (a job queue keyed per run id, a single consumer) or by
a pass-through when the deployment is single-writer by construction. What
you must not do is leave the default in place, which reaches the
Postgres-only lock_run/3 and raises mid-run.
The four conformance cases those three callbacks generate carry
@tag :postgres, so such a host runs the shipped suite green and honest:
mix test --exclude postgres #=> 31 tests, 0 failures, 4 excluded
Running on a backend that is not Postgres is the full guide: what is Postgres-only and why, how to write the strategy, what declining costs (durable subcharts and the run listings refuse rather than break), and how to verify your own setup.
0.7.0 needs V03 of the package DDL, and the order matters: run the
migration first, then deploy the new code. outcome_blob is an
unconditional field on the generated runs schema, so 0.7.0 against a V02
database fails on every query that touches the runs table, not only on
the fan-out write that introduced the column. The reverse order is safe:
V03 on a database still served by 0.6.x adds a column nobody writes and
an index nobody's query needs yet.
A host already on V02 picks V03 up with an ordinary migration of its own:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceOutcomeBlob do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 3)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 3)
end
A host whose first migration is capped - up(for: MyApp.Persistence, version: 2), which is what keeps a fresh clone and an already-migrated
database on the same sequence of steps - caps its rollback the same way,
with from: 2:
defmodule MyApp.Repo.Migrations.AddStatifierPersistence do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, version: 2)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, from: 2)
end
Leaving that down uncapped is the failure this ceiling exists to
prevent: Ecto rolls migrations back newest first, so mix ecto.rollback --all runs the V03 migration's down and then this one's, which without
from: starts at V03 again and fails on a column that is already gone
(no such column: outcome_blob). Rolling back a single step is
unaffected either way.
V03 does two things, and only one of them is cheap.
The outcome_blob column is a nullable :binary added to the runs
table. Postgres adds a nullable column with no default as a catalog-only
change, so this part is fast whatever the table's size. It takes the
configured :blob_type with the other three blob columns, so a
:blob_type whose underlying database type is not binary needs the same
hand-written ALTER this README's encryption section already describes
for those three - now for four columns, not three.
The metadata GIN index is the part to plan for. V03's up/1
issues a plain CREATE INDEX, not CREATE INDEX CONCURRENTLY: it
takes a SHARE lock on the runs table for the whole build, which blocks
every INSERT, UPDATE and DELETE against that table until the index
is finished. Reads are unaffected. On a small or idle runs table this is
imperceptible. On a large one it is an outage of every write the runs
table takes - which, for a host stepping runs durably, means every step
of every run.
How long that is depends on the row count, the width of the metadata
maps, and the server, so measure rather than guess. (sp-461 is a
separate measurement issue and not a number for this build: it measures
the settlement read cost against a GIN-indexed metadata column at
increasing fan-out widths.)
The concurrent build is what a host with a large runs table wants, and
0.8.0 ships it as V04 - see below.
The index is not optional in effect: without it, every fan-out child
completion asks whether its N siblings are terminal with a jsonb
containment query, and each one is a sequential scan of the whole runs
table.
On an Ecto adapter that is not Postgres, the index is skipped. GIN
and jsonb_path_ops are Postgres spellings, so up/1 creates the index
only when the migration's repo runs on Ecto.Adapters.Postgres, and
down/1 drops it under the same condition. Everything else in V03 - the
outcome_blob column included - is created on every adapter, which is
what lets a SQLite host run this package's DDL at all. (In 0.7.0 it could
not: the index raised, the whole migration rolled back, and the column
went with it. Fixed in 0.7.1.)
What is skipped with the index is what the index served. Both metadata
queries this package issues -
StatifierPersistence.Storage.Ecto.list_runs_by_metadata/2 and the
status projection list_run_states_by_metadata/2 - are jsonb
containment SQL, which a non-Postgres backend does not parse. So on such
an adapter the Ecto adapter declares no metadata support: a metadata:
map at create is refused with {:error, :metadata_unsupported}, the two
listings refuse with {:error, :child_listing_unsupported} and
{:error, :run_states_unsupported} - and the two raw adapter callbacks
behind them answer {:error, :metadata_unsupported} rather than issuing
SQL the backend cannot parse, for a host that reaches them directly - and
a durable subchart or a fan-out
over that store is refused at open rather than started and left with
children nothing can settle. Storing, loading, stepping and resuming runs
are unaffected. Per-run locking is a separate Postgres-only surface -
lock_run/3 is pg_advisory_xact_lock plus SELECT ... FOR UPDATE -
and is tracked in sp-5lm.
0.8.0 adds V04, whose whole job is to rebuild V03's metadata GIN index
with CREATE INDEX CONCURRENTLY - same name, same expression, same
jsonb_path_ops opclass, built without the SHARE lock. It ships in the
same versioned helper as everything else, so a fresh database picks it up
with the one-call recipe and needs nothing else.
The one thing V04 cannot do for you is turn off the transaction it runs
in. CREATE INDEX CONCURRENTLY cannot run inside a transaction block,
and Ecto reads @disable_ddl_transaction and @disable_migration_lock
from the module Ecto.Migrator runs - your migration, not a module
it delegates to. So V04 wants a migration of its own:
defmodule MyApp.Repo.Migrations.RebuildStatifierPersistenceMetadataIndex do
use Ecto.Migration
@disable_ddl_transaction true
@disable_migration_lock true
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 4)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 4)
end
Called from an ordinary transactional migration instead, V04 leaves V03's index in place and does nothing else. That is deliberate rather than a failure mode: the index it would have built is the one already there, under the same name, and raising would break the one-call recipe every fresh database and test harness uses, where a plain build on an empty runs table costs nothing. It logs a warning when the runs table already holds rows, which is the case where the plain build did block writes and the two attributes are what you were missing.
down/1 for V04 does nothing at all: what it leaves behind is V03's
index, and V03's down/1 is what drops it. A non-transactional migration
has no rollback, so if the rebuild is interrupted, re-run the migration -
the drop is drop_if_exists, so it clears a missing or an invalid
leftover either way.
A host with a large runs table that has not yet reached V03 is the
one case V04 does not solve by itself, because V03 still builds the index
plainly on the way past. Such a host adds the outcome_blob column by
hand, skipping V03's helper call entirely:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceOutcomeBlob do
use Ecto.Migration
def up do
alter table("runs") do
add(:outcome_blob, :binary, null: true)
end
end
end
substituting your configured table name and prefix, and then runs the V04
migration above for the index. That is a smaller hand-written migration
than 0.7.x asked for - the index half is V04's now - and the warning
about never calling up(from: 3) afterwards still applies: V03's
create/1 is a plain create, not create_if_not_exists, so a second
run fails on the index that is already there.
On an Ecto adapter that is not Postgres, V04 is a no-op, both directions, under the same adapter check V03 uses: there is no index to rebuild, because V03 created none. Such a host needs neither attribute.
A run record carries engine identities and opaque blobs. Nothing on it
answers the question a multi-tenant host asks first - "list the runs for
scope X" - so ADR-0006 adds one optional, opaque metadata map to a run,
stored beside it and handed back unchanged.
Take a card-processing host running a myapp:authorize / myapp:capture
chart, one run per payment attempt, and a support screen that lists every
run for one processor account. Tag the run at create with the account ids
the host already keys its own tables by:
{:ok, run, _machine_state} =
StatifierPersistence.Runs.create(store, payment_id, machine,
executor: MyApp.Executor,
metadata: %{
"tenant_id" => "acct_01H8X",
"processor_account_id" => "pacct_4471"
}
)
and read them back with an equality match on every pair:
{:ok, runs} =
StatifierPersistence.Storage.Ecto.list_runs_by_metadata(store.opts, %{
"processor_account_id" => "pacct_4471"
})
Equality on all given pairs is the whole query surface: no ranges, no
partial matches, no ordering guarantee. Anything richer is a query you
write against your own column - the table name is yours to configure, so
that is a supported thing to do. The V02 migration adds the column as
nullable jsonb with no index of its own, because which pairs you query
by is your call. V03 (above) adds the one containment index this
package's own settlement query needs, a GIN jsonb_path_ops index on the
whole column; an expression index on particular keys, or the wider
jsonb_ops operator set, is still yours to add when the volume asks.
Two rules come with it.
Identities only, never personal data. Keys and values are host
identities - a tenant id, a subject-entity id, a correlation id - and
never a name, an email address, a postal address, a card number, or any
other personal or cardholder data. This is a rule of the contract, not
advice: :blob_type encryption (below) covers the three blob columns and
does not reach this one, so anything you file here is at rest in the clear
no matter how the blobs are configured. The map is opaque to this package
by design, so nothing here can inspect a value and reject it - the rule is
kept by you.
An adapter may refuse it. An adapter that cannot store the map refuses
a non-empty one at the create with {:error, :metadata_unsupported}, so
you learn on the first call rather than finding a silently dropped scope
later. An empty or absent map is never refused. The shipped in-memory and
Ecto adapters both support it; a third-party adapter that does not is
still conformant, and the conformance suite tests both answers. The Ecto
adapter refuses at the same point for a value jsonb cannot hold - a
tuple, an atom, a pid, or a binary that is not valid UTF-8 - rather than
storing something that is not what you handed it. The map is write-once:
it is set at create and a later step or abandonment carries it forward
untouched.
A run is the only thing this package scopes for you, and only through that
map. A chart is not: StatifierPersistence.Storage.save_chart/3 keys a
chart on its content hash alone, so two tenants storing byte-identical
charts share one chart row. Tenant-qualify your own per-chart rows in your
own tables - folding a namespace into the hash would change what a chart's
identity is, which is statifier-ex's contract and not an option this
package offers.
A durably stepped run stores a chart, a position and a run record, and none of them is an input. The position is the run's current configuration, overwritten on every step, so by construction the history an offline replay needs is destroyed by the mechanism that makes the run durable.
ADR-0010 adds an optional per-run input log to close that gap. It is
opt-in by export, exactly as the metadata map is: an adapter that
exports supports_input_log?/1 and answers true keeps one, and an
adapter that does not stores no inputs and behaves exactly as it did
before. Nothing refuses a run over it - a diagnostic facility must
not break the run it is diagnosing - so ask before you rely on it:
StatifierPersistence.Storage.input_log_supported?(store)
The Ecto adapter keeps the log on every backend, in the V05 table. Each
entry is the %Statifier.Event{} the interpreter was handed, verbatim,
stamped with the public door it entered by and a dense zero-based
ordinal:
{:ok, entries} = StatifierPersistence.Runs.inputs(store, "run_1")
Enum.map(entries, &{&1.seq, &1.door, &1.event.name})
#=> [{0, "step", "advance"}, {1, "answer_parent", "done.invoke.call"}]
One log belongs to one run. A durable subchart's child is an ordinary
run, so it has its own log; the parent's holds the answer it saw at the
answer_parent door, and not the child's inputs. Only inputs an
interpreter actually saw are recorded - a delivery to a terminal run, or
to an invocation the chart has since cancelled, is discarded and appends
nothing, because a replay that applied it would produce a different run
than the one that happened.
Turning the log on is a data-retention decision, not a debugging
switch. Chart, position and identity blobs are engine-shaped; an
event's data is your own values, and this is the first thing this
package stores that can hold personal or cardholder data. So input_blob
is a blob in the :blob_type sense below, and it is not the metadata
map, which stays in the clear by design. Bound how much accumulates with
a per-run cap declared where every other adapter setting is:
{:ok, store} =
StatifierPersistence.Storage.new(
StatifierPersistence.Storage.Ecto,
persistence: MyApp.Persistence,
input_log_cap: 500
)
The default is :infinity; a bounded default would be this package
silently truncating your log. Past the cap the log closes itself: the
last slot is written as a marker entry whose event is nil, every
later append is refused, and the step itself succeeds and the run carries
on. The marker is the point - a truncated log that looked complete would
satisfy every check a replay makes while replaying a run that never
happened.
The replay itself is statifier_ui's
(StatifierUI.Trace.Replay.from_events/4); ADR-0010 decision 8 names the
mapping from a stored entry to what that function takes, and nothing in
this package depends on statifier_ui to say so.
use StatifierPersistence.Ecto hard-codes :binary for its payload blob
columns (identity_blob, chart_blob, position_blob, outcome_blob,
and the input log's input_blob) by default - plain bytea,
byte-identical round trip, nothing extra. Pass :blob_type to put a
custom Ecto type on those columns instead, and encryption at rest needs
no wrapping adapter:
defmodule MyApp.Persistence do
use StatifierPersistence.Ecto,
repo: MyApp.Repo,
blob_type: MyApp.EncryptedBlob
end
:blob_type accepts a bare module implementing Ecto.Type, or a
{module, opts} tuple for an Ecto.ParameterizedType. It reaches only
those payload columns: keys and lookup columns (content_hash,
session_id, run_id, status, failure, and the input log's seq
and door) always stay plain, because the identity guard, the unique
indexes and the log's ordering depend on reading them back verbatim.
The shape a production MyApp.EncryptedBlob needs is a vault-backed or
envelope-encrypting Ecto.Type - dump/1 encrypts on the way in,
load/1 decrypts on the way out. This package takes no position on
which key-management scheme backs it; that choice belongs to the host.
To prove the shape without any encryption dependency, here is a minimal
Ecto.Type that reversibly transforms every byte (not encryption - a
stand-in to show the wiring):
defmodule MyApp.ReversibleBlob do
use Ecto.Type
@mask 0xA5
def type, do: :binary
def cast(binary) when is_binary(binary), do: {:ok, binary}
def cast(_other), do: :error
def dump(binary) when is_binary(binary), do: {:ok, transform(binary)}
def dump(_other), do: :error
def load(binary) when is_binary(binary), do: {:ok, transform(binary)}
def load(_other), do: :error
defp transform(binary) do
for <<byte <- binary>>, into: <<>>, do: <<Bitwise.bxor(byte, @mask)>>
end
end
The shipped migrations always emit :binary (bytea) for the payload
blob columns and do not read :blob_type. A :blob_type whose
underlying database type is still binary - an envelope-encrypting type
that dumps to and loads from raw bytes, like the sketch above - needs
no DDL change. A :blob_type that dumps to a different underlying type
(text, jsonb, a Postgres domain) needs you to alter those columns
yourself; the migrations helper does not do it for you.
The suite includes database-backed tests against a real Postgres server -
ADR-0005 rejects a skip tag for when one is absent, so mix quality and
mix test both need one reachable. Start it once with:
docker compose up -d db
which brings up postgres:17 on localhost:5432 with user/password
postgres. Override host, port, user, password, or database name with the
PGHOST, PGPORT, PGUSER, PGPASSWORD, and PGDATABASE env vars (see
config/test.exs for the defaults) if a server is already running
elsewhere.
docs/restart-demo.md walks through the demo embedder that drives this
package's whole surface across a simulated restart with no Session
process: persist mid-run with a pending durable timer and an in-flight
async invocation, drop everything volatile, cold-boot from the run id
alone, and finish with zero duplicate side effects and a replay that
reproduces the path. The executable version lives in
test/statifier_persistence/demo/restart_demo_test.exs (and its
Postgres variant beside it).
The persisted-position story is already specified upstream, and this package is one consumer of it rather than the definition of it:
docs/persistence.mdin statifier-ex covers what MachineState contains, the interned-index hazard, chart identity, and the resume recipe.- ADR-0052 there records the rules: a persisted position is only meaningful against the exact chart revision that produced it, so every load is guarded by the Machine identity / content-hash. Loading a position against the wrong revision does not error - it silently resumes the wrong configuration.
- ADR-0060 records the resume API: the
:resumeoption onSession.start_link/2, the pure-core rehydration path, and what a resume deliberately does not restore (in-flight delayed-send timers and live invoked children).
Read all three before adding code here.
In scope:
- A storage-adapter behaviour: save/load of MachineState snapshots (or Recordings), guarded by the Machine identity so a position can never be loaded against the wrong chart revision.
- Run lifecycle as a library: create/step/complete/fail, with a serialization guarantee per run so concurrent event deliveries to one run are ordered.
- The load -> handle_event -> execute effects -> persist loop, with effect execution delegated to the host.
- An Ecto adapter shipping schemas and migrations for chart definitions, versions, and runs; the host supplies the Repo and any tenancy columns.
Out of scope: domain actions, authoring UI, and job scheduling - statifier_oban owns timers and async work.