🌋 hekla

A single-app event-sourcing runtime you write in heklang, over the Dynamic Consistency Boundary.

Business logic is plain source text: commands validate input, replay the history a decision depends on, and append events; projectors consume events into queryable SQLite read models; effects react to events with durable, replay-safe side effects. There is no build step, because there is nothing to compile. Deploy is restart.

heklang is pure and sandboxed, so determinism is structural rather than policed: a command cannot call out, a projector cannot decrypt, and a fold cannot read a clock, each because of what kind of declaration it is rather than because something checks. That is what lets a projector rebuild and an effect replay reproduce exactly what they did the first time, and what lets the runtime give effects a Temporal-style journal so a crash mid-arm resumes without re-firing what already happened.

hekla runs on tephra for the event log and SQLite for read models. It is a rewrite of umari, which expressed the same model as WASM component modules.

What a project looks like

// events/order.hk
subject Customer(Int)
subject Shop(Int)

event @order.placed {
  order_id: Uuid,
  customer_id: Customer,
  shop_id: Shop,
  // Encrypted under a key scoped to the customer, so erasing them is one key delete.
  // Optional because an erased column reads back absent, and a type that cannot be
  // absent could not say so.
  email: String? @subject(customer_id) @max(200),
}

// commands/place-order.hk
refusal SoldOut "this shop's launch allocation is gone"

command PlaceOrder(order_id: Uuid, customer_id: Customer, shop_id: Shop, email: String?) {
  // `fold` is a read declaration, not a binding: it names a slice of the log, folds
  // it, and that slice is what the append conditions on. What you folded is what you
  // conflict on, so a concurrent write inside it loses rather than races.
  //
  // Narrow: this one order, so a retry of the same id is a no-op.
  fold placed: Bool = false
    on @order.placed(order_id) => true

  // Wide on purpose: a launch allocation is a rule about every order in the shop.
  fold sold: Int = 0
    on @order.placed(shop_id) => sold + 1

  if placed {
    return
  }
  if sold >= 100 {
    reject SoldOut
  }

  emit @order.placed { order_id, customer_id, shop_id, email }
}

POST /commands/PlaceOrder runs that: the route is the declared name, not the file. Add a projector and GET /read/{Projector}/{Entity}/{key} serves what it built. Both routes are generated from the declarations, with an OpenAPI document and a reference UI at /docs.

Install

cargo install hekla

Or nix run git+https://git.tqwewe.com/tephra/hekla to run it without installing anything, and nix build git+https://git.tqwewe.com/tephra/hekla for the binary alone.

One binary is the whole runtime: it serves the API, runs the scenarios, and does the key management. tephra is embedded as a library and SQLite is bundled, so there is no server to stand up beside it and nothing to point it at. A project is a directory of .hk files, and examples/orders in this repository is a working one.

To drive the runtime from your own Rust rather than the CLI, cargo add hekla: the binary is a shim over the library, and everything it does is public.

Run it

hekla check examples/orders   # static analysis, for CI and pre-commit
hekla test examples/orders    # scenarios under tests/

HEKLA_MASTER_KEY=$(head -c 32 /dev/urandom | base64) \
  hekla serve examples/orders # the API on 127.0.0.1:8080

check reports the compiler's diagnostics plus what only hekla knows: that a declaration sits in the directory its kind requires, that a read model can be keyed and indexed the way the read API needs, and two warnings about a boundary too broad or too narrow to do its job. Also hekla erase and hekla rotate for key management, hekla plan for what a deploy would change (and, with --replay, would do), hekla project for folding an undeployed projector over the log to answer a question once, hekla rewind for taking an effect back over history it has already processed, and hekla verify for the invariant sweep below.

From a checkout the same three are cargo run -- check examples/orders and so on. The repository is also a flake: nix build for the binary, nix flake check for the suite.

Planning a deploy

The event log is append-only and a read model is rebuilt from it, so a deploy is not a thing you undo. hekla plan says what one would change before it changes it.

hekla plan . --data-dir /srv/hekla/data

compared 6 declaration(s) against what is deployed
  behaviour command DoA (commands/a.hk)
  behaviour command DoB (commands/b.hk)
  contract  projector UserStats (projectors/user-stats.hk)
  projector UserStats rebuilds from zero, redoing 12481 position(s)
  because `guard ShopIsConnected` changed: DoA, DoB
0 added, 0 removed, 3 changed; 1 projector(s) would rebuild

behaviour means the declaration does something different behind a contract that did not move, so nothing outside the program can tell; contract means what is visible outside changed. Both come from heklang's digest, which hashes the lowered program, so reformatting a file or renaming a local is not a change and a corrected handler body is.

A const, refusal or guard is spliced into what names it and has no row of its own, so editing one moves every hash that reaches it. The report groups those and names the cause rather than leaving a wall of diffs.

Without --replay it reads the recorded declarations and the read models and nothing else, so it opens no event log at all, with one exception: when the diff says an event, record or enum moved it reads the oldest stored event of each declared type, to answer whether the deploy could still read its own history. Either way it takes no data-directory lock, so it runs against a directory a server has open. It exits 0 whether or not anything would change, and --json carries the whole plan for a deploy gate.

A declaration this program could not read is reported with the credentials it has not got, in the same words, because both are a deploy that would refuse to start rather than a deploy that would change something.

Would it still do the same thing?

That diff says an effect changed. It cannot say whether the change matters, and "would this now send a different HTTP request" is what the deploy actually turns on. --replay answers it.

hekla plan . --data-dir /srv/hekla/data --replay

  effect NotifyCustomer @ 4812: it reached a call the recorded run never made (http.post #0)
replayed 312 invocation(s) across 2 affected effect(s); 310 reproduce, 2 diverge
this project retains 7 day(s) of journals; anything older was reclaimed before the replay could see it
0 added, 0 removed, 1 changed; 0 projector(s) would rebuild, 2 recorded invocation(s) would diverge

Every recorded invocation of every affected effect is re-run against the candidate code and the journal the original run left behind. Nothing is mocked: the journal holds the responses that run really received, so a candidate that branches differently on a response reaches a call the journal has no entry for, and that miss is the finding. Nothing is sent, appended or erased.

Affected means the effect's own digest changed or it names something whose digest changed: a module fn, an event it handles, a record or enum either of those carries. Each of those is a declaration of its own, so editing the helper that builds a URL, or adding @subject(...) to a field the arm binds, leaves the effect's own hash exactly where it was, and a check that looked only at that hash would miss both.

This half opens the log, through a read-only follower that takes no lock and creates nothing, so it too runs against a deployment serving traffic. It pins one committed prefix when it opens, so an invocation the live server records after that is left out rather than replayed against an event the reader cannot see.

What it cannot see is counted or named rather than assumed away: an invocation whose subject has been erased (the plaintext the handler branched on is gone, by design); one an operator skipped, since nothing ran it to an end and its journal is the prefix of a run that stopped where it wedged; one whose record could not be read at all, because a busy op-DB is not evidence about the candidate; one retention already reclaimed (row and journal go together, so it is invisible rather than skipped, and the window is printed instead) or reclaims while the replay is reading it; and whatever --replay-limit drops, which defaults to the 1000 most recent invocations per effect and names any effect it bit. The baseline is also only what is running: rows written by a version this deploy is not replacing are still on disk, and replaying those would report a difference the running code already has.

An effect that reveals needs HEKLA_MASTER_KEY, decided per effect rather than per project. Without one those invocations are counted and the rest still replay, so a CI job can plan against production without holding the production key. A key that is present and cannot unwrap what is stored degrades the same way and says so, rather than throwing away a diff that is already computed.

A gate reads --json, where divergences and coverage are both null when no replay ran: an empty divergence list would be a clean replay result, and nothing should read one off a run that never opened the log. secrets is never null, because it is always computable, and a deploy whose credentials this machine cannot supply is not an empty plan.

Asking the log a question

The read API serves what a deployed projector chose to materialise. For the question you want to ask once, hekla project folds an undeployed projector over the log and prints its rows, without deploying anything or waiting for a rebuild.

cat > /tmp/by-customer.hk <<'EOF'
projector ByCustomer {
  entity PerCustomer { customer_id: Customer @key, orders: Int }
  on @order.placed { customer_id } { patch PerCustomer[customer_id] { orders: .orders + 1 } }
}
EOF

# Against whichever data directory already holds a log; a fresh checkout has none.
hekla project /tmp/by-customer.hk examples/orders --data-dir /srv/hekla/data

The scratch file is compiled with the project, so it typechecks against the deployed events and can call the project's own helpers. It reads the log through a read-only follower, so like plan it takes no lock and runs against a server that is serving traffic, and the rows are a snapshot at the position it reports. Nothing is recorded: no declaration, no read model on disk, no route.

Deployment credentials

A project declares what it needs and the deployment supplies it:

secret DISCORD_WEBHOOK

[secrets]
DISCORD_WEBHOOK = { env = "DISCORD_WEBHOOK_URL" }
# or { file = "/run/secrets/..." }; a name absent here reads HEKLA_SECRET_<NAME>

The language guarantees a credential reaches a url, a header value or a request body and nothing observable: not a log line, not an event, not a read model, not a fold. hekla adds the other half. Every surface it serves shows {SECRET:NAME} rather than a value, including the journal key and a wedge message, so a rotation moves no hash and leaves no journal entry orphaned. hekla serve refuses to start when a required credential is unset, naming every missing one at once, and hekla secrets answers the same question on its own for a pre-deploy gate.

Rotating is changing the source and restarting. There is no _PREVIOUS list: a master key needs one because stored data is wrapped under it, and a credential wraps nothing.

Checking the invariants

The event log is append-only, so the bugs worth hunting are the ones you cannot undo: an event that should never have been appended, an effect that fired twice, a read model that quietly disagrees with its own history. hekla verify checks the properties those rest on, against whatever state a deployment actually reached rather than against cases someone thought to write down.

# Verify a stopped instance, or a `hekla backup` of a running one. Take the copy with
# that rather than with `cp -r`: a hand-copied live directory is not consistent (SQLite
# WAL, and the key store against the log), so the sweep could report divergence the copy
# caused, the worst possible false positive for a tool whose whole value is being believed.
cargo run -- verify . --data-dir data   # non-zero exit on any violation, for CI or a nightly job

A backup holds no read models, so a sweep over one reports checked 0 projector(s): rebuild equivalence needs a live model to compare a rebuild against, and verify builds none (it opens the directory quiescent, starting no projector and no effect). What it does check is the half a bad copy would break: that the log opens, and that every recorded invocation still replays.

Giving it the other half means running the deployment from the copy, which is not a routine check. A boot resumes the invocations that were in flight when the backup was taken, performing their calls for real: webhooks sent again, and an erase carried out. Do that only with the credentials taken away, which hekla turns into a refusal rather than a surprise, and never against a copy whose effects can still reach production.

It checks that every projector rebuilt from position 0 matches the live one row for row, and that every recorded effect invocation still replays without performing anything. The replay runs against a sealed host that can only read the journal and can never send, so verifying an invocation is structurally incapable of repeating it.

serve --verify runs the per-operation half continuously: every completed invocation is replayed as it finishes. A component that breaks an invariant is quarantined (it stops advancing, its reads return 503, and /status names what broke) while the rest of the runtime keeps serving. Turn it on permanently with [verify] enabled = true in hekla.toml.

One process at a time: a runtime takes an exclusive lock on the whole data directory, before it opens anything in it. tephra refuses a second writer on the log by itself, but the operational database and the read models have no such rule, and by the time a second process reached the log it would already have migrated one and started building the others.

Backing it up

cargo run -- backup . /backups/hekla              # while the server is serving
cargo run -- backup . /backups/hekla              # again: copies only what changed
cargo run -- verify . --data-dir /backups/hekla   # and the copy replays

The first argument is the deployment, named either by its data directory (/var/lib/hekla) or by the project that holds one at <dir>/data. There is no --data-dir: the other subcommands have one because they need the code and the data together and only a project can say where the code is, and a backup needs no project at all.

The log can be copied from under the writer safely, because a segment is never rewritten and a batch counts only if it ends in a commit marker, so any copy is a committed prefix. What an operator reaching for cp, rsync or Borg cannot know is that the order is forced, and that the order alone is not enough.

The key store has to be at least as new as the log. A missing subject key is indistinguishable from an erasure, by design, so a key store copied first makes everything appended in between read as though those customers had asked to be forgotten, with nothing in any log to say otherwise. The effect journal has the opposite constraint: a restored log carries on at head + 1, and an invocation is recorded against a position, so a record left above the head makes a different event look like one that has already been handled, and its effect silently never runs. Both of those live in hekla.db, so there is no order that satisfies both, and the second is repaired after the copy rather than ordered for.

So hekla backup copies the log, snapshots the operational database, and then lowers every position the copy records to the head of the log it ended up with, telling you what that discarded. It takes no lock on the source, loads no project and needs no master key. Read models are left out, because a restore rebuilds them from the log.

The target is an ordinary data directory: check it with hekla verify, or serve from it. Run the command again over the same target to update it in place, which is also what carries an erasure across: the state is replaced wholesale, so there is no older key store in it to resurrect a key from. A dated copy you keep yourself holds the key store as of its date, which is the one thing a backup cannot make right. An erasure driven by an effect re-applies itself on a restore (the request is in the log, and erase is journaled, so a replay that finds no entry performs it); hekla erase has to be re-run.

The backup is inert without the master key it was taken under, which the summary and the manifest both name. Keep that somewhere else.

A run that dies part way through leaves the target marked complete: false, because in that window its log has grown past the key store beside it and a restore would read everything in between as erased. The next run repairs it, and serve and verify both say so if they are pointed at one first, since a sweep over it would otherwise pass: a key that is absent is a legitimate state.

Watching it run

/status says how things are right now. GET /metrics says the same in Prometheus text format, plus the things a snapshot cannot carry: how often a command was refused, how long a projector has been behind, whether an effect's lag is falling or stuck.

curl -s localhost:8080/metrics | grep hekla_effect_

Series are named hekla_* and cover the log head, each module's position, lag and state, command outcomes by refusal code, DCB conflict retries, projector rebuilds, and the failure modes that are particular to this runtime: wedged lanes, terminal skips, quarantine, and the positions rule 15 suppressed or collapsed. docs/monitoring/hekla-alerts.yml has alerting rules for the ones worth being woken up for, with the reasoning for each threshold beside it.

Every label is a declaration: a command, projector, entity, effect or event name, a refusal code, or a fixed outcome word. Never a lane key, and that is about erasure rather than cardinality: a lane key is a partition key, /status may name it because it is a live view an erasure passes through, and a scrape may not because it is a copy taken somewhere hekla erase cannot reach. So the metric counts wedged lanes and /admin/effects/{name} names them.

Like the rest of the surface, /metrics is unauthenticated and the bind address is the boundary.

Learn more

• heklang/docs/ is the language: commands, projectors, effects, folds, sealed content, tests. hekla does not repeat it.
• AUTHORING.md is everything hekla adds around it: the directory convention, the envelope, the generated HTTP surface, the CLI, and how subject encryption really works underneath.
• ARCHITECTURE.md covers the design and the alternatives that were rejected.
• ROADMAP.md tracks what is done and what is next.

License

Licensed under either of Apache-2.0 or MIT at your option.

hekla was built with AI use and careful review.