Follower mode: a sapedbd that subscribes to another, applies what it says, and refuses writes

Status

Done

Version

1.0.0

Component

server

Priority

High

Blocked by

SAPE-26 — cleared

Blocks

—

Shipped

SAPEDB_FOLLOW='sapedb://acme:pw@leader:7433/main?sig=…'   # this daemon becomes a follower
SAPEDB_FOLLOW_INSECURE=1                                  # dial the leader without TLS

Account and database come from the connection string itself. One connection string works for both roles, because the signature covers account, password and database name and deliberately does not cover the host — which is what makes criterion 3's control case clean: the very same Invoke is sent to both machines.

Criterion 4 answered the way the ticket hoped, and that erases the trap rather than balancing it

There is no position file. store.Apply already writes the log counter into the same tree, in the same transaction as the documents it changes, and the follower commits once — so the data and the place it has reached become durable together or not at all. The resume point is the database's own LatestLSN(), plus one.

That removes both halves of the classic bug instead of trading them off. Crash after applying and before recording the position: impossible, they are one write. Crash after recording and before applying: impossible, same reason. Reconnect and be sent an entry already held: Apply ignores anything at or below the latest.

Verified by mutation, run in a copy of the tree under /tmp — never on the real tree:

Two mutations survived, and they are declared rather than buried, because a survivor that looks like a hole and is not is exactly what a mutation report exists to distinguish:

• Applying every entry twice stays green — every branch of Apply is idempotent. So "a repeat" is not a failure this code can produce, and saying so is more honest than claiming a test measured it.
• Letting the put branch record a Change of its own stays green — takeLSN returns exactly the entry being applied, which is then overwritten by the leader's verbatim copy. Not a reason to loosen the guard; the reason the guard can stay where it is.

Criterion 2, the primary one, with its numbers

602 entries on the leader; the follower killed at entry 200 — a number read back out of its own file after death, with the test failing if it did not land strictly between zero and the total; 40 more entries written while it was dead; restarted on the same directory. Compared entry by entry (what the leader has and the follower does not, where the two disagree, and any gap in the numbering) with the per-field document comparison layered on top.

Before concluding anything, the test counts which Kinds actually travelled — put, delete, declare and operation must each be above zero or it fails. An agreement between two empty sets is the failure this guards against.

What it turned up — reading the catalogue is a write

store.WhatIsHere and store.Explore both record a ChangeRead entry — on purpose, because the audit trail was never meant to stop at reads. An entry is an entry, so a read-only follower refuses both. The real consequence: sapedb shell pointed at a follower cannot list its collections. Invoking a read-only operation still works, because that records nothing — and today that is the only way to read a follower. Said out loud in the CHANGELOG, the README and the godoc rather than left to be discovered. Filed as ISS-26.

Description

A sapedbd that connects to another as a client, subscribes from the LSN it is already holding, applies each change through SAPE-26's wiring, and refuses every write sent to it directly. This is the smallest thing that can be called "a follower": one upstream, read-only, catches up after being offline.

Why now — read this before asking why replication skipped the usual triage

By this project's own rule — does it lose data, break a promise on a public surface, or stop somebody installing and running the thing — replication does not qualify for 1.0.0 on its own merits. Nothing about sapedb today loses data, breaks a promise, or fails to install for lack of a follower. This ticket and SAPE-26 are in 1.0.0 because the user chose to put them there, as a scope decision, not because they passed the triage every other ticket in this backlog passed. Writing that plainly here is what lets whoever reads this backlog later understand how it was actually built, rather than inferring a triage reason that was never applied.

Measured 2026-09-20, sapedb 248fa02c. A case-insensitive, literal-string search (grep -riF, so a term spelled across multiple source lines or built from single-character literals would still be found) for follower, master, slave and standby across every non-test .go file returns zero files for each of the four terms. replica returns 17 hits, every one of them in a comment describing what the log format is for — none is an identifier, a type, or a mode. leader returns 33 hits, and every one of them is about the file leader (internal/pager/led.go, internal/store/parts.go): the single .sapedb file that partition files (.parts) are recorded against, unrelated to replication. There is no follower mode, no partial one, and no naming collision waiting to happen with a different meaning of "leader" already in use for something else — worth knowing before choosing a name.

The change log was designed with this exact feature in mind, which is why SAPE-26's Apply already exists: internal/store/log.go's own comments name the three traps a follower has to survive, in these words — "a replica that loses its place and rewinds is a normal event, not an emergency", "off-by-one wrong means a replica that silently skips a change", and "a replica with a gap in it is a replica that is wrong in a way nothing will notice later". Criterion 2 above is aimed directly at those three sentences, and the third mutation in the table is the one that proves the aim landed.

Scope, cut deliberately narrow

Follower is read-only. Replication is asynchronous — a follower is behind by however long the network and its own apply loop take, and nothing here promises otherwise. There is no automatic promotion of a follower to a primary. There is no conflict handling, because a read-only follower with one upstream never has a conflict to handle. All four are v1.1 material; naming them here is what keeps this ticket from growing into them by accident. All four held.

Acceptance criteria

1. A follower started against a primary with existing data catches up, and a get issued against the follower returns exactly what the same get returns on the primary. Met, across two real sapedbd processes, compared per field in both directions.
2. Kill the follower mid-stream and restart it, compared change by change. Met — see the numbers above.
3. A write sent directly to the follower is refused, with a reason. Met — code read_only, message naming the leader, with a positive control first (the same write succeeds on the leader) and a second control (the follower answers a read-only invoke correctly, so it is not simply refusing everything).
4. The follower's replication position survives a restart, and where it lives is written down here. Met — and the answer is the one this criterion hoped for: nowhere separate. Proven three ways, including listing the data directory and failing if any file exists beyond main.sapedb and main.parts — that is, no pointer file at all.

Debt left behind

• The follower commits once per entry, so a batch of 512 from the leader's feed is 512 commits and 512 syncs. Nobody has needed it faster.
• too_far_behind still cannot occur against a real sapedbd — see ISS-23. follow.Run handles it, but only an in-process test that sets its own ceiling reaches that path, and no fake was built to pretend otherwise.
• A follower has no operator shell — ISS-26.
• The follower goroutine writes to the announce channel alongside Serve. That pattern predates this ticket; it now has one more writer.

Out of scope

• Automatic promotion or failover. Named above; repeated here because it is the one most tempting to add "while we're in here".
• Conflict resolution and multi-primary. A follower in this release has exactly one upstream and accepts no writes of its own.
• Replication lag monitoring or alerting. A follower can be asked its own LatestLSN(); building a dashboard around that is separate work.
• More than one follower reading from the same primary at once. Nothing here should prevent it later, but proving it works is not this ticket's criterion.