deleteRange — remove a stretch of keys, with a ceiling
Asked whether a batch could delete a few thousand rows atomically, this project's planner answered
no, and gave as the reason that steps is a field on the declaration so a batch's shape
is fixed when declared. That is true and it was the wrong reason.
It describes how the driver models a batch. It says nothing about what the engine may do. The two are different layers, and the promise this project is built on — that cost is declared and nobody writes an expression — constrains only the layer a client speaks to.
The proof was already here. A rollup is the engine computing: it keeps a count and a sum per group and maintains them incrementally. The driver never lets anybody write that as an expression — it lets them declare this rollup. Engine computes, driver declares, and the promise is intact. Any primitive that can be exposed the same way is available on the same terms.
The two questions are whether the cost can be bounded and whether it is safe for the server. Both
answer yes, and the machinery is already in place: tree.Ascend walks a key range and
Collection.DeleteBy removes one document. This is those two things in a loop with a
ceiling.
The cost is not what "delete a range" sounds like, and the ticket should say so.
Reading Collection.remove: each document has to be read before it can go,
because its index entries are derived from its contents; then every entry is deleted, then
contribute(tree, document, -1) adjusts the rollups, then the document itself goes, then
the change is recorded. So the cost is
limit × (one document read + its index entries + its rollup contribution + one log entry)
It is linear in limit and every unit is bounded, which is exactly what makes it
declarable. It is not a cheap truncation of a key range, and anybody who reads the name and assumes
otherwise will be surprised by the log.
Two consequences to decide rather than discover. N deletions write N log entries,
which lands on ISS-23 — no operator can cap retention, so deleting a
large file's worth of rows grows the log by that much. And a key range may span partitions, since
into(key) picks a tree per key; whether one call may cross that boundary is a decision,
not an implementation detail.
The operator shell already has what this needs: get, scan and
count are ad-hoc typed access, available to an operator and not to a client. A
deleteRange beside them is the same shape and the same audience — somebody holding
the connection, cleaning up.
It should read like scan, because it walks the same way: from and
to with the same exclusive forms, limit with the same meaning, and the same
answer when it stops early. scan already prints
"… and more: this stopped at the limit", and that sentence is what makes a cursor usable
without guessing.
As an operation, with three parameters and no expression:
{ "action": "deleteRange", "collection": "chunk",
"input": [ {"name":"from","type":"string","required":true},
{"name":"to","type":"string","required":true} ],
"from": {"terms":[{"arg":"from"}], "exclusive": true},
"to": {"terms":[{"arg":"to"}]},
"limit": 500 }
The shape is fixed at declare time, the ceiling is in the declaration where every other ceiling on this store lives, and the caller supplies bounds rather than behaviour. The cost envelope (SAPE-8) can state it before anything runs, like any other operation.
It returns what a caller needs to continue: how many went, the last key removed, and whether it
stopped at the limit. That is a cursor, and it is the same one scan already hands back.
These three primitives are the first things in this store that walk a range at run time. Everything
before them is bounded by construction: a declaration cannot express a loop, because a step pins
operation@version, version numbers only ever rise, and a pinned reference resolves only
to a version that already exists — so a cycle would need a version to exist before it was
declared. The reference graph is a directed acyclic graph because nothing can build anything else,
which is why ceiling() can compute a total at declare time and why nothing counts during
a run.
That property does not survive a thing that walks. So the limit on this operation is a ceiling the host counts against while walking, not a number the host takes on trust, and it is counted per run of the operation rather than per underlying call.
What happens at the ceiling was wrong on this page, and the code is right. This
paragraph used to end: “At the ceiling the operation stops and refuses; it does not return
a shortened answer, because a truncated result is indistinguishable from a complete one.”
That sentence was written for SAPE-34 and pasted into all three of these
tickets unchanged. It is true of a digest — a hash of part of a range looks exactly like a hash
of all of it — and false here, because this answer carries the last key
removed and a flag saying it stopped early. Three other places on this page say so plainly: the shell
section asks for “the same answer when it stops early”, the driver section calls
it “a cursor, and the same one scan already hands back”, and
acceptance criterion 1 measures “a thousand rows in pages”, which no refusal
could satisfy.
The decisive argument is not textual. When the ceiling is reached the rows are already gone. Refusing would not un-remove them; it would only withhold from the caller which ones went. That is the silent failure this project spends its documentation arguing against, dressed as strictness. So: the operation stops, and the answer it returns says it stopped. The ceiling still binds — it is still counted by the host, still per run, and it still refuses to go further.
The reason to write this down rather than assume it is measured, in pipelines/tasks/0071,
measurement W7. An external operation declaring limit 50, correctly
sandboxed and correctly signed, served 50,000,000 rows in one call — one
million calls of a host function that served fifty each — because the host checked every
individual call and never the total. A signature proves whose binary it is. A sandbox proves it does
not escape. Neither proves what it costs. That gap is not open today because nothing
can loop; it opens the moment something can.
What is counted here is rows removed. Each one costs a document read, its index entries, its rollup contribution and a log entry, so the row count is the honest unit and the log grows with it.
Landed in cfa5e52, CI green on both jobs. Twenty-one new top-level tests and twenty-six
mutations, each one watched going red.
One mutation survived the first pass, and the guard was fixed rather than the report.
Removing contribute(…, -1) from remove() — so a delete stops
decrementing the rollups it contributed to — left the test green. The assertion was
relative: it compared a deleteRange store against a store where the same rows
were deleted one at a time. The mutation breaks both paths identically, so two wrong answers agreed
with each other. The fix is an absolute check: the rollup is now compared against the
rows actually still in the collection, recomputed by walking them. Re-run, it goes red. This is the
rule about a guard not reading its expectation from the thing it guards, failing in a form that looks
like diligence — the comparison was real, it was just comparing two copies of the same mistake.
And the per-run ceiling is only measurable across a partition boundary. A mutation giving each partition file a fresh budget — the literal W7 shape — leaves all twelve single-partition tests green and is caught by exactly one test, the one built to span two trees. SAPE-34 recorded the same thing an hour earlier. On a one-partition store the two counters cannot be told apart, so a green result there says nothing about this at all.
Access.Kind carried a stated rule: “Nothing writes: a change to a production
database should be something somebody wrote down, reviewed and can run again, which is a declared
operation.” The shell section of this ticket overturns it, deliberately — a
deleteRange for “somebody holding the connection, cleaning up”. The
comment now keeps the old rule in view instead of deleting it, and names the case it never covered:
the operator at two in the morning clearing rows that should not be there. Declaring an operation is
the right answer when those rows will be back next week and the wrong one when they are the
incident.
It is not a hole in the rule for the same reason none of the reads are: a typed delete is the
Operation it would have to be declared as, held to validateOperation, run
down perform() like any other, and handed back by declare — nothing
can be typed there that could not be written down, and every removal is in the change log under the
operator's name. But this is the line SAPE-35 exists to decide,
and it has moved before that decision was taken. Whoever answers SAPE-35 should know they are
answering about a shell that already writes.
walkKeys's undecodable-key refusal has no test. It carries the error
out rather than stopping silently, because a silent stop on a delete would report “removed
three, nothing more to do” and the caller would page straight past the damaged row. A primary
key could not be corrupted from outside the package to exercise it. A guard nobody has
watched fail, said plainly.Result.Changed reports 1 for a retried range delete of any size, because
perform()'s dedup path sets it that way for any repeated write. Already wrong for
batch; now wrong here too. Pre-existing, not widened, and not fixed — its own
ticket.bool range still reads as empty (ISS-39). On a
read that returns nothing; on a delete it means “removed nothing, nothing more to
do” about a range somebody meant to clear. Inherited rather than introduced, and worse here.contribute, and it is the reason this ticket is not simply a
loop somebody writes in an afternoon.Store.Drop, which exists.