← 1.0.0 scope SAPE-32

deleteRange — remove a stretch of keys, with a ceiling

Status
Done
Shipped in
1.0.0 — cfa5e52
Component
server
Priority
Medium — the machinery exists; this is assembly, not invention
Blocked by
—
Blocks
—

Why this ticket exists, and the correction behind it

Asked whether a batch could delete a few thousand rows atomically, this project's planner answered no, and gave as the reason that steps is a field on the declaration so a batch's shape is fixed when declared. That is true and it was the wrong reason.

It describes how the driver models a batch. It says nothing about what the engine may do. The two are different layers, and the promise this project is built on — that cost is declared and nobody writes an expression — constrains only the layer a client speaks to.

The proof was already here. A rollup is the engine computing: it keeps a count and a sum per group and maintains them incrementally. The driver never lets anybody write that as an expression — it lets them declare this rollup. Engine computes, driver declares, and the promise is intact. Any primitive that can be exposed the same way is available on the same terms.

Layer one — should the engine have it?

The two questions are whether the cost can be bounded and whether it is safe for the server. Both answer yes, and the machinery is already in place: tree.Ascend walks a key range and Collection.DeleteBy removes one document. This is those two things in a loop with a ceiling.

The cost is not what "delete a range" sounds like, and the ticket should say so. Reading Collection.remove: each document has to be read before it can go, because its index entries are derived from its contents; then every entry is deleted, then contribute(tree, document, -1) adjusts the rollups, then the document itself goes, then the change is recorded. So the cost is

limit × (one document read + its index entries + its rollup contribution + one log entry)

It is linear in limit and every unit is bounded, which is exactly what makes it declarable. It is not a cheap truncation of a key range, and anybody who reads the name and assumes otherwise will be surprised by the log.

Two consequences to decide rather than discover. N deletions write N log entries, which lands on ISS-23 — no operator can cap retention, so deleting a large file's worth of rows grows the log by that much. And a key range may span partitions, since into(key) picks a tree per key; whether one call may cross that boundary is a decision, not an implementation detail.

Layer two — how the shell offers it

The operator shell already has what this needs: get, scan and count are ad-hoc typed access, available to an operator and not to a client. A deleteRange beside them is the same shape and the same audience — somebody holding the connection, cleaning up.

It should read like scan, because it walks the same way: from and to with the same exclusive forms, limit with the same meaning, and the same answer when it stops early. scan already prints "… and more: this stopped at the limit", and that sentence is what makes a cursor usable without guessing.

Layer three — how the driver declares it

As an operation, with three parameters and no expression:

{ "action": "deleteRange", "collection": "chunk",
  "input": [ {"name":"from","type":"string","required":true},
             {"name":"to","type":"string","required":true} ],
  "from": {"terms":[{"arg":"from"}], "exclusive": true},
  "to":   {"terms":[{"arg":"to"}]},
  "limit": 500 }

The shape is fixed at declare time, the ceiling is in the declaration where every other ceiling on this store lives, and the caller supplies bounds rather than behaviour. The cost envelope (SAPE-8) can state it before anything runs, like any other operation.

It returns what a caller needs to continue: how many went, the last key removed, and whether it stopped at the limit. That is a cursor, and it is the same one scan already hands back.

The ceiling is counted, not trusted

These three primitives are the first things in this store that walk a range at run time. Everything before them is bounded by construction: a declaration cannot express a loop, because a step pins operation@version, version numbers only ever rise, and a pinned reference resolves only to a version that already exists — so a cycle would need a version to exist before it was declared. The reference graph is a directed acyclic graph because nothing can build anything else, which is why ceiling() can compute a total at declare time and why nothing counts during a run.

That property does not survive a thing that walks. So the limit on this operation is a ceiling the host counts against while walking, not a number the host takes on trust, and it is counted per run of the operation rather than per underlying call.

What happens at the ceiling was wrong on this page, and the code is right. This paragraph used to end: “At the ceiling the operation stops and refuses; it does not return a shortened answer, because a truncated result is indistinguishable from a complete one.” That sentence was written for SAPE-34 and pasted into all three of these tickets unchanged. It is true of a digest — a hash of part of a range looks exactly like a hash of all of it — and false here, because this answer carries the last key removed and a flag saying it stopped early. Three other places on this page say so plainly: the shell section asks for “the same answer when it stops early”, the driver section calls it “a cursor, and the same one scan already hands back”, and acceptance criterion 1 measures “a thousand rows in pages”, which no refusal could satisfy.

The decisive argument is not textual. When the ceiling is reached the rows are already gone. Refusing would not un-remove them; it would only withhold from the caller which ones went. That is the silent failure this project spends its documentation arguing against, dressed as strictness. So: the operation stops, and the answer it returns says it stopped. The ceiling still binds — it is still counted by the host, still per run, and it still refuses to go further.

The reason to write this down rather than assume it is measured, in pipelines/tasks/0071, measurement W7. An external operation declaring limit 50, correctly sandboxed and correctly signed, served 50,000,000 rows in one call — one million calls of a host function that served fifty each — because the host checked every individual call and never the total. A signature proves whose binary it is. A sandbox proves it does not escape. Neither proves what it costs. That gap is not open today because nothing can loop; it opens the moment something can.

What is counted here is rows removed. Each one costs a document read, its index entries, its rollup contribution and a log entry, so the row count is the honest unit and the log grows with it.

What shipped, and the guard that had to be rebuilt

Landed in cfa5e52, CI green on both jobs. Twenty-one new top-level tests and twenty-six mutations, each one watched going red.

One mutation survived the first pass, and the guard was fixed rather than the report. Removing contribute(…, -1) from remove() — so a delete stops decrementing the rollups it contributed to — left the test green. The assertion was relative: it compared a deleteRange store against a store where the same rows were deleted one at a time. The mutation breaks both paths identically, so two wrong answers agreed with each other. The fix is an absolute check: the rollup is now compared against the rows actually still in the collection, recomputed by walking them. Re-run, it goes red. This is the rule about a guard not reading its expectation from the thing it guards, failing in a form that looks like diligence — the comparison was real, it was just comparing two copies of the same mistake.

And the per-run ceiling is only measurable across a partition boundary. A mutation giving each partition file a fresh budget — the literal W7 shape — leaves all twelve single-partition tests green and is caught by exactly one test, the one built to span two trees. SAPE-34 recorded the same thing an hour earlier. On a one-partition store the two counters cannot be told apart, so a green result there says nothing about this at all.

It writes from the shell, and that reverses a sentence

Access.Kind carried a stated rule: “Nothing writes: a change to a production database should be something somebody wrote down, reviewed and can run again, which is a declared operation.” The shell section of this ticket overturns it, deliberately — a deleteRange for “somebody holding the connection, cleaning up”. The comment now keeps the old rule in view instead of deleting it, and names the case it never covered: the operator at two in the morning clearing rows that should not be there. Declaring an operation is the right answer when those rows will be back next week and the wrong one when they are the incident.

It is not a hole in the rule for the same reason none of the reads are: a typed delete is the Operation it would have to be declared as, held to validateOperation, run down perform() like any other, and handed back by declare — nothing can be typed there that could not be written down, and every removal is in the change log under the operator's name. But this is the line SAPE-35 exists to decide, and it has moved before that decision was taken. Whoever answers SAPE-35 should know they are answering about a shell that already writes.

Named debt

Acceptance criteria

  1. A range of keys is removed up to the declared limit, and the answer names the last key removed and whether more remain — measured by deleting a thousand rows in pages and reading the collection back empty.
  2. Indexes and rollups agree afterwards. A rollup over the deleted range reads the same as if every row had been deleted one at a time. This is the assertion that would catch a fast path that skipped contribute, and it is the reason this ticket is not simply a loop somebody writes in an afternoon.
  3. The cost envelope reports the declared limit, and a caller can read it before invoking.
  4. The shell command and the declared operation produce the same result on the same range, because two spellings of one primitive that disagree is worse than one spelling.
  5. Every new guard watched failing.

Out of scope