deleteRange — remove a stretch of keys, with a ceiling
Asked whether a batch could delete a few thousand rows atomically, this project's planner answered
no, and gave as the reason that steps is a field on the declaration so a batch's shape
is fixed when declared. That is true and it was the wrong reason.
It describes how the driver models a batch. It says nothing about what the engine may do. The two are different layers, and the promise this project is built on — that cost is declared and nobody writes an expression — constrains only the layer a client speaks to.
The proof was already here. A rollup is the engine computing: it keeps a count and a sum per group and maintains them incrementally. The driver never lets anybody write that as an expression — it lets them declare this rollup. Engine computes, driver declares, and the promise is intact. Any primitive that can be exposed the same way is available on the same terms.
The two questions are whether the cost can be bounded and whether it is safe for the server. Both
answer yes, and the machinery is already in place: tree.Ascend walks a key range and
Collection.DeleteBy removes one document. This is those two things in a loop with a
ceiling.
The cost is not what "delete a range" sounds like, and the ticket should say so.
Reading Collection.remove: each document has to be read before it can go,
because its index entries are derived from its contents; then every entry is deleted, then
contribute(tree, document, -1) adjusts the rollups, then the document itself goes, then
the change is recorded. So the cost is
limit × (one document read + its index entries + its rollup contribution + one log entry)
It is linear in limit and every unit is bounded, which is exactly what makes it
declarable. It is not a cheap truncation of a key range, and anybody who reads the name and assumes
otherwise will be surprised by the log.
Two consequences to decide rather than discover. N deletions write N log entries,
which lands on ISS-23 — no operator can cap retention, so deleting a
large file's worth of rows grows the log by that much. And a key range may span partitions, since
into(key) picks a tree per key; whether one call may cross that boundary is a decision,
not an implementation detail.
The operator shell already has what this needs: get, scan and
count are ad-hoc typed access, available to an operator and not to a client. A
deleteRange beside them is the same shape and the same audience — somebody holding
the connection, cleaning up.
It should read like scan, because it walks the same way: from and
to with the same exclusive forms, limit with the same meaning, and the same
answer when it stops early. scan already prints
"… and more: this stopped at the limit", and that sentence is what makes a cursor usable
without guessing.
As an operation, with three parameters and no expression:
{ "action": "deleteRange", "collection": "chunk",
"input": [ {"name":"from","type":"string","required":true},
{"name":"to","type":"string","required":true} ],
"from": {"terms":[{"arg":"from"}], "exclusive": true},
"to": {"terms":[{"arg":"to"}]},
"limit": 500 }The shape is fixed at declare time, the ceiling is in the declaration where every other ceiling on this store lives, and the caller supplies bounds rather than behaviour. The cost envelope (SAPE-8) can state it before anything runs, like any other operation.
It returns what a caller needs to continue: how many went, the last key removed, and whether it
stopped at the limit. That is a cursor, and it is the same one scan already hands back.
These three primitives are the first things in this store that walk a range at run time. Everything
before them is bounded by construction: a declaration cannot express a loop, because a step pins
operation@version, version numbers only ever rise, and a pinned reference resolves only
to a version that already exists — so a cycle would need a version to exist before it was
declared. The reference graph is a directed acyclic graph because nothing can build anything else,
which is why ceiling() can compute a total at declare time and why nothing counts during
a run.
That property does not survive a thing that walks. So the limit on this operation is a ceiling the host counts against while walking, not a number the host takes on trust, and it is counted per run of the operation rather than per underlying call. At the ceiling the operation stops and refuses; it does not return a shortened answer, because a truncated result is indistinguishable from a complete one.
The reason to write this down rather than assume it is measured, in pipelines/tasks/0071,
measurement W7. An external operation declaring limit 50, correctly
sandboxed and correctly signed, served 50,000,000 rows in one call — one
million calls of a host function that served fifty each — because the host checked every
individual call and never the total. A signature proves whose binary it is. A sandbox proves it does
not escape. Neither proves what it costs. That gap is not open today because nothing
can loop; it opens the moment something can.
What is counted here is rows removed. Each one costs a document read, its index entries, its rollup contribution and a log entry, so the row count is the honest unit and the log grows with it.
contribute, and it is the reason this ticket is not simply a
loop somebody writes in an afternoon.Store.Drop, which exists.