← 1.0.0 scope SAPE-31

A type for bytes, because a string silently is not one

Status
To do
Component
server
Priority
High — it is a correctness gap, not a space one
Blocked by
SAPE-29
Blocks
—

What was measured

A field declared string, given a value containing the byte 0xFF, through a real daemon: 

  sent   61 ff 62        ("a", 0xFF, "b")
  stored 61 ef bf bd 62  ("a", U+FFFD, "b")
  answer changed 1

Accepted. Not refused, not warned about. The byte is replaced with the Unicode replacement character on the way in, and the original is unrecoverable. A string is UTF-8 text and the wire encoding enforces that; it is not a byte container and never was.

One thing this measurement does not say. A first attempt sent a NUL byte and saw the value truncated, which looked like a second bug. It was the test harness: a process argument cannot carry a NUL, so the value was cut at the exec boundary and never reached the database. Reported here because the wrong version of this ticket would have carried it.

Why this is correctness and not economy

The obvious workaround is base64 in a string, and it works. But that makes base64 mandatory rather than conventional, and three things follow that a caller is not told:

The question that prompted this ticket was whether sapedb has an equivalent of Postgres BYTEA. It does not, and the honest form of that answer is this ticket rather than a paragraph telling people to encode it themselves.

What it needs

  1. A wire representation. JSON has no byte string, so the value travels as base64 — but declared as bytes, so the store decodes it once and stores bytes. The difference from today is that the encoding is the protocol's job rather than the caller's, and a value that is not valid base64 is refused by name instead of stored as text.
  2. A tag in the key encoding, sorting bytewise, so a byte prefix range scan is expressible. The existing string encoding already escapes 0x00 and terminates with 0x01; bytes need the same treatment or a length prefix, and which one is a decision because it changes what sorts before what.
  3. A size ceiling. Still the server protecting itself from a client rather than a consequence of the declared-cost promise — that framing stands. But one already exists at the frame layer: protocol.MaxPayload is 16 MiB, enforced when a frame is written and again when one is read, and its comment gives the reason — a length field is a promise about an allocation, and this is the limit on how much a peer can make the server believe. So the gap is narrower than this ticket first claimed: what is missing is a ceiling on a single value or document, and a ceiling on what one database may hold in total. A 16 MiB frame already bounds one message; it bounds nothing about ten thousand of them. The driver may check early as a convenience, but the server is where it has to be enforced, because a check only the client performs is not a limit. The earlier version of this item was wrong to say that no cap existed: the search that went looking for one never included the constant's own name, MaxPayload, and a search that found nothing was read as a thing that is not there.
  4. A mapping in the TypeScript generator — Uint8Array is the obvious emit, and the obvious emit should still be written down.

Acceptance criteria

  1. The three bytes 61 ff 62 go in and come out identical, through a real daemon, read back from the file. That exact value, because it is the one measured to corrupt today.
  2. A value that is not valid base64 in a bytes field is refused by name, not stored.
  3. An index over a bytes field sorts bytewise: 00 before 7f before ff, which base64 text does not.
  4. A string field still refuses nothing it accepts today. This ticket adds a type; it does not tighten the old one — and if the group decides string should reject invalid UTF-8, that is a separate decision with a deadline, because tightening cannot happen after the tag.
  5. Every new guard watched failing.

Out of scope