RBE Feature Proposal: Frame Delimiting and the Extent Customization Point¶

Summary¶

Give the frame-delimiting model one vocabulary and one extension seam.

Today rbe answers how does a frame find its end? structurally, with payload_extent_of<H, P>, and answers what does it cost to know its length? implicitly — length() exists on every frame and means something different per category. The two questions are independent, and conflating them is what makes a counted frame unrepresentable, a custom frame classify by accident, and many<T> constrained on the wrong property.

The proposal has three parts:

1. name the two axes — mechanism (structural, compile time) and cost (how length() is obtained);
2. turn step 2 of frame construction (where does the payload end?) into a real customization point with a stated contract, so a terminator-delimited, composite-discriminant or body-length protocol swaps one step instead of rewriting make();
3. decide what length() means for each category, and what the library checks versus assumes.

Related: Framing Design Notes §3 (length resolution), §4 (self-delimiting and buffer-delimited frames), §5 (any_id), §6 (unknown ids) and §11 (custom frames), which this document supersedes on the points listed in the Decision Log.

Motivation¶

rbe::frame<H, P> covers the protocols whose end is in the header or in the type. Three shapes in the repo today are outside that premise, and each one is already paying for it:

• Terminator-delimited (cboe TOP): the end is found by scanning for a sentinel. Supported only by writing a whole frame view by hand — example/markets/cboe.hpp:975 and a near-identical copy in tests/runtime/test_dsrl_frame.cpp:294 both re-implement steps 1 and 3 in order to change step 2.
• Counted (frame<HeaderWithCount, many<U>>): the header says how many elements follow. The frame does know where it ends, yet it classifies as buffer_delimited and its length() returns the whole buffer — tests/runtime/test_dsrl_frame.cpp:461 carries the open TODO and navigates with payload.remainder() precisely because length() cannot be used.
• Composite / in-body discriminants (OPRA): the extent follows from two header fields, or from the first two bytes of the payload.

The underlying claim, against the structural premise of notes §3/§4: a frame's delimiting is not always a function of header_type and payload_type. Once that is accepted, the classification needs an input it does not have today, and that input is the customization point.

Inconsistencies in the code today¶

Two invariants the design below leans on, confirmed in the code and written down nowhere:

• base_frame::payload() returns by value, so a stored many hands out a fresh cursor on every call. Payload re-traversal — what an O(n) distance() would need — is therefore already sound.
• header<H>::delimits_payload and explicitly_delimited_frame<F> are equivalent given that the buffer_delimited case is handled first (a length field always wins in payload_extent_of), which is why the swap at dsrl/frame.hpp:81 is behaviour-preserving.

Design¶

The two axes¶

Axis A — mechanism (compile time; structural, or declared where structure cannot say). How is the end found?

Axis B — cost. Given a constructed frame, what does length() cost? The criterion is not "calculable from the type in a number of reads fixed by the type" — that wording excludes line_view, which stores a scanned length and is perfectly O(1) afterwards. The criterion is:

the constructed frame knows its length without traversing anything further.

That admits two ways of getting there — eager (pay the scan in make() and store the result: cboe TOP) and cheap (read a field, add static sizes) — and excludes exactly one shape: a counted payload of variable-size elements, whose length is only knowable by walking the elements.

The scan in the eager case is unavoidable, not hidden cost: the payload is handed an already trimmed span, so somebody has to find the boundary first. The total cost stays linear in the bytes consumed, so there is no per-element O(n) smuggled into iteration.

Bifurcation 1 — where the extent customization point lives¶

These are not all mutually exclusive: 1e composes with any of the others (the annotation lowers to a library-provided policy), and 1c survives under 1a for frames that abandon the shape. Recommendation, if a single answer is wanted: 1a, with 1e layered on later for the terminator case, and 1c kept as the escape hatch.

Bifurcation 2 — what length() means, and how axis B is expressed¶

2a and 2b are the real fork: is the extent always resolved eagerly, and therefore always O(1), or is "non-measurable" a first-class state the library supports? The answer also decides fork 3.

Bifurcation 3 — the counted case¶

3b and 3c compose: take the arithmetic when U has a static stride and the walk otherwise, both behind one counted_extent. 3a is a legitimate default — the user pays no scan they did not ask for — as long as it is documented as a choice rather than left as the only option.

Sub-fork: who consumes payload_count? Today the frame reads it in step 3 and injects it into many's constructor (dsrl/payload.hpp:127,148). Under 3b/3c the count is also extent information, so either (i) the extent rule reads it and many keeps being told as well, or (ii) the count becomes purely extent information and many simply stops at the extent it was handed. (ii) is cleaner but changes the behaviour the new test pins down.

Bifurcation 4 — the result type of step 2¶

Bifurcation 5 — many's stop reason¶

done() has three causes and has_seen_partial() one bit (#9).

• 5a keep it and document the count caveat (status quo bug).
• 5b an enum — consumed / count_reached / incomplete / malformed — with has_seen_partial() kept as stop() == incomplete.
• 5c two predicates: pending() (the count is not satisfied) and truncated() (bytes are left that do not parse), with has_seen_partial() as either of them.

Either of 5b/5c also needs bytes consumed (consumed(), or a length() on many, which #14 argues for anyway): that is what a counted_extent rule (3c) calls, and what would make many a measurable payload.

Bifurcation 6 — how the classification learns about a custom rule¶

• 6a a static constexpr bool self_delimiting on the vocabulary type (notes §11, Decided).
• 6b the extent policy declares it — free under 1a.
• 6c infer it from the view's API. Brittle: line_view already looks self-delimiting by accident.
• 6d a payload_extent::custom enumerator, so the walk in is_self_delimiting() has a branch to land on.

Independently of the fork: make is_self_delimiting() enumerate every case instead of extent != buffer_end (#5, notes §4), and reorder payload_extent_of to test is_any / is_many before wirable (#6).

Bifurcation 7 — who validates the extent¶

• 7a the library, at the call site: report n > rest.size(), reject a zero-length frame, refuse a non-advancing many::next().
• 7b the rule's narrow contract, asserted in hardened builds only.
• 7c both: the wide path (make) reports and the narrow path (the constructor) asserts — the split the rest of the library already uses.

7c is the house style. 7a's many::next() guard is needed either way, since a non-advancing step is an infinite loop rather than a wrong answer.

The contract for an extent rule¶

Whatever fork 1 resolves to, the rule is a function of (header view, bytes after the header), and the compiler cannot check any of this. Hence the laws:

• L1 — Purity. The result depends only on the header's bytes and on rest's bytes. Two calls over equal bytes give equal results. Required because make() and the narrow constructor must agree, and because a re-traversal (distance(), a second payload()) must reach the same boundary.
• L2 — Prefix stability. If the rule yields n for rest, it yields n for every buffer that extends rest; and if it yields incomplete for rest, it yields incomplete for every prefix of rest. Appending bytes never moves a boundary that was already found, and truncating never completes one. This is the law #12 breaks, and the reason the LF-run scan is a bug rather than a design choice: the answer must not depend on how TCP split the stream.
• L3 — Range. 0 <= n <= rest.size(), and header.length() + n > 0. Zero is a valid payload extent (an rbe::empty payload) but never a valid frame length — a zero-length frame is what makes many loop forever.
• L4 — Three states, with meaning. incomplete = these bytes are a prefix of a frame, wait for more; malformed = no continuation can make this a frame, resync; otherwise, the extent. incomplete must not be returned once rest already covers the bound the rule itself declares.
• L5 — The extent is the authority. The frame's length is header.length() + n. The payload view built over those bytes may narrow itself within n — a proxy to its wire size, an any to its candidate — so payload_span() is the full n while payload().as_span() may be shorter. The LF padding of a TOP line lives in exactly that gap.
• L6 — Independence from the payload (recommended, not required). A rule that ignores the id is what lets iteration continue past an id this build does not know; a rule that depends on the id inherits dispatch_delimited's failure mode (notes §6).
• L7 — The declared axis A must match the rule. A rule that consults rest.size() to decide the boundary is not self-delimiting, whatever it declares. Unverifiable, and the one that silently breaks many.

What the library does enforce (fork 7): the range check at the call site, the non-advancing guard in many::next(), and the narrow-path asserts in hardened builds.

Alternatives Considered¶

• Make payload_extent_of recognise custom frames structurally. It cannot: the delimiting of a custom frame is not in its types. That is what forces fork 6 to exist at all.
• Derive a frame from dsrl::frame and shadow make / length. Forgetting length() iterates wrongly in silence; base_frame without primitives makes it a compile error (notes §11).
• A protected resolve() on base_frame. It only customizes length() in practice; replacing a step still means copying frame's private code.
• Caching the resolved extent lazily behind const (a mutable member). Rejected on the same grounds as the any index in notes §5 and the lazy offset table in §8: compute it in the constructor.
• Deframing inside frame — byte-stuffed protocols (SLIP, COBS, HDLC). Still out of scope: their boundary comes from a scanner over the stream, not from a frame's own bytes, and they fit as "outermost frame over what the deframer returned". Note that an extent rule does bring the plain terminator case (no escaping, no stuffing) inside frame, which is a narrower claim than general deframing.
• rbe::any_frame<H, Msgs...> for the composite-discriminant case. The id is just one header field; a rule that needs another one reads it off header().

Open Questions¶

1. Fork 1: which seam, and does 1e (a terminated_by annotation) come now or later?
2. Fork 2: eager-and-stored (2a) or measurable_frame + distance() (2b)? This is the one decision the rest hang off.
3. Fork 3: is a counted frame self-delimiting by default (3b/3c) or only on opt-in (3a)? And does many keep being handed the count, or stop at the extent it was given?
4. Fork 4: does expected<size_type, extent_error> stay inside step 2, or does the incomplete/malformed distinction propagate to frame::make, header::make, any::make and the deserialize entry points? It is the same hole as has_seen_partial().
5. Does many gain a length of its own (consumed() / length())? Required by 3c, and it would make many::as_span() mean what it means everywhere else (#14).
6. Naming: the enum payload_extent versus the function payload_extent (#8). Candidates: extent_source / delimiting_kind for the enum, payload_bounds / extent_of for the function.
7. Does the vocabulary rbe::many<T> get the delimiting constraint, so that the diagnostic lands where the user wrote the type (#13)?
8. Do the delimiting concepts keep applying to lowered types, now that a policy (1a) or an opt-in (1c) lives at one level only? The "lowering never changes delimiting" invariant (same_extent_when_lowered) has to be restated.
9. Length units — words, element counts — still unresolved (notes §3). A counted extent is the first case where the unit is not bytes.

Decision Log¶

• Already implemented: payload_extent_of<H, P> as the single compile-time classification; self_delimiting / buffer_delimited / explicitly_delimited / implicitly_delimited / dispatch_delimited; public building blocks for steps 1–3; base_frame with no primitives; dsrl::many constrained on self_delimiting_frame; payload_count limiting many's iteration (5c59426); many::remainder() as the forward-navigation idiom for buffer-delimited frames.
• Already decided, not implemented (notes §4, §11): many requiring self-delimiting elements at the vocabulary level too; an explicit per-enumerator is_self_delimiting(); the self_delimiting opt-in; rewriting cboe::top::line as a vocabulary serder.
• This proposal adds: the two axes as named vocabulary; counted and custom as first-class mechanisms; the extent rule as a customization point with laws L1–L7; the incomplete/malformed distinction; library-side extent validation.
• Confirmed by the code, not by the docs: payload() returning by value already makes payload re-traversal sound, which is what an O(n) distance() needs.