HTML Tokenizer — Byte Stream to Token Array, cc0 Dark Lane (sigil-apps de47df3)

apps/web/html_tokenizer.sg (sigil-apps de47df3) is the HTML byte-stream tokenizer for sigilOS browser — the first stage of the parse pipeline. It takes raw HTML source bytes and produces a flat token array of (type, offset, length) triples that layout_box.sg consumes to build the box tree. The tokenizer is written in Sigil arm-el0 and holds the cc0 dark lane until the cc0 runtime restarts — when cc0 ships, the cc0 parser takes over this lane permanently and html_tokenizer.sg becomes the validation reference. State machine: INIT→TAG→NAME→ATTR→ATTRN→ATTRE→ENDNAME. All five token types exercised: StartTag, EndTag, Text, Attr, SelfClose.

Token model

Source HTML in src_buf at 0x400000. Token array at tok_buf at 0x500000. Each token is 12 bytes:

Bytes	Field	Description
0–3	`type (i32)`	1=StartTag, 2=EndTag, 3=Text, 4=Attr, 5=SelfClose
4–7	`off (i32)`	Byte offset into `src_buf` where this token's content starts
8–11	`len (i32)`	Length in bytes of this token's content

For StartTag/EndTag/SelfClose: off+len spans the tag name (e.g. h1, img). For Text: spans the text content (whitespace-only tokens trimmed). For Attr: off+len spans the attribute name; the value (if present, after =) is emitted as a separate Attr token immediately following.

The layout_box layer reads tok_buf sequentially: StartTag creates a new node, Text creates an inline text node, Attr sets properties on the current node, EndTag/SelfClose close the current scope.

State machine

INIT

Scanning body text. On <: if next byte is / → go to ENDNAME (EndTag), else go to TAG. On any other byte: accumulate into Text token. On whitespace-only accumulation: trim (emit no token).

TAG

Reading past the <, collecting tag name bytes. On first non-< byte: start name collection, go to NAME.

NAME

Accumulating tag-name bytes (a-z, A-Z, 0-9, -). On space/tab: emit StartTag token for collected name, go to ATTR. On >: emit StartTag, go to INIT. On / then >: emit SelfClose, go to INIT.

ATTR

Between attributes. Skip whitespace. On >: go to INIT. On / then >: emit SelfClose, go to INIT. On a name byte: go to ATTRN to collect attribute name.

ATTRN

Collecting attribute name bytes. On =: emit Attr token for name collected so far, go to ATTRE. On space or >: emit Attr (name only, no value), go to ATTR or INIT.

ATTRE

After =, expect opening quote (" or '). On ": go to collecting attribute value (read until matching "), emit Attr token for value, go to ATTR.

ENDNAME

After </. Collect tag name. On >: emit EndTag token, go to INIT.

Test document — all token types exercised

Test HTML:

<h1>Hello</h1><p class="lead">World</p><img src="x.png" alt="X"><a href="/go">Link</a>

Token stream:

#	Type	Content
0	StartTag	`h1`
1	Text	`Hello`
2	EndTag	`h1`
3	StartTag	`p`
4	Attr (name)	`class`
5	Attr (value)	`lead`
6	Text	`World`
7	EndTag	`p`
8	SelfClose	`img`
9	Attr (name)	`src`
10	Attr (value)	`x.png`
11	Attr (name)	`alt`
12	Attr (value)	`X`
13	StartTag	`a`
14	Attr (name)	`href`
15	Attr (value)	`/go`
16	Text	`Link`
17	EndTag	`a`

All 5 token types hit: StartTag (h1, p, a), EndTag (h1, p, a), Text (Hello, World, Link), Attr (class=lead, src=x.png, alt=X, href=/go), SelfClose (img).

cc0 lane ownership

The comment in the source is explicit: "cc0 owns this lane permanently; Apps holds it until cc0 restarts." When the cc0/JS runtime lands (0.7.0 sprint), the cc0 HTML parser takes over tok_buf and feeds layout_box.sg directly. The html_tokenizer.sg arm-el0 implementation then becomes:

A validation reference for the cc0 parser output
A fallback for non-JS contexts (terminal browser, accessibility mode)
A test fixture for layout_box.sg integration

The tok_buf / src_buf ABI (0x400000 / 0x500000, 12B token triples) is fixed — cc0 writes the same format, layout_box reads the same format. No ABI change when cc0 takes over.

html_tokenizer.sg (arm-el0, de47df3)
  └── src_buf @ 0x400000   ← raw HTML bytes
  └── tok_buf @ 0x500000   ← flat (type, off, len) triples, 12B each
        ↓
  layout_box.sg            ← reads tok_buf sequentially, builds box tree

  [cc0 runtime, 0.7.0]
  └── cc0 HTML parser      ← takes over tok_buf write; same ABI
  └── html_tokenizer.sg    ← becomes validation reference + fallback