SIGILOS · BUILD JOURNAL

Build Journal

The metal bring-up saga — getting sigilOS to render and run on real hardware.

Build Journal — The Metal Bring-Up Saga

This is the engineering log of getting sigilOS to render on real hardware — not QEMU, not OVMF, but @grio's physical x86-64 box and a stack of Raspberry Pis. It's the messy part: the part where "PASS in the emulator" and "works on metal" turn out to be different claims, and the gap between them is where the real bugs live.

Two tracks ran in parallel: the x86 login bring-up (VER series) and the Pi login bring-up (BUILD series). Both converged on the same lesson — the emulator lies by being too forgiving.

Track 1 — x86: pink → green → rects-on-metal

The x86 login screen had to render correctly on @grio's GOP framebuffer (1280×800, pixel format 1) before anything else mattered. The bring-up was a bisection, one VER at a time, each isolating one more variable.

VER Result on metal What it proved
VER=4 pink + frozen framebuffer reached, but wrong color order + a hang
VER=8 pink, frozen still wrong — the format guess was off
VER=9 solid green, stable ✅ correct color (G=0xFF format-robust in both RGBX+BGRX), stable paint before any xHCI. Serial M3IN V9GRN w=1280 h=800 fmt=1
VER=10 green + login-card → colors WRONG + freeze the login-card render path breaks — bisection target found
VER=11 login card raw-rects render PERFECT on metal ✅ raw-rect packing + geometry correct: green bg, navy card, blue title bar, pink+cyan accent rects, two input bars. Stable, no freeze. Freeze isolated to glyph/text
VER=12 🎉 FULL LOGIN SCREEN ON METAL ✅ glyph text back on the proven card — "sigilOS" titlebar, "sign in to continue", grioghar(admin) + dev(standard) user cards with avatars, password field, Sign-in button. Crisp, correct colors, stable. Serial V11:ABCDEF → FNT:0GHIJKLMN → V12LGN

VER=12 is the milestone: @grio's photo shows the complete, correct sigilOS login screen rendering on real x86-64 hardware. The x86 real-metal render gate is closed. The bisection paid off — VER=11 proved the rects, VER=12 added glyphs (backed by cc0's bounded font_raster, 43eb721) and fixed the VER=10 freeze at its root.

The last x86 item is VER=13 — xHCI keyboard, so @grio can type the password and actually sign in. Drivers shipped hid_keyboard_xhci (b071c0e); wiring it into the x86 login (Enable-Slot → Address → HID report) is the final step to an interactive x86 desktop on metal.

The keyboard grind — VER=13 → 15

Rendering was the first half. Typing is the second, and it reran the exact same QEMU-vs-metal story.

VER QEMU (OVMF) @grio's metal What it added
VER=13 (fabbfbf) signs in ✅ login renders, no keyboard Enable-Slot → Address-Device → Config-EP → HID-boot-report; keystroke → password field → verify → sign-in
VER=14 (b241690) signs in keyboard plugged, port-connect still RED PORTSC.PP port-power + numeric on-screen port readout
VER=15 (1545dc3) signs in enum still not completing scan all 8 root ports, longer CCS settle + re-poll

The signature again: passes in QEMU, stalls on metal. QEMU auto-powers its xHCI root ports; real hardware needs explicit PORTSC.PP (port-power bit) plus a debounce settle before the connect-status bit goes valid. @grio confirmed a keyboard physically plugged in, yet marker 1 (port-connect) stayed red — the controller wasn't seeing the device. VER=14 set port power; VER=15 widened the port scan and settle window. As of this writing the x86 keyboard is the active metal blocker — the render is done, the input path is the last mile.

The Pi UART grind — BUILD 18 → 19, and the on-screen-debug rule

The Pi track kept fighting the same UART. cc0's fix v2 (faf6c54, seed bb49744a) — D-cache off, MMU on — landed on main, and BUILD 19 (a7e6a33e) built against it. On @grio's Pi-4B metal: the screen renders the login (so D-cache-off + MMU-on preserved the framebuffer path and the firmware map), but the kernel's UART is still silent at 115200. Firmware UART works, kernel UART doesn't — the fault is narrowing but not yet closed, and after too many one-variable-per-boot cycles the Director called it: one build that closes the whole fix space (no-reinit + DSB + flow-off + FIFO-drain) rather than another blind bisection step.

Out of that grind came a standing rule (grioghar) that reshapes how we debug bring-up: all debug/diagnostic codes print ON-SCREEN, not just to UART. The framebuffer is the universal debug surface — it works on every board whether or not a serial cable is attached, and x86 has no serial at all. Every stage marker, register value, and enum step now renders as a compact persistent on-screen overlay (the kernel's boot_diag ring + Video's boot_readout lattice + the per-stage corner readouts). The reason is concrete: @grio debugged VER=14/15's keyboard enum by reading colored squares off the screen — there was no other channel. The UART silence on Pi made the same point from the other direction. The screen is the one output you can always trust to reach the human.

🎉 The Pi UART win — it was the GPIO mux

And the on-screen rule immediately paid for itself. The Director cracked the Pi UART by reading the PL011 register grid off a photo of @grio's screen — and the values told the story: the kernel had only ever read GPFSEL1, never written it. It trusted the firmware's GPIO pin-multiplexing instead of selecting ALT0 (the PL011 function) for the UART's TX/RX pins itself. The cached-MMIO fix was real and necessary — but it un-cached writes to a UART whose pins weren't muxed to the UART. Two independent faults stacked: uncached (the cache fix) and mapped to the right peripheral function (the GPIO fix). BUILD 20 (000755d6) dropped its own pl011_init entirely and inherited the firmware's already-correct PL011 + pin-mux — and KKKK PILOGINB20 ok=1 came over the FTDI at 115200. The Pi serial-on-metal gate is closed. Pi now renders and talks.

The unifying pattern: cached MMIO on both arches

The deepest result of the whole saga is that the x86 and Pi bugs turned out to be the same bug. When the x86 keyboard gave the same failure across VER=14, 15, and 16 — three different downstream fixes, identical result — the Director made the call: an unchanging failure under changing downstream code means the fault is upstream of all of it. The xHCI controller was never running, because EFI hands off the xHCI BAR cached, exactly like the Pi UART. VER=17 (ffeb86b) forces it uncached — cc0 only exposes CR0/CR4 (no CR3 walk, no wrmsr/MTRR/PAT), so the fix uses CR0.CD (cache-disable) before controller init. Same disease, two arches, one named cause: MMIO must be uncached, and the emulator never enforced it. That's the through-line of the entire bring-up — QEMU models neither MMIO cacheability nor GPIO muxing strictly, so every bug that depended on them was invisible until metal.

A side-effect: the build harness went all-Sigil

The 15-minute compile that made every metal iteration painful had a second problem: the metal build harness was Python (build.py + keystone-engine arm_emit), a standing violation of the project's build-tools-in-Sigil directive. grioghar called both out as one issue, and cc0 closed it: tools/build_native_cc0.sh + tools/metal_build.sh now drive the native cc0 backend to emit bootable Pi/x86 images directly — zero Python, zero keystone, no interpreter in the emit path (cc0 059a62e). The CI gate proves it: stage E has native cc0 emit a raw Pi kernel with its own backend and asserts it's not an ELF. The grind that exposed the cache/mux bugs also forced the toolchain to finish self-hosting its own image emit — the slow Python path is gone, and so is the last Python in the build.

The keyboard, by LED — and how it was won

Update (June 29): the Pi 4B keyboard now types on metal. @grio pressed a letter and the character appeared on the login screen — ~100 build-flash-cold-cycle iterations later. The section below is the method that got us there; the full layer-by-layer story is its own post: The Pi4B Keyboard Lives.

With both boards rendering the login, the last mile is input — and the breakthrough in how we debug it came from grioghar: watch the keyboard's lock-LED. A USB keyboard flashes its lock-LEDs the instant VBUS is applied, before any enumeration. That single observable splits the entire problem:

Reading the LED off @grio's hardware split the two boards cleanly:

Board LED Diagnosis
x86 lights ✅ VBUS good (it's hardware, live at POST). x86 is an enumeration problem — drive RUN → PORTSC.PP → CCS → reset → Enable-Slot → Address-Device → GET_DESC → SET_CONFIG → ConfigEP → SET_PROTOCOL(boot) → interrupt-IN reports
Pi 4B dark VL805 isn't driving VBUS → the xHCI isn't actually brought up. Power problem — PCIe RC up → VL805 firmware → xHCI RUN → PORTSC.PP before any enum can matter

This produced the working method for the whole keyboard push: address every layer per build, not one per boot — power/VBUS, controller init (CNR→HCRST→RUN), port (PP→debounce→CCS→reset), enumeration (the full chain above) — and instrument each stage on-screen (the kernel's PORT PWR Y/N banner, Drivers' xhci_trace_cell per-substage prints) so @grio reads exactly which layer stalled. The 15-minute-build era of one-variable-per-boot is what made that discipline non-negotiable.

That discipline carried the Pi 4B all the way home. With the LED confirming VBUS, the failure was never a mystery — just a stack of named walls, each with an on-screen probe pointed at it: off-bus VL805 (NOTIFY-last reinit kept it on the PCIe bus), AddressDevice AD24 split-transaction error (drop the HS→FS clamp + fix EP0 max-packet-size → AD01), garbled bytes (a missing pre-doorbell dsb() let the controller read a stale ring; plus FIFO pacing), the it's-actually-a-hub turn (DVCL=0x09 — the real keyboard hangs off port 4 at full speed behind the hub's Transaction Translator, so hub enumeration and TT think-time had to be built from scratch), an EP0 control-endpoint STALL (RESET_ENDPOINT + Set-TR-Dequeue recovery + a two-stage max-read), a wrong-interface bind on a composite device (select the int-IN endpoint on bInterfaceProtocol==1), and finally babble→short-packet (size the report buffer to MPS, accept the cc=13 short-packet as success). Then RPT=0800 decoded to 'e' — real scancodes, real letters, drawn into the password field. (One honest correction logged along the way: a suspected dead-code return 0 blocking the x86 keyboard turned out to be a stale checkout — the live x86 image ships the xHCI path inline, no early return.)

The new frontier is the same hub with a mouse alongside the keyboard — multi-device enumeration behind one VIA HS hub. Early metal runs surfaced a speed-misdetect (a stale heartbeat read mis-flagging an FS device as Low-Speed, since fixed) and a remaining TT/hub-slot-context gap on AddressDevice for the second device. The render, the UART, and the keyboard each fell to the same loop; the mouse is next.

Track 2 — Pi: BUILD 12 → cached-MMIO → BUILD 18

The Pi track hit a subtler wall. The kernel loaded and rendered the login screen (we could see it), but produced zero serial output on @grio's metal — while the exact same image printed its K proof-of-life characters fine in QEMU.

That divergence was the diagnosis. Code root-caused it: cached MMIO. The UART's memory-mapped registers were being mapped cacheable, so writes sat in the cache instead of reaching the hardware. QEMU doesn't model MMIO cacheability strictly, so it "worked" there — the emulator being too forgiving, exactly the recurring theme.

The fix lived not in any .sg driver but in cc0's ARM boot stub (metal_entry). And it took two tries to get right:

The deeper lesson here: "uncached" and "unmapped" are not the same thing, and conflating them cost a build cycle. The bisection discipline that worked on x86 — change one variable, observe, narrow — is exactly what untangled it: screen renders + UART silent is a conclusive signature (RAM reachable, MMIO not), not an ambiguous failure.

Track 3 — Lumen: the desktop, and the compiler that lied

With the screen rendering, the UART talking, and the keyboard typing, the last mile was the desktop — a cold-boot one-shot that auto-logs in and brings up the full Lumen compositor, on metal. It produced the strangest bug of the whole bring-up, and the most fitting.

The wall was the compiler — the Python one. The auto-login path hung at the same spot on every boot: it painted PAL=1, then froze. The gate was provably correct — pi_autologin() literally returns 1 — yet the build wedged in id_pw_digest, the very next function. The split that cracked it: builds emitted by the Python reference interpreter hung (DBLD=155/156/157); the one built by the native cc0 backend did not. The interpreter's code generator was miscompiling that function — a deterministic defect, not logic, not chance. The fix was the instruction grioghar had just given the whole fleet: use the native compiler, not Python. Rebuilt native on the Proxmox runners, the hang vanished and AUTO fired on metal for the first time (DBLD=158, 7f530647).

Then a trail of codegen hazards, read off the screen. Past auto-login, the desktop's first paint hit a cluster of hangs in the window-chrome renderer — each localized the same way the keyboard was, by painting a trace marker before every step and reading which was last: TD3 (the welcome-window spawn), LA2 (lma_draw_chrome, the title/border), DC2 (lma_draw_dot, the traffic-light buttons). The common root was a single AArch64 codegen hazard — a decrement-past-zero loop (while X >= 0 { X = X - 1 }) the backend mis-emitted into a non-terminating loop — found and fixed (Video 11b3a27), then swept for across every repo's source (a sibling negative-start pattern, var X = 0 - N, audited alongside it).

LMDN. With the last render bug cornered, the call was to stop chasing the welcome window, gate it off, and prove the bare desktop paints. DBLD=164 (36fea04f, native, zero Python) did exactly that: AUTO → TD3 → LMDN, then a steady render loop — gradient wallpaper, status bar, dock, DMA-presented, no fault. Confirmed over UART on the Pi 4B, and then by grioghar on the panel in front of him: “I see Lumen.” The full story is its own post: The Lumen Desktop Is Up On Metal.

This track added a second axis to the journal's recurring lesson. The emulator-vs-metal gap had a sibling all along: an interpreter-vs-native-compiler gap. “PASS on the interpreter” was no more sufficient than “PASS in QEMU” — the native compiler is now the floor, and the toolchain itself became something to verify. (The same week, cc0 went native on Apple Silicon too — 896a833 — taking the Python interpreter fully off the critical path.)

What the saga forced into the architecture

The bring-up didn't just fix bugs — it changed the design.

The unified Pi image. Three Pi boards (3B, 4B, 5) with three peripheral bases (0x3F / 0xFE / 0x107C), three USB stacks (dwc2 / VL805-xHCI / RP1-xHCI), and different clocks. Rather than three SD cards, the directive (grioghar) is one kernel on one SD that detects the SoC at boot and scales up from the Pi-3B floor. Kernel shipped the foundation (6c22408): boardsel.sg binds the right HAL by peripheral base; cap_tier.sg sets the capability tier (Pi3 floor → Pi4 +xHCI/HDMI/PCIe → Pi5 +RP1/compute; unknown → floor). Verified CAPT pi3=1 pi4=1 pi5=1 unk=1. This makes the project's standing "scale with hardware" rule literal in the boot path.

The boot-diagnostics launcher. The metal-bringup pain became a feature: the Ex Machina boot animation now doubles as a BIOS-style hardware readout — each detected component lights a node on a capability-topology lattice, and every boot stage emits a structured, AI-analyzable log entry (Kernel boot_diag.sg + panic_log.sg; Video boot_readout.sg; FS /logs/boot-<n>.log sink, fs_logs.sg shipped fe55737). The next time something diverges between QEMU and metal, the board tells you what it detected and where it stalled.

The recurring lesson

Three separate bugs — x86 color format, x86 glyph freeze, Pi cached-MMIO — and one root cause behind two of them: QEMU is too forgiving. It tolerates a pixel format guess, it doesn't enforce MMIO cacheability. "PASS in the emulator" is necessary but not sufficient; the Director standing rule — QEMU screendump required on every test pass — is the floor, and real-metal proof is a distinct 1.0 gate above it.

The bring-up is not done — VER=12 glyph rendering on x86, BUILD 18 confirmation on Pi metal, and the cached-normal MMIO cleanup are open. But the hardest part is behind us: we can render correctly on real x86 metal, and we can get serial on real Pi metal. The emulator-to-metal gap is now a list of named, cornered bugs instead of a fog.

Journal maintained by Sigil-Docs. Updated as the bring-up advances.