Convert the remaining expect-based test scripts (phase1-6 runtime, usb,
wifi, bluetooth, dbus, greeter, live-iso, posix) to the stdlib python
harness, completing the migration started in 9ec00d4c81. Each script now
prefers booting the live ISO when present (falling back to harddrive.img
with snapshot=on), and retries up to 5 times when QEMU exits early or is
killed (rc 3 / 137) — the host-contention failure mode recorded in the
2026-07-20 runtime-proof status.
qemu-login-expect.py: treat an incomplete step sequence as a failure even
when a pass marker was seen partway through — partial sequences must not
count as a pass.
Post-review fixes:
- Phase 2 no longer breaks early when only --fail_marker is set: the
'if not args.pass_marker: break' guard skipped all output reads, so a
fail marker was never detected (exit 0 instead of 1). Now breaks only
when neither pass nor fail markers are configured.
- docstring referenced --pass/--fail; argparse registers
--pass_marker/--fail_marker. Align the docstring.
- retry echo said 'attempt N/3' but the loop limit is 5 attempts; say /5.
- warn and skip malformed steps (missing expect:/send: prefix) instead of
spinning in select until timeout.
IRQ plan + validation matrix: the login-based QEMU proofs (PS/2 + serio,
monotonic timer, IOMMU first-use, USB storage BOT) now run through
local/scripts/qemu-login-expect.py (stdlib python) instead of the host expect
tool. MSI-X (virtio-net) and xHCI interrupt-driven mode were re-confirmed on
the current redbear-mini ISO. Re-confirmation of the login-based legs with the
new harness is pending an uncontended host (parallel QEMU workload kills QEMU
processes and stalls guest boots). Also record the fbcond Performing-handoff
boot freeze observed with -vga std (separate graphics-path issue).
Drop the host expect dependency from the PS/2, timer, IOMMU, and USB
storage (BOT) runtime proofs. local/scripts/qemu-login-expect.py drives
the guest over serial-on-stdio with ordered expect/send steps, per-step
timeouts (matching expect's per-pattern semantics), and pass/fail markers.
It distinguishes premature QEMU death (exit 3) from genuine check failure
(exit 1) so the scripts retry external interruptions without retrying real
failures. Scripts use headless -display none -vga none (fbcond serial
mirror), a 3600s per-step timeout, and a retry loop (max 5 attempts).
All six validation scripts (xhci-irq, msix, ps2, timer, iommu,
usb-storage) booted build/<arch>/<config>/harddrive.img, which is only
produced by older make-all flows and goes stale (the Jul-17 image).
Running them against it measured the wrong tree — the same trap that
produced the false '+rb0.3.0' bootloader reading and the first hub-test
failure. Each now prefers build/<arch>/<config>.iso via -cdrom and only
falls back to harddrive.img when no ISO exists. extra.img and
usb-storage.img remain as data drives in both modes.
Correct the P3-A runtime-validation note and matrix row: the
intermittent apparent boot freeze is startup latency under load (guest
clock ~10-15x slower than wall), not a deadlock or an xhcid bug. A short
50-60s test timeout occasionally cuts the spawn+enumerate sequence,
producing a 'frozen' log while the guest is still running — proven by
18 boots (one apparent short-timeout loss), a 12s log-quiet 'freeze'
the guest resumed from, and every >=180s run enumerating fully. No fix
required beyond adequate test timeouts; no speculative kernel-deadlock
fix was implemented.
The usb-kbd behind the QEMU usb-hub now enumerates fully: debounce ->
port reset -> enable -> second debounce -> pre-attach -> attach ->
ATTACH 5.2 -> slot 3 -> addressed -> descriptors read -> HID class match
-> 'Loading subdriver "USB HID" for port 5.2' (class 3.1 proto 1,
keyboard boot protocol).
Root cause of the final stall: an unconditional SetPortFeature
(PORT_INDICATOR) for every connected+enabled port NAK-hung on QEMU's
indicator-less hub (no control-transfer timeout), blocking the flow
before the post-enable debounce. Now gated on
wHubCharacteristics.HUB_CHAR_PORTIND like Linux has_indicators.
test-usb-hub-qemu.sh: check #5 now requires the hub-child subdriver
line ('Loading subdriver "USB HID" for port <root>.<child>') so a
root-port HID (tablet) can no longer satisfy it — closes a false-
positive hole. Timeout raised to 360s for slow TT-path driver bring-up.
Docs (plan P3-A runtime validation + matrix): full hub + hub-child
chain recorded as QEMU-proven; remaining items (intermittent reactor
startup race, slow TT-path Evaluate Context) noted.
Hub enumeration is quick, but a hub-child's class driver then performs
its own descriptor/control transfers on the slow emulated TT path, and
the guest clock runs several times slower than wall time under load.
180s cut the boot before a hub-child HID driver could finish
registering after being spawned.
USB-IMPLEMENTATION-PLAN.md P3-A: record the 2026-07-19 QEMU runtime
validation (test-usb-hub-qemu.sh) — full usbhubd chain proven
(descriptor, EP1 change detection, initial scan, debounce, reset to
port-enable), the five latent bugs it surfaced and that were fixed
(scheme attach bootstrap, bitmap off-by-one, missing initial scan,
unbounded EP1 wait, debounce clear deadlock), QEMU's ClearPortFeature
stall handled gracefully by P2-C recovery, and the remaining gap:
hub-child device enumeration (usb-kbd to HID driver) blocked on
xhcid's hub-child attach path (debug-level; TT/route investigation)
plus an intermittent xhcid IRQ-reactor startup race.
HARDWARE-VALIDATION-MATRIX.md: hub row to QEMU-validated for the
enumeration chain, with the hub-child gap noted.
Serial output from dozens of concurrent daemons interleaves mid-line,
so full-sentence matches like 'usbhubd: 8 port(s) detected' fail even
when the content is present (observed mangled as 'usbhubd: 548: port(s)
detected'). Match short fragments ('port(s) detected', 'change
detection', 'registered producer') that survive interleaving.
Runtime log output is user-visible system surface; cross-references to
other operating systems belong in code comments and commit messages
(per project convention), not in runtime logs. The debounce parameters
and the comment-level references stay unchanged.
The 'USB HUB driver spawned' log line is emitted before setup_logging
and never reaches the serial console, so check #2 could never pass.
Use the hub-descriptor read line (emitted only after XhciClientHandle
open + 8-port descriptor fetch succeed) as the spawn proof, and fold
the old duplicate descriptor check into it.
Cat 1 discovery treated every local/recipes/*\/source/Cargo.toml as an
in-house crate, including git-fetched upstream sources like brush
(git = github.com/reubeno/brush). Rewriting upstream crate versions to
the branch version broke upstream-internal version requirements
(brush-shell was forced to 0.3.1 while brush required ^0.4.0 —
'failed to select a version for the requirement brush-shell = ^0.4.0').
Guard: skip any crate whose Cargo.toml is not tracked by the parent
repo. In-house crates are tracked trees (no independent .git); fetched
sources are gitignored and untracked. Restores correct classification:
75 crates checked (12 fetched-source crates now excluded), drift 0,
brush cooks successfully again.
- sync-versions.sh: 11 brush workspace crates 0.x -> 0.3.1 (Cat 1
in-house version policy).
- test-usb-hub-qemu.sh: boot the live ISO (canonical build-redbear.sh
artifact) instead of harddrive.img, which is only produced by older
make-all flows and goes stale — a stale Jul-17 image with a
pre-version-sync bootloader (+rb0.3.0) masked the current build and
crashed the first proof attempt. Falls back to harddrive.img only
when no ISO exists.
The script cd's into the fork dir for the clean-tree check (line ~202)
and never returns to PROJECT_ROOT. Every subsequent relative
'cd "$fork_dir"' subshell then fails with 'No such file or directory',
making single-fork and multi-fork runs always FAIL at the upstream-ref
resolution step. Make fork_dir absolute at assignment so all subshell
cd/git calls work regardless of current directory.
Found while attempting the base fork sync (verify-fork-functions gate).
Track base pointer to bd379e07 (usbhubd debounce/reset-wait/U0-wait/
wHubDelay + SET_ISOCH_DELAY, HubDescriptorV3 device_removable fix).
USB-IMPLEMENTATION-PLAN.md: P3-A marked CORE COMPLETE with the full
Linux equivalence inventory and documented follow-ups (USB 2.0 bitmaps,
SS.Inactive warm-reset loop, multi-TT, hub LPM). Hub gap-table row
updated (249 -> ~700 LoC with Linux state machine).
HARDWARE-VALIDATION-MATRIX.md: hub row updated — enumeration core
complete with 14 unit tests; runtime hub proof tracked under P8-B.
Track base pointer to d20557f7 (xHCI 1.1+ feature gating: LEC/U3C/CIC
gates, HW LPM substrate with protocol-caps HLC detection, PORTPMSC
register-target fix, bogus HCCPARAMS1-HLC accessor removed).
Mark P2-B COMPLETE in USB-IMPLEMENTATION-PLAN.md with the full gating
inventory and Linux 7.1 cross-references. Per-device L1 enablement
(BESL, MEL Evaluate Context) explicitly deferred to P3 hub work.
Expand XhciControllerQuirkFlags from 7 to all 51 flags defined in
Linux 7.1 xhci.h:1587-1649, each using 1u64 << N matching the kernel
bit positions. Add XhciQuirkEntry struct with vendor/device/revision
matching and a WILDCARD constant for catch-all rules.
Create xhci_table.rs — the canonical controller quirk table ported
from Linux 7.1 xhci-pci.c:xhci_pci_quirks() (lines 251-517). Covers
~85 entries across all vendors (Fresco Logic, NEC, AMD, ATI, Intel,
ETRON, Renesas, VIA, CDNS, Phytium, ASMedia, TI, Broadcom, Cavium,
Zhaoxin). 38 unit tests verify flag assignment for every vendor family.
Update lookup_xhci_controller_quirks_full() signature to accept
revision: u8 and hci_version: u16, enabling the two table entries
that depend on hci_version (AMD_0x96_HOST, >= 0x120 spec rule).
Bugs fixed vs xhcid's prior self-contained table:
- Intel 0x9c31 duplicate removed; WildcatPoint corrected to 0x9cb1
- Panther Point corrected to 0x1e31
- VIA VL805 corrected to LPM_SUPPORT + TRB_OVERFETCH
- Phytium corrected to RESET_ON_RESUME
- ASMedia 0x1142 corrected to NO_64BIT_SUPPORT + ASMEDIA_MODIFY_FLOWCONTROL
- Intel 0x9a13 moved off RESET_TO_DEFAULT list
xhcid now consumes this canonical path via a thin re-export shim
(base submodule pointer bumped to 9fc1947d).
Reference: local/reference/linux-7.1/drivers/usb/host/xhci-pci.c
local/reference/linux-7.1/drivers/usb/host/xhci.h
Tier 1 message fixes (Part 5.3):
- M18/M19 (build-redbear.sh:523-524): reworded patch-application messages
to reflect local fork model (forks have patches committed; recipe patches
for non-fork recipes applied atomically by repo fetch)
- M46 (build-redbear.sh:923): 'immutable archives' → 'local forks'
Tier 2 doc consolidation (Part 7):
- GRUB-INTEGRATION-PLAN.md: updated all 'make all CONFIG_NAME=' commands
to canonical './local/scripts/build-redbear.sh' equivalents
- SLEEP-IMPLEMENTATION-PLAN.md: archived (referenced 0.2.4 branch, pre-0.3.1)
Assessment doc Part 7 updated: all Tier 1 (items 1-6) and Tier 2 (items 7-13)
marked as verified complete with per-item status.
All Q1-Q10 quality improvement opportunities from Part 6.2 and all Tier 1-3
items from Part 7 are now resolved.
Q1 from assessment Part 6.2 — closes the largest robustness gap (Mechanism #2):
corrupt cached pkgar that passes mtime check was previously not detected
until runtime.
Publish side (repo_builder.rs):
- Accumulates BLAKE3 hash of each published file (pkgar, toml, dep_hashes,
auto_deps) into a BinaryStoreManifest
- Writes <recipe>.manifest.toml alongside published artifacts in repo/
Restore side (cook_build.rs):
- BinaryStoreManifest struct with read() returning Result<Option<Self>, String>
(same pattern as DepHashes: Ok(None) for missing = backward compat,
Err for corrupt TOML = loud WARN + skip)
- After restoring from binary store, verifies each file's BLAKE3 against
the manifest. On mismatch or missing file: WARN + all_restored=false
(forces rebuild). Missing manifest = backward compatible (older cookbook
published without one).
Helper (fs.rs):
- compute_file_blake3_hex(): 64KB chunked BLAKE3 hash, avoids loading
entire pkgar into memory
Tests: 3 new (roundtrip, missing-file Ok(None), corrupt-TOML Err).
Total: 38/38 pass. cargo check clean. No new clippy warnings.
Assessment doc Part 6.2/6.3/7 updated: Q1-Q6 all marked resolved.
Auto-correction table updated: transient network failure (Q2 retry) and
corrupt cached pkgar (Q1 BLAKE3 manifest) moved from 'does NOT auto-correct'
to 'auto-corrects'.
Tier 3 code improvements (from BUILD-SYSTEM-ASSESSMENT-2026-07-18 Part 6.2):
- Q2: Add run_command_with_retry() helper with exponential backoff (1s, 2s, 4s).
Wrap download_wget and git clone with 3 attempts. Git clone closure cleans
partial tmp dir between retries (git refuses non-empty clone targets).
- Q3: Preserve auto_deps.toml alongside dep_hashes.toml in binary store publish
(repo_builder.rs). Restore path prefers the preserved copy, falling back to
declared-depends-only reconstruction only when the preserved copy is absent
(e.g. published by an older cookbook). Preserves full ELF dynamic dep graph.
- Q4: DepHashes::read now returns Result<Option<Self>, String> discriminating
'missing' (legitimate first build, fall back to mtime) from 'corrupt' (loud
WARN + force rebuild). Closes a silent-swallow gap where a corrupt
dep_hashes.toml produced an incorrect mtime-fallback rebuild.
- Q6: Standardize error reporting in repo.rs nonstop path to {:#} format,
matching the pattern established in the earlier Tier 1 message commit.
Tier 1 message fixes (remaining items from assessment Part 5.3/5.4):
- M26: build-redbear.sh:569 '30-60 minutes' -> config-dependent range
- A6/A8: Makefile container messages add remediation hint
- B3: mk/config.mk sccache hint adds install guidance
- H2: mk/qemu.mk Unsupported ARCH lists supported values
- mk/redbear.mk comment 'fully-patched' -> 'versioned'
Verification: cargo check 0 errors, cargo test --lib 35/35 passed,
cargo clippy 0 new warnings vs baseline, make -n live + validate clean.
Phase 1 remediation of the build-system assessment:
- Makefile: wire 'validate' target into build/live/reimage flows; surface
lint-config and init-service validators as a first-class gate.
- mk/disk.mk: add 'validate' target running lint-config, init-service
validator, and file-ownership validator; suppress noisy unmount warnings
that masked real failures.
- mk/redbear.mk: add source-fingerprint tracking so integrate-redbear.sh
re-runs when local/recipes, local/Assets, or local/firmware change.
- src/cook/cook_build.rs: atomic dep_hashes.toml write (tmp + rename) to
prevent torn-write cache corruption; binary-store restore now checks
dep_hashes before silent restore; fix production bug in
collect_files_recursive that silently dropped subdirectories whose
name matched an exclude pattern.
- src/cook/fetch.rs, src/cook/fetch_repo.rs: harden atomic patch
application and protected-recipe gating.
- AGENTS.md, local/AGENTS.md, local/docs/COLLISION-DETECTION-STATUS.md:
sync collision-detection status to 'implemented (Phase 15.0)' now that
CollisionTracker is wired across all four installer layers
(installer submodule pointer tracked separately in 13cc6fb0c3).
Verified: cargo check (0 errors), cargo test --lib (35/35 passed),
cargo clippy (0 new warnings vs baseline), make -n validate, make -n live.
vasilito