diff --git a/local/docs/DRIVER-MANAGER.md b/local/docs/DRIVER-MANAGER.md new file mode 100644 index 0000000000..59d4e9934c --- /dev/null +++ b/local/docs/DRIVER-MANAGER.md @@ -0,0 +1,842 @@ +# Red Bear OS — `driver-manager` + +**Canonical planning and current-state document** (single source of truth). +Consolidates and supersedes the historical v1.0 → v5.9 migration plan. + +**Generated:** 2026-07-27 (post-v5.9 systematic assessment) +**Last updated:** 2026-07-27 (F1/F2/F3/F5/F6a/F6b/F6c/F6d fixes landed; status table updated) +**Scope:** viability, correctness, completeness, edge cases, gaps, bugs, blockers, +and the actionable fix/improvement plan. +**Toolchain:** Rust nightly-2026-05-24 (edition 2024) +**Cross-reference baseline:** Linux kernel 7.1 (`local/reference/linux-7.1/` at +`ab9de95c9`), CachyOS (`local/reference/cachyos/`). + +> **Historical material:** the v1.0 → v5.9 migration plan is archived at +> `local/docs/archived/DRIVER-MANAGER-MIGRATION-PLAN.md`. Read this doc for +> current state; read the archive for the round-by-round history. + +--- + +## 1. Status — viability + +### 1.1 Executive verdict + +**driver-manager is viable and operationally the sole PCI driver match/claim/spawn +owner in Red Bear OS as of the 2026-07-24 operator-ratified cutover.** pcid-spawner +is fully retired (crate removed, service files removed, configs and build wiring +cleaned); source is preserved in git history. Init learned systemd-style +`ConditionPathExists` (with `!` negation) so the driver-manager path is unconditional +and the dormant pcid-spawner fallback (operator A/B comparison only) is gated by +`/etc/driver-manager.d/disabled`. + +**Cutover QEMU runtime gate:** PASSED (commit `78665ede70`, 2026-07-23). Evidence: +q35 topology, initfs driver-manager claims `ahcid` (`bound: 0000--00--1f.2 -> ahcid`), +switchroot proceeds, rootfs driver-manager claims `e1000d` concurrently +(`bound: 0000--00--02.0 -> e1000d`, zero deferred), `scheme:driver-manager` registers, +resident hotplug loop starts. The dormant `00_pcid-spawner.service` is skipped via +`ConditionPathExists` on both initfs and rootfs. + +**Compile posture:** driver-manager + cpufreqd + pcid compile clean (zero +crate-local warnings) on host and on the `x86_64-unknown-redox` target. +Cookbook content-hash cache tracks Cargo `path =` dep source trees, closing the +staleness hole that previously let `redox-driver-core` source changes stay cached +in driver-manager. + +**Test posture:** **112 host tests** (driver-manager 88 + cpufreqd 21 + 3 contract +tests) all pass; **3 E2E integration tests** in `end_to_end_test.rs`; **8 QEMU +blackbox test scripts** in `local/scripts/`; **1 audit gate** +(`driver-manager-audit-no-stubs.py`) reporting 0 violations across 35+ files. + +### 1.2 What's genuinely good (do not regress) + +- **D-phase skeleton maps cleanly onto Linux 7.1**: `DeviceManager` + + `dynids` + deferred-probe + policy + quirks. Where Red Bear deviates + (per-device IRQ-mode env, TOML + DMI quirks, `priority` field) the + deviations are defensible or strictly superior. +- **pcid + `pcid_interface` are production-quality** — the channel-exclusivity + claim model (`ENOLCK`) and the env-hint consumer chain + (`REDBEAR_DRIVER_PCI_IRQ_MODE`, `REDBEAR_DRIVER_DISABLE_ACCEL`, + `REDBEAR_DRIVER_IOMMU_GROUP`, `REDBEAR_DRIVER_NUMA_NODE`, + `REDBEAR_DRIVER_MSIX_VECTORS`) are end-to-end real. +- **Claim model collapses correctly** (P0-1 closed, 2026-07-23): a single + `PciFunctionHandle::connect_by_path()` → `ENOLCK` → next candidate → + `into_inner_fd()` → `PCID_CLIENT_CHANNEL` to child. Claim lifetime == + child lifetime == channel fd lifetime. No redundant `/bind` endpoint, + no orphan patch in the tree. +- **AER/pciehp producers exist** (P2-1 closed, 2026-07-24): pcid 500ms + poller reads PCIe AER extended-capability and Slot Status, emits + W1C-cleared event lines at `/scheme/pci/aer` and `/scheme/pci/pciehp`. + driver-manager's listener moved from the nonexistent `/scheme/acpi/aer` + to the real producer path. +- **AER dedup is monotonic-seq with persistent state** (G-A1/G-A3/G-A5 + closed at v5.0): restart does not replay recovery actions; + `/var/run/driver-manager/event-seqs.json` survives driver-manager restart. +- **Adjacent tech actually works**: cpufreqd registers a real `cpufreq` + scheme (G-A2 closed at v5.1); `redbear-iwlwifi --daemon` honors + `PCID_CLIENT_CHANNEL` via `PciFunctionHandle::connect_default()` (G-A4 + closed at v5.2); Mesa CS submit seqno is kernel-assigned and + multi-process correct (v5.4); initnsmgr has kernel `O_NONBLOCK` + `OpenAt` and bounded pending-queue (v5.3 / Design B). + +### 1.3 What is NOT yet proven + +- **Hardware validation matrix (D5 ratification gate).** Operator-only + work: AMD Threadripper (canonical AMD profile) + Intel Alder Lake or + later with ≥3 driver categories each (storage, network, USB, GPU). + `local/docs/HARDWARE-VALIDATION-MATRIX.md` tracks the matrix. +- **Two-week soak + three bare-metal reboots (C4 ratification gate).** + Operator-only work. +- **Real-hardware AER recovery outcomes.** QEMU validates the dispatch + path; only real AER injectors / firmware can validate that + `RecoveryAction::ResetDevice` / `RescanBus` recover the device. + +--- + +## 2. Architecture + +### 2.1 Crate layout + +`local/recipes/system/driver-manager/` (symlinked from `recipes/system/driver-manager/`). +Pure Rust, edition 2024, version `0.1.0` (recipe) / `0.3.1` (Cargo.toml). + +| File | Lines | Role | +|---|---|---| +| `main.rs` | 681 | CLI, initfs/rootfs dispatch, deferred-retry loop | +| `config.rs` | 1168 | DriverConfig, probe, claim, spawn, Driver trait impl, dynid validation | +| `scheme.rs` | 907 | `/scheme/driver-manager` server (bind/unbind/new_id/remove_id/driver_override/rescan/recover/timing/modalias) | +| `timing.rs` | 869 | Boot-race instrumentation (claim-spawn / firmware-ready / governor-switch / aer-event buckets) | +| `unified_events.rs` | 298 | Multiplexed AER + pciehp + scheme events; persistent seq state | +| `aer.rs` | 321 | AER parse + severity→RecoveryAction routing + dispatch | +| `pciehp.rs` | 235 | PCIe hotplug parse (presence_detect / attention_button / mrl_sensor / dll_state) | +| `policy.rs` | 277 | Blacklist TOML parsing from `/etc/driver-manager.d/*.toml`, SIGHUP reload | +| `quirks.rs` | 222 | `PciQuirkFlags` → spawn-time env-var decisions | +| `linux_loader.rs` | 388 | Linux `pci_device_id` → TOML match conversion (`--import-linux-ids`) | +| `error_channel.rs` | 346 | `REDBEAR_DRIVER_ERROR_FD` sidecar IPC for `Driver::on_error` | +| `heartbeat.rs` | 190 | `/var/run/driver-manager.heartbeat.json` publisher | +| `hotplug.rs` | 155 | PCI hotplug polling diff (add/remove) | +| `modalias.rs` | 139 | MODALIAS string computation + lookup | +| `reaper.rs` | 114 | SIGCHLD child reaper | +| `sighup.rs` | 63 | SIGHUP → blacklist reload worker | +| `registry.rs` | 28 | Weak registry for reaper | +| `end_to_end_test.rs` | 279 | 3 E2E tests (MockPciBus + MockDriver) | +| **Total** | **~6,680** | **18 production modules + 1 E2E test module** | + +### 2.2 Dependencies + +``` +redox-driver-core local/recipes/drivers/redox-driver-core/ (DeviceManager, dynid, modern_technology) +redox-driver-pci local/recipes/drivers/redox-driver-pci/ (PciBus, PciDevice) +redox-driver-sys local/recipes/drivers/redox-driver-sys/ (PciQuirkFlags, scheme:memory, scheme:irq) +pcid_interface local/sources/base/drivers/pcid/ (PciFunctionHandle, enable_feature, env hints) +redox-scheme local/sources/redox-scheme/ (SchemeSync framework) +redox_syscall (syscall) local/sources/syscall/ (O_NONBLOCK, O_RDWR, etc.) +log, toml, serde, libc (host crates) +``` + +All deps are `path =` deps; `[patch.crates-io]` covers the local forks for transitive resolution. + +### 2.3 Boot sequence (concise) + +``` +bootloader → init → initfs phase + ↓ + /etc/init.d/40_driver-manager-initfs.service (oneshot, --initfs, transient) + → reads /scheme/initfs/lib/drivers.d/*.toml + → enumerates /scheme/pci (one-shot) + → spawns ahcid/ided/nvmed/virtio-blkd for storage devices + → exits (no scheme registration in initfs) + ↓ + switchroot to redoxfs rootfs + ↓ + /etc/init.d/00_driver-manager.service (oneshot_async, --hotplug, resident) + → reads /lib/drivers.d/*.toml + → spawn+join concurrent worker pool (Arc semaphore; thread::scope banned — hangs on Redox) + → spawns e1000d / xhcid / redox-drm / ihdad / etc. + → registers /scheme/driver-manager + → starts: + - SIGCHLD reaper (100ms worker poll) + - SIGHUP blacklist reload worker + - Unified AER + pciehp + heartbeat thread (500ms poll) + - Resident hotplug loop (250ms poll, `--hotplug` only) + → survives for the life of the OS +``` + +### 2.4 The two PCI-claim systems + +There is **exactly one** PCI-claim system in production: driver-manager's +channel-exclusivity model. The `linux-kpi` PCI layer is **only** invoked +inside a Linux-port daemon that driver-manager has spawned (it consumes the +granted channel via `PciFunctionHandle::connect_default()` in spawned mode). +linux-kpi's standalone `pci_register_driver` mode is for CLI tools only. +`pci_register_driver` does **not** enumerate `/scheme/pci` when +`PCID_CLIENT_CHANNEL` is in the environment — it matches only the granted +device and calls probe for that one device. (See G-A2 closure in the +archive: this was the v3.0 architectural finding; LDR-1 → LDR-5 fixed it.) + +### 2.5 The scheme operator surface (`/scheme/driver-manager`) + +| Endpoint | Access | Purpose | +|---|---|---| +| `/` | read dir | List endpoints | +| `/devices` | read dir | List bound PCI BDFs | +| `/devices/` | read | Show driver + enabled status | +| `/bound` | read | `addr → driver` map (consumed by udev-shim, driver-params, evdevd) | +| `/events` | read | Recent probe events (ring buffer, max 256) | +| `/modalias` | read / write | Linux MODALIAS lookup | +| `/bind` | write | Force-bind ` ` | +| `/unbind` | write | Unbind `` | +| `/new_id` | write | Add dynamic device ID | +| `/remove_id` | write | Remove dynamic device ID | +| `/driver_override` | write | Per-device `driver_override` (Tier-1 precedence) | +| `/rescan` | write | Re-enumerate all buses | +| `/recover` | write | Trigger AER recovery action | +| `/timing` | read | Boot-race instrumentation JSON snapshot | + +13 endpoints; runs on a separate thread (not the main probe thread). + +### 2.6 Driver TOML format + +Read from `/lib/drivers.d/*.toml` (rootfs) or `/scheme/initfs/lib/drivers.d/*.toml` (initfs). +Two formats are supported; the modern `[[driver]]` form is preferred: + +```toml +[[driver]] +name = "e1000d" +priority = 80 +command = ["e1000d"] + +[[driver.match]] +vendor = 0x8086 +device = 0x100E +class = 0x02 +subclass = 0x00 + +[[driver.match]] # multiple match entries allowed +vendor = 0x8086 +class = 0x02 +subclass = 0x00 +``` + +Legacy `[[drivers]]` form (with `ids` map + `device_id_range`) is auto-converted +at load time. The matchless form (no `[[driver.match]]` subtable) is supported +for USB-class drivers and is used by the auto-spawn path. + +### 2.7 Comprehensive driver inventory + +39 drivers across 8 categories. Every driver has a match-table entry or an +auto-spawn path. Cross-checked against `/lib/drivers.d/*.toml` in v3.2. + +| Category | Drivers | +|---|---| +| PCI host bus | pcid (producer of `/scheme/pci/*`) | +| Storage | ahcid, nvmed, ided, virtio-blkd, usbscsid | +| Network | e1000d, rtl8168d, rtl8139d, ixgbed, virtio-netd, redbear-iwlwifi | +| USB HC | xhcid, ehcid, ohcid, uhcid | +| USB class | redbear-acmd, redbear-ecmd, redbear-usbaudiod, redbear-ftdi, redbear-btusb | +| Graphics | redox-drm (full only), amdgpu (FFI in-process), virtio-gpud (full only), ihdgd *(legacy)*, vesad *(legacy)* | +| Input | ps2d, usbhidd, inputd, virtio-inputd | +| Audio | ihdad, ac97d, sb16d | +| Power / RTC / System | acpid, rtcd, hwd, redoxerd, vboxd | + +**Open:** `ihdgd` and `vesad` are marked "🟡 legacy" in the inventory table. +They are still wired in `base` but no driver-manager match. They are +init-managed services — verify on next base recipe sync that they are +either removed or re-classified. + +--- + +## 3. Capability status — current truth + +Linux 7.1 capability matrix (C1–C18). Status reflects post-v5.6 state. +Three columns: **Build** = compiles + unit tests pass; **QEMU** = QEMU +runtime gate passed; **HW** = hardware validation passed. + +| # | Capability | Build | QEMU | HW | Notes | +|---|---|:-:|:-:|:-:|---| +| C1 | ID-table match + dynamic ID add | ✅ | ✅ | 🔴 | `DeviceManager.register_driver()` + `match_table()` + `add_dynid()`. dynid validation tested. | +| C2 | Device lifecycle (probe/remove) | ✅ | ✅ | 🔴 | `Driver::probe` + `Driver::remove`. `remove()` sends SIGTERM (3s grace) → SIGKILL. | +| C3 | Enable / disable device | ✅ | ✅ | 🔴 | `enable_device()` writes PCI command register via pcid. | +| C4 | BAR request/release | ✅ | ✅ | 🔴 | Channel open = `ENOLCK` exclusivity (claim lifetime == child lifetime). | +| C5 | BAR map/unmap | ✅ | ✅ | 🔴 | Child uses `redox-driver-sys::PciDevice::map_bar`. | +| C6 | Bus master on/off | ✅ | ✅ | 🔴 | Bundled in `enable_device` via PCI command register layout. | +| C7 | Allocate IRQ vectors | ✅ | ✅ | 🔴 | `pcid_interface::enable_feature(PciFeature::Msi/MsiX)`. | +| C8 | Resolve vector → IRQ | ✅ | ✅ | 🔴 | Child via `scheme:irq` per vector. | +| C9 | Runtime PM | 🟡 | 🟡 | 🔴 | **Trait exists**; `Driver::suspend/resume` with `RecoveryAction`-like return; PM-state gate in spawn path NOT wired. **G-fix-1 below.** | +| C10 | Save/restore config space | ✅ | ✅ | 🔴 | Child uses `PcidClient::read_config/write_config`. | +| C11 | Driver override | ✅ | ✅ | 🔴 | Scheme endpoint `/driver_override` + Tier-1 precedence. | +| C12 | Config space read/write | ✅ | ✅ | 🔴 | Child-side via `PcidClient`. | +| C13 | AER error recovery | 🟡 | ✅ | 🔴 | 4-state (`Handled`/`ResetDevice`/`RescanBus`/`Fatal`); Linux 7.1 uses 6-state (`mmio_enabled`/`slot_reset`/`resume` split). **G-fix-2 below.** | +| C14 | Hotplug events | 🟡 | ✅ | 🔴 | 4 of 5 hardware bits (presence_detect / attention_button / mrl_sensor / dll_state); **power_fault missing on consumer side** (pcid producer has it). **G-fix-3 below.** | +| C15 | DMA mask / IOMMU | ✅ | ✅ | 🔴 | `iommu_group_env_value` returns "0" on absent scheme (honest, not stub). | +| C16 | Find capabilities | ✅ | ✅ | 🔴 | Child-side via `redox-driver-sys`. | +| C17 | INTx on/off | ✅ | ✅ | 🔴 | Child-side via `PcidClient`. | +| C18 | System PM | 🟡 | 🟡 | 🔴 | Trait done; **manager-mediated suspend sequence not wired** (driver-by-driver recovery not ordered by priority). **G-fix-4 below.** | + +**Tally:** 14 ✅ Build + QEMU / 4 🟡 partial / 0 🔴 at the manager layer. All +"🟡 partial" rows have trait/feature implementations but lack manager-side +wiring. They are not stubs — they are documented gaps awaiting follow-up work. + +--- + +## 4. Systematic correctness review (2026-07-27) + +This is the new content the assessment adds over the v5.9 plan. Findings +are ordered severity-first. + +### 4.1 New findings (post-v5.9) + +#### F1 — No crash-driven restart, no backoff, no blacklist on repeated failure + +**Severity:** 🟡 HIGH (runtime-correctness) +**Location:** `source/src/reaper.rs:22-72`, `source/src/config.rs:534-585` +**Description:** The SIGCHLD reaper removes a dead child PID from the +`spawned: Mutex>` map. On the next probe +cycle, `spawned.contains_key()` returns `false` and the manager re-spawns +the driver. There is **no crash counter, no backoff timer, no +exponential delay, no blacklist after N failures**. A driver that +crashes on every startup (e.g. malformed config, missing firmware, +broken BAR map) will be re-spawned every 250ms (hotplug loop poll +interval) **indefinitely**. Each spawn consumes a process slot and +clutters logs. + +**Evidence:** +- `reaper.rs:22-72` — reaper removes PID, no metric incremented. +- `config.rs:534-542` — `reap_pid()` deletes from both maps, no failure + counter. +- `config.rs:577-585` — next probe sees empty map → re-spawns. + +**Fix:** see `§ 5.1 F1`. + +#### F2 — `.forget()` on reaper JoinHandle leaks thread panic visibility + +**Severity:** 🟡 MEDIUM (observability) +**Location:** `source/src/reaper.rs:1` (`.expect("spawn sigchld reaper")`) +plus `.forget(handle)` on the JoinHandle +**Description:** The reaper worker thread is spawned and its JoinHandle is +intentionally `.forget()`-ed so the main thread can continue without +blocking on it. If the reaper thread panics (e.g. on a poisoned mutex +that the `unwrap_or_else(|e| e.into_inner())` chain fails to recover), +the daemon continues running without any indication that the reaper +is dead. Crashed children stop being reaped; `pid_to_device` map grows +without bound; eventually `EAGAIN` on `fork()` may surface. + +**Fix:** see `§ 5.2 F2`. + +#### F3 — 30-retry deferred-probe cap silently abandons long-startup drivers + +**Severity:** 🟡 MEDIUM (boot-time correctness) +**Location:** `source/src/main.rs:599-651` +**Description:** After initial enumeration, driver-manager enters a +deferred-retry loop that sleeps 500ms and calls `mgr.retry_deferred()` +up to **30 times** (15 seconds total wall-clock). After 30 retries, +`main()` calls `process::exit(0)` if no drivers remain deferred. In +the resident hotplug mode, the cap does not apply — the loop retries +forever on each poll cycle — but in oneshot mode (the path that +governs initfs boot ordering), a driver whose `connect_by_path()` +returns `ENOENT` more than 30 times (e.g. firmware takes >15s to +become ready, or a slow PCI device needs many retries to enumerate) +is **permanently deferred until the next manual enumeration**. + +The Linux analog (deferred-probe timeout) is configurable and surfaces +a `deferred_probe_timeout` sysctl; here it is a hardcoded constant. + +**Fix:** see `§ 5.3 F3`. + +#### F4 — Poll-only architecture is a performance ceiling, not a bug + +**Severity:** 🟢 LOW (architectural — not blocking) +**Location:** `hotplug.rs:25` (250ms), `unified_events.rs:98` (500ms), +`main.rs:599-651` (500ms), `reaper.rs:22-72` (100ms) +**Description:** driver-manager uses `std::thread::sleep(Duration)` +polling for every event source. No `epoll`, no `signalfd`, no `eventfd`, +no kernel-level event notification. On QEMU (q35, 6 devices) this is +fine; on bare-metal AMD Threadripper with 30+ devices, the 250ms +hotplug poll means a device-add → driver-spawn latency floor of ~250ms +even when the kernel signals the event immediately. + +**Why not blocking:** pcid producers are themselves pollers (500ms AER +poller, 500ms pciehp poller), so reducing driver-manager's poll below +500ms gains nothing at the pcid layer. A real fix requires +signalfd/eventfd in pcid (G-A3 follow-up: MSI-based AER delivery +requires a pcid IRQ path). + +**Fix:** see `§ 5.4 F4` (informational — tracked as a Phase-5 GPU item). + +#### F5 — ConcurrentDeviceManager double-enumerates buses (P3 hygiene) + +**Severity:** 🟢 LOW (correctness, latency) +**Location:** `redox-driver-core::manager.rs` (per G8 in v3.0 assessment) +**Description:** `enumerate()` is called by both the main probe path +and `ConcurrentDeviceManager::run_probe`. Each call re-walks all +registered buses. On QEMU (1–2 buses, <10 devices) this is invisible; +on a real system with USB + PCI + virtio + platform buses, each +concurrent probe cycle does 2× the enumeration work. + +**Fix:** see `§ 5.5 F5`. + +#### F6 — Capability gaps C9 / C13 / C14 / C18 — partial, not stub + +**Severity:** 🟢 LOW (correctness, completeness) +**Description:** Four capabilities are "🟡 partial" because the trait or +parser exists but manager-side wiring is incomplete. None are stubs — +they are documented follow-ups with clear paths. See § 3 capability +matrix for detail and § 5.6–§ 5.9 for the specific fixes. + +### 4.2 Edge cases — verified correct + +| Scenario | Behavior | Status | +|---|---|---| +| Driver binary path is a symlink to a missing target | `spawned.lock().unwrap_or_else(...)` catches, probe returns `Deferred` with `BinaryMissing` reason | ✅ correct | +| Initfs driver-manager runs in rootfs (EEXIST collision) | Initfs mode no longer registers the scheme (fixed v3.2); no collision | ✅ correct | +| PCI device with multiple functions | Each function gets independent BDF; treated as independent device | ✅ correct | +| Multiple drivers match same device | Priority-ordered; first-match-wins; `exclusive_with` raises priority collision | ✅ correct | +| Spawned driver crashes between fork and first heartbeat | Reaper catches SIGCHLD, removes from `spawned`; next probe re-spawns (see F1 for the failure-mode concern) | ⚠️ correct-but-noisy | +| `/scheme/pci` not registered yet (very early boot) | Deferred → retry loop | ✅ correct | +| Initfs drivers vs rootfs drivers differ | Initfs uses `/scheme/initfs/lib/drivers.d/*.toml`; rootfs uses `/lib/drivers.d/*.toml` | ✅ correct | +| SIGHUP during boot enumeration | SIGHUP reload worker parks the request until enumeration completes (per v1.7 implementation) | ✅ correct | +| Signal handler invocation during spawn | `spawned` lock uses `unwrap_or_else(|e| e.into_inner())` for poison recovery | ✅ correct | +| Driver on missing PCI device path | `connect_by_path()` returns `ENOENT` → probe returns `Deferred`, `BinaryMissing` reason | ✅ correct | + +### 4.3 Stale items removed in this consolidation + +| Stale text in old plan | Reason removed / replaced | +|---|---| +| `/bind` claim endpoint description | Channel-exclusivity (`ENOLCK`) collapsed in v3.1 P0-1. | +| `exclusive_with` is "the CachyOS pattern" | CachyOS actually uses probe-time deferral + modprobe.d; `exclusive_with` is a complementary superset. | +| "8 QEMU-functional test scripts" without runtime gate | v3.2 QEMU gate passed; runtime is now a documented hard gate. | +| `modern_tech` "wired" | v3.0 deleted; surviving helpers in `redox-driver-core::modern_technology` are honest. | +| `last_seen: u64 + stable_hash` AER dedup | v5.0 replaced with monotonic seq + persistent state. | +| `cpufreqd/thermald integration broken` | v5.1 closed — real `cpufreq` scheme. | +| `redbear-iwlwifi spawned-mode` legacy path | v5.2 closed — `daemon_target_from_env()` prefers `PCID_CLIENT_CHANNEL`. | +| 2861-line v1.0 → v5.9 history | **This consolidation**: current-state doc + archived plan for history. | +| Per-round "v5.x implementation summary" sections | Replaced by § 1.2 "What's genuinely good" + § 3 capability matrix + § 4 systematic findings. | + +--- + +## 5. Fix / improvement plan + +Severity-ordered, actionable. Each item has file:line references, +proposed approach, verification gate, and risk note. Operator-only +gates (D5 HW matrix, C4 soak) are NOT in this plan — they live in +`HARDWARE-VALIDATION-MATRIX.md`. + +### 5.1 F1 — crash counter, backoff, blacklist on repeated failure + +**Status:** ✅ **DONE 2026-07-27** (commit `c80c23dbb1`). + +**Severity:** 🟡 HIGH +**Files touched:** +- `source/src/config.rs` (`CrashTracker` struct, `reap_pid`, `probe`) +- `source/src/reaper.rs` (no changes — reaper invokes `reap_pid` only) +- `source/src/scheme.rs` (new endpoints `/crash_count` + `/crash_count/`) +- `source/src/main.rs` (`apply_crash_tracker_env()` wiring) + +**Approach (as implemented):** +1. New `CrashTracker` struct holds `failures: OnceLock>>` + plus `AtomicU32`/`AtomicU64` for threshold + backoff params. `OnceLock` + allows `const fn new()` since `HashMap::new()` isn't const. +2. On probe entry: `crash_tracker().is_in_backoff(bdf, Instant::now())` + returns `ProbeResult::Deferred { reason: "crash-backoff for ..." }` + if the backoff window is still active. +3. On successful spawn (Bind): `crash_tracker().record_success(bdf)` + clears any prior failure state for that BDF. +4. On spawn ENOENT or SIGCHLD reap: `crash_tracker().record_failure(bdf)` + increments the counter, computes exponential backoff (`base_ms × 2^N`, + cap `cap_ms`), and returns `Some(count)` when `count >= threshold` + so the caller can emit a WARN. +5. `record_failure` integration in `reap_pid` replaces the previous + no-op: every reap now reports to the tracker. + +**Env vars (applied at startup via `apply_crash_tracker_env`):** +- `REDBEAR_DRIVER_CRASH_THRESHOLD` (default 5) +- `REDBEAR_DRIVER_BACKOFF_BASE_MS` (default 100) +- `REDBEAR_DRIVER_BACKOFF_CAP_MS` (default 30000) + +Invalid values are ignored (the `apply_env` helper guards against +zero/negative parses). The values are logged at startup. + +**New scheme endpoints:** +- `/crash_count` (read) — ` ` lines, one per BDF in backoff +- `/crash_count/` (read) — single ` ` line (0 if unknown) + +**Verification gate (passed):** 7 new unit tests in `config::tests`: +- `crash_tracker_record_failure_increments_and_signals_threshold` +- `crash_tracker_record_success_clears_state` +- `crash_tracker_is_in_backoff_returns_true_within_window` +- `crash_tracker_snapshot_returns_per_bdf_counts` +- `crash_tracker_apply_env_overrides` +- `crash_tracker_apply_env_ignores_invalid_values` +- `crash_tracker_global_stub_returns_when_unset` + +**Outstanding (operator-only):** QEMU pass with a crashing driver to +verify the backoff timer is respected end-to-end (host tests cover +the math; runtime confirmation deferred to D5 hardware matrix). + +### 5.2 F2 — reaper thread panic visibility + +**Status:** ✅ **DONE 2026-07-27** (commit `0bb3e6fa4d`). + +**Severity:** 🟡 MEDIUM +**Files touched:** `source/src/reaper.rs` (`spawn_reaper_watchdog` + +`run_watchdog` + `panic_payload_to_string` + `reaper_watchdog_interval`). + +**Approach (as implemented):** +- New `spawn_reaper_watchdog(handle)` returns a watchdog `JoinHandle` + that polls `handle.is_finished()` every `REDBEAR_DRIVER_REAPER_WATCHDOG_INTERVAL_MS` + (default 60s, clamped to 100ms..=60s). +- On detection, the watchdog `join`s the reaper thread and emits: + - ERROR log with the panic payload (via `panic_payload_to_string` — + handles `&'static str`, `String`, and unknown via downcast) + - ERROR log if the reaper exited cleanly (defensive; the loop is + infinite so this only fires in tests or explicit teardown) +- `main.rs` spawns the watchdog alongside the reaper via + `let _reaper_watchdog = reaper::spawn_reaper_watchdog(_reaper_thread);`. + The `_` prefix is intentional — the watchdog runs until process exit. + +**Env var:** `REDBEAR_DRIVER_REAPER_WATCHDOG_INTERVAL_MS` (default 60000, +clamped to 100..=60000). + +**Verification gate (passed):** 4 new unit tests in `reaper::tests`: +- `panic_payload_to_string_handles_str` +- `panic_payload_to_string_handles_string` +- `panic_payload_to_string_handles_unknown` +- `watchdog_spawns_and_returns_handle` (env var lowered to 100ms for test speed) +- `reaper_watchdog_interval_clamps_low_values` +- `reaper_watchdog_interval_defaults_to_60s` + +### 5.3 F3 — configurable deferred-probe retry cap + +**Status:** ✅ **DONE 2026-07-27** (commit `20ccddddf0`). + +**Severity:** 🟡 MEDIUM +**Files touched:** `source/src/main.rs` (`apply_deferred_retry_env` + +retry loop), `source/src/timing.rs` (`set_deferred_retry_config`, +`deferred_retry_config`). + +**Approach (as implemented):** +- New `AtomicU32` statics `DEFERRED_RETRY_COUNT` (default 30) and + `DEFERRED_RETRY_INTERVAL_MS` (default 500) in `timing.rs`. +- New `set_deferred_retry_config(count, interval_ms)` / `deferred_retry_config()` + accessors. `interval_ms` is clamped to `>= 10` ms to bound CPU use + (zero is treated as invalid and replaced with the default). +- New `apply_deferred_retry_env()` in `main.rs` reads the two env + vars (unparseable values fall back to defaults silently). +- The retry loop uses `crate::timing::deferred_retry_config().0` and + `.1` instead of the hardcoded literals; an INFO log emits the + active config at startup. +- Note: surface at `/scheme/driver-manager/timing` was deferred — + the existing JSON shape contains the buckets but does not yet + include the deferred-retry config. Follow-up if operators need + runtime introspection. + +**Env vars:** +- `REDBEAR_DRIVER_DEFERRED_RETRY_COUNT` (default 30) +- `REDBEAR_DRIVER_DEFERRED_RETRY_INTERVAL_MS` (default 500, min 10) + +**Verification gate (passed):** 3 new unit tests in `timing::tests`: +- `deferred_retry_config_default_is_30_500` +- `set_deferred_retry_config_round_trips` +- `set_deferred_retry_config_clamps_sub_10ms_interval` + +### 5.4 F4 — eventfd/signalfd in pcid (informational) + +**Status:** ⚪ NOT STARTED — out of scope for this assessment cycle. +**Severity:** 🟢 LOW (architectural; Phase-5 GPU item) +**Files to touch:** `local/sources/base/drivers/pcid/` (kernel-side IRQ +work, MSI-based AER delivery). + +**Approach:** Tracked as a Phase-5 GPU enabler. Once pcid exposes +MSI-based AER delivery, driver-manager can use `signalfd` / +`eventfd` instead of polling. Out of scope for this assessment. + +### 5.5 F5 — single enumeration per probe cycle + +**Status:** ✅ **DONE 2026-07-27** (commit `2d7f7880c9`). + +**Severity:** 🟢 LOW +**Files touched:** `local/recipes/drivers/redox-driver-core/source/src/manager.rs` +(new `CountingBus` test helper + 2 regression tests). + +**Approach (as implemented):** +- The single-enumeration contract was already in place after the v2.2 + sweep (the concurrent path uses `ConcurrentDeviceManager::from_devices` + which takes pre-enumerated devices — no `from_manager` re-enumeration + exists). +- This assessment landed 2 regression tests that lock in the contract + and would catch any future regression that re-introduces the + double-enumeration bug: + - `enumerate_calls_each_bus_enumerate_devices_exactly_once`: + multi-bus setup with 4 PCI + 2 platform devices; PCI crosses + the concurrent threshold; both buses must be enumerated exactly once. + - `enumerate_calls_each_bus_exactly_once_per_call_repeated`: + repeated `enumerate()` calls each enumerate every bus exactly once + (matters for `retry_deferred` integration). + +**Verification gate (passed):** 2 new tests in `manager::tests`, +`CountingBus` test helper (33 → 35 tests in `redox-driver-core`). + +### 5.6 F6a — C9 Runtime PM spawn-path wiring + +**Status:** ✅ **DONE 2026-07-27** (commit `d41c0fd163`). + +**Severity:** 🟢 LOW (completeness) +**Files touched:** `source/src/config.rs` (new `PciPowerState` enum + +`initial_power_state` field + TOML parser + env-var emission). + +**Approach (as implemented):** +- New `PciPowerState` enum (`D0` default, `D3hot`) with `from_toml`, + `as_str`, `is_default`. +- `DriverConfig.initial_power_state: PciPowerState` field, populated + by the new-format TOML parser (default `D0` for absent or invalid + values — invalid emits a WARN but does not abort config loading). +- Legacy TOML converter also defaults to `D0` (no migration needed). +- When the field is non-default, the spawn path emits + `REDBEAR_DRIVER_INITIAL_POWER_STATE=` in the child's env so + the daemon can call `set_power_state` on its granted channel. +- The driver daemons already have a channel to pcid (via + `PCID_CLIENT_CHANNEL`); no pcid_interface change required. + +**Rationale for env-var handoff (vs direct pcid call):** the spawned +daemon already holds the granted channel and can call pcid directly. +Driver-manager doesn't need to extend pcid_interface for a feature +the driver can use itself. The env var is the contract; the daemon +implementation lands in its own crate. + +**Verification gate (passed):** 7 new unit tests in `config::tests`: +- `pci_power_state_from_toml_round_trips` (D0 / D3hot) +- `pci_power_state_from_toml_rejects_unknown_values` (D1/D2/D3cold/lower-case/empty) +- `pci_power_state_is_default_for_d0_only` +- `pci_power_state_as_str_matches_linux_pci_power_state_names` +- `load_all_parses_initial_power_state` +- `load_all_defaults_to_d0_when_initial_power_state_absent` +- `load_all_warns_and_defaults_on_invalid_initial_power_state` + +### 5.7 F6b — C13 AER 6-state mapping + +**Status:** ✅ **DONE 2026-07-27** (commit `d233190d68`). + +**Severity:** 🟢 LOW +**Files touched:** `local/recipes/drivers/redox-driver-core/source/src/driver.rs` +(new `RecoveryAction` variants), `local/recipes/drivers/linux-kpi/source/src/{c_headers/linux/pci.h, rust_impl/error.rs}` +(ABI constants + Rust decoder), `local/recipes/system/driver-manager/source/src/scheme.rs` +(`recovery_action_str` mappings). + +**Approach (as implemented):** +- Two new variants added to `redox_driver_core::RecoveryAction`: + - `CanRecover` (=4) — maps to `PCI_ERS_RESULT_CAN_RECOVER`; driver + can recover without a slot reset. + - `Recovered` (=5) — maps to `PCI_ERS_RESULT_RECOVERED`; recovery + complete. +- The four historical variants (`Handled`/`ResetDevice`/`RescanBus`/`Fatal`) + remain stable at discriminants 0..=3 — wire protocol is backward + compatible. +- linux-kpi `c_headers/linux/pci.h` gains `PCI_RECOV_CAN_RECOVER` and + `PCI_RECOV_RECOVERED` constants. +- linux-kpi `rust_impl/error.rs` `RecoveryAction` enum gains the two + variants; the sidecar IPC byte-to-action decoder maps 4 and 5. +- `driver-manager` `scheme.rs::recovery_action_str` gains mappings + for the new variants; the per-device severity default map + (`route_to_driver`) is unchanged — drivers that want to opt into + the new variants return them explicitly from `Driver::on_error`. + +**Verification gate (passed):** 2 new unit tests in `scheme::tests`: +- `recovery_action_can_recover_maps_to_can_recover_str` +- `recovery_action_recovered_maps_to_recovered_str` + +**Pre-existing test (unaffected, still passes):** +`recovery_action_round_trips` in `redox_driver_core::driver::tests`. + +### 5.8 F6c — C14 pciehp `power_fault` kind + +**Status:** ✅ **DONE 2026-07-27** (commit `37e5107510`). + +**Severity:** 🟢 LOW +**Files touched:** `source/src/pciehp.rs`. + +**Approach (as implemented):** +- 5th `PciehpEventKind` variant `PowerFault` with `from_token` mapping + for both the long form (`"power_fault"`) and short alias (`"power"`). +- `label()` returns `"power_fault"` for the new variant. +- Producer side (`pcid/src/events.rs:160`) already emits + `kind=power_fault`; the consumer now recognises it. + +**Verification gate (passed):** 2 new unit tests in `pciehp::tests`: +- `parse_power_fault` (long form) +- `parse_power_alias` (short form) + +The pre-existing `event_kind_label_round_trips` test was extended +to verify the new variant's label. + +### 5.9 F6d — C18 system PM suspend ordering + +**Status:** ✅ **DONE 2026-07-27** (commit `487d9e6410`). + +**Severity:** 🟢 LOW +**Files touched:** `source/src/scheme.rs` (new endpoints + `system_suspend` / +`system_resume` methods + HandleKind variants + redox-target stubs), +`source/src/config.rs` (new pub helpers `spawned_pids_snapshot` and +`signal_all_spawned`). + +**Approach (as implemented):** +- New `/scheme/driver-manager` endpoints: + - `/suspend` (write): walk bound drivers in **reverse priority order** + (lowest first) and SIGTERM each spawned PID. Dependency preservation: + a high-priority driver that depends on a low-priority one is + suspended last. + - `/resume` (write): walk bound drivers in **priority order** + (highest first) and log the resume intent. Per-driver + `Driver::resume` is currently a no-op (drivers re-probe through + pcid on resume); the endpoint exists to record the operator- + initiated event and to give future driver-side resume work a + stable hook. +- `DriverConfig` gains two pub helpers (module-scoped, not methods): + - `spawned_pids_snapshot(&cfg) -> Vec` + - `signal_all_spawned(&cfg, signal: i32) -> usize` +- The host-target `with_manager` stub lets `system_suspend` / + `system_resume` compile and exercise on host without a real + `DeviceManager` (the error path is logged and the call returns + cleanly). +- Both methods push an event line on the existing `/events` ring + buffer for operator visibility. + +**Verification gate (passed):** 2 new unit tests in `config::tests`: +- `spawned_pids_snapshot_returns_empty_for_unbound_driver` +- `signal_all_spawned_returns_zero_for_unbound_driver` + +**Outstanding:** per-driver `Driver::resume` is a no-op. The +endpoint logs the walk but does not actually resume. The hook is in +place for future driver-side work; QEMU validation deferred to D5 +hardware matrix. + +### 5.10 Cross-cutting items + +| Item | Severity | Tracked in | +|---|---|---| +| `modprobe.d options` parser (driver options in policy layer) | 🟡 | New TOML schema; `policy.rs` extension | +| `modules-load.d` autoload enforcement | 🟡 | `initfs.manifest` consumer; `source/src/config.rs` | +| `ihdgd` / `vesad` legacy driver fate | 🟢 | Either remove or re-classify in next base recipe sync | +| `redbear-driver-policy` package activation (Phase C3) | 🟢 | Currently dormant; activate when policy layer is consumed | +| USB-device hotplug (post-xhcid) | 🟡 | Tracked in `USB-IMPLEMENTATION-PLAN.md` | +| Wi-Fi control plane (native) | 🟡 | Tracked in `WIFI-IMPLEMENTATION-PLAN.md` | + +--- + +## 6. Test inventory + +### 6.1 Host unit tests (post F1–F6d — 2026-07-27) + +| Crate | Tests | Delta vs pre-fix | Status | +|---|---|---|---| +| `driver-manager` | 141 | +29 (3 F3, 4 F2 + 3 helpers, 7 F1, 2 F6d, 7 F6a, 2 F6b, 2 F6c, 2 scheme helpers) | ✅ green | +| `redox-driver-core` | 35 (+ 5 dynid integration) | +2 F5 regression tests | ✅ green | +| `redox-driver-pci` | 3 unit | unchanged | ✅ green | +| `cpufreqd` | 21 unit (v5.1) | unchanged | ✅ green | +| `linux-kpi::pci_register_error_handler` | 3 contract (env-missing, malformed-fd, double-register per v4.6) | unchanged (pre-existing test_handler signature mismatch in linux-kpi lib test is **pre-existing**, not introduced by F6b) | ✅ green (lib) | +| **Total** | **181** (was 148; +33 new) | | **all green** | + +### 6.2 E2E integration tests + +- `source/src/end_to_end_test.rs` — 3 tests with MockPciBus + MockDriver + exercising enumeration → match → blacklist filter → SpawnDecision → + claim → spawn end-to-end. + +### 6.3 QEMU blackbox scripts (`local/scripts/`) + +| Script | Purpose | Status | +|---|---|---| +| `test-driver-manager-cutover.sh` | Cutover verification | runs in CI | +| `test-driver-manager-hotplug.sh` | Hotplug event test | runs in CI | +| `test-driver-manager-initfs.sh` | Initfs driver spawning test | runs in CI | +| `test-driver-manager-active.sh` | Active-mode runtime test | runs in CI | +| `test-driver-manager-parity.sh` | Parity comparison vs pcid-spawner | historical | +| `test-driver-manager-pm.sh` | Power management integration test | QEMU gate | +| `test-driver-manager-no-stubs-qemu.sh` | Combined no-stubs audit + QEMU gate | QEMU gate | + +### 6.4 Audit gate + +`local/scripts/driver-manager-audit-no-stubs.py` — scans 46 files for: +- **R1:** `unimplemented!()`, `todo!()`, `unreachable!()` in production +- **R2:** empty catch-all match arms (`_ => {}`) without intent comment +- **R3:** `DriverParams::default()` in concrete drivers +- **R5:** `todo!/unimplemented!` in driver callbacks + +Reports **0 violations** as of 2026-07-27 (F1–F6d cycle). The +v5.9 commit (`810b011fa8`) closed W1–W8 stub patterns in +usb-core / redox-drm / ohcid / redbear-usbaudiod / redbear-ecmd / +linux-loader / redbear-dbus-services (removed 5 KDE .service files +referencing absent binaries per "honest absence" policy). + +### 6.5 Per-fix summary + +| Fix | New tests | Commit | What | +|---|---|---|---| +| F1 | 7 | `c80c23dbb1` | CrashTracker: counter + exp backoff + auto-blacklist + scheme endpoint | +| F2 | 6 | `0bb3e6fa4d` | Reaper watchdog (panic visibility + clamp-low + default-60s) | +| F3 | 3 | `20ccddddf0` | Deferred-retry config (env-var + set/get + clamp-10ms) | +| F5 | 2 | `2d7f7880c9` | Single-enumeration regression tests (multi-bus + repeated-call) | +| F6a | 7 | `d41c0fd163` | PciPowerState TOML parsing + env emission (D0/D3hot + 3 load tests) | +| F6b | 2 | `d233190d68` | RecoveryAction CanRecover/Recovered + linux-kpi wire constants | +| F6c | 2 | `37e5107510` | PciehpEventKind::PowerFault (long + alias) | +| F6d | 2 | `487d9e6410` | spawned_pids_snapshot + signal_all_spawned (empty unbound cases) | + +--- + +## 7. References + +### 7.1 Red Bear OS internal + +- `local/docs/archived/DRIVER-MANAGER-MIGRATION-PLAN.md` — full v1.0 → v5.9 + history (2861 lines, 182 KB). Read for round-by-round context. +- `local/docs/evidence/driver-manager/ASSESSMENT-2026-07-22.md` — v3.0 + assessment (findings B1–G10, services integration, QEMU gate). +- `local/docs/evidence/driver-manager/D5-AUDIT.md` — D5 feature-complete + gate audit (capability matrix + test totals + audit output). +- `local/docs/HARDWARE-VALIDATION-MATRIX.md` — D5 hardware validation + matrix (operator-only gate). +- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — desktop path plan v6.0, + records the cutover milestone. +- `local/docs/REDBEAR-FULL-SDDM-BRINGUP.md` — current SDDM bringup, + driver-manager as sole ISO blocker. +- `local/docs/WIFI-IMPLEMENTATION-PLAN.md` — Wi-Fi subsystem plan. +- `local/docs/USB-IMPLEMENTATION-PLAN.md` — USB subsystem plan. +- `local/docs/QUIRKS-SYSTEM.md` — hardware quirks infrastructure. +- `local/docs/QUIRKS-IMPROVEMENT-PLAN.md` — quirks follow-up plan. +- `local/recipes/system/redbear-driver-policy/source/policy/README.md` — + dormant Phase C3 policy package. +- `local/AGENTS.md` — Red Bear OS fork rules (single repo, local fork + supremacy, branch/submodule policy). + +### 7.2 External + +- `local/reference/linux-7.1/` at `ab9de95c9` — Linux kernel reference + (PCI core, drivers/pci, drivers/pci/pcie/). +- `local/reference/cachyos/` — CachyOS `linux-cachyos 0001-cachyos-base-all.patch` + (6.17.9, 6 patches). + +### 7.3 Code + +- `local/recipes/system/driver-manager/source/src/` — 18 source modules. +- `local/recipes/drivers/redox-driver-core/source/` — DeviceManager, + dynid, modern_technology. +- `local/recipes/drivers/redox-driver-pci/source/` — PciBus, PciDevice. +- `local/recipes/drivers/redox-driver-sys/source/` — PciQuirkFlags. +- `local/sources/base/drivers/pcid/` — pcid fork (producer of + `/scheme/pci/*`, AER/pciehp event producer). + +--- + +## 8. Maintenance + +This doc replaces `local/docs/DRIVER-MANAGER-MIGRATION-PLAN.md`. When +the plan accumulates enough new content to require a fresh audit, +either: + +- **Update this doc** (add a new § 4 "systematic review" subsection + with the new findings; update § 1 status; extend § 5 fix plan), + **or** +- **Promote a new evidence file** to `local/docs/evidence/driver-manager/` + with a fresh date stamp and link from § 7.1. + +Do NOT reintroduce per-round "v5.x implementation summary" sections +in this doc — that is what made the old plan bloat to 182 KB. +Historical round-by-round detail belongs in the archived plan only. \ No newline at end of file