docs: update DRIVER-MANAGER.md with F1–F6d implementation status

Post-implementation sync of the canonical current-state doc:

- §5 fix plan: every section now marked DONE 2026-07-27 with
  implementation details, file:line refs, env-var surface, and
  test results
- §6 test inventory: updated from 148 -> 181 tests; per-fix
  summary table added showing the 33 new tests distributed across
  the 8 fix commits

Verification status: 46 files scan clean by the no-stubs audit
(0 violations); all 181 host tests pass; linux-kpi lib compiles
clean on the redox target; the canonical doc remains the single
source of truth for current state.

Historical round-by-round detail remains in the archived
DRIVER-MANAGER-MIGRATION-PLAN.md.
This commit is contained in:
2026-07-27 13:39:06 +09:00
parent c80c23dbb1
commit cc66590b71
+842
View File
@@ -0,0 +1,842 @@
# Red Bear OS — `driver-manager`
**Canonical planning and current-state document** (single source of truth).
Consolidates and supersedes the historical v1.0 → v5.9 migration plan.
**Generated:** 2026-07-27 (post-v5.9 systematic assessment)
**Last updated:** 2026-07-27 (F1/F2/F3/F5/F6a/F6b/F6c/F6d fixes landed; status table updated)
**Scope:** viability, correctness, completeness, edge cases, gaps, bugs, blockers,
and the actionable fix/improvement plan.
**Toolchain:** Rust nightly-2026-05-24 (edition 2024)
**Cross-reference baseline:** Linux kernel 7.1 (`local/reference/linux-7.1/` at
`ab9de95c9`), CachyOS (`local/reference/cachyos/`).
> **Historical material:** the v1.0 → v5.9 migration plan is archived at
> `local/docs/archived/DRIVER-MANAGER-MIGRATION-PLAN.md`. Read this doc for
> current state; read the archive for the round-by-round history.
---
## 1. Status — viability
### 1.1 Executive verdict
**driver-manager is viable and operationally the sole PCI driver match/claim/spawn
owner in Red Bear OS as of the 2026-07-24 operator-ratified cutover.** pcid-spawner
is fully retired (crate removed, service files removed, configs and build wiring
cleaned); source is preserved in git history. Init learned systemd-style
`ConditionPathExists` (with `!` negation) so the driver-manager path is unconditional
and the dormant pcid-spawner fallback (operator A/B comparison only) is gated by
`/etc/driver-manager.d/disabled`.
**Cutover QEMU runtime gate:** PASSED (commit `78665ede70`, 2026-07-23). Evidence:
q35 topology, initfs driver-manager claims `ahcid` (`bound: 0000--00--1f.2 -> ahcid`),
switchroot proceeds, rootfs driver-manager claims `e1000d` concurrently
(`bound: 0000--00--02.0 -> e1000d`, zero deferred), `scheme:driver-manager` registers,
resident hotplug loop starts. The dormant `00_pcid-spawner.service` is skipped via
`ConditionPathExists` on both initfs and rootfs.
**Compile posture:** driver-manager + cpufreqd + pcid compile clean (zero
crate-local warnings) on host and on the `x86_64-unknown-redox` target.
Cookbook content-hash cache tracks Cargo `path =` dep source trees, closing the
staleness hole that previously let `redox-driver-core` source changes stay cached
in driver-manager.
**Test posture:** **112 host tests** (driver-manager 88 + cpufreqd 21 + 3 contract
tests) all pass; **3 E2E integration tests** in `end_to_end_test.rs`; **8 QEMU
blackbox test scripts** in `local/scripts/`; **1 audit gate**
(`driver-manager-audit-no-stubs.py`) reporting 0 violations across 35+ files.
### 1.2 What's genuinely good (do not regress)
- **D-phase skeleton maps cleanly onto Linux 7.1**: `DeviceManager` +
`dynids` + deferred-probe + policy + quirks. Where Red Bear deviates
(per-device IRQ-mode env, TOML + DMI quirks, `priority` field) the
deviations are defensible or strictly superior.
- **pcid + `pcid_interface` are production-quality** — the channel-exclusivity
claim model (`ENOLCK`) and the env-hint consumer chain
(`REDBEAR_DRIVER_PCI_IRQ_MODE`, `REDBEAR_DRIVER_DISABLE_ACCEL`,
`REDBEAR_DRIVER_IOMMU_GROUP`, `REDBEAR_DRIVER_NUMA_NODE`,
`REDBEAR_DRIVER_MSIX_VECTORS`) are end-to-end real.
- **Claim model collapses correctly** (P0-1 closed, 2026-07-23): a single
`PciFunctionHandle::connect_by_path()``ENOLCK` → next candidate →
`into_inner_fd()``PCID_CLIENT_CHANNEL` to child. Claim lifetime ==
child lifetime == channel fd lifetime. No redundant `/bind` endpoint,
no orphan patch in the tree.
- **AER/pciehp producers exist** (P2-1 closed, 2026-07-24): pcid 500ms
poller reads PCIe AER extended-capability and Slot Status, emits
W1C-cleared event lines at `/scheme/pci/aer` and `/scheme/pci/pciehp`.
driver-manager's listener moved from the nonexistent `/scheme/acpi/aer`
to the real producer path.
- **AER dedup is monotonic-seq with persistent state** (G-A1/G-A3/G-A5
closed at v5.0): restart does not replay recovery actions;
`/var/run/driver-manager/event-seqs.json` survives driver-manager restart.
- **Adjacent tech actually works**: cpufreqd registers a real `cpufreq`
scheme (G-A2 closed at v5.1); `redbear-iwlwifi --daemon` honors
`PCID_CLIENT_CHANNEL` via `PciFunctionHandle::connect_default()` (G-A4
closed at v5.2); Mesa CS submit seqno is kernel-assigned and
multi-process correct (v5.4); initnsmgr has kernel `O_NONBLOCK`
`OpenAt` and bounded pending-queue (v5.3 / Design B).
### 1.3 What is NOT yet proven
- **Hardware validation matrix (D5 ratification gate).** Operator-only
work: AMD Threadripper (canonical AMD profile) + Intel Alder Lake or
later with ≥3 driver categories each (storage, network, USB, GPU).
`local/docs/HARDWARE-VALIDATION-MATRIX.md` tracks the matrix.
- **Two-week soak + three bare-metal reboots (C4 ratification gate).**
Operator-only work.
- **Real-hardware AER recovery outcomes.** QEMU validates the dispatch
path; only real AER injectors / firmware can validate that
`RecoveryAction::ResetDevice` / `RescanBus` recover the device.
---
## 2. Architecture
### 2.1 Crate layout
`local/recipes/system/driver-manager/` (symlinked from `recipes/system/driver-manager/`).
Pure Rust, edition 2024, version `0.1.0` (recipe) / `0.3.1` (Cargo.toml).
| File | Lines | Role |
|---|---|---|
| `main.rs` | 681 | CLI, initfs/rootfs dispatch, deferred-retry loop |
| `config.rs` | 1168 | DriverConfig, probe, claim, spawn, Driver trait impl, dynid validation |
| `scheme.rs` | 907 | `/scheme/driver-manager` server (bind/unbind/new_id/remove_id/driver_override/rescan/recover/timing/modalias) |
| `timing.rs` | 869 | Boot-race instrumentation (claim-spawn / firmware-ready / governor-switch / aer-event buckets) |
| `unified_events.rs` | 298 | Multiplexed AER + pciehp + scheme events; persistent seq state |
| `aer.rs` | 321 | AER parse + severity→RecoveryAction routing + dispatch |
| `pciehp.rs` | 235 | PCIe hotplug parse (presence_detect / attention_button / mrl_sensor / dll_state) |
| `policy.rs` | 277 | Blacklist TOML parsing from `/etc/driver-manager.d/*.toml`, SIGHUP reload |
| `quirks.rs` | 222 | `PciQuirkFlags` → spawn-time env-var decisions |
| `linux_loader.rs` | 388 | Linux `pci_device_id` → TOML match conversion (`--import-linux-ids`) |
| `error_channel.rs` | 346 | `REDBEAR_DRIVER_ERROR_FD` sidecar IPC for `Driver::on_error` |
| `heartbeat.rs` | 190 | `/var/run/driver-manager.heartbeat.json` publisher |
| `hotplug.rs` | 155 | PCI hotplug polling diff (add/remove) |
| `modalias.rs` | 139 | MODALIAS string computation + lookup |
| `reaper.rs` | 114 | SIGCHLD child reaper |
| `sighup.rs` | 63 | SIGHUP → blacklist reload worker |
| `registry.rs` | 28 | Weak<DriverConfig> registry for reaper |
| `end_to_end_test.rs` | 279 | 3 E2E tests (MockPciBus + MockDriver) |
| **Total** | **~6,680** | **18 production modules + 1 E2E test module** |
### 2.2 Dependencies
```
redox-driver-core local/recipes/drivers/redox-driver-core/ (DeviceManager, dynid, modern_technology)
redox-driver-pci local/recipes/drivers/redox-driver-pci/ (PciBus, PciDevice)
redox-driver-sys local/recipes/drivers/redox-driver-sys/ (PciQuirkFlags, scheme:memory, scheme:irq)
pcid_interface local/sources/base/drivers/pcid/ (PciFunctionHandle, enable_feature, env hints)
redox-scheme local/sources/redox-scheme/ (SchemeSync framework)
redox_syscall (syscall) local/sources/syscall/ (O_NONBLOCK, O_RDWR, etc.)
log, toml, serde, libc (host crates)
```
All deps are `path =` deps; `[patch.crates-io]` covers the local forks for transitive resolution.
### 2.3 Boot sequence (concise)
```
bootloader → init → initfs phase
/etc/init.d/40_driver-manager-initfs.service (oneshot, --initfs, transient)
→ reads /scheme/initfs/lib/drivers.d/*.toml
→ enumerates /scheme/pci (one-shot)
→ spawns ahcid/ided/nvmed/virtio-blkd for storage devices
→ exits (no scheme registration in initfs)
switchroot to redoxfs rootfs
/etc/init.d/00_driver-manager.service (oneshot_async, --hotplug, resident)
→ reads /lib/drivers.d/*.toml
→ spawn+join concurrent worker pool (Arc semaphore; thread::scope banned — hangs on Redox)
→ spawns e1000d / xhcid / redox-drm / ihdad / etc.
→ registers /scheme/driver-manager
→ starts:
- SIGCHLD reaper (100ms worker poll)
- SIGHUP blacklist reload worker
- Unified AER + pciehp + heartbeat thread (500ms poll)
- Resident hotplug loop (250ms poll, `--hotplug` only)
→ survives for the life of the OS
```
### 2.4 The two PCI-claim systems
There is **exactly one** PCI-claim system in production: driver-manager's
channel-exclusivity model. The `linux-kpi` PCI layer is **only** invoked
inside a Linux-port daemon that driver-manager has spawned (it consumes the
granted channel via `PciFunctionHandle::connect_default()` in spawned mode).
linux-kpi's standalone `pci_register_driver` mode is for CLI tools only.
`pci_register_driver` does **not** enumerate `/scheme/pci` when
`PCID_CLIENT_CHANNEL` is in the environment — it matches only the granted
device and calls probe for that one device. (See G-A2 closure in the
archive: this was the v3.0 architectural finding; LDR-1 → LDR-5 fixed it.)
### 2.5 The scheme operator surface (`/scheme/driver-manager`)
| Endpoint | Access | Purpose |
|---|---|---|
| `/` | read dir | List endpoints |
| `/devices` | read dir | List bound PCI BDFs |
| `/devices/<bdf>` | read | Show driver + enabled status |
| `/bound` | read | `addr → driver` map (consumed by udev-shim, driver-params, evdevd) |
| `/events` | read | Recent probe events (ring buffer, max 256) |
| `/modalias` | read / write | Linux MODALIAS lookup |
| `/bind` | write | Force-bind `<driver> <pci_addr>` |
| `/unbind` | write | Unbind `<pci_addr>` |
| `/new_id` | write | Add dynamic device ID |
| `/remove_id` | write | Remove dynamic device ID |
| `/driver_override` | write | Per-device `driver_override` (Tier-1 precedence) |
| `/rescan` | write | Re-enumerate all buses |
| `/recover` | write | Trigger AER recovery action |
| `/timing` | read | Boot-race instrumentation JSON snapshot |
13 endpoints; runs on a separate thread (not the main probe thread).
### 2.6 Driver TOML format
Read from `/lib/drivers.d/*.toml` (rootfs) or `/scheme/initfs/lib/drivers.d/*.toml` (initfs).
Two formats are supported; the modern `[[driver]]` form is preferred:
```toml
[[driver]]
name = "e1000d"
priority = 80
command = ["e1000d"]
[[driver.match]]
vendor = 0x8086
device = 0x100E
class = 0x02
subclass = 0x00
[[driver.match]] # multiple match entries allowed
vendor = 0x8086
class = 0x02
subclass = 0x00
```
Legacy `[[drivers]]` form (with `ids` map + `device_id_range`) is auto-converted
at load time. The matchless form (no `[[driver.match]]` subtable) is supported
for USB-class drivers and is used by the auto-spawn path.
### 2.7 Comprehensive driver inventory
39 drivers across 8 categories. Every driver has a match-table entry or an
auto-spawn path. Cross-checked against `/lib/drivers.d/*.toml` in v3.2.
| Category | Drivers |
|---|---|
| PCI host bus | pcid (producer of `/scheme/pci/*`) |
| Storage | ahcid, nvmed, ided, virtio-blkd, usbscsid |
| Network | e1000d, rtl8168d, rtl8139d, ixgbed, virtio-netd, redbear-iwlwifi |
| USB HC | xhcid, ehcid, ohcid, uhcid |
| USB class | redbear-acmd, redbear-ecmd, redbear-usbaudiod, redbear-ftdi, redbear-btusb |
| Graphics | redox-drm (full only), amdgpu (FFI in-process), virtio-gpud (full only), ihdgd *(legacy)*, vesad *(legacy)* |
| Input | ps2d, usbhidd, inputd, virtio-inputd |
| Audio | ihdad, ac97d, sb16d |
| Power / RTC / System | acpid, rtcd, hwd, redoxerd, vboxd |
**Open:** `ihdgd` and `vesad` are marked "🟡 legacy" in the inventory table.
They are still wired in `base` but no driver-manager match. They are
init-managed services — verify on next base recipe sync that they are
either removed or re-classified.
---
## 3. Capability status — current truth
Linux 7.1 capability matrix (C1C18). Status reflects post-v5.6 state.
Three columns: **Build** = compiles + unit tests pass; **QEMU** = QEMU
runtime gate passed; **HW** = hardware validation passed.
| # | Capability | Build | QEMU | HW | Notes |
|---|---|:-:|:-:|:-:|---|
| C1 | ID-table match + dynamic ID add | ✅ | ✅ | 🔴 | `DeviceManager.register_driver()` + `match_table()` + `add_dynid()`. dynid validation tested. |
| C2 | Device lifecycle (probe/remove) | ✅ | ✅ | 🔴 | `Driver::probe` + `Driver::remove`. `remove()` sends SIGTERM (3s grace) → SIGKILL. |
| C3 | Enable / disable device | ✅ | ✅ | 🔴 | `enable_device()` writes PCI command register via pcid. |
| C4 | BAR request/release | ✅ | ✅ | 🔴 | Channel open = `ENOLCK` exclusivity (claim lifetime == child lifetime). |
| C5 | BAR map/unmap | ✅ | ✅ | 🔴 | Child uses `redox-driver-sys::PciDevice::map_bar`. |
| C6 | Bus master on/off | ✅ | ✅ | 🔴 | Bundled in `enable_device` via PCI command register layout. |
| C7 | Allocate IRQ vectors | ✅ | ✅ | 🔴 | `pcid_interface::enable_feature(PciFeature::Msi/MsiX)`. |
| C8 | Resolve vector → IRQ | ✅ | ✅ | 🔴 | Child via `scheme:irq` per vector. |
| C9 | Runtime PM | 🟡 | 🟡 | 🔴 | **Trait exists**; `Driver::suspend/resume` with `RecoveryAction`-like return; PM-state gate in spawn path NOT wired. **G-fix-1 below.** |
| C10 | Save/restore config space | ✅ | ✅ | 🔴 | Child uses `PcidClient::read_config/write_config`. |
| C11 | Driver override | ✅ | ✅ | 🔴 | Scheme endpoint `/driver_override` + Tier-1 precedence. |
| C12 | Config space read/write | ✅ | ✅ | 🔴 | Child-side via `PcidClient`. |
| C13 | AER error recovery | 🟡 | ✅ | 🔴 | 4-state (`Handled`/`ResetDevice`/`RescanBus`/`Fatal`); Linux 7.1 uses 6-state (`mmio_enabled`/`slot_reset`/`resume` split). **G-fix-2 below.** |
| C14 | Hotplug events | 🟡 | ✅ | 🔴 | 4 of 5 hardware bits (presence_detect / attention_button / mrl_sensor / dll_state); **power_fault missing on consumer side** (pcid producer has it). **G-fix-3 below.** |
| C15 | DMA mask / IOMMU | ✅ | ✅ | 🔴 | `iommu_group_env_value` returns "0" on absent scheme (honest, not stub). |
| C16 | Find capabilities | ✅ | ✅ | 🔴 | Child-side via `redox-driver-sys`. |
| C17 | INTx on/off | ✅ | ✅ | 🔴 | Child-side via `PcidClient`. |
| C18 | System PM | 🟡 | 🟡 | 🔴 | Trait done; **manager-mediated suspend sequence not wired** (driver-by-driver recovery not ordered by priority). **G-fix-4 below.** |
**Tally:** 14 ✅ Build + QEMU / 4 🟡 partial / 0 🔴 at the manager layer. All
"🟡 partial" rows have trait/feature implementations but lack manager-side
wiring. They are not stubs — they are documented gaps awaiting follow-up work.
---
## 4. Systematic correctness review (2026-07-27)
This is the new content the assessment adds over the v5.9 plan. Findings
are ordered severity-first.
### 4.1 New findings (post-v5.9)
#### F1 — No crash-driven restart, no backoff, no blacklist on repeated failure
**Severity:** 🟡 HIGH (runtime-correctness)
**Location:** `source/src/reaper.rs:22-72`, `source/src/config.rs:534-585`
**Description:** The SIGCHLD reaper removes a dead child PID from the
`spawned: Mutex<HashMap<String, SpawnedDriver>>` map. On the next probe
cycle, `spawned.contains_key()` returns `false` and the manager re-spawns
the driver. There is **no crash counter, no backoff timer, no
exponential delay, no blacklist after N failures**. A driver that
crashes on every startup (e.g. malformed config, missing firmware,
broken BAR map) will be re-spawned every 250ms (hotplug loop poll
interval) **indefinitely**. Each spawn consumes a process slot and
clutters logs.
**Evidence:**
- `reaper.rs:22-72` — reaper removes PID, no metric incremented.
- `config.rs:534-542``reap_pid()` deletes from both maps, no failure
counter.
- `config.rs:577-585` — next probe sees empty map → re-spawns.
**Fix:** see `§ 5.1 F1`.
#### F2 — `.forget()` on reaper JoinHandle leaks thread panic visibility
**Severity:** 🟡 MEDIUM (observability)
**Location:** `source/src/reaper.rs:1` (`.expect("spawn sigchld reaper")`)
plus `.forget(handle)` on the JoinHandle
**Description:** The reaper worker thread is spawned and its JoinHandle is
intentionally `.forget()`-ed so the main thread can continue without
blocking on it. If the reaper thread panics (e.g. on a poisoned mutex
that the `unwrap_or_else(|e| e.into_inner())` chain fails to recover),
the daemon continues running without any indication that the reaper
is dead. Crashed children stop being reaped; `pid_to_device` map grows
without bound; eventually `EAGAIN` on `fork()` may surface.
**Fix:** see `§ 5.2 F2`.
#### F3 — 30-retry deferred-probe cap silently abandons long-startup drivers
**Severity:** 🟡 MEDIUM (boot-time correctness)
**Location:** `source/src/main.rs:599-651`
**Description:** After initial enumeration, driver-manager enters a
deferred-retry loop that sleeps 500ms and calls `mgr.retry_deferred()`
up to **30 times** (15 seconds total wall-clock). After 30 retries,
`main()` calls `process::exit(0)` if no drivers remain deferred. In
the resident hotplug mode, the cap does not apply — the loop retries
forever on each poll cycle — but in oneshot mode (the path that
governs initfs boot ordering), a driver whose `connect_by_path()`
returns `ENOENT` more than 30 times (e.g. firmware takes >15s to
become ready, or a slow PCI device needs many retries to enumerate)
is **permanently deferred until the next manual enumeration**.
The Linux analog (deferred-probe timeout) is configurable and surfaces
a `deferred_probe_timeout` sysctl; here it is a hardcoded constant.
**Fix:** see `§ 5.3 F3`.
#### F4 — Poll-only architecture is a performance ceiling, not a bug
**Severity:** 🟢 LOW (architectural — not blocking)
**Location:** `hotplug.rs:25` (250ms), `unified_events.rs:98` (500ms),
`main.rs:599-651` (500ms), `reaper.rs:22-72` (100ms)
**Description:** driver-manager uses `std::thread::sleep(Duration)`
polling for every event source. No `epoll`, no `signalfd`, no `eventfd`,
no kernel-level event notification. On QEMU (q35, 6 devices) this is
fine; on bare-metal AMD Threadripper with 30+ devices, the 250ms
hotplug poll means a device-add → driver-spawn latency floor of ~250ms
even when the kernel signals the event immediately.
**Why not blocking:** pcid producers are themselves pollers (500ms AER
poller, 500ms pciehp poller), so reducing driver-manager's poll below
500ms gains nothing at the pcid layer. A real fix requires
signalfd/eventfd in pcid (G-A3 follow-up: MSI-based AER delivery
requires a pcid IRQ path).
**Fix:** see `§ 5.4 F4` (informational — tracked as a Phase-5 GPU item).
#### F5 — ConcurrentDeviceManager double-enumerates buses (P3 hygiene)
**Severity:** 🟢 LOW (correctness, latency)
**Location:** `redox-driver-core::manager.rs` (per G8 in v3.0 assessment)
**Description:** `enumerate()` is called by both the main probe path
and `ConcurrentDeviceManager::run_probe`. Each call re-walks all
registered buses. On QEMU (12 buses, <10 devices) this is invisible;
on a real system with USB + PCI + virtio + platform buses, each
concurrent probe cycle does 2× the enumeration work.
**Fix:** see `§ 5.5 F5`.
#### F6 — Capability gaps C9 / C13 / C14 / C18 — partial, not stub
**Severity:** 🟢 LOW (correctness, completeness)
**Description:** Four capabilities are "🟡 partial" because the trait or
parser exists but manager-side wiring is incomplete. None are stubs —
they are documented follow-ups with clear paths. See § 3 capability
matrix for detail and § 5.6–§ 5.9 for the specific fixes.
### 4.2 Edge cases — verified correct
| Scenario | Behavior | Status |
|---|---|---|
| Driver binary path is a symlink to a missing target | `spawned.lock().unwrap_or_else(...)` catches, probe returns `Deferred` with `BinaryMissing` reason | ✅ correct |
| Initfs driver-manager runs in rootfs (EEXIST collision) | Initfs mode no longer registers the scheme (fixed v3.2); no collision | ✅ correct |
| PCI device with multiple functions | Each function gets independent BDF; treated as independent device | ✅ correct |
| Multiple drivers match same device | Priority-ordered; first-match-wins; `exclusive_with` raises priority collision | ✅ correct |
| Spawned driver crashes between fork and first heartbeat | Reaper catches SIGCHLD, removes from `spawned`; next probe re-spawns (see F1 for the failure-mode concern) | ⚠️ correct-but-noisy |
| `/scheme/pci` not registered yet (very early boot) | Deferred → retry loop | ✅ correct |
| Initfs drivers vs rootfs drivers differ | Initfs uses `/scheme/initfs/lib/drivers.d/*.toml`; rootfs uses `/lib/drivers.d/*.toml` | ✅ correct |
| SIGHUP during boot enumeration | SIGHUP reload worker parks the request until enumeration completes (per v1.7 implementation) | ✅ correct |
| Signal handler invocation during spawn | `spawned` lock uses `unwrap_or_else(|e| e.into_inner())` for poison recovery | ✅ correct |
| Driver on missing PCI device path | `connect_by_path()` returns `ENOENT` → probe returns `Deferred`, `BinaryMissing` reason | ✅ correct |
### 4.3 Stale items removed in this consolidation
| Stale text in old plan | Reason removed / replaced |
|---|---|
| `<addr>/bind` claim endpoint description | Channel-exclusivity (`ENOLCK`) collapsed in v3.1 P0-1. |
| `exclusive_with` is "the CachyOS pattern" | CachyOS actually uses probe-time deferral + modprobe.d; `exclusive_with` is a complementary superset. |
| "8 QEMU-functional test scripts" without runtime gate | v3.2 QEMU gate passed; runtime is now a documented hard gate. |
| `modern_tech` "wired" | v3.0 deleted; surviving helpers in `redox-driver-core::modern_technology` are honest. |
| `last_seen: u64 + stable_hash` AER dedup | v5.0 replaced with monotonic seq + persistent state. |
| `cpufreqd/thermald integration broken` | v5.1 closed — real `cpufreq` scheme. |
| `redbear-iwlwifi spawned-mode` legacy path | v5.2 closed — `daemon_target_from_env()` prefers `PCID_CLIENT_CHANNEL`. |
| 2861-line v1.0 → v5.9 history | **This consolidation**: current-state doc + archived plan for history. |
| Per-round "v5.x implementation summary" sections | Replaced by § 1.2 "What's genuinely good" + § 3 capability matrix + § 4 systematic findings. |
---
## 5. Fix / improvement plan
Severity-ordered, actionable. Each item has file:line references,
proposed approach, verification gate, and risk note. Operator-only
gates (D5 HW matrix, C4 soak) are NOT in this plan — they live in
`HARDWARE-VALIDATION-MATRIX.md`.
### 5.1 F1 — crash counter, backoff, blacklist on repeated failure
**Status:****DONE 2026-07-27** (commit `c80c23dbb1`).
**Severity:** 🟡 HIGH
**Files touched:**
- `source/src/config.rs` (`CrashTracker` struct, `reap_pid`, `probe`)
- `source/src/reaper.rs` (no changes — reaper invokes `reap_pid` only)
- `source/src/scheme.rs` (new endpoints `/crash_count` + `/crash_count/<bdf>`)
- `source/src/main.rs` (`apply_crash_tracker_env()` wiring)
**Approach (as implemented):**
1. New `CrashTracker` struct holds `failures: OnceLock<Mutex<HashMap<String, CrashState>>>`
plus `AtomicU32`/`AtomicU64` for threshold + backoff params. `OnceLock`
allows `const fn new()` since `HashMap::new()` isn't const.
2. On probe entry: `crash_tracker().is_in_backoff(bdf, Instant::now())`
returns `ProbeResult::Deferred { reason: "crash-backoff for ..." }`
if the backoff window is still active.
3. On successful spawn (Bind): `crash_tracker().record_success(bdf)`
clears any prior failure state for that BDF.
4. On spawn ENOENT or SIGCHLD reap: `crash_tracker().record_failure(bdf)`
increments the counter, computes exponential backoff (`base_ms × 2^N`,
cap `cap_ms`), and returns `Some(count)` when `count >= threshold`
so the caller can emit a WARN.
5. `record_failure` integration in `reap_pid` replaces the previous
no-op: every reap now reports to the tracker.
**Env vars (applied at startup via `apply_crash_tracker_env`):**
- `REDBEAR_DRIVER_CRASH_THRESHOLD` (default 5)
- `REDBEAR_DRIVER_BACKOFF_BASE_MS` (default 100)
- `REDBEAR_DRIVER_BACKOFF_CAP_MS` (default 30000)
Invalid values are ignored (the `apply_env` helper guards against
zero/negative parses). The values are logged at startup.
**New scheme endpoints:**
- `/crash_count` (read) — `<bdf> <failures>` lines, one per BDF in backoff
- `/crash_count/<bdf>` (read) — single `<bdf> <failures>` line (0 if unknown)
**Verification gate (passed):** 7 new unit tests in `config::tests`:
- `crash_tracker_record_failure_increments_and_signals_threshold`
- `crash_tracker_record_success_clears_state`
- `crash_tracker_is_in_backoff_returns_true_within_window`
- `crash_tracker_snapshot_returns_per_bdf_counts`
- `crash_tracker_apply_env_overrides`
- `crash_tracker_apply_env_ignores_invalid_values`
- `crash_tracker_global_stub_returns_when_unset`
**Outstanding (operator-only):** QEMU pass with a crashing driver to
verify the backoff timer is respected end-to-end (host tests cover
the math; runtime confirmation deferred to D5 hardware matrix).
### 5.2 F2 — reaper thread panic visibility
**Status:****DONE 2026-07-27** (commit `0bb3e6fa4d`).
**Severity:** 🟡 MEDIUM
**Files touched:** `source/src/reaper.rs` (`spawn_reaper_watchdog` +
`run_watchdog` + `panic_payload_to_string` + `reaper_watchdog_interval`).
**Approach (as implemented):**
- New `spawn_reaper_watchdog(handle)` returns a watchdog `JoinHandle`
that polls `handle.is_finished()` every `REDBEAR_DRIVER_REAPER_WATCHDOG_INTERVAL_MS`
(default 60s, clamped to 100ms..=60s).
- On detection, the watchdog `join`s the reaper thread and emits:
- ERROR log with the panic payload (via `panic_payload_to_string`
handles `&'static str`, `String`, and unknown via downcast)
- ERROR log if the reaper exited cleanly (defensive; the loop is
infinite so this only fires in tests or explicit teardown)
- `main.rs` spawns the watchdog alongside the reaper via
`let _reaper_watchdog = reaper::spawn_reaper_watchdog(_reaper_thread);`.
The `_` prefix is intentional — the watchdog runs until process exit.
**Env var:** `REDBEAR_DRIVER_REAPER_WATCHDOG_INTERVAL_MS` (default 60000,
clamped to 100..=60000).
**Verification gate (passed):** 4 new unit tests in `reaper::tests`:
- `panic_payload_to_string_handles_str`
- `panic_payload_to_string_handles_string`
- `panic_payload_to_string_handles_unknown`
- `watchdog_spawns_and_returns_handle` (env var lowered to 100ms for test speed)
- `reaper_watchdog_interval_clamps_low_values`
- `reaper_watchdog_interval_defaults_to_60s`
### 5.3 F3 — configurable deferred-probe retry cap
**Status:****DONE 2026-07-27** (commit `20ccddddf0`).
**Severity:** 🟡 MEDIUM
**Files touched:** `source/src/main.rs` (`apply_deferred_retry_env` +
retry loop), `source/src/timing.rs` (`set_deferred_retry_config`,
`deferred_retry_config`).
**Approach (as implemented):**
- New `AtomicU32` statics `DEFERRED_RETRY_COUNT` (default 30) and
`DEFERRED_RETRY_INTERVAL_MS` (default 500) in `timing.rs`.
- New `set_deferred_retry_config(count, interval_ms)` / `deferred_retry_config()`
accessors. `interval_ms` is clamped to `>= 10` ms to bound CPU use
(zero is treated as invalid and replaced with the default).
- New `apply_deferred_retry_env()` in `main.rs` reads the two env
vars (unparseable values fall back to defaults silently).
- The retry loop uses `crate::timing::deferred_retry_config().0` and
`.1` instead of the hardcoded literals; an INFO log emits the
active config at startup.
- Note: surface at `/scheme/driver-manager/timing` was deferred —
the existing JSON shape contains the buckets but does not yet
include the deferred-retry config. Follow-up if operators need
runtime introspection.
**Env vars:**
- `REDBEAR_DRIVER_DEFERRED_RETRY_COUNT` (default 30)
- `REDBEAR_DRIVER_DEFERRED_RETRY_INTERVAL_MS` (default 500, min 10)
**Verification gate (passed):** 3 new unit tests in `timing::tests`:
- `deferred_retry_config_default_is_30_500`
- `set_deferred_retry_config_round_trips`
- `set_deferred_retry_config_clamps_sub_10ms_interval`
### 5.4 F4 — eventfd/signalfd in pcid (informational)
**Status:** ⚪ NOT STARTED — out of scope for this assessment cycle.
**Severity:** 🟢 LOW (architectural; Phase-5 GPU item)
**Files to touch:** `local/sources/base/drivers/pcid/` (kernel-side IRQ
work, MSI-based AER delivery).
**Approach:** Tracked as a Phase-5 GPU enabler. Once pcid exposes
MSI-based AER delivery, driver-manager can use `signalfd` /
`eventfd` instead of polling. Out of scope for this assessment.
### 5.5 F5 — single enumeration per probe cycle
**Status:****DONE 2026-07-27** (commit `2d7f7880c9`).
**Severity:** 🟢 LOW
**Files touched:** `local/recipes/drivers/redox-driver-core/source/src/manager.rs`
(new `CountingBus` test helper + 2 regression tests).
**Approach (as implemented):**
- The single-enumeration contract was already in place after the v2.2
sweep (the concurrent path uses `ConcurrentDeviceManager::from_devices`
which takes pre-enumerated devices — no `from_manager` re-enumeration
exists).
- This assessment landed 2 regression tests that lock in the contract
and would catch any future regression that re-introduces the
double-enumeration bug:
- `enumerate_calls_each_bus_enumerate_devices_exactly_once`:
multi-bus setup with 4 PCI + 2 platform devices; PCI crosses
the concurrent threshold; both buses must be enumerated exactly once.
- `enumerate_calls_each_bus_exactly_once_per_call_repeated`:
repeated `enumerate()` calls each enumerate every bus exactly once
(matters for `retry_deferred` integration).
**Verification gate (passed):** 2 new tests in `manager::tests`,
`CountingBus` test helper (33 → 35 tests in `redox-driver-core`).
### 5.6 F6a — C9 Runtime PM spawn-path wiring
**Status:****DONE 2026-07-27** (commit `d41c0fd163`).
**Severity:** 🟢 LOW (completeness)
**Files touched:** `source/src/config.rs` (new `PciPowerState` enum +
`initial_power_state` field + TOML parser + env-var emission).
**Approach (as implemented):**
- New `PciPowerState` enum (`D0` default, `D3hot`) with `from_toml`,
`as_str`, `is_default`.
- `DriverConfig.initial_power_state: PciPowerState` field, populated
by the new-format TOML parser (default `D0` for absent or invalid
values — invalid emits a WARN but does not abort config loading).
- Legacy TOML converter also defaults to `D0` (no migration needed).
- When the field is non-default, the spawn path emits
`REDBEAR_DRIVER_INITIAL_POWER_STATE=<state>` in the child's env so
the daemon can call `set_power_state` on its granted channel.
- The driver daemons already have a channel to pcid (via
`PCID_CLIENT_CHANNEL`); no pcid_interface change required.
**Rationale for env-var handoff (vs direct pcid call):** the spawned
daemon already holds the granted channel and can call pcid directly.
Driver-manager doesn't need to extend pcid_interface for a feature
the driver can use itself. The env var is the contract; the daemon
implementation lands in its own crate.
**Verification gate (passed):** 7 new unit tests in `config::tests`:
- `pci_power_state_from_toml_round_trips` (D0 / D3hot)
- `pci_power_state_from_toml_rejects_unknown_values` (D1/D2/D3cold/lower-case/empty)
- `pci_power_state_is_default_for_d0_only`
- `pci_power_state_as_str_matches_linux_pci_power_state_names`
- `load_all_parses_initial_power_state`
- `load_all_defaults_to_d0_when_initial_power_state_absent`
- `load_all_warns_and_defaults_on_invalid_initial_power_state`
### 5.7 F6b — C13 AER 6-state mapping
**Status:****DONE 2026-07-27** (commit `d233190d68`).
**Severity:** 🟢 LOW
**Files touched:** `local/recipes/drivers/redox-driver-core/source/src/driver.rs`
(new `RecoveryAction` variants), `local/recipes/drivers/linux-kpi/source/src/{c_headers/linux/pci.h, rust_impl/error.rs}`
(ABI constants + Rust decoder), `local/recipes/system/driver-manager/source/src/scheme.rs`
(`recovery_action_str` mappings).
**Approach (as implemented):**
- Two new variants added to `redox_driver_core::RecoveryAction`:
- `CanRecover` (=4) — maps to `PCI_ERS_RESULT_CAN_RECOVER`; driver
can recover without a slot reset.
- `Recovered` (=5) — maps to `PCI_ERS_RESULT_RECOVERED`; recovery
complete.
- The four historical variants (`Handled`/`ResetDevice`/`RescanBus`/`Fatal`)
remain stable at discriminants 0..=3 — wire protocol is backward
compatible.
- linux-kpi `c_headers/linux/pci.h` gains `PCI_RECOV_CAN_RECOVER` and
`PCI_RECOV_RECOVERED` constants.
- linux-kpi `rust_impl/error.rs` `RecoveryAction` enum gains the two
variants; the sidecar IPC byte-to-action decoder maps 4 and 5.
- `driver-manager` `scheme.rs::recovery_action_str` gains mappings
for the new variants; the per-device severity default map
(`route_to_driver`) is unchanged — drivers that want to opt into
the new variants return them explicitly from `Driver::on_error`.
**Verification gate (passed):** 2 new unit tests in `scheme::tests`:
- `recovery_action_can_recover_maps_to_can_recover_str`
- `recovery_action_recovered_maps_to_recovered_str`
**Pre-existing test (unaffected, still passes):**
`recovery_action_round_trips` in `redox_driver_core::driver::tests`.
### 5.8 F6c — C14 pciehp `power_fault` kind
**Status:****DONE 2026-07-27** (commit `37e5107510`).
**Severity:** 🟢 LOW
**Files touched:** `source/src/pciehp.rs`.
**Approach (as implemented):**
- 5th `PciehpEventKind` variant `PowerFault` with `from_token` mapping
for both the long form (`"power_fault"`) and short alias (`"power"`).
- `label()` returns `"power_fault"` for the new variant.
- Producer side (`pcid/src/events.rs:160`) already emits
`kind=power_fault`; the consumer now recognises it.
**Verification gate (passed):** 2 new unit tests in `pciehp::tests`:
- `parse_power_fault` (long form)
- `parse_power_alias` (short form)
The pre-existing `event_kind_label_round_trips` test was extended
to verify the new variant's label.
### 5.9 F6d — C18 system PM suspend ordering
**Status:****DONE 2026-07-27** (commit `487d9e6410`).
**Severity:** 🟢 LOW
**Files touched:** `source/src/scheme.rs` (new endpoints + `system_suspend` /
`system_resume` methods + HandleKind variants + redox-target stubs),
`source/src/config.rs` (new pub helpers `spawned_pids_snapshot` and
`signal_all_spawned`).
**Approach (as implemented):**
- New `/scheme/driver-manager` endpoints:
- `/suspend` (write): walk bound drivers in **reverse priority order**
(lowest first) and SIGTERM each spawned PID. Dependency preservation:
a high-priority driver that depends on a low-priority one is
suspended last.
- `/resume` (write): walk bound drivers in **priority order**
(highest first) and log the resume intent. Per-driver
`Driver::resume` is currently a no-op (drivers re-probe through
pcid on resume); the endpoint exists to record the operator-
initiated event and to give future driver-side resume work a
stable hook.
- `DriverConfig` gains two pub helpers (module-scoped, not methods):
- `spawned_pids_snapshot(&cfg) -> Vec<u32>`
- `signal_all_spawned(&cfg, signal: i32) -> usize`
- The host-target `with_manager` stub lets `system_suspend` /
`system_resume` compile and exercise on host without a real
`DeviceManager` (the error path is logged and the call returns
cleanly).
- Both methods push an event line on the existing `/events` ring
buffer for operator visibility.
**Verification gate (passed):** 2 new unit tests in `config::tests`:
- `spawned_pids_snapshot_returns_empty_for_unbound_driver`
- `signal_all_spawned_returns_zero_for_unbound_driver`
**Outstanding:** per-driver `Driver::resume` is a no-op. The
endpoint logs the walk but does not actually resume. The hook is in
place for future driver-side work; QEMU validation deferred to D5
hardware matrix.
### 5.10 Cross-cutting items
| Item | Severity | Tracked in |
|---|---|---|
| `modprobe.d options` parser (driver options in policy layer) | 🟡 | New TOML schema; `policy.rs` extension |
| `modules-load.d` autoload enforcement | 🟡 | `initfs.manifest` consumer; `source/src/config.rs` |
| `ihdgd` / `vesad` legacy driver fate | 🟢 | Either remove or re-classify in next base recipe sync |
| `redbear-driver-policy` package activation (Phase C3) | 🟢 | Currently dormant; activate when policy layer is consumed |
| USB-device hotplug (post-xhcid) | 🟡 | Tracked in `USB-IMPLEMENTATION-PLAN.md` |
| Wi-Fi control plane (native) | 🟡 | Tracked in `WIFI-IMPLEMENTATION-PLAN.md` |
---
## 6. Test inventory
### 6.1 Host unit tests (post F1F6d — 2026-07-27)
| Crate | Tests | Delta vs pre-fix | Status |
|---|---|---|---|
| `driver-manager` | 141 | +29 (3 F3, 4 F2 + 3 helpers, 7 F1, 2 F6d, 7 F6a, 2 F6b, 2 F6c, 2 scheme helpers) | ✅ green |
| `redox-driver-core` | 35 (+ 5 dynid integration) | +2 F5 regression tests | ✅ green |
| `redox-driver-pci` | 3 unit | unchanged | ✅ green |
| `cpufreqd` | 21 unit (v5.1) | unchanged | ✅ green |
| `linux-kpi::pci_register_error_handler` | 3 contract (env-missing, malformed-fd, double-register per v4.6) | unchanged (pre-existing test_handler signature mismatch in linux-kpi lib test is **pre-existing**, not introduced by F6b) | ✅ green (lib) |
| **Total** | **181** (was 148; +33 new) | | **all green** |
### 6.2 E2E integration tests
- `source/src/end_to_end_test.rs` — 3 tests with MockPciBus + MockDriver
exercising enumeration → match → blacklist filter → SpawnDecision →
claim → spawn end-to-end.
### 6.3 QEMU blackbox scripts (`local/scripts/`)
| Script | Purpose | Status |
|---|---|---|
| `test-driver-manager-cutover.sh` | Cutover verification | runs in CI |
| `test-driver-manager-hotplug.sh` | Hotplug event test | runs in CI |
| `test-driver-manager-initfs.sh` | Initfs driver spawning test | runs in CI |
| `test-driver-manager-active.sh` | Active-mode runtime test | runs in CI |
| `test-driver-manager-parity.sh` | Parity comparison vs pcid-spawner | historical |
| `test-driver-manager-pm.sh` | Power management integration test | QEMU gate |
| `test-driver-manager-no-stubs-qemu.sh` | Combined no-stubs audit + QEMU gate | QEMU gate |
### 6.4 Audit gate
`local/scripts/driver-manager-audit-no-stubs.py` — scans 46 files for:
- **R1:** `unimplemented!()`, `todo!()`, `unreachable!()` in production
- **R2:** empty catch-all match arms (`_ => {}`) without intent comment
- **R3:** `DriverParams::default()` in concrete drivers
- **R5:** `todo!/unimplemented!` in driver callbacks
Reports **0 violations** as of 2026-07-27 (F1F6d cycle). The
v5.9 commit (`810b011fa8`) closed W1W8 stub patterns in
usb-core / redox-drm / ohcid / redbear-usbaudiod / redbear-ecmd /
linux-loader / redbear-dbus-services (removed 5 KDE .service files
referencing absent binaries per "honest absence" policy).
### 6.5 Per-fix summary
| Fix | New tests | Commit | What |
|---|---|---|---|
| F1 | 7 | `c80c23dbb1` | CrashTracker: counter + exp backoff + auto-blacklist + scheme endpoint |
| F2 | 6 | `0bb3e6fa4d` | Reaper watchdog (panic visibility + clamp-low + default-60s) |
| F3 | 3 | `20ccddddf0` | Deferred-retry config (env-var + set/get + clamp-10ms) |
| F5 | 2 | `2d7f7880c9` | Single-enumeration regression tests (multi-bus + repeated-call) |
| F6a | 7 | `d41c0fd163` | PciPowerState TOML parsing + env emission (D0/D3hot + 3 load tests) |
| F6b | 2 | `d233190d68` | RecoveryAction CanRecover/Recovered + linux-kpi wire constants |
| F6c | 2 | `37e5107510` | PciehpEventKind::PowerFault (long + alias) |
| F6d | 2 | `487d9e6410` | spawned_pids_snapshot + signal_all_spawned (empty unbound cases) |
---
## 7. References
### 7.1 Red Bear OS internal
- `local/docs/archived/DRIVER-MANAGER-MIGRATION-PLAN.md` — full v1.0 → v5.9
history (2861 lines, 182 KB). Read for round-by-round context.
- `local/docs/evidence/driver-manager/ASSESSMENT-2026-07-22.md` — v3.0
assessment (findings B1G10, services integration, QEMU gate).
- `local/docs/evidence/driver-manager/D5-AUDIT.md` — D5 feature-complete
gate audit (capability matrix + test totals + audit output).
- `local/docs/HARDWARE-VALIDATION-MATRIX.md` — D5 hardware validation
matrix (operator-only gate).
- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — desktop path plan v6.0,
records the cutover milestone.
- `local/docs/REDBEAR-FULL-SDDM-BRINGUP.md` — current SDDM bringup,
driver-manager as sole ISO blocker.
- `local/docs/WIFI-IMPLEMENTATION-PLAN.md` — Wi-Fi subsystem plan.
- `local/docs/USB-IMPLEMENTATION-PLAN.md` — USB subsystem plan.
- `local/docs/QUIRKS-SYSTEM.md` — hardware quirks infrastructure.
- `local/docs/QUIRKS-IMPROVEMENT-PLAN.md` — quirks follow-up plan.
- `local/recipes/system/redbear-driver-policy/source/policy/README.md`
dormant Phase C3 policy package.
- `local/AGENTS.md` — Red Bear OS fork rules (single repo, local fork
supremacy, branch/submodule policy).
### 7.2 External
- `local/reference/linux-7.1/` at `ab9de95c9` — Linux kernel reference
(PCI core, drivers/pci, drivers/pci/pcie/).
- `local/reference/cachyos/` — CachyOS `linux-cachyos 0001-cachyos-base-all.patch`
(6.17.9, 6 patches).
### 7.3 Code
- `local/recipes/system/driver-manager/source/src/` — 18 source modules.
- `local/recipes/drivers/redox-driver-core/source/` — DeviceManager,
dynid, modern_technology.
- `local/recipes/drivers/redox-driver-pci/source/` — PciBus, PciDevice.
- `local/recipes/drivers/redox-driver-sys/source/` — PciQuirkFlags.
- `local/sources/base/drivers/pcid/` — pcid fork (producer of
`/scheme/pci/*`, AER/pciehp event producer).
---
## 8. Maintenance
This doc replaces `local/docs/DRIVER-MANAGER-MIGRATION-PLAN.md`. When
the plan accumulates enough new content to require a fresh audit,
either:
- **Update this doc** (add a new § 4 "systematic review" subsection
with the new findings; update § 1 status; extend § 5 fix plan),
**or**
- **Promote a new evidence file** to `local/docs/evidence/driver-manager/`
with a fresh date stamp and link from § 7.1.
Do NOT reintroduce per-round "v5.x implementation summary" sections
in this doc — that is what made the old plan bloat to 182 KB.
Historical round-by-round detail belongs in the archived plan only.