RedBear-OS/local/docs/INIT-NAMESPACE-MANAGER-SCALABILITY-PLAN.md

# Red Bear OS Init Namespace Manager Scalability Plan

## Purpose

This document is the **canonical analysis and execution plan** for a systemic reliability defect
in the boot/console path: the single-threaded, blocking design of the init namespace manager
(`initnsmgr`) turns any transient slowness in *one* scheme provider into a *whole-system* wedge.

It was written after root-causing the long-standing "`redbear-mini` never reaches a usable brush
login" symptom (16+ prior investigation sessions with shifting, incorrect theories). The immediate
trigger was a concrete acpid bug (fixed — see below), but the reason a single stuck daemon could
freeze the entire system is the architecture described here. That amplifier is still present.

When another document discusses mini/console login reliability, boot slowness, "flaky boot hangs",
or the namespace/scheme-open path, prefer this file for the current robustness judgment and the
execution order.

## Ground state (2026-07-21)

- **`redbear-mini` logs into `brush` and executes commands.** Verified by framebuffer ground truth:
  `Red Bear login: user` → MOTD → `user@redbear: $` → `echo RB=$((21*2))=OK` → `RB=42=OK`. When it
  wins the race, it is fast (login prompt ~12s, brush prompt ~16s in QEMU q35/KVM).
- The dominant blocker — an acpid AML-mutex self-deadlock — is **fixed** (`local/sources/base`
  commit `d78fd44a`). See "Trigger 1" below.
- A **residual, architectural** flakiness remains: under load, boot occasionally wedges at the
  fbcond→vesad display handoff (`display.vesa:v2/2`). This is not a vesad bug; it is the namespace
  manager amplifying vesad's transient unreadiness. See "The amplifier" and "Execution plan".

## The two layers

### Trigger 1 — acpid AML mutex (FIXED, `d78fd44a`)

`cpufreqd` reads `/scheme/acpi/processor/CPUn/pss` (ACPI P-state data). Evaluating that AML runs
`_ACQ` on an ACPI mutex. The `acquire` handler in
`local/sources/base/drivers/acpid/src/aml_physmem.rs` had two bugs:

1. **Timeout units.** ACPI `_ACQ` timeouts are milliseconds; `0xFFFF` is the spec's "wait forever"
   (ACPICA `ACPI_WAIT_FOREVER`, `include/acpi/actypes.h:459`). The code multiplied the value by
   1000, treating it as seconds — `0xFFFF` became ~18 hours.
2. **No ownership / no recursion.** ACPI mutexes are recursive: the owning thread may re-acquire,
   each `Acquire` paired with a `Release` (ACPICA `acpi_ex_acquire_mutex_object`,
   `drivers/acpi/acpica/exmutex.c:140`, `acquisition_depth`). The code tracked only a "held" set
   with no owner, so a nested acquire by the single AML thread waited for a release that only the
   waiting thread itself could perform — a self-deadlock.

Fixed by tracking `(owner ThreadId, recursion depth)` per mutex, treating a nested acquire by the
owner as a depth bump, interpreting the timeout as milliseconds, and bounding any wait to 5s so a
misbehaving AML method can never freeze the scheme-serving thread. Verified against
`local/reference/linux-7.1` ACPICA.

### The amplifier — `initnsmgr` head-of-line blocking (NOT fixed)

`local/sources/base/bootstrap/src/initnsmgr.rs` runs the init namespace manager. Every process in a
restricted login namespace (i.e. every login shell, and much of init's own spawn path) resolves
paths through it: the kernel routes an open of `scheme:path` in that namespace to the manager, which
proxies to the real provider. Two properties make this fragile:

- It is **single-threaded**: one `run()` loop, `next_request` → `handle_sync` → `write_response`.
- Its `open_scheme_resource` does a **blocking `syscall::openat`** on the provider's capability fd,
  *inside the dispatch loop*.

So when any provider is briefly not servicing its socket (descheduled under load, mid-`tick`, in a
one-shot init window), the manager's `openat` blocks, the loop stops, and **every open in the
system queues behind it forever**. This is textbook head-of-line blocking. It also explains mini's
historical slowness: ~2452 namespace opens serialize through that one thread per boot.

acpid was merely the trigger with ~100% hit probability. With acpid fixed, the next-most-likely
trigger (vesad's display handoff) surfaces at a lower rate. **Fixing individual daemons is a
whack-a-mole; the amplifier is the real defect.**

Diagnostic proof (reusable): the console/log stack is itself a victim of the wedge, so a diagnostic
that writes `/scheme/debug` or re-opens a scheme goes silent exactly when the wedge starts. Cut
through it with (1) the QEMU debug console (I/O port `0xE9`, `qemu -debugcon file:`) after acquiring
port-IO rights via the pcid handshake, and (2) pre-opening the kernel scheme fds once and re-reading
via `lseek(0)`+`read`. A per-open trace in `initnsmgr` (to a debug fd from
`UPPER_FDTBL_TAG + GlobalSchemes::Debug`) pinpointed the wedged open as `acpi:processor/CPU3/pss`
(then, post-fix, `display.vesa:v2/2`).

## Co-victim audit (2026-07-21): other daemons on the same chain

A systematic audit of all 18 scheme-serving daemons for the acpid failure class (a single-threaded
daemon that can wait unboundedly in its serving thread) found:

- **acpid was the only *software* unbounded wait** in the boot-critical path (the AML mutex bug —
  fixed). No other boot-path scheme handler contains an unbounded software wait or a reentrant
  blocking open/read/call of another scheme.
- **Hardware busy-waits** (`rtcd` RTC UIP bit, `ps2d` PS/2 `OUTPUT_FULL`, `sb16d`/`ihdad` DSP/RIRB,
  `ihdgd`/`ahcid`/`nvmed`/`bcm2835` register polls) are either hardware-bounded (the bit clears in
  microseconds) or live in daemons that pcid does not spawn without the matching device (no audio on
  QEMU q35, etc.). Not the acpid class. (They are still a robustness item under the project WARNING
  POLICY — unbounded on paper — but adding timeouts to ~15 sites blind, without a reproducing hang,
  risks regressions and is out of scope here.)
- **`ucsid` is the one remaining co-victim worth noting.** It consumes `/scheme/acpi/*` in
  `build_state()` *before* it publishes its scheme, and in `config/redbear-mini.toml` it is the only
  boot-critical, acpi-dependent unit with a **blocking** init type (`type = { scheme = "ucsi" }`,
  while every other acpi consumer there is `oneshot_async`). Before the acpid fix, a stalled acpid
  would have hung ucsid's init and thus boot. It already degrades gracefully on `EAGAIN`
  (`discover_ucsi_devices` returns empty), but `EAGAIN` only covers "acpid is busy", not "acpid is
  wedged" (open blocks, per the kernel corollary below). With acpid fixed it works.

  **Do not simply flip ucsid to `oneshot_async`** to satisfy the "optional hardware must not block
  boot" rule (`bare-boot-blocking-init-units`): `ready_sync_scheme` hands the scheme cap fd to init
  via the INIT_NOTIFY pipe (`daemon/src/lib.rs:130` → `ready_with_fd`), so init registers the `ucsi`
  scheme *on the daemon's behalf* — only the `{scheme=…}` init type does that. `oneshot_async` would
  leave `ucsi` unregistered (consumers get `ENODEV`). The correct hardening is a source refactor:
  publish the scheme immediately with a lazy/empty state, then run the heavy ACPI/i2c discovery in
  the background after `ready_sync_scheme`. It is defense-in-depth against an already-fixed trigger,
  not an active bug, so it must be done with runtime validation, not blind at session end.

## The kernel corollary — `open` ignores `O_NONBLOCK`

The manager cannot even *defer* a slow open today, because in Redox an `open()` with `O_NONBLOCK`
still blocks: `O_NONBLOCK` governs later reads, not the open itself. This is documented in-tree at
`local/sources/base/drivers/inputd/src/lib.rs:139` ("this can block indefinitely") and enforced by
`local/sources/kernel/src/scheme/user.rs` `UserInner::call_inner`, which unconditionally `.block()`s
the caller until the provider replies (no `O_NONBLOCK` check on the request/response path).

Notably, **fbcond already has handoff-retry logic that would work if open returned `EAGAIN`** — its
retry never fires today only because the first open blocks instead of returning.

## Execution plan (fix EITHER to break the chain; both are worth doing)

Ordered by blast-radius / safety. Neither was attempted at the end of the discovering session:
both live in the most critical components (kernel, `no_std` bootstrap), where an error means "the
system does not boot at all", and the discovering host was under heavy uncontrolled external load,
making clean validation impossible. They must be done deliberately, with framebuffer-ground-truth
validation on an idle host or real hardware.

### Option A — make `initnsmgr` not block the dispatch loop (preferred)

Two shapes, in increasing order of change:

1. **Deferred/event-driven open.** Attempt the provider `openat` non-blocking; on would-block, park
   `(tag, cap_fd, reference, flags)` and return without responding, then complete and
   `write_response` when the provider becomes ready. Requires the kernel corollary (Option C) so the
   open can actually report would-block, plus a redox-scheme deferred-response path.
2. **Worker offload.** Keep the dispatch loop responsive by handing the blocking `openat` to a small
   worker pool; respond from the worker via `socket.write_response(Response::…, tag)` (`Socket` is an
   fd wrapper → `Send`; `CallRequest`→`Tag` is `Send`). Cost: `bootstrap` is `#![no_std]`, so this
   needs a real thread primitive (`redox_rt::thread::rlct_clone_impl` — raw stack/TCB/TLS setup),
   which is the risky part.

This is the direct answer to the operator's SMP question: the parallelism that matters here is
**fault isolation of the namespace-open path**, not throughput. AML execution itself must stay
serialized (ACPI requires it; Linux does the same with a global interpreter mutex) — the thing that
must not serialize is the *transport*: serving the scheme socket must not stall behind one slow open.

### Option B — bound provider unreadiness at the source

Ensure boot-critical providers (notably vesad) are servicing their scheme socket before anything can
open them (e.g. tighten the inputd→driver handoff so the handoff signal follows, not precedes, the
provider entering its event loop). Narrower, but only closes known triggers; it does not remove the
amplifier for the *next* slow daemon.

### Option C — honor `O_NONBLOCK` on `open` in the kernel

Make `UserInner::call_inner` (open opcode) return `EAGAIN` instead of blocking when `O_NONBLOCK` is
set and the provider has not yet taken/answered the request (with the existing cancellation
protocol). This is the cleanest enabler for Option A.1 and immediately activates fbcond's existing
retry, but it changes scheme-open semantics system-wide — the widest blast radius, so it needs the
most care and the strongest validation.

## acpid hardening (2026-07-21): the daemon-local half of the same problem

`acpid` is the same shape as `initnsmgr` in miniature: single-threaded, serving a scheme (`acpi`)
on the very thread that evaluates AML. A slow or stuck AML method stops it answering scheme
requests, which is what let the mutex bug wedge the whole system. Six hardening items were
identified; five are **done** (safe, verifiable, reduce or bound the ways acpid can stall the
transport), one is **designed-but-deferred** (the same deferred-response rewrite as Option A above,
in a boot-critical daemon).

**Done (`local/sources/base/drivers/acpid`):**

- **AML mutex recursion + ownership + timeout units** (`aml_physmem.rs`, the original fix): a nested
  acquire by the owning thread bumps a depth instead of self-deadlocking; timeout read as
  milliseconds; any wait bounded to 5s. Mirrors ACPICA `exmutex.c:140`.
- **`stall()` bounded per ACPICA** (`aml_physmem.rs`): refuse `>255us` and warn `>100us` instead of
  an unbounded CPU busy-spin on the serving thread (ACPICA `exsystem.c:129-147`).
- **`release` owner-check** (`aml_physmem.rs`): a release of a not-currently-held mutex, or by a
  non-owning thread, is logged (ACPICA `AE_NOT_ACQUIRED` / owner-mismatch, `exmutex.c:287,376`)
  rather than silently corrupting depth.
- **Static processor-method cache** (`scheme.rs`): `_PSS`/`_PSD`/`_CST`/`_CPC` are fixed after boot
  but cpufreqd polls them; cache the first evaluation so later reads never re-run the AML
  interpreter under the global lock. This removes acpid's dominant *recurring* head-of-line source.
- **Panic-free scheme path** (`acpi.rs`): every `release_global_lock().expect(...)` on the
  AML-evaluation path became log-and-continue, and a `result.ok()?.unwrap()` that panicked on
  `Ok(None)` (absent method) became `?`. A scheme-daemon panic kills the `acpi` scheme and wedges
  every consumer, so the serving path must not panic on odd firmware.
- **Observability** (`aml_physmem.rs`, `acpi.rs`): log AML evaluations ≥50ms and mutex-acquire
  timeouts — the early-warning signal before a stall becomes a wedge.

**Deferred (#1 — decouple AML execution from the transport):** make acpid keep answering the `acpi`
scheme while a slow AML method runs, by rewriting its loop from `process_requests_nonblocking` onto
the low-level `next_request` → park the `CallRequest` `Tag` → run AML on a single serialized worker
→ `write_response(tag)` when done. AML must stay serialized (ACPI requires it); only the transport
is parallelized. This is exactly Option A's deferred-response shape, in a boot-critical daemon, and
cannot be cleanly validated under the current external load — so it is designed here, not done
blind. With the cache and bounds above, acpid's remaining exposure is a *single* slow/first AML
evaluation (bounded to seconds, not infinite), which is the tail this rewrite would close.

## Validation expectations

- Framebuffer screendump (QMP) is ground truth; the serial mirror is racy and often silent on a
  healthy boot. Harness pattern: QMP `send-key` login + periodic `screendump`.
- Because the failure is a race, **one boot proves nothing** — always compare rates across N≥5 boots
  on an idle host (external load, e.g. a background `opencode` at ~290% CPU, contaminates every
  measurement and inflates the failure rate).
- Success criterion for this plan: N≥10 consecutive clean boots to a working brush login on an idle
  host, and no `display.vesa:v2/*`-class wedge under induced provider delay.

## Related

- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — canonical desktop path; console/login is its floor.
- `local/docs/IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` — acpid/ACPI robustness context.
- Memory: `mini-login-acpid-mutex-rootcause`, `fbcond-handoff-console-wedge`.