# initnsmgr Concurrency Design — worker-offload of the blocking open **Status:** Design (not implemented). Companion to `INIT-NAMESPACE-MANAGER-SCALABILITY-PLAN.md`, which states *why* this is needed; this document states *how*. Implementation must be validated on an idle host or real hardware — see "Validation". ## Problem restated (precisely) `local/sources/base/bootstrap/src/initnsmgr.rs` runs the init namespace manager as a single thread: ``` run(): loop { req = socket.next_request() // one request resp = req.handle_sync(&mut scheme) // synchronous — runs the handler inline socket.write_response(resp) } ``` Every path resolution in a restricted namespace (every login shell, much of init's own spawn path) is proxied here. The only handler that can block is `openat` → `open_scheme_resource`, which does a **blocking `syscall::openat(cap_fd, …)`** to the *provider* daemon. If that provider is briefly not servicing its socket (descheduled under load, mid-`tick`, in a one-shot startup window), the `openat` blocks, the loop stops, and **every** other request queues behind it. The wedge surfaces at whatever boot stage was in flight (e.g. "ahcid" in the logs) — that stage is the *symptom location*, not the cause. The cause is head-of-line blocking on a single serving thread. All the other handlers are fast, in-memory operations that must **not** be parallelized (they mutate shared namespace state): `dup` (ForkNs / ShrinkPermissions / IssueRegister), `unlinkat`, `on_close`, `on_sendfd`, `getdents`, `fstat`, and the `namespace:`/`""` (list) branches of `openat`. Only the `open_scheme_resource` branch — a blocking call to another daemon — needs to move off the loop. ## Design A — worker-offload (self-contained in bootstrap; recommended first) Keep resolution serialized; move only the blocking `openat` to a small worker pool. The dispatcher resolves the target `cap_fd` under a lock (fast), then hands the *blocking* call to a worker and keeps accepting requests. The worker replies out-of-band when the open completes. ### Building blocks (all verified available in the no_std bootstrap) - **`redox_rt::sync::Mutex`** (`relibc/redox-rt/src/sync.rs`) — futex-based, `Send + Sync` for `T: Send`. Replaces the current `Rc>` sharing. - **`FdGuard` is `#[repr(transparent)]` over `usize`** (`redox-rt/src/proc.rs:792`) — trivially `Send`. The namespace already holds scheme caps as `Arc`; a worker clones the `Arc` (not the fd), so `FdGuard`'s `Drop` (which closes the fd) fires only when the last reference is gone — no double-close. - **`Socket`** (redox-scheme) is an fd wrapper; `write_response(&self, …)` takes `&self`, and `CallRequest → Tag` is `Send`. Share the socket as `Arc` so a worker can reply. - **`redox_rt::thread::rlct_clone_impl`** (`redox-rt/src/thread.rs`) — the raw thread primitive. **This is the hard blocker, and it is worse than first estimated.** `rlct_clone_impl(stack, tcb: &RtTcb)` requires a *fully-constructed TCB for the new thread*. relibc's `pthread_create` builds that TCB with `Tcb::new(tls_len)` (`relibc/src/pthread/mod.rs:167`), pushes the entry/arg/tcb/shim onto a freshly `mmap`ed stack (`:177-201`), calls `rlct_clone`, and the `new_thread_shim` (`:218`) activates the TCB (TLS) + installs the signal handler before jumping to the entry point. **bootstrap has none of this**: it runs `redox_rt::initialize_freestanding(this_thr_fd)` (`exec.rs:71`), which sets up exactly ONE TCB (`RtTcb::current()`); there is no `Tcb::new`, no TLS allocator, no thread shim in the freestanding path. So Design A needs, as a prerequisite, either: 1. **Port a minimal thread-spawn helper into `redox_rt`'s freestanding path** — allocate a stack, build a new `RtTcb`, wire the TLS masters pointers, and provide a shim — the low-level, arch-specific core of `pthread_create`, but without relibc `std`. This is the real cost, and it is deep `unsafe` in the earliest-boot component. 2. **Or use worker PROCESSES, not threads.** bootstrap already forks its three services via `spawn` (`exec.rs:290`). But processes do not share `Arc>`, so the dispatcher would have to pass each resolved `cap_fd` + reply `Tag` to a worker process over a pipe/socket and the worker replies on the shared scheme socket — more moving parts than threads. **Consequence:** the "just call rlct_clone" framing was too optimistic. Given this, **Design B (below) is now the more attractive first step** — it needs no bootstrap threads at all. Design A remains the cleaner end-state *if* a freestanding thread-spawn helper is added to `redox_rt` first (that helper is independently useful and should be its own task). ### State changes ```rust // Before: single-threaded interior mutability namespace: Rc> // After: shareable across dispatcher + workers namespace: Arc> ``` `Namespace.schemes` is already `HashMap>` — `Arc` is `Send`, so the map is `Send` once the outer cell is a `Mutex`. The `NamespaceScheme.handles`/`next_id` bookkeeping stays on the dispatcher thread (never touched by workers). ### Work item + queue ```rust struct OpenWork { tag: Tag, // reply target (Send) cap_fd: Arc, // provider scheme cap (Send; refcount keeps it alive) reference: String, // path within the provider scheme flags: usize, fcntl_flags: u32, } // Shared, bounded queue + futex wakeup (no std::mpsc in no_std): struct WorkQueue { inner: redox_rt::sync::Mutex>, // futex word bumped on push; workers futex-wait on it when the queue is empty. } ``` Bounded (e.g. 64). On overflow the dispatcher falls back to handling the open inline (degrades to today's behavior for that one request rather than dropping it) — never unbounded growth. ### Dispatcher openat path ```rust // openat, scheme != "namespace" and != "" (list): let cap_fd = { let ns = ns_access.namespace.lock(); // fast: hashmap lookup ns.get_scheme_fd(scheme).cloned() // Arc clone, released with the lock }; let Some(cap_fd) = cap_fd else { return Err(ENODEV) }; queue.push(OpenWork { tag: req.tag(), cap_fd, reference, flags, fcntl_flags }); // DO NOT write a response here — the worker will. Return a "deferred" marker so // run() skips write_response for this request. ``` This needs the low-level request API (`next_request` → keep the `CallRequest`/`Tag`, respond later) rather than `handle_sync`, which always writes a response. redox-scheme already exposes `Tag` / `Response::return_external_fd(fd, tag)` for exactly this. ### Worker loop ```rust loop { let work = queue.pop_blocking(); // futex-wait when empty let res = syscall::openat(work.cap_fd.as_raw_fd(), &work.reference, work.flags, work.fcntl_flags as usize); let resp = match res { Ok(fd) => Response::return_external_fd(fd, work.tag), Err(e) => Response::err(e.errno, work.tag), }; let _ = socket.write_response(resp, SignalBehavior::Restart); // Arc } ``` Results arrive out of order — fine, the kernel matches by `tag`. If the client died meanwhile, its `on_close` already ran on the dispatcher; the late `write_response` targets a dead tag and the kernel discards it (must be verified to be a no-op, not an error). ### Concurrency invariants - **Namespace mutation stays serialized** under the `Mutex`; workers never touch namespace state, only a cloned `cap_fd`. So there is no ordering hazard between a `fork`/`register` and an in-flight open — the open captured its `cap_fd` before being queued. - **Provider serialization is unchanged**: each provider still serializes its own requests; we only stop *initnsmgr* from serializing *unrelated* providers behind one slow one. - **Worker count**: 2–4. This is fault-isolation, not throughput — enough that a couple of slow providers cannot stall the rest. ## Design B — kernel `O_NONBLOCK` on open + single-thread deferred (no bootstrap threads) Avoids the no_std thread bring-up entirely, at the cost of a kernel change with a wider blast radius. 1. **Kernel**: make `UserInner::call_inner` (`kernel/src/scheme/user.rs`) honor `O_NONBLOCK` on the open opcode — return `EAGAIN` instead of `.block()`ing when the provider has not taken/answered the request, using the existing cancellation path. 2. **initnsmgr** stays single-threaded and event-driven (the `RawEventQueue` pattern acpid already uses): try `openat` with `O_NONBLOCK`; on `EAGAIN`, park `(tag, cap_fd, reference, flags)` in a pending list, subscribe to the provider fd's readiness, and continue `next_request`. On readiness, retry and `write_response(tag)`. 3. **Bonus**: this immediately activates fbcond's existing handoff-retry (it already opens with `O_NONBLOCK` and retries; today the retry never fires because the first open blocks). B is architecturally cleaner (no threads in the earliest-boot component) but changes scheme-open semantics for *every* scheme in the system, so it needs the strongest system-wide validation. ## Recommendation (revised 2026-07-22 after the thread-spawn investigation) The original recommendation was "start with A". After confirming that bootstrap's freestanding `redox_rt` has **no thread-spawn machinery** (no `Tcb::new`, no TLS allocator, no thread shim — see the `rlct_clone_impl` note above), the ordering changes: - **Step 1 (Send refactor) is done and stands regardless of A vs B** — `Arc>` is a strict improvement and a prerequisite for A; it is inert (single-threaded) until A lands. Keep it (currently `local/patches/wip-initnsmgr/step1-send-refactor.patch`), boot-validate on an idle host, and commit. - **Prefer Design B (kernel `O_NONBLOCK` on open + single-thread deferred) as the first functional step.** It needs no bootstrap threads, reuses the `RawEventQueue` pattern acpid already runs, and immediately activates fbcond's existing retry. Its cost is a kernel scheme-open change with a system-wide blast radius — so it needs strong validation — but it avoids the deepest `unsafe` in the earliest-boot component. - **Design A remains the cleaner end-state**, but only after a **freestanding thread-spawn helper** is added to `redox_rt` as its own, independently-useful task. Do not attempt A's worker bring-up by open-coding TCB/TLS setup inside initnsmgr. Net: **Step 1 → (idle validation + commit) → Design B for the functional fix → Design A later** once `redox_rt` grows a freestanding thread helper. All of this still requires an idle host or real hardware; none of it can be validated under the current external load. ## Staged implementation plan (A) 1. **Refactor sharing, no behavior change**: `Rc>` → `Arc>`, still fully synchronous (lock/handle/unlock inline). Build + boot: must be identical to today. This isolates the mechanical Send refactor from the concurrency change. 2. **Thread bring-up spike**: bring up ONE worker thread in bootstrap that does nothing but log a heartbeat via the debug fd, and prove it survives boot. This de-risks the hardest part alone. 3. **Introduce the queue + single worker**, route only `open_scheme_resource` through it, deferred response. Keep the inline fallback on queue-full. Boot + measure. 4. **Scale to N=2–4 workers**, add the bounded-queue overflow fallback and dead-tag handling. 5. **Load test**: N≥10 boots on an idle host, plus an induced-slow-provider test (a provider that sleeps before servicing its socket) to confirm one slow provider no longer wedges the rest. Each step is independently bootable and revertable. ## Validation - Framebuffer screendump (QMP) is ground truth; the serial mirror is racy. - **One boot proves nothing** — the failure is a race; compare rates across N≥10 boots **on an idle host**. External load (e.g. a background `opencode` at 200–290% CPU) contaminates every measurement and must be absent for a verdict. - Success: N≥10 consecutive clean boots to a working brush login, and no head-of-line wedge under an induced slow-provider. ## Risk / rollback - Blast radius of A is initnsmgr only; revert = restore one file + the `submodule/base` gitlink. - The thread bring-up is the sole high-risk element; step 2 isolates it before any concurrency logic. - Do NOT ship any step that was only validated under external load.