RedBear-OS/local/docs/INITNSMGR-CONCURRENCY-DESIGN.md

# initnsmgr Concurrency Design — worker-offload of the blocking open

**Status:** Design (not implemented). Companion to
`INIT-NAMESPACE-MANAGER-SCALABILITY-PLAN.md`, which states *why* this is needed; this document
states *how*. Implementation must be validated on an idle host or real hardware — see "Validation".

## Problem restated (precisely)

`local/sources/base/bootstrap/src/initnsmgr.rs` runs the init namespace manager as a single thread:

```
run(): loop {
    req = socket.next_request()          // one request
    resp = req.handle_sync(&mut scheme)  // synchronous — runs the handler inline
    socket.write_response(resp)
}
```

Every path resolution in a restricted namespace (every login shell, much of init's own spawn path)
is proxied here. The only handler that can block is `openat` → `open_scheme_resource`, which does a
**blocking `syscall::openat(cap_fd, …)`** to the *provider* daemon. If that provider is briefly not
servicing its socket (descheduled under load, mid-`tick`, in a one-shot startup window), the
`openat` blocks, the loop stops, and **every** other request queues behind it. The wedge surfaces at
whatever boot stage was in flight (e.g. "ahcid" in the logs) — that stage is the *symptom location*,
not the cause. The cause is head-of-line blocking on a single serving thread.

All the other handlers are fast, in-memory operations that must **not** be parallelized (they mutate
shared namespace state): `dup` (ForkNs / ShrinkPermissions / IssueRegister), `unlinkat`, `on_close`,
`on_sendfd`, `getdents`, `fstat`, and the `namespace:`/`""` (list) branches of `openat`. Only the
`open_scheme_resource` branch — a blocking call to another daemon — needs to move off the loop.

## Design A — worker-offload (self-contained in bootstrap; recommended first)

Keep resolution serialized; move only the blocking `openat` to a small worker pool. The dispatcher
resolves the target `cap_fd` under a lock (fast), then hands the *blocking* call to a worker and
keeps accepting requests. The worker replies out-of-band when the open completes.

### Building blocks (all verified available in the no_std bootstrap)

- **`redox_rt::sync::Mutex<T>`** (`relibc/redox-rt/src/sync.rs`) — futex-based, `Send + Sync` for
  `T: Send`. Replaces the current `Rc<RefCell<Namespace>>` sharing.
- **`FdGuard` is `#[repr(transparent)]` over `usize`** (`redox-rt/src/proc.rs:792`) — trivially
  `Send`. The namespace already holds scheme caps as `Arc<FdGuard>`; a worker clones the `Arc`
  (not the fd), so `FdGuard`'s `Drop` (which closes the fd) fires only when the last reference is
  gone — no double-close.
- **`Socket`** (redox-scheme) is an fd wrapper; `write_response(&self, …)` takes `&self`, and
  `CallRequest → Tag` is `Send`. Share the socket as `Arc<Socket>` so a worker can reply.
- **`redox_rt::thread::rlct_clone_impl`** (`redox-rt/src/thread.rs`) — the raw thread primitive.
  **This is the risky part**: bootstrap is `#![no_std]` and does not link relibc's `pthread`, so a
  worker thread must be brought up by hand (stack `mmap`, TCB alloc/init, TLS, then `rlct_clone_impl`)
  — the minimal subset of what `relibc/src/pthread/mod.rs:203` + `platform/redox/mod.rs:1089` do.
  Budget this as the main implementation cost.

### State changes

```rust
// Before: single-threaded interior mutability
namespace: Rc<RefCell<Namespace>>

// After: shareable across dispatcher + workers
namespace: Arc<redox_rt::sync::Mutex<Namespace>>
```

`Namespace.schemes` is already `HashMap<String, Arc<FdGuard>>` — `Arc<FdGuard>` is `Send`, so the
map is `Send` once the outer cell is a `Mutex`. The `NamespaceScheme.handles`/`next_id` bookkeeping
stays on the dispatcher thread (never touched by workers).

### Work item + queue

```rust
struct OpenWork {
    tag: Tag,                 // reply target (Send)
    cap_fd: Arc<FdGuard>,     // provider scheme cap (Send; refcount keeps it alive)
    reference: String,        // path within the provider scheme
    flags: usize,
    fcntl_flags: u32,
}

// Shared, bounded queue + futex wakeup (no std::mpsc in no_std):
struct WorkQueue {
    inner: redox_rt::sync::Mutex<VecDeque<OpenWork>>,
    // futex word bumped on push; workers futex-wait on it when the queue is empty.
}
```

Bounded (e.g. 64). On overflow the dispatcher falls back to handling the open inline (degrades to
today's behavior for that one request rather than dropping it) — never unbounded growth.

### Dispatcher openat path

```rust
// openat, scheme != "namespace" and != "" (list):
let cap_fd = {
    let ns = ns_access.namespace.lock();      // fast: hashmap lookup
    ns.get_scheme_fd(scheme).cloned()          // Arc clone, released with the lock
};
let Some(cap_fd) = cap_fd else { return Err(ENODEV) };

queue.push(OpenWork { tag: req.tag(), cap_fd, reference, flags, fcntl_flags });
// DO NOT write a response here — the worker will. Return a "deferred" marker so
// run() skips write_response for this request.
```

This needs the low-level request API (`next_request` → keep the `CallRequest`/`Tag`, respond later)
rather than `handle_sync`, which always writes a response. redox-scheme already exposes `Tag` /
`Response::return_external_fd(fd, tag)` for exactly this.

### Worker loop

```rust
loop {
    let work = queue.pop_blocking();           // futex-wait when empty
    let res = syscall::openat(work.cap_fd.as_raw_fd(),
                              &work.reference, work.flags, work.fcntl_flags as usize);
    let resp = match res {
        Ok(fd) => Response::return_external_fd(fd, work.tag),
        Err(e) => Response::err(e.errno, work.tag),
    };
    let _ = socket.write_response(resp, SignalBehavior::Restart); // Arc<Socket>
}
```

Results arrive out of order — fine, the kernel matches by `tag`. If the client died meanwhile, its
`on_close` already ran on the dispatcher; the late `write_response` targets a dead tag and the
kernel discards it (must be verified to be a no-op, not an error).

### Concurrency invariants

- **Namespace mutation stays serialized** under the `Mutex`; workers never touch namespace state,
  only a cloned `cap_fd`. So there is no ordering hazard between a `fork`/`register` and an in-flight
  open — the open captured its `cap_fd` before being queued.
- **Provider serialization is unchanged**: each provider still serializes its own requests; we only
  stop *initnsmgr* from serializing *unrelated* providers behind one slow one.
- **Worker count**: 2–4. This is fault-isolation, not throughput — enough that a couple of slow
  providers cannot stall the rest.

## Design B — kernel `O_NONBLOCK` on open + single-thread deferred (no bootstrap threads)

Avoids the no_std thread bring-up entirely, at the cost of a kernel change with a wider blast radius.

1. **Kernel**: make `UserInner::call_inner` (`kernel/src/scheme/user.rs`) honor `O_NONBLOCK` on the
   open opcode — return `EAGAIN` instead of `.block()`ing when the provider has not taken/answered
   the request, using the existing cancellation path.
2. **initnsmgr** stays single-threaded and event-driven (the `RawEventQueue` pattern acpid already
   uses): try `openat` with `O_NONBLOCK`; on `EAGAIN`, park `(tag, cap_fd, reference, flags)` in a
   pending list, subscribe to the provider fd's readiness, and continue `next_request`. On readiness,
   retry and `write_response(tag)`.
3. **Bonus**: this immediately activates fbcond's existing handoff-retry (it already opens with
   `O_NONBLOCK` and retries; today the retry never fires because the first open blocks).

B is architecturally cleaner (no threads in the earliest-boot component) but changes scheme-open
semantics for *every* scheme in the system, so it needs the strongest system-wide validation.

## Recommendation

Start with **A**. It is contained in one component and does not change kernel or redox-scheme
semantics for the rest of the system, so a regression is bounded to initnsmgr and reversible by
reverting one file. Its cost is the manual thread bring-up in no_std; budget that explicitly.
Consider **B** later as the cleaner long-term shape, or if upstream Redox moves scheme-open toward
non-blocking semantics anyway.

## Staged implementation plan (A)

1. **Refactor sharing, no behavior change**: `Rc<RefCell<Namespace>>` → `Arc<sync::Mutex<Namespace>>`,
   still fully synchronous (lock/handle/unlock inline). Build + boot: must be identical to today.
   This isolates the mechanical Send refactor from the concurrency change.
2. **Thread bring-up spike**: bring up ONE worker thread in bootstrap that does nothing but log a
   heartbeat via the debug fd, and prove it survives boot. This de-risks the hardest part alone.
3. **Introduce the queue + single worker**, route only `open_scheme_resource` through it, deferred
   response. Keep the inline fallback on queue-full. Boot + measure.
4. **Scale to N=2–4 workers**, add the bounded-queue overflow fallback and dead-tag handling.
5. **Load test**: N≥10 boots on an idle host, plus an induced-slow-provider test (a provider that
   sleeps before servicing its socket) to confirm one slow provider no longer wedges the rest.

Each step is independently bootable and revertable.

## Validation

- Framebuffer screendump (QMP) is ground truth; the serial mirror is racy.
- **One boot proves nothing** — the failure is a race; compare rates across N≥10 boots **on an idle
  host**. External load (e.g. a background `opencode` at 200–290% CPU) contaminates every measurement
  and must be absent for a verdict.
- Success: N≥10 consecutive clean boots to a working brush login, and no head-of-line wedge under an
  induced slow-provider.

## Risk / rollback

- Blast radius of A is initnsmgr only; revert = restore one file + the `submodule/base` gitlink.
- The thread bring-up is the sole high-risk element; step 2 isolates it before any concurrency logic.
- Do NOT ship any step that was only validated under external load.