RedBear-OS/local/docs/INITNSMGR-CONCURRENCY-DESIGN.md

initnsmgr Concurrency Design — worker-offload of the blocking open

Status: Design (not implemented). Companion to INIT-NAMESPACE-MANAGER-SCALABILITY-PLAN.md, which states why this is needed; this document states how. Implementation must be validated on an idle host or real hardware — see "Validation".

Problem restated (precisely)

local/sources/base/bootstrap/src/initnsmgr.rs runs the init namespace manager as a single thread:

run(): loop {
    req = socket.next_request()          // one request
    resp = req.handle_sync(&mut scheme)  // synchronous — runs the handler inline
    socket.write_response(resp)
}

Every path resolution in a restricted namespace (every login shell, much of init's own spawn path) is proxied here. The only handler that can block is openat → open_scheme_resource, which does a blocking syscall::openat(cap_fd, …) to the provider daemon. If that provider is briefly not servicing its socket (descheduled under load, mid-tick, in a one-shot startup window), the openat blocks, the loop stops, and every other request queues behind it. The wedge surfaces at whatever boot stage was in flight (e.g. "ahcid" in the logs) — that stage is the symptom location, not the cause. The cause is head-of-line blocking on a single serving thread.

All the other handlers are fast, in-memory operations that must not be parallelized (they mutate shared namespace state): dup (ForkNs / ShrinkPermissions / IssueRegister), unlinkat, on_close, on_sendfd, getdents, fstat, and the namespace:/"" (list) branches of openat. Only the open_scheme_resource branch — a blocking call to another daemon — needs to move off the loop.

Design A — worker-offload (self-contained in bootstrap; recommended first)

Keep resolution serialized; move only the blocking openat to a small worker pool. The dispatcher resolves the target cap_fd under a lock (fast), then hands the blocking call to a worker and keeps accepting requests. The worker replies out-of-band when the open completes.

Building blocks (all verified available in the no_std bootstrap)

• redox_rt::sync::Mutex<T> (relibc/redox-rt/src/sync.rs) — futex-based, Send + Sync for T: Send. Replaces the current Rc<RefCell<Namespace>> sharing.

• FdGuard is #[repr(transparent)] over usize (redox-rt/src/proc.rs:792) — trivially Send. The namespace already holds scheme caps as Arc<FdGuard>; a worker clones the Arc (not the fd), so FdGuard's Drop (which closes the fd) fires only when the last reference is gone — no double-close.

• Socket (redox-scheme) is an fd wrapper; write_response(&self, …) takes &self, and CallRequest → Tag is Send. Share the socket as Arc<Socket> so a worker can reply.

• redox_rt::thread::rlct_clone_impl (redox-rt/src/thread.rs) — the raw thread primitive. This is the hard blocker, and it is worse than first estimated. rlct_clone_impl(stack, tcb: &RtTcb) requires a fully-constructed TCB for the new thread. relibc's pthread_create builds that TCB with Tcb::new(tls_len) (relibc/src/pthread/mod.rs:167), pushes the entry/arg/tcb/shim onto a freshly mmaped stack (:177-201), calls rlct_clone, and the new_thread_shim (:218) activates the TCB (TLS) + installs the signal handler before jumping to the entry point. bootstrap has none of this: it runs redox_rt::initialize_freestanding(this_thr_fd) (exec.rs:71), which sets up exactly ONE TCB (RtTcb::current()); there is no Tcb::new, no TLS allocator, no thread shim in the freestanding path. So Design A needs, as a prerequisite, either:

  1. Port a minimal thread-spawn helper into redox_rt's freestanding path — allocate a stack, build a new RtTcb, wire the TLS masters pointers, and provide a shim — the low-level, arch-specific core of pthread_create, but without relibc std. This is the real cost, and it is deep unsafe in the earliest-boot component.
  2. Or use worker PROCESSES, not threads. bootstrap already forks its three services via spawn (exec.rs:290). But processes do not share Arc<Mutex<Namespace>>, so the dispatcher would have to pass each resolved cap_fd + reply Tag to a worker process over a pipe/socket and the worker replies on the shared scheme socket — more moving parts than threads.

  Consequence: the "just call rlct_clone" framing was too optimistic. Given this, Design B (below) is now the more attractive first step — it needs no bootstrap threads at all. Design A remains the cleaner end-state if a freestanding thread-spawn helper is added to redox_rt first (that helper is independently useful and should be its own task).

State changes

// Before: single-threaded interior mutability
namespace: Rc<RefCell<Namespace>>

// After: shareable across dispatcher + workers
namespace: Arc<redox_rt::sync::Mutex<Namespace>>

Namespace.schemes is already HashMap<String, Arc<FdGuard>> — Arc<FdGuard> is Send, so the map is Send once the outer cell is a Mutex. The NamespaceScheme.handles/next_id bookkeeping stays on the dispatcher thread (never touched by workers).

Work item + queue

struct OpenWork {
    tag: Tag,                 // reply target (Send)
    cap_fd: Arc<FdGuard>,     // provider scheme cap (Send; refcount keeps it alive)
    reference: String,        // path within the provider scheme
    flags: usize,
    fcntl_flags: u32,
}

// Shared, bounded queue + futex wakeup (no std::mpsc in no_std):
struct WorkQueue {
    inner: redox_rt::sync::Mutex<VecDeque<OpenWork>>,
    // futex word bumped on push; workers futex-wait on it when the queue is empty.
}

Bounded (e.g. 64). On overflow the dispatcher falls back to handling the open inline (degrades to today's behavior for that one request rather than dropping it) — never unbounded growth.

Dispatcher openat path

// openat, scheme != "namespace" and != "" (list):
let cap_fd = {
    let ns = ns_access.namespace.lock();      // fast: hashmap lookup
    ns.get_scheme_fd(scheme).cloned()          // Arc clone, released with the lock
};
let Some(cap_fd) = cap_fd else { return Err(ENODEV) };

queue.push(OpenWork { tag: req.tag(), cap_fd, reference, flags, fcntl_flags });
// DO NOT write a response here — the worker will. Return a "deferred" marker so
// run() skips write_response for this request.

This needs the low-level request API (next_request → keep the CallRequest/Tag, respond later) rather than handle_sync, which always writes a response. redox-scheme already exposes Tag / Response::return_external_fd(fd, tag) for exactly this.

Worker loop

loop {
    let work = queue.pop_blocking();           // futex-wait when empty
    let res = syscall::openat(work.cap_fd.as_raw_fd(),
                              &work.reference, work.flags, work.fcntl_flags as usize);
    let resp = match res {
        Ok(fd) => Response::return_external_fd(fd, work.tag),
        Err(e) => Response::err(e.errno, work.tag),
    };
    let _ = socket.write_response(resp, SignalBehavior::Restart); // Arc<Socket>
}

Results arrive out of order — fine, the kernel matches by tag. If the client died meanwhile, its on_close already ran on the dispatcher; the late write_response targets a dead tag and the kernel discards it (must be verified to be a no-op, not an error).

Concurrency invariants

• Namespace mutation stays serialized under the Mutex; workers never touch namespace state, only a cloned cap_fd. So there is no ordering hazard between a fork/register and an in-flight open — the open captured its cap_fd before being queued.
• Provider serialization is unchanged: each provider still serializes its own requests; we only stop initnsmgr from serializing unrelated providers behind one slow one.
• Worker count: 2–4. This is fault-isolation, not throughput — enough that a couple of slow providers cannot stall the rest.

Design B — kernel O_NONBLOCK on open + single-thread deferred (no bootstrap threads)

Avoids the no_std thread bring-up entirely, at the cost of a kernel change with a wider blast radius.

1. Kernel: make UserInner::call_inner (kernel/src/scheme/user.rs) honor O_NONBLOCK on the open opcode — return EAGAIN instead of .block()ing when the provider has not taken/answered the request, using the existing cancellation path.
2. initnsmgr stays single-threaded and event-driven (the RawEventQueue pattern acpid already uses): try openat with O_NONBLOCK; on EAGAIN, park (tag, cap_fd, reference, flags) in a pending list, subscribe to the provider fd's readiness, and continue next_request. On readiness, retry and write_response(tag).
3. Bonus: this immediately activates fbcond's existing handoff-retry (it already opens with O_NONBLOCK and retries; today the retry never fires because the first open blocks).

B is architecturally cleaner (no threads in the earliest-boot component) but changes scheme-open semantics for every scheme in the system, so it needs the strongest system-wide validation.

Recommendation (revised 2026-07-22 after the thread-spawn investigation)

The original recommendation was "start with A". After confirming that bootstrap's freestanding redox_rt has no thread-spawn machinery (no Tcb::new, no TLS allocator, no thread shim — see the rlct_clone_impl note above), the ordering changes:

• Step 1 (Send refactor) is done and stands regardless of A vs B — Arc<Mutex<Namespace>> is a strict improvement and a prerequisite for A; it is inert (single-threaded) until A lands. Keep it (currently local/patches/wip-initnsmgr/step1-send-refactor.patch), boot-validate on an idle host, and commit.
• Prefer Design B (kernel O_NONBLOCK on open + single-thread deferred) as the first functional step. It needs no bootstrap threads, reuses the RawEventQueue pattern acpid already runs, and immediately activates fbcond's existing retry. Its cost is a kernel scheme-open change with a system-wide blast radius — so it needs strong validation — but it avoids the deepest unsafe in the earliest-boot component.
• Design A remains the cleaner end-state, but only after a freestanding thread-spawn helper is added to redox_rt as its own, independently-useful task. Do not attempt A's worker bring-up by open-coding TCB/TLS setup inside initnsmgr.

Net: Step 1 → (idle validation + commit) → Design B for the functional fix → Design A later once redox_rt grows a freestanding thread helper. All of this still requires an idle host or real hardware; none of it can be validated under the current external load.

Staged implementation plan (A)

1. Refactor sharing, no behavior change: Rc<RefCell<Namespace>> → Arc<sync::Mutex<Namespace>>, still fully synchronous (lock/handle/unlock inline). Build + boot: must be identical to today. This isolates the mechanical Send refactor from the concurrency change.
2. Thread bring-up spike: bring up ONE worker thread in bootstrap that does nothing but log a heartbeat via the debug fd, and prove it survives boot. This de-risks the hardest part alone.
3. Introduce the queue + single worker, route only open_scheme_resource through it, deferred response. Keep the inline fallback on queue-full. Boot + measure.
4. Scale to N=2–4 workers, add the bounded-queue overflow fallback and dead-tag handling.
5. Load test: N≥10 boots on an idle host, plus an induced-slow-provider test (a provider that sleeps before servicing its socket) to confirm one slow provider no longer wedges the rest.

Each step is independently bootable and revertable.

Validation

• Framebuffer screendump (QMP) is ground truth; the serial mirror is racy.
• One boot proves nothing — the failure is a race; compare rates across N≥10 boots on an idle host. External load (e.g. a background opencode at 200–290% CPU) contaminates every measurement and must be absent for a verdict.
• Success: N≥10 consecutive clean boots to a working brush login, and no head-of-line wedge under an induced slow-provider.

Risk / rollback

• Blast radius of A is initnsmgr only; revert = restore one file + the submodule/base gitlink.
• The thread bring-up is the sole high-risk element; step 2 isolates it before any concurrency logic.
• Do NOT ship any step that was only validated under external load.