RedBear-OS/local/docs/MULTITHREADING-COMPREHENSIVE-ASSESSMENT-AND-PLAN.md

# Red Bear OS — Multi-Threading Comprehensive Assessment and Implementation Plan

**Date:** 2026-07-02 (initial assessment); 2026-07-02 (Phase 0c patch recovery complete)
**Scope:** Full-stack multi-threading audit: hardware/SMP, kernel scheduler, kernel futex, kernel syscall ABI, relibc pthreads, userspace threading correctness and performance
**Status:** Authoritative — supersedes `archived/KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` and `archived/SCHEDULER-REVIEW-FINAL.md` for all threading matters
**Validation levels:** `builds` → `enumerates` → `usable` → `validated` → `hardware-validated`

---

## UPDATE — Phase 0c Patch Recovery (2026-07-02)

The assessment in §1 below was accurate as of the time of writing. The Phase 0c
patch recovery was then executed in the same session, landing the following
commits on the local kernel fork (`local/sources/kernel/`):

| Commit | Effect |
|--------|--------|
| `ed3f0e1` | **P6-futex-sharding**: replaces single global `Mutex<L1, FutexList>` with 64-shard hash table |
| `5fb42fc` | **RUN_QUEUE_COUNT pre-flight**: defines `pub const RUN_QUEUE_COUNT: usize = 40;` (was missing from patch chain) |
| `cbf051e` | **P7-cache-affine-context (manual)**: surgically inserts `SchedPolicy` enum, `SCHED_PRIORITY_LEVELS`, helper functions, and 9 new Context fields (last_cpu, sched_policy, sched_rt_priority, sched_rr_ticks_consumed, sched_static_prio, sched_rr_quantum, vruntime, futex_pi_*, PhysicalAddress import) |
| `f7652fc` | **P5-context-mod-sched + P8-percpu-sched + P8-percpu-wiring**: PerCpuSched struct with SyncUnsafeCell-wrapped per-CPU run queues, get_percpu_block helper, full per-CPU scheduler wiring in switch.rs (pick_next_from_queues, pick_next_from_global_queues, select_next_context) |
| `7fc8bbf` | **P8-initial-placement + P9-numa-topology + P9-proc-lock-ordering**: least-loaded-CPU spawn, NUMA topology hints, proc scheme lock order fix |
| `327c150` | **`set_sched_policy` + `set_sched_other_prio`**: missing Context methods called by proc scheme handles |
| `e8ec916` | **fadt usize/u32 type mismatch fix**: changes FADT_MIN_SIZE constants to `u32` to match `Sdt::length()` |
| `4789d54` | **SchedPolicy/Name/Priority proc scheme handles**: adds `/proc/<tid>/{name, sched-policy, priority}` paths and read/write handlers |

**Upstream check (bg_27f3578a, 2026-07-02):** verified that `gitlab.redox-os.org/redox-os/kernel`
master (commit `aa7e7d2f44ba7cd9d1b007d37db139b345d46b8a`) has **NONE** of these features. The
local fork is the sole implementation. No upstream cherry-picks are available.

**Plan-vs-actual state:**

| Plan claim (Section 1) | Actual state after Phase 0c |
|---|---|
| "Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine)" | ✅ All 5 features now present. Per-CPU `PerCpuSched` with `run_queues`, `steal_work()`, `migrate_one_context()`, `maybe_balance_queues()`, `vruntime` CFS-style weighting, `last_cpu` cache-affine vruntime bonus, `SchedPolicy::Fifo`/`RoundRobin` RT scanning in `pick_next_from_queues` |
| "Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET)" | 🟡 **Sharding done** (64-shard hash). REQUEUE/PI/robust/WAKE_OP/BITSET still missing. |
| "relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent" | 🟡 relibc fork only has the `pthread_cond_signal` POSIX fix so far. `sched_*`, robust, PI still pending in relibc. |
| "❌ Missing from relibc: CPU affinity API, Thread naming" | 🟡 **Kernel side done** — `/proc/<tid>/{sched-policy, name, priority}` handles. relibc pthread_setname_np / pthread_setaffinity_np / sched_setscheduler still pending. |
| "cargo check has 1 pre-existing error" | ✅ **Fixed** — `cargo check` now exits 0 with 0 errors. |

**Phase 0c status: kernel side complete (all 8 of 8 applicable kernel P5–P9 patches
re-applied or made obsolete by the existing refactored scheduler). Remaining work
is in the relibc fork (Phase 0e) and the futex-REQUEUE/PI/robust work (Phase 1).**

The detailed analysis in §1–§9 below is preserved as historical record. The status
column "🚧 Missing" in §1 should be re-read as "now present in the local kernel
fork, pending relibc userspace wiring."

---

---

## 1. Executive Summary

### The Critical Finding — Lost Threading Work

The P5–P9 scheduler and futex enhancement work (documented as "complete" in the archived
plans) was **lost during the local fork migration** (2026-06). The local forks at
`local/sources/kernel/` and `local/sources/relibc/` were created from **upstream Redox
baselines** that did NOT include the Red Bear enhancement patches. The patches exist in
`local/patches/kernel/` and `local/patches/relibc/` but are **not wired into the recipes**
(both `recipe.toml` files use `path = "..."` with no `patches = [...]` list).

**Impact:** The running kernel has:
- Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine)
- Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET)
- relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent

**Recovery:** 13 of 18 kernel P5–P9 patches apply cleanly to the current fork. 5 fail due to
patch-chain dependencies (they expect earlier patches applied first). The bulk of the work is
recoverable by re-applying patches to the forks and committing them.

### What Actually Works Today

| Layer | Status | Detail |
|-------|--------|--------|
| **SMP boot** | ✅ Solid | INIT→SIPI sequence correct, per-CPU PCR via GS_BASE, x2APIC support |
| **Context switching** | ✅ Solid | FPU/SIMD/AVX state save via XSAVE, FSBASE/GSBASE swap (FSGSBASE or MSR), correct callee-saved register save |
| **TLB shootdown protocol** | ✅ Correct | AtomicBool flag + IPI + ack counter with `fence(SeqCst)` race prevention |
| **Basic thread lifecycle** | ✅ Functional | pthread_create/join/detach/exit through proc scheme + redox_rt clone |
| **Basic synchronization** | ✅ Functional | Futex-backed mutex, condvar, rwlock, barrier, spinlock, once |
| **TLS** | ✅ Functional | ELF PT_TLS + pthread_key_create/getspecific/setspecific |
| **Per-CPU data** | ✅ Functional | PercpuBlock via GS_BASE, all per-CPU state accessible |
| **Signal delivery** | ✅ Functional | Shared-memory Sigcontrol pages, per-thread masks, trampoline |
| **Scheduler algorithm** | 🚧 Basic DWRR | 40 priority levels, geometric weights, cooperative preemption (3-tick quantum) |
| **Futex operations** | 🚧 Basic only | WAIT/WAIT64/WAKE with single global mutex |
| **SMP load balancing** | ❌ Missing | No work stealing, no migration, contexts stuck on birth CPU |
| **RT scheduling** | ❌ Missing | No SCHED_FIFO/SCHED_RR, no kernel policy dispatch |
| **Futex REQUEUE** | ❌ Missing | Condvar broadcast causes thundering herd |
| **Robust mutexes** | ❌ Missing | Thread death while holding mutex → permanent deadlock |
| **PI futexes** | ❌ Missing | No priority inheritance → priority inversion risk |
| **CPU affinity API** | ❌ Missing from relibc | Kernel supports sched_affinity field but no userspace API |
| **Thread naming** | ❌ Missing from relibc | Kernel supports name field but no userspace API |
| **Per-page TLB flush** | ❌ Missing | `invalidate_all()` = full CR3 reload on every shootdown |
| **NUMA awareness** | ❌ Missing | No SRAT/SLIT, no proximity domains, flat memory model |
| **IRQ balancing** | ❌ Missing | All legacy IRQs hardwired to BSP |

---

## 2. Layer-by-Layer Assessment

### 2.1 Hardware / SMP Layer

**Files:** `src/acpi/madt/arch/x86.rs`, `src/arch/x86_shared/start.rs`,
`src/arch/x86_shared/device/local_apic.rs`, `src/arch/x86_shared/device/ioapic.rs`,
`src/arch/x86_shared/ipi.rs`, `src/arch/x86_shared/interrupt/ipi.rs`, `src/percpu.rs`,
`src/arch/x86_shared/gdt.rs`

**Verdict: Functional foundation, performance gaps.**

| Component | Status | Detail |
|-----------|--------|--------|
| AP boot (INIT/SIPI) | ✅ validated | Correct trampoline at 0x8000, per-AP PCR/IDT/stack allocation |
| x2APIC mode | ✅ builds | Detected via CPUID, MSR-based access, APIC ID detection |
| Per-CPU PCR via GS_BASE | ✅ validated | `PercpuBlock::current()` reads from PCR, SWAPGS protocol correct |
| IPI send/receive | ✅ functional | 5 IPI kinds (Wakeup/Tlb/Switch/Pit/Profile), broadcast + unicast |
| TLB shootdown | ✅ correct | AtomicBool + IPI + ack with `fence(SeqCst)` race prevention |
| TLB granularity | ❌ coarse | Full CR3 reload (`mov cr3, cr3`) on every shootdown — no INVLPG |
| TLB broadcast | 🚧 sequential | Iterates CPUs individually, doesn't use ICR "all excluding self" shorthand |
| IRQ routing | ❌ BSP-only | Legacy I/O APIC entries hardcode `dest: bsp_apic_id` |
| NUMA | ❌ absent | No SRAT/SLIT, no proximity domains |
| SMT/HT topology | ❌ absent | No cache hierarchy, no hyperthread awareness |
| Idle loop | ✅ functional | MWAIT with deepest C-state or HLT fallback |
| W^X for trampoline | 🚧 minor | Trampoline page briefly W+X, unmapped after AP boot |

### 2.2 Kernel Scheduler Layer

**Files:** `src/context/switch.rs`, `src/context/mod.rs`, `src/context/context.rs`,
`src/context/timeout.rs`

**Verdict: Correct but primitive — DWRR only, no SMP balancing, no RT classes.**

**Algorithm:** Deficit Weighted Round Robin (DWRR)
- 40 priority levels, each a `VecDeque<WeakContextRef>`
- Geometric weights: `SCHED_PRIO_TO_WEIGHT[i] ≈ 1.25^i` (88761 → 15)
- Per-CPU `balance` accumulator drives dequeue decisions
- Quantum: 3 PIT ticks (~12.2ms) per scheduling round
- Cooperative preemption: `preempt_locks > 0` disables preemption

**Global locks:**
- `RUN_CONTEXTS: Mutex<L1, RunContextData>` — all 40 priority queues under one L1 lock
- `IDLE_CONTEXTS: Mutex<L2, VecDeque<WeakContextRef>>` — sleeping contexts
- `CONTEXT_SWITCH_LOCK: AtomicBool` — global CAS spinlock serializing all context switches

**What's missing (all was in lost P5–P9 work):**

| Gap | Lost Patch | Recoverable? |
|-----|-----------|-------------|
| Per-CPU run queues (eliminate global L1) | P6-percpu-runqueues, P8-percpu-sched, P8-percpu-wiring | ✅ applies cleanly |
| Work stealing | P8-work-stealing | ❌ needs rebase (depends on per-CPU wiring) |
| Initial placement (least-loaded CPU) | P8-initial-placement | ✅ applies cleanly |
| Load balancing | P8-load-balance (absorbed) | needs verification |
| Vruntime tracking + min-vruntime selection | P6-vruntime-switch | ✅ applies cleanly |
| SchedPolicy enum (FIFO/RR/Other) | P5-sched-rt-policy | ✅ applies cleanly |
| RT scheduling dispatch | P5-sched-rt-policy | ✅ applies cleanly |
| Cache-affine scheduling | P7-cache-affine-switch | ✅ applies cleanly |
| NUMA topology hints | P9-numa-topology | ✅ applies cleanly |

### 2.3 Kernel Futex Layer

**File:** `src/syscall/futex.rs`

**Verdict: Baseline only — critical operations missing for desktop workloads.**

| Operation | Status | Impact of Absence |
|-----------|--------|-------------------|
| `FUTEX_WAIT` (32-bit) | ✅ | — |
| `FUTEX_WAIT64` (64-bit) | ✅ | — |
| `FUTEX_WAKE` | ✅ | — |
| `FUTEX_REQUEUE` | ❌ returns EINVAL | `pthread_cond_broadcast` wakes ALL waiters (thundering herd) |
| `FUTEX_CMP_REQUEUE` | ❌ not defined | Same + atomicity gap |
| `FUTEX_WAKE_OP` | ❌ not defined | glibc mutex fast path unavailable |
| `FUTEX_WAIT_BITSET` | ❌ not defined | `pselect`/`ppoll` optimization unavailable |
| `FUTEX_WAKE_BITSET` | ❌ not defined | Targeted wake unavailable |
| `FUTEX_LOCK_PI` / `UNLOCK_PI` | ❌ not defined | Priority inversion unprotected |
| Robust futex list | ❌ not defined | Thread death → permanent deadlock |
| Futex sharding (per-futex lock) | ❌ single global L1 mutex | All futex ops on all CPUs contend on one lock |
| Process-private futexes | ❌ global table | Unnecessary cross-process visibility |

**Architecture:**
```
static FUTEXES: Mutex<L1, FutexList>  // single global lock
type FutexList = HashMap<PhysicalAddress, Vec<FutexEntry>>
```

Physical address is the key (enables cross-address-space futex via MAP_SHARED).
Virtual address + Weak<AddrSpaceWrapper> used for CoW disambiguation.

**Recoverable work (lost patches):**

| Feature | Lost Patch | Applies? |
|---------|-----------|----------|
| 64-shard hash table | P6-futex-sharding | ✅ cleanly |
| FUTEX_REQUEUE + CMP_REQUEUE | P8-futex-requeue | ❌ needs rebase |
| PI futex (LOCK_PI/UNLOCK_PI/TRYLOCK_PI) | P8-futex-pi | ❌ needs rebase |
| PI CAS fix | P9-futex-pi-cas-fix | ❌ needs rebase |
| Robust futex list | P8-futex-robust | ❌ needs rebase |

The 4 failing patches likely fail because they depend on sharding (P6-futex-sharding) being
applied first. Apply in order: P6-sharding → P8-requeue → P8-pi → P8-robust → P9-pi-cas-fix.

### 2.4 Kernel Syscall ABI Layer

**Files:** `src/syscall/mod.rs`, `src/syscall/futex.rs`, `src/syscall/time.rs`,
`src/syscall/process.rs`, `local/sources/syscall/src/number.rs`, `src/scheme/proc.rs`

**Verdict: Minimal surface — most threading done via proc scheme, not syscalls.**

The kernel defines only ~35 syscall numbers. Threading-relevant ones:

| Syscall | Status | Notes |
|---------|--------|-------|
| `SYS_FUTEX` (240) | ✅ partial | WAIT/WAIT64/WAKE only |
| `SYS_YIELD` (158) | ✅ | `context::switch()` + signal handler |
| `SYS_FMAP` (900) | ✅ | Anonymous + file-backed mmap |
| `SYS_FUNMAP` (92) | ✅ | munmap |
| `SYS_MPROTECT` (125) | ✅ | |
| `SYS_MREMAP` (155) | ✅ | |
| `SYS_NANOSLEEP` (162) | ✅ | EINTR-aware |
| `SYS_CLOCK_GETTIME` (265) | ✅ partial | REALTIME + MONOTONIC only |

**Threading done via proc scheme (not syscalls):**

| Operation | Mechanism |
|-----------|-----------|
| Thread/process creation | `proc:` scheme: open "new-context", share addr_space + files via kdup |
| waitpid | `proc:` scheme: `EVENT_READ` on context fd |
| getpid/gettid | `proc:` scheme: read "attrs" handle |
| kill/tkill | `proc:` scheme: `ForceKill` / `Interrupt` ContextVerb |
| CPU affinity | `proc:` scheme: write "sched-affinity" handle |
| Priority | `proc:` scheme: write "attrs" prio field |
| Signal setup | `proc:` scheme: write "sighandler" + shared Sigcontrol pages |
| TLS base (FSBASE) | `proc:` scheme: write "regs/env" EnvRegisters |

**Completely missing syscalls (no number, no handler):**
`clone`, `fork`, `vfork`, `waitpid`, `wait4`, `kill`, `tkill`, `tgkill`, `arch_prctl`,
`set_thread_area`, `set_tid_address`, `set_robust_list`, `get_robust_list`,
`sched_setaffinity`, `sched_getaffinity`, `sched_setscheduler`, `sched_getparam`,
`sigaction`, `sigprocmask`, `sigpending`, `sigsuspend`, `sigtimedwait`,
`timer_create`, `timer_settime`, `timer_delete`, `timerfd_create`,
`getrusage`, `setrlimit`, `getrlimit`, `times`

### 2.5 relibc Pthread Layer

**Files:** `src/pthread/mod.rs`, `src/sync/*.rs`, `src/header/pthread/*.rs`,
`src/header/sched/mod.rs`, `src/ld_so/tcb.rs`, `src/platform/redox/mod.rs`

**Verdict: Core pthreads solid, scheduling/robust/PI absent, several POSIX gaps.**

#### Fully Working (futex-backed)

| API Group | Backend | Notes |
|-----------|---------|-------|
| `pthread_create/join/detach/exit` | redox_rt clone + Waitval | Stack via mmap, TLS via Tcb::new() |
| `pthread_cancel/setcancelstate/testcancel` | SIGRT_RLCT_CANCEL (33) | Deferred cancellation only |
| `pthread_mutex_*` (normal/recursive/errorcheck) | AtomicU32 CAS + futex_wait/wake | 3-state: unlocked/locked/waiters |
| `pthread_cond_*` | Two-counter futex design | CLOCK_REALTIME only (monotonic = stub) |
| `pthread_rwlock_*` | AtomicU32 + futex | Reader count + WAITING_WR bit |
| `pthread_barrier_*` | Mutex + Cond | gen_id wrapping counter |
| `pthread_spin_*` | AtomicI32 CAS | No futex, pure spinning |
| `pthread_once` | 3-state futex (UNINIT→INITING→INIT) | |
| `pthread_key_create/getspecific/setspecific/delete` | BTreeMap global + thread_local values | Destructor iteration per POSIX |
| `pthread_sigmask` | Delegates to sigprocmask | |
| `pthread_kill` | redox_rt::rlct_kill | |
| `pthread_atfork` | Thread-local LinkedList hooks | |
| ELF TLS (`__thread` / `#[thread_local]`) | PT_TLS + Tcb | Static + dynamic DTV for dlopen |
| `pthread_attr_*` (getters/setters) | RlctAttr struct | |

#### Stubs / No-ops / Missing

| API | Status | Root Cause |
|-----|--------|------------|
| `sched_get_priority_max/min` | `todo!()` | Kernel has no scheduling policy API |
| `sched_getparam/setparam` | `todo!()` | Same |
| `sched_setscheduler` | `todo!()` | Same |
| `sched_rr_get_interval` | `todo!()` | Same |
| `pthread_setschedparam` | No-op (returns Ok) | Kernel ignores policy |
| `pthread_setschedprio` | No-op (returns Ok) | Kernel ignores priority change |
| `pthread_getschedparam` | `todo!()` | |
| `pthread_getcpuclockid` | ENOENT | No per-thread CPU clock |
| `pthread_mutex_consistent` | `todo_skip!` | Robust mutex not implemented |
| `pthread_mutex_getprioceiling` | `todo_skip!` | Priority ceiling not implemented |
| `pthread_mutex_setprioceiling` | `todo_skip!` | Same |
| `pthread_mutexattr_setprotocol` (PRIO_INHERIT) | Accepted, no-op | PI futex missing |
| `pthread_mutexattr_setrobust` (ROBUST) | Accepted, no-op | Robust futex missing |
| `pthread_cond_init` CLOCK_MONOTONIC | `todo_skip!` | |
| `pthread_cond_signal` | Calls broadcast (wakes ALL) | Missing FUTEX_REQUEUE optimization |
| `pthread_setaffinity_np` | Not defined | |
| `pthread_getaffinity_np` | Not defined | |
| `pthread_setname_np` | Not defined | |
| `pthread_getname_np` | Not defined | |
| `pthread_setcanceltype` | Always returns DEFERRED | ASYNC not tracked |
| Guard pages | Attribute stored, not mapped | No PROT_NONE page before stack |
| PTHREAD_KEYS_MAX limit | Not checked | |

---

## 3. Gap Classification

### 3.1 Correctness Gaps (Must Fix — Silent Data Corruption or Deadlock)

| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| C1 | **No robust mutexes** | Thread death while holding mutex → permanent deadlock for all waiters | Kernel: robust futex list + relibc: pthread_mutex_consistent |
| C2 | **No PI futexes** | Priority inversion: low-prio thread blocks high-prio thread indefinitely | Kernel: FUTEX_LOCK_PI/UNLOCK_PI + relibc: mutexattr_setprotocol |
| C3 | **`pthread_cond_signal` wakes ALL** | Correctness: wastes CPU. Performance: thundering herd on every signal | relibc: use true wake(1) — may need FUTEX_REQUEUE |
| C4 | **`fork()` not thread-safe** | `pthread_atfork` hooks exist but child inherits locked mutexes | relibc: implement atfork child handlers properly |

### 3.2 Performance Gaps (Must Fix for Desktop Responsiveness)

| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| P1 | **No SMP load balancing** | Cores sit idle while others are overloaded | Kernel: work stealing + initial placement |
| P2 | **No futex sharding** | Single global L1 mutex for ALL futex ops on ALL CPUs | Kernel: 64-shard hash table |
| P3 | **No FUTEX_REQUEUE** | `pthread_cond_broadcast` wakes all → thundering herd | Kernel: REQUEUE + CMP_REQUEUE |
| P4 | **Full TLB flush on every shootdown** | Per-page mprotect/munmap flushes entire TLB on all cores | Kernel: INVLPG-based selective flush |
| P5 | **Global context switch lock** | Serialization bottleneck beyond ~8 cores | Kernel: per-CPU context switch (needs per-CPU run queues) |
| P6 | **All IRQs to BSP** | CPU 0 handles all interrupts, cache thrash, latency | Kernel: IRQ steering in I/O APIC + MSI/MSI-X dest field |
| P7 | **No RT scheduling** | Audio/compositor threads can't get priority | Kernel: SchedPolicy + RT dispatch + relibc: sched_setscheduler |

### 3.3 POSIX Completeness Gaps (Must Fix for Application Compatibility)

| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| X1 | `sched_*` all `todo!()` | Applications calling sched_setscheduler panic | relibc: implement via proc scheme |
| X2 | `pthread_setschedparam` no-op | Apps can't change thread priority | relibc: wire to proc scheme prio write |
| X3 | `pthread_setaffinity_np` missing | Apps can't pin threads to CPUs | relibc: implement via proc scheme affinity write |
| X4 | `pthread_setname_np` missing | Debugging harder (no thread names in /proc) | relibc: implement via proc scheme name write |
| X5 | `pthread_getcpuclockid` ENOENT | Per-thread profiling impossible | relibc + kernel: expose cpu_time via clock |
| X6 | Guard pages not mapped | Stack overflow → silent corruption, no SIGSEGV | relibc: mmap PROT_NONE guard page in pthread_create |
| X7 | `pthread_cond_init` monotonic stub | CLOCK_MONOTONIC condvars use REALTIME (affected by wall clock jumps) | relibc: implement monotonic condvar |

---

## 4. Implementation Plan

### Phase 0: Patch Recovery — Re-Apply Lost Threading Work (Week 1–2)

**Goal:** Recover the P5–P9 work that was lost during the local fork migration.

**This is the highest-priority phase — it restores ~6 months of work with minimal new code.**

#### 0.1 — Re-apply kernel scheduler patches to local fork

Apply in dependency order to `local/sources/kernel/`:

| Order | Patch | Status | Action |
|-------|-------|--------|--------|
| 1 | P6-futex-sharding | ✅ applies | Commit directly |
| 2 | P6-percpu-runqueues | ✅ applies | Commit directly |
| 3 | P8-percpu-sched | ✅ applies | Commit directly |
| 4 | P8-percpu-wiring | ✅ applies | Commit directly |
| 5 | P8-initial-placement | ✅ applies | Commit directly |
| 6 | P5-sched-rt-policy | ✅ applies | Commit directly |
| 7 | P5-context-mod-sched | ✅ applies | Commit directly |
| 8 | P6-vruntime-switch | ✅ applies | Commit directly |
| 9 | P7-cache-affine-switch | ✅ applies | Commit directly |
| 10 | P9-numa-topology | ✅ applies | Commit directly |
| 11 | P9-proc-lock-ordering | ✅ applies | Commit directly |
| 12 | P8-work-stealing | ❌ needs rebase | Rebase against 1–11, then apply |
| 13 | P8-futex-requeue | ❌ needs rebase | Rebase against P6-sharding (#1), then apply |
| 14 | P8-futex-pi | ❌ needs rebase | Rebase against #13, then apply |
| 15 | P8-futex-robust | ❌ needs rebase | Rebase against #14, then apply |
| 16 | P9-futex-pi-cas-fix | ❌ needs rebase | Rebase against #14, then apply |
| 17 | P7-scheduler-improvements | ❌ needs rebase | Rebase against 1–11, then apply |

**Verification after each patch:**
```bash
cd local/sources/kernel
cargo check  # must pass
```

#### 0.2 — Re-apply relibc threading patches to local fork

Apply to `local/sources/relibc/`:

| Patch | Action |
|-------|--------|
| P3-threads.patch | ✅ applies — commit |
| P3-barrier-smp-futex (from absorbed/) | Verify already in fork; if not, apply |
| P3-pthread-signal-races (from absorbed/) | Verify already in fork |
| P3-pthread-yield (from absorbed/) | Verify already in fork |
| P5-robust-mutexes (from absorbed/) | Verify; re-apply if missing |
| P5-robust-mutex-enotrec-fix (from absorbed/) | Same |
| P5-sched-api (from absorbed/) | Same |
| P7-pthread-affinity (from absorbed/) | Same |
| P7-pthread-setname (from absorbed/) | Same |
| P7-setpriority (from absorbed/) | Same |
| P9-spin-and-barrier (from absorbed/) | Same |
| P9-spin-fix (from absorbed/) | Same |
| P3-semaphore-comprehensive | ✅ applies |

**Verification:**
```bash
cd local/sources/relibc
make all  # must pass
touch relibc && make prefix  # rebuild prefix with new libc
```

#### 0.3 — Build and smoke test

```bash
export REDBEAR_ALLOW_PROTECTED_FETCH=1
./local/scripts/build-redbear.sh --upstream redbear-mini
make qemu  # verify boot + basic operation
```

**Success criteria:** redbear-mini boots, multi-threaded daemons (pcid, xhcid) start, no kernel panic.

---

### Phase 1: Futex Completeness (Week 2–4)

**Goal:** Close the futex operation gaps that affect correctness and performance.

**Depends on:** Phase 0 complete (sharding applied first).

#### 1.1 — FUTEX_REQUEUE + FUTEX_CMP_REQUEUE

**Kernel:** `src/syscall/futex.rs`
- Add `FUTEX_REQUEUE` and `FUTEX_CMP_REQUEUE` to the futex dispatcher
- Implement: move up to `val` waiters from addr1 → addr2, optionally compare `*addr1 == val2`
- Requires locking TWO shards (acquire both in deterministic order to avoid deadlock)

**relibc:** `src/sync/cond.rs`
- Change `pthread_cond_broadcast` to use `FUTEX_REQUEUE` (move waiters from condvar futex to mutex futex)
- Change `pthread_cond_signal` to wake exactly 1 (not all)

**Impact:** Eliminates thundering herd on every `pthread_cond_broadcast`. Major win for Qt event loop, KWin compositor, Mesa worker threads.

#### 1.2 — PI Futexes (FUTEX_LOCK_PI / FUTEX_UNLOCK_PI / FUTEX_TRYLOCK_PI)

**Kernel:** `src/syscall/futex.rs`
- Add `PiState` tracking per futex: owner context + waiter list with priorities
- On `LOCK_PI` block: boost owner's priority to waiter's priority
- On `UNLOCK_PI`: restore original priority, wake highest-priority waiter
- Requires kernel RT scheduling (Phase 0.1 #6–7: P5-sched-rt-policy)

**relibc:** `src/sync/pthread_mutex.rs`
- Implement `PTHREAD_PRIO_INHERIT` protocol path using PI futex
- Replace `todo_skip!` in `pthread_mutex_consistent` with real implementation

#### 1.3 — Robust Futex List

**Kernel:** `src/syscall/futex.rs` + `src/context/context.rs`
- Add `robust_list_head: Option<usize>` to `Context` struct
- Implement `set_robust_list` / `get_robust_list` via proc scheme or syscall
- On thread exit (`exit_this_context`): walk robust list, set `FUTEX_OWNER_DIED` bit, wake one waiter with `EOWNERDEAD`

**relibc:** `src/sync/pthread_mutex.rs`
- Implement robust list registration in `pthread_mutex_lock`
- Implement `pthread_mutex_consistent`: clear `EOWNERDEAD` state
- Replace `todo_skip!` with real implementation

#### 1.4 — FUTEX_WAKE_OP

**Kernel:** `src/syscall/futex.rs`
- Implement atomic op + wake: perform op on addr2, then wake up to `val` waiters on addr1
- Operations: set, add, or, andn, xor, with comparison condition

**Impact:** glibc mutex fast path optimization. Not critical for relibc but helps ported glibc-linked binaries.

---

### Phase 2: SMP Scheduling Quality (Week 3–6)

**Goal:** Make multi-core actually distribute work.

**Depends on:** Phase 0 complete (per-CPU queues applied).

#### 2.1 — Work stealing (recover + fix)

**Kernel:** `src/context/switch.rs`
- On `select_next_context()` empty local queue: steal from victim CPU
- Pick victim by round-robin, steal highest-priority runnable context
- Limit steal batch size (1–2 contexts per steal attempt)
- Send `IpiKind::Wakeup` to target CPU if stealing woke it from idle

**Recovery:** P8-work-stealing needs rebase against per-CPU wiring.

#### 2.2 — Load balancing (recover + verify)

**Kernel:** `src/context/switch.rs`
- Periodic balance trigger (every N ticks or when queue depth difference > threshold)
- Migrate contexts from overloaded CPU to most-idle CPU
- Respect `sched_affinity` mask during migration

**Recovery:** P8-load-balance is in absorbed/ — verify it's in the fork after Phase 0.

#### 2.3 — Reschedule IPI

**Kernel:** `src/arch/x86_shared/ipi.rs` + `src/context/switch.rs`
- When waking a context on a different CPU, send `IpiKind::Switch` to that CPU
- Currently the Switch IPI exists but is not used by the scheduler

#### 2.4 — Per-page TLB flush (INVLPG)

**Kernel:** `rmm/src/arch/x86_64.rs` + `src/context/memory.rs`
- Add `invalidate_page(addr)` using `invlpg` instruction
- Modify `Flusher` to track individual pages and use INVLPG when ≤ N pages affected
- Fall back to CR3 reload only for large-scale invalidations

**Impact:** Every `mprotect`/`mmap`/`munmap` on a multi-threaded process currently flushes the ENTIRE TLB on every core. This is one of the most impactful single fixes.

#### 2.5 — TLB broadcast optimization

**Kernel:** `src/percpu.rs`
- Replace per-CPU sequential `shootdown_tlb_ipi(Some(id))` loop with ICR "all excluding self" (destination shorthand 0b11)
- Single IPI + global ack counter instead of N individual IPIs + N ack counters

---

### Phase 3: RT Scheduling (Week 4–6)

**Goal:** Allow applications to request real-time scheduling for latency-sensitive threads.

**Depends on:** Phase 0 (SchedPolicy applied) + Phase 2 (per-CPU queues).

#### 3.1 — Kernel RT scheduling dispatch

**Kernel:** `src/context/switch.rs` (from P5-sched-rt-policy — recovered in Phase 0)
- `select_next_context()` passes:
  1. SCHED_FIFO contexts (highest RT priority first, no preemption within same prio)
  2. SCHED_RR contexts (highest RT priority first, round-robin within same prio)
  3. SCHED_OTHER contexts (existing DWRR/vruntime)
- SCHED_RR quantum: configurable per-context (default 100ms)

#### 3.2 — relibc sched_* API completion

**relibc:** `src/header/sched/mod.rs`

Replace ALL `todo!()` stubs:

| Function | Implementation |
|----------|---------------|
| `sched_getscheduler(pid)` | Read policy from proc scheme attrs |
| `sched_setscheduler(pid, policy, param)` | Write policy + RT priority via proc scheme |
| `sched_getparam(pid, param)` | Read RT priority from proc scheme |
| `sched_setparam(pid, param)` | Write RT priority via proc scheme |
| `sched_get_priority_max(policy)` | Return 99 for FIFO/RR, 0 for OTHER |
| `sched_get_priority_min(policy)` | Return 1 for FIFO/RR, 0 for OTHER |
| `sched_rr_get_interval(pid, tp)` | Return SCHED_RR quantum (100ms default) |

#### 3.3 — pthread_setschedparam wiring

**relibc:** `src/pthread/mod.rs`
- Replace `set_sched_param` no-op with real proc scheme call
- Replace `set_sched_priority` no-op with real proc scheme call

---

### Phase 4: POSIX Pthread Completeness (Week 5–8)

**Goal:** Close remaining POSIX gaps that block application compatibility.

**Depends on:** Phase 0 + Phase 3 (for sched API).

#### 4.1 — pthread_setaffinity_np / pthread_getaffinity_np

**relibc:** `src/header/pthread/mod.rs` + `src/header/sched/mod.rs`
- Implement using proc scheme "sched-affinity" write/read
- Define `cpu_set_t` type and `CPU_SET/CPU_CLR/CPU_ZERO/CPU_ISSET` macros

#### 4.2 — pthread_setname_np / pthread_getname_np

**relibc:** `src/header/pthread/mod.rs`
- Implement using proc scheme name write/read (kernel already supports 32-char name field)

#### 4.3 — pthread_cond_init CLOCK_MONOTONIC

**relibc:** `src/sync/cond.rs`
- Replace `todo_skip!` with real monotonic clock support
- Store clock choice in cond struct, use `CLOCK_MONOTONIC` for deadline calculations

#### 4.4 — Guard pages

**relibc:** `src/pthread/mod.rs`
- In `pthread_create`, when allocating stack via mmap:
  - Map `[stack_base, stack_base + guard_size)` with `PROT_NONE`
  - Map `[stack_base + guard_size, stack_base + guard_size + stack_size)` with `PROT_READ | PROT_WRITE`
- On thread exit, munmap both regions

#### 4.5 — pthread_getcpuclockid

**relibc:** `src/header/pthread/mod.rs`
- Return `CLOCK_THREAD_CPUTIME_ID` (requires kernel support — add clock to `clock_gettime`)

**Kernel:** `src/syscall/time.rs`
- Add `CLOCK_THREAD_CPUTIME_ID` → read `context.cpu_time`

#### 4.6 — PTHREAD_KEYS_MAX enforcement

**relibc:** `src/header/pthread/tls.rs`
- Check `NEXTKEY` against `PTHREAD_KEYS_MAX` (1024) before allocating

---

### Phase 5: IRQ Steering and NUMA (Week 8–12)

**Goal:** Distribute interrupt load and respect memory locality.

**Depends on:** Phase 2 (per-CPU infrastructure).

#### 5.1 — IRQ steering

**Kernel:** `src/arch/x86_shared/device/ioapic.rs` + `src/arch/x86_shared/idt.rs`
- Change I/O APIC redirection `dest` from `bsp_apic_id` to round-robin or RSS hash
- Add per-CPU legacy IRQ handlers in IDT (not just BSP)
- For MSI/MSI-X: set destination CPU in Message Address register

#### 5.2 — NUMA topology discovery

**Kernel:** `src/acpi/` (from P9-numa-topology — recovered in Phase 0)
- Parse SRAT (Static Resource Affinity Table) for proximity domains
- Parse SLIT (System Locality Distance Information Table) for inter-node distances
- Store `NumaTopology` in kernel for O(1) scheduling lookups

#### 5.3 — NUMA-aware memory allocation

**Kernel:** `src/memory/` + frame allocator
- Track frame NUMA node in `Frame` or `PageInfo`
- On allocation, prefer frames from requesting CPU's NUMA node
- Fallback to remote node when local node is exhausted

---

## 5. Dependency Chain

```
Phase 0 (Patch Recovery) ← BLOCKING FOR ALL OTHERS
    │
    ├──► Phase 1 (Futex Completeness)
    │       │
    │       ├──► 1.1 REQUEUE ──► condvar performance
    │       ├──► 1.2 PI ──► priority inversion fix (needs Phase 3.1)
    │       ├──► 1.3 Robust ──► deadlock prevention
    │       └──► 1.4 WAKE_OP ──► glibc compat
    │
    ├──► Phase 2 (SMP Scheduling)
    │       │
    │       ├──► 2.1 Work stealing ──► core utilization
    │       ├──► 2.2 Load balancing ──► fair distribution
    │       ├──► 2.3 Reschedule IPI ──→ cross-CPU wakeup
    │       ├──► 2.4 Per-page TLB ──► mmap/mprotect performance
    │       └──► 2.5 TLB broadcast ──► IPI efficiency
    │
    ├──► Phase 3 (RT Scheduling)
    │       │
    │       ├──► 3.1 Kernel RT dispatch (from Phase 0)
    │       ├──► 3.2 relibc sched_* API ──► POSIX compat
    │       └──► 3.3 pthread_setschedparam ──► app priority control
    │
    ├──► Phase 4 (POSIX Pthread Completeness)
    │       │
    │       ├──► 4.1 Affinity API ──► CPU pinning
    │       ├──► 4.2 Thread naming ──► debuggability
    │       ├──► 4.3 Monotonic condvar ──► clock correctness
    │       ├──► 4.4 Guard pages ──► stack overflow detection
    │       ├──► 4.5 CPU clock ──► per-thread profiling
    │       └──► 4.6 Keys max ──► resource limit
    │
    └──► Phase 5 (IRQ + NUMA)
            │
            ├──► 5.1 IRQ steering ──► interrupt distribution
            ├──► 5.2 NUMA topology ──► (from Phase 0)
            └──► 5.3 NUMA allocator ──► memory locality
```

**Parallel work possible:**
- Phase 1 + Phase 2 + Phase 3 can run in parallel after Phase 0
- Phase 4 items are independent of each other
- Phase 5 depends on Phase 2 but not on Phase 1/3/4

---

## 6. Validation Plan

### 6.1 Build Evidence

| Check | Command |
|-------|---------|
| Kernel compiles | `make r.kernel` |
| relibc compiles | `make r.relibc` |
| Prefix rebuilt | `touch relibc kernel && make prefix` |
| Full OS builds | `./local/scripts/build-redbear.sh redbear-mini` |

### 6.2 Runtime Evidence (QEMU)

| Test | Verification |
|------|-------------|
| Multi-threaded boot | `make qemu QEMUFLAGS="-smp 4"` — all 4 CPUs active |
| pthread smoke test | Guest: compile + run simple pthread_create/join/mutex test |
| Work stealing | Guest: spawn 8 threads on 4-CPU QEMU, verify all CPUs utilized |
| Futex REQUEUE | Guest: condvar broadcast benchmark — waiters wake in ≤2 batches, not N |
| PI futex | Guest: priority inversion test — high-prio thread unblocked within 1 tick |
| Robust mutex | Guest: kill thread holding mutex, verify EOWNERDEAD recovery |
| RT scheduling | Guest: SCHED_FIFO thread preempts SCHED_OTHER within 100μs |
| CPU affinity | Guest: pin thread to CPU 1, verify it never runs on CPU 0 |
| Thread naming | Guest: `cat /scheme/proc/*/name` shows set names |
| Guard pages | Guest: overflow stack, verify SIGSEGV (not silent corruption) |
| TLB efficiency | Guest: mprotect benchmark — compare TLB miss rate before/after |

### 6.3 Validation Scripts (to create)

```bash
local/scripts/test-threading-qemu.sh          # Comprehensive threading smoke test
local/scripts/test-futex-requeue-qemu.sh      # REQUEUE-specific test
local/scripts/test-futex-pi-qemu.sh           # PI futex test
local/scripts/test-futex-robust-qemu.sh       # Robust mutex test
local/scripts/test-sched-rt-qemu.sh           # RT scheduling latency test
local/scripts/test-sched-balance-qemu.sh      # Load balancing on multi-vCPU
local/scripts/test-threading-baremetal.sh     # Bare metal multi-threaded stress
```

---

## 7. Estimated Effort

| Phase | Duration | New Code | Recovery | Dependencies |
|-------|----------|----------|----------|-------------|
| Phase 0: Patch Recovery | 1–2 weeks | Minimal (rebase 5 patches) | 13 patches apply directly | None |
| Phase 1: Futex Completeness | 2–3 weeks | REQUEUE impl + WAKE_OP | PI/robust from P8 patches | Phase 0 |
| Phase 2: SMP Scheduling | 3–4 weeks | TLB INVLPG + broadcast opt | Work stealing from P8 | Phase 0 |
| Phase 3: RT Scheduling | 1–2 weeks | relibc sched_* API | RT dispatch from P5 | Phase 0 |
| Phase 4: POSIX Pthread | 2–3 weeks | Affinity/naming/guard/clock | Partial from P7 patches | Phase 0, 3 |
| Phase 5: IRQ + NUMA | 3–4 weeks | IRQ steering + NUMA allocator | NUMA topology from P9 | Phase 0, 2 |

**Total:** 12–18 weeks with 1–2 developers. Phase 0 alone recovers the majority of the value in 1–2 weeks.

---

## 8. Integration with Existing Plans

| Plan | Relationship |
|------|-------------|
| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | **Consumer** — Phase 3 (KWin) needs PI futex + RT scheduling; Phase 2 (compositor) needs work stealing |
| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | **Sibling** — IRQ steering (Phase 5.1) belongs to both plans |
| `DRM-MODERNIZATION-EXECUTION-PLAN.md` | **Consumer** — GPU worker threads benefit from load balancing + affinity |
| `IMPLEMENTATION-MASTER-PLAN.md` | **Parent** — this plan covers the kernel threading substrate |
| `CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md` | **Sibling** — overlaps on scheduler/IRQ delivery |

---

## 9. Bottom Line

The Red Bear OS threading stack is **functional for basic single-threaded and lightly-threaded
workloads**. The SMP boot, context switching, TLB shootdown, and basic futex operations are
correct.

The **critical problem** is that 6 months of threading enhancement work (P5–P9 patches) was
lost during the local fork migration. This work exists as patch files that apply cleanly to
the current fork — **Phase 0 (Patch Recovery) is the single highest-ROI action**.

After Phase 0, the remaining gaps are:
1. **Futex REQUEUE/PI/robust** — for condvar performance and deadlock prevention
2. **SMP work stealing + load balancing** — for multi-core utilization
3. **RT scheduling** — for audio/compositor thread priority
4. **POSIX pthread completeness** — for application compatibility
5. **IRQ steering + NUMA** — for multi-socket performance

The **desktop-critical path** (KWin responsiveness) requires Phases 0–3. The
**server-critical path** (multi-socket, NUMA) adds Phase 5. Phase 4 (POSIX completeness)
benefits all paths but is not desktop-blocking.