diff --git a/AGENTS.md b/AGENTS.md index 1f9a946c26..e4825390b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -458,34 +458,46 @@ On every build, the cookbook MUST: b. Emit a clear actionable error: `LOCAL FORK OUTDATED: local/sources/ is at vX.Y.Z but vA.B.C is required by ` c. Do NOT silently proceed with a broken build. -**Remediation (automatic, when invoked)**: The `local/scripts/bump-fork.sh` -script (or equivalent `repo bump-fork ` command) MUST: -1. Fetch the upstream source at the required version. -2. Apply all Red Bear patches from `local/patches//` using the - atomic patch application mechanism (see "Atomic Patch Application" below). -3. Update the version field in the local fork's `Cargo.toml`. -4. Commit the result to the `submodule/` branch in the RedBear-OS repo. -5. Rebuild the affected packages. +**Remediation (canonical: `bump-release.sh`)**: The +`local/scripts/bump-release.sh` orchestrator is the canonical fork-bump path +(see `local/docs/RELEASE-BUMP-WORKFLOW.md`). For each fork it resolves the +latest upstream release tag (excluding tags already reachable from the fork's +HEAD — ancestor tags are history, not upgrades) and: +1. Rebases the fork onto the new upstream tag via + `upgrade-forks.sh --to=` (net-diff reapply with cherry-pick fallback, + then verifies no upstream functions were dropped). +2. Updates the version field in the local fork's `Cargo.toml` to + `+rb`. +3. Updates `local/fork-upstream-map.toml` column 3 in place. +4. Regenerates the fork's `Cargo.lock`. +5. Prints the exact commit sequence for the operator (fork commits to + `submodule/` + the parent map commit). It never commits itself. +6. The stale-prefix and source-fingerprint machinery then rebuilds affected + packages on the next `build-redbear.sh` run. + +`local/scripts/bump-fork.sh` is **DEPRECATED** (guarded by +`REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1`): its wholesale-replacement model +discards committed fork work not present as patch files. Kept for historical +reference only — do not use it. **Symlink aliasing**: `recipes//` MUST be a symlink to `../../../local/sources//` when a local fork exists. This ensures backward compatibility for recipes that reference the `recipes/` path while still preferring the local fork as the source of truth. -**Implementation status**: Detection is fully implemented across 5 tools: +**Implementation status**: Detection is fully implemented across the +release-bump toolchain: +- `bump-release.sh`: canonical release-bump orchestrator (fork source+label coherence) - `sync-versions.sh`: Cat 0 (cookbook) + Cat 1 (in-house) + Cat 2 (fork +rb suffix) versions - `verify-fork-versions.sh`: Cat 2 fork supremacy + content check vs upstream tag - `verify-patch-content.py`: orphan-patch detection + operator-decision automation -- `bump-fork.sh` + `upgrade-forks.sh`: fork rebase + patch reapplication +- `upgrade-forks.sh`: fork rebase + patch reapplication (`--to=`, diverged guard) - `push-fork-branches.sh`: operator-reviewed fork push helper +- `refresh-fork-upstream-map.sh`: append-only upstream-tag maintenance for the fork map +- `check-external-versions.sh` + `bump-graphics-recipes.sh`: external desktop-stack version check + source bumps -See `local/docs/HOOKS.md` for the full tool inventory (10 tools as of Phase 9.2). -process is: -1. `cd local/sources/` -2. Edit `Cargo.toml` version field to match upstream -3. `git pull` or fetch upstream at the new tag -4. Reapply all `local/patches//*.patch` (see AGENTS.md "Atomic Patch Application") -5. Commit to `submodule/` branch +See `local/scripts/TOOLS.md` for the full tool inventory (15 tools as of +Round 14) and `local/docs/HOOKS.md` for the opt-in git hooks. ### Offline-First By Default diff --git a/README.md b/README.md index 1d02abca5d..809ca603e7 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,7 @@ Red Bear OS operates under strict discipline. Full policies: [`local/AGENTS.md`] - [Patch Preservation Audit](local/docs/PATCH-PRESERVATION-AUDIT-2026-07-12.md) — orphan-patch governance, Rounds 1-6 audit results, SUPERSEDED.md log - [Collision Detection Status](local/docs/COLLISION-DETECTION-STATUS.md) — runtime collision-detection current state - [Hooks](local/docs/HOOKS.md) — opt-in git hooks (pre-push safety net, etc.) -- [Build Tools](local/scripts/TOOLS.md) — 10-tool reference (patch-status, sync, verify, collision, etc.) +- [Build Tools](local/scripts/TOOLS.md) — 15-tool reference (patch-status, sync, verify, collision, release-bump, etc.) - [Fork Push Status](local/docs/fork-push-status/2026-07-12-Round-9-phase-8.3.md) — fork-branch push results + base deadlock (updated Round 9) - [Wi‑Fi Plan](local/docs/WIFI-IMPLEMENTATION-PLAN.md) — Wireless architecture and driver plan - [Bluetooth Plan](local/docs/BLUETOOTH-IMPLEMENTATION-PLAN.md) — Bluetooth stack design diff --git a/docs/07-RED-BEAR-OS-IMPLEMENTATION-PLAN.md b/docs/07-RED-BEAR-OS-IMPLEMENTATION-PLAN.md index 21c79d7284..bf834a6f64 100644 --- a/docs/07-RED-BEAR-OS-IMPLEMENTATION-PLAN.md +++ b/docs/07-RED-BEAR-OS-IMPLEMENTATION-PLAN.md @@ -411,7 +411,7 @@ The current subsystem plans to treat as first-class are: - `local/docs/WIFI-IMPLEMENTATION-PLAN.md` - `local/docs/BLUETOOTH-IMPLEMENTATION-PLAN.md` - `local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md` — implemented credential syscalls + kernel robustness -- `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` +- `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` - `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — **canonical comprehensive plan** The older architecture/roadmap docs under `docs/01`–`docs/05` remain useful, but they should be diff --git a/docs/README.md b/docs/README.md index 2c9df15463..7d7dff60c6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -66,7 +66,7 @@ console-to-KDE plan. - `../local/docs/IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` — PCI/IRQ quality, MSI/MSI-X - `../local/docs/DRM-MODERNIZATION-EXECUTION-PLAN.md` — DRM-focused execution (subsystem detail) - `../local/docs/WAYLAND-IMPLEMENTATION-PLAN.md` — Wayland compositor (subsystem detail) -- `../local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — relibc IPC surface +- `../local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — relibc IPC surface - `../local/docs/GREETER-LOGIN-IMPLEMENTATION-PLAN.md` — greeter/login design - `../local/docs/DBUS-INTEGRATION-PLAN.md` — D-Bus architecture - `../local/docs/SCRIPT-BEHAVIOR-MATRIX.md` — script guarantees and non-guarantees diff --git a/local/AGENTS.md b/local/AGENTS.md index 3189298164..b915ad41f7 100644 --- a/local/AGENTS.md +++ b/local/AGENTS.md @@ -718,6 +718,44 @@ newer upstream release. The `--check` mode is suitable for CI gates and preflight checks. +### Release-bump orchestrator (canonical) + +`./local/scripts/bump-release.sh` is the canonical entry point for a Red Bear +OS release bump. It DECIDES what moves and DELEGATES execution to existing +single-purpose tools: + +- **no args** — label sync (`sync-versions.sh --no-regen`) + fork upstream + report + external version report. Fast, mostly local. +- `--with-sources` — additionally rebase eligible Cat 2 forks onto newer + upstream tags via `upgrade-forks.sh --to=`, update the fork + `Cargo.toml` label + `local/fork-upstream-map.toml` column 3 in place. +- `--with-external` — additionally run the rewritten map-driven + `bump-graphics-recipes.sh` for the Qt6/KF6/Plasma desktop stack. +- `--check`, `--dry-run`, `--no-fetch`, `--regen`, `--force-diverged`, + `--fork=` — see `local/scripts/TOOLS.md` for the full flag matrix. + +Fork routing is by map mode: `snapshot`/`tracked` → source-bump eligible; +`diverged` (kernel/bootloader/installer) → report-only (`--force-diverged` +escape hatch); `base` (tag=`main`) → always suffix-only. Nothing ever +auto-commits; the orchestrator prints the exact `git add`/`git commit` +sequence for the parent repo and each affected `submodule/` branch. + +### Branch-switch automation (opt-in post-checkout hook) + +`./local/scripts/post-checkout-version-sync.sh` is an **opt-in** git hook +that runs the label-only half of a release bump automatically when an +operator checks out a semver release branch. Guards: branch checkout only, +anchored semver branch name, no rebase/cherry-pick/merge in progress, +`REDBEAR_NO_AUTO_SYNC` unset, clean working tree. It NEVER does network, +lockfile regen, source upgrades, or commits, and never blocks a checkout. +Bypass: `REDBEAR_NO_AUTO_SYNC=1 git checkout `. Install via +`./local/scripts/install-git-hooks.sh`. + +**Full runbook** (4-step release model, fork decision tree, bootloader +`git replace --graft` note, lockfile-regen sequencing, commit-sequence +conventions): `local/docs/RELEASE-BUMP-WORKFLOW.md`. **Tool inventory:** +`local/scripts/TOOLS.md`. **Hook details:** `local/docs/HOOKS.md`. + ### Fork authorship attribution Every Cat 2 fork (`local/sources//Cargo.toml`) that carries Red Bear @@ -1022,14 +1060,20 @@ As of 2026-06, the following core components are built from **local forks** in | Component | Local fork path | Recipe | |-------------|------------------------------------------|-------------------------------------------------------| -| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (git URL) | -| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (git URL) | -| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` | -| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (+ Cargo.toml dep) | -| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` | -| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` | +| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (path = ...) | +| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (path = ...) | +| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` (path = ...) | +| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (path = ..., + Cargo.toml dep) | +| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` (path = ...) | +| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` (path = ...) | | **base** | **`local/sources/base/`** (path source) | **`recipes/core/base/recipe.toml` (path = ...) ** | +All seven recipe'd components use `path = "../../../local/sources/"`; +the cookbook symlinks `recipes/core//source` to the fork worktree. The +remaining forks — `syscall`, `libredox`, `redox-scheme` — have **no recipe**: +they are consumed purely as Cargo `path` dependencies (plus `[patch.crates-io]`) +by the sibling forks (base, relibc, …). See "Local fork dependency rule". + ### Why local forks for these components? 1. **Critical bug fixes** — these components had multiple broken patches that drifted diff --git a/local/docs/ACPI-FORK-SYNC-STRATEGY-2026-06-30.md b/local/docs/ACPI-FORK-SYNC-STRATEGY-2026-06-30.md deleted file mode 100644 index f89083ff5c..0000000000 --- a/local/docs/ACPI-FORK-SYNC-STRATEGY-2026-06-30.md +++ /dev/null @@ -1,536 +0,0 @@ -# ACPI Fork-Sync Strategy — 2026-06-30 - -This document captures the upstream-fork survey for the ACPI fixes. -Based on the user's direction ("our kernel must bump too"), the strategy -is to **synchronize the local Red Bear kernel+base forks with upstream -Redox main, then implement the remaining gaps on top of the synchronized -foundation**. - -## Upstream commits worth pulling in - -### Kernel (`gitlab.redox-os.org/redox-os/kernel`) - -| SHA | Title | Author | Date | Gap it addresses | Verdict | -|---|---|---|---|---|---| -| `f49c7d99` | RSDP validation and fixing a few clippy lints | Speedy_Lex | 2026-05-05 | **Gap #1: RSDP checksum validation** | ✅ Direct fix | -| `678a8c56` | Fail softly on bad RSDP | Speedy_Lex | 2026-05-05 | Complements #1 | ✅ Robustness | -| `6b55ffea` | Make rsdp pointer NonNull | Speedy_Lex | 2026-05-05 | Type-safety refactor | ✅ Trivial merge | -| `8011f3f6` | Simplify acpi scheme (MR #613) | 4lDO2 | 2026-06-02 | **Gap #8: AcpiScheme fevent + restructures kernel-side ACPI** | 🚧 Major refactor | -| `a47b3a8d` | Add timeout for events | Wildan M | 2026-05-30 | Shutdown timeout support | ✅ Direct | -| `50c68cd7` | Remove ACPI search from the kernel | (recent) | (recent) | Architectural | 🚧 Couples with #613 | -| `f3394673` | Remove dead_code lint allow for ACPI | (recent) | (recent) | Cleanup | ✅ Trivial | -| `62f220b1` | Use cfg! rather than #[cfg] for ACPI enabling | (recent) | (recent) | Cleanup | ✅ Trivial | - -### Base (`gitlab.redox-os.org/redox-os/base`) - -| SHA | Title | Author | Date | Gap it addresses | Verdict | -|---|---|---|---|---|---| -| `9dd6901d` | acpid: setrens before ready(), fixing deadlock | 4lDO2 | 2026-06-05 | **acpid startup correctness** | ✅ 4-line drop-in fix | -| `f2f834d4` | Update ACPI crate | Wildan M | 2026-04-27 | **Bumps jackpot51/acpi to redox-6.x fork** | 🚧 Requires amlserde enum update | -| `2045364b` | Merge `simplify_acpi` (MR #275) | Jeremy Soller | 2026-06-02 | **Pairs with kernel MR #613** | 🚧 Major refactor | - -### Kernel MR #613: "Simplify acpi scheme" - -**Description** (from gitlab): "Complexity can be reduced somewhat by not providing a -filesystem-like interface for RXSDT+shutdown. If file-based interfaces should exist -for the rsdt/xsdt, it can be done by userspace. Reduces kernel size by 4%, -presumably due to the removed hashmap." - -**What it does:** -- Removes `/scheme/kernel.acpi/rxsdt` and `/scheme/kernel.acpi/kstop` files -- Replaces with a single FD-based `call()` interface using `AcpiVerb` enum -- `AcpiVerb::ReadRxsdt` returns rxsdt bytes -- `AcpiVerb::RegisterKstop` registers a kstop listener -- Uses `bitflags!` for handle permission tracking -- Uses new `Mutex` from `crate::sync::ordered` -- Adds `EXISTS_KSTOP_HANDLE` atomic to track handler presence -- Simplifies `kfpath` requirement - -**Required dependency bumps:** -- `redox_syscall 0.7.4 → 0.8.1` (we're 2 versions behind upstream's 0.8.1) -- Pulls in new `AcpiVerb` enum, `CallFlags` enum -- Uses `crate::sync::ordered::Mutex` (new in upstream) - -### Base MR #275: "Use simplified kernel ACPI interface" - -**Description** (from gitlab): "Switches to the `SYS_CALL`-based kernel ACPI interface. -Needs https://gitlab.redox-os.org/redox-os/kernel/-/merge_requests/613" - -**Required user-space changes** (in acpid, hwd): -- Replace `fs::read("/scheme/kernel.acpi/rxsdt")` with `Fd::open("/scheme/kernel.acpi").call_ro(..., AcpiVerb::ReadRxsdt)` -- Replace `File::open("/scheme/kernel.acpi/kstop")` with `Fd::open(...).call_ro(..., AcpiVerb::RegisterKstop)` -- Adds `[patch.crates-io] redox_syscall = { git = ".../syscall.git" }` in workspace Cargo.toml -- Adds `redox_syscall = "0.8.1"` to hwd's Cargo.toml -- Splits `AmlSerdeReferenceKind::LocalOrArg` into 4 separate variants (`Local`, `Arg`, `Index`, `Named`) - -## Coupling matrix - -Pulling in the upstream changes requires coordinated bumps: - -``` -Kernel Cargo.toml: redox_syscall 0.7.4 → 0.8.1 -Kernel src/scheme/acpi.rs: rewrite using AcpiVerb + bitflags + Mutex -Kernel src/syscall/debug.rs: add _ => Err(EINVAL) catch-all (per MR #613 src/scheme/proc.rs diff) - -Base Cargo.toml: redox_syscall 0.7.4 → 0.8.1 + patch.crates-io -Base drivers/acpid: rewrite using Fd::open + call_ro(AcpiVerb::ReadRxsdt/RegisterKstop) -Base drivers/hwd: rewrite using Fd::open + call_ro(AcpiVerb::ReadRxsdt); add redox_syscall dep -Base drivers/amlserde: enum variant split (LocalOrArg → Local/Arg/Index/Named) -Base drivers/acpid/src/scheme.rs: may need Fd-based kstop register instead of open("/scheme/kernel.acpi/kstop") - -All other base drivers: bump redox_syscall transitive dep -``` - -## Recommended execution order - -This is **NOT** a 1-day fix. It's a multi-day fork-sync. Recommended sequence: - -### Phase A — Kernel fork-sync (1 session) -1. **Pull MR #613** into `local/sources/kernel/`: - - `git fetch` (if remote configured) or apply patches manually - - Update `Cargo.toml`: `redox_syscall 0.7.4 → 0.8.1` - - Port `src/scheme/acpi.rs` to the new call() interface - - Add `_ => Err(EINVAL)` catch-all in `src/scheme/proc.rs` (per upstream diff) - - Build kernel with `cargo build --release --target x86_64-unknown-none` - - Verify kernel still compiles -2. **Pull `f49c7d99` + `678a8c56` + `6b55ffea`** — these are trivial RSDP improvements that ride on the same kernel bump. - -### Phase B — Base fork-sync (1-2 sessions) -1. **Pull `9dd6901d`** — 4-line deadlock fix, drop-in. -2. **Pull MR #275 / `2045364b`**: - - Update workspace `Cargo.toml`: `redox_syscall 0.8.0 → 0.8.1` + `[patch.crates-io]` - - Update `acpid/src/main.rs`: Fd-based rxsdt + kstop - - Update `hwd/src/backend/acpi.rs`: Fd-based rxsdt - - Update `hwd/Cargo.toml`: add `redox_syscall = "0.8.1"` -3. **Pull `f2f834d4`** — ACPI crate update: - - Workspace `Cargo.toml`: add `acpi = { git = "...redox-os/acpi.git", branch = "redox-6.x" }` - - `drivers/acpid/Cargo.toml`: `acpi.workspace = true` - - `drivers/amlserde/Cargo.toml`: `acpi.workspace = true` - - `drivers/amlserde/src/lib.rs`: split `AmlSerdeReferenceKind::LocalOrArg` into 4 variants - - Verify `acpid` builds against new acpi crate. - -### Phase C — Apply remaining gaps (1 session each) -After Phase A+B, the new infrastructure (Fd-based call interface, updated acpi crate) makes these gaps easier: -- **Gap #1 RSDP validation**: already pulled from upstream `f49c7d99`. -- **Gap #5 SLP_TYPb write**: now in `acpid/acpi.rs:679` — straightforward 3-line addition. -- **Gap #3 AML Mutex stubs**: the new acpi crate may have fixed these. Check after pulling. -- **Gap #4 S1-S4 sleep**: requires `_PTS`/`_WAK` AML method evaluation — non-trivial. -- **Gap #6 parse_lnk_irc range validation**: independent of upstream. -- **Gap #7 thermal/power enumeration**: independent of upstream. - -## Risk register - -- **Phase A kernel bump risk: HIGH.** `redox_syscall 0.7.4 → 0.8.1` is a 2-version jump. The new `Mutex` and `ordered` module may not exist in our kernel. Build failures likely. -- **Phase B base bump risk: MEDIUM.** MR #275 already has working code in acpid and hwd; just needs porting. -- **amlserde enum breakage risk: MEDIUM.** Every consumer of `AmlSerdeReferenceKind` must be updated. Need to find all callers. -- **Sync conflict risk with Red Bear's own changes**: Our `local/sources/base` has Red Bear commits (`76e0928` etc.). Pulling upstream may require rebasing. - -## Out of scope for this session - -- Implementing the full `_PTS`/`_WAK` AML semantics for S1-S4 (gap #4) -- DMAR root-cause investigation (gap #2) -- 35 minor TODO/FIXME/panic markers -- LegacyBackend implementation -- AML region handlers for SMBus/IPMI/GeneralPurposeBus - -## Decision - -User said "our kernel must bump too." This is the right call — the upstream -changes are coupled and pulling them in one piece is much cleaner than -backporting individual fixes that would otherwise be incompatible with the -new infrastructure. - -**Recommendation:** Start with Phase A (kernel fork-sync) in this session. -If that succeeds (kernel builds), proceed to Phase B (base fork-sync). -If anything in Phase A breaks irrecoverably, document and defer. - ---- - -## Phase A outcome — 2026-06-30 - -**Status: ✅ COMPLETE. Kernel compiles and links.** - -Inner-kernel commit: `4f2a043 kernel: re-sync ACPI subsystem with upstream master` - -Changes applied (11 files, +136/-278): - -1. **`Cargo.toml`** — `redox_syscall` from `version = "0.7.4"` to - `git = "https://gitlab.redox-os.org/redox-os/syscall.git"`. The crates.io - 0.8.1 release predates the `AcpiVerb` enum that MR #613 introduced, - so a git ref is required to get the latest syscall interface. - -2. **`src/acpi/rsdp.rs`** — full rewrite with RSDP validation: - - signature check `"RSD PTR "` - - 20-byte base checksum - - extended checksum for `revision >= 2` - - `NonNull` pointer type instead of `*const u8` - -3. **`src/startup/mod.rs`** — `acpi_rsdp()` returns `Option>`. - -4. **`src/acpi/mod.rs`** — `init()` takes `Option>`. - -5. **`src/scheme/acpi.rs`** — full rewrite to upstream MR #613. Drops the - `/scheme/kernel.acpi/` filesystem in favor of a single `Fd::open` + - `call()` interface with `AcpiVerb` verbs (`ReadRxsdt`, `CheckShutdown`). - Uses `HandleBits` bitflags, atomic `EXISTS_KSTOP_HANDLE`, `Mutex`. - -6. **`src/scheme/mod.rs`** — `KernelScheme::kcall` signature changed from - `id: usize` to `fds: &[usize]`. `kfpath` now defaults to `EOPNOTSUPP`. - -7. **`src/scheme/{memory,proc,user}.rs`** — `kcall` impls updated for new - trait signature. - -8. **`src/scheme/proc.rs`** — added `_ => Err(EINVAL)` catch-all in - `kcall` dispatch to handle new `ProcSchemeVerb` variants - (`RegsInt`, `RegsFloat`, `RegsEnv`, `SchedAffinity`, `Start`) that - the gitlab syscall crate adds. - -9. **`src/syscall/fs.rs`** — `SYS_CALL` dispatcher passes `&[number]` to - `scheme.kcall()` to match new signature. - -10. **`Makefile`** — removed `-Z json-target-spec` flag (no longer - needed; was added by commit 4cb9d80 to compensate for our nightly - being too old, but our nightly is now too NEW for that flag). - -**Verified by:** `make` in `local/sources/kernel/` with cross-toolchain -in PATH. Kernel binary `kernel` produced at 1.2 MiB. - -**Not pushed to remote** — the inner kernel fork's remote is -`gitlab.redox-os.org/redox-os/kernel.git` (upstream Redox, not Red -Bear's gitea). This is the same footgun documented as build-system -issue #11. The change is durable in the inner fork's commit -`4f2a043` and will be reflected in the outer Red Bear repo's -`local/sources/kernel/` submodule pointer when the outer repo -is updated. - -**Gap #1 closed:** RSDP checksum validation now active in the kernel. -**Gap #8 closed:** `AcpiScheme` now uses an atomic + bitflags approach -that wakes waiters immediately (the new `kcall` interface is event-driven -by design). - -**Next step:** Phase B (base fork-sync) — port acpid + hwd + amlserde to -the new Fd::open + call() interface, and bump `redox_syscall 0.7.4 → 0.8.1` -in the workspace `Cargo.toml`. - ---- - -## Phase B outcome — 2026-06-30 - -**Status: ✅ COMPLETE. Cookbook builds; ISO produced.** - -Inner-base commit: `ae57fe3 base: re-sync ACPI userspace with upstream master` - -Changes applied (8 files, +54/-23): - -1. **`Cargo.toml` (workspace)** — added `acpi` workspace dep pointing to - the gitlab redox-os/acpi fork at branch `redox-6.x`. Switched - `redox_syscall` from crates.io `0.8.1` to a git ref of - gitlab.redox-os.org/redox-os/syscall.git, plus - `[patch.crates-io] redox_syscall = { git = ".../syscall.git" }` to - redirect crates.io consumers. - -2. **`drivers/acpid/Cargo.toml`** — `acpi.workspace = true`. - -3. **`drivers/amlserde/Cargo.toml`** — `acpi.workspace = true`. - -4. **`drivers/hwd/Cargo.toml`** — added `redox_syscall.workspace = true`. - -5. **`drivers/amlserde/src/lib.rs`** — split `AmlSerdeReferenceKind::LocalOrArg` - into `Local`, `Arg`, `Index`, `Named` to match the new gitlab acpi - crate's `ReferenceKind` enum (which already split the old combined - variant). Both the `to_serde` and `from_serde` match arms updated. - -6. **`drivers/acpid/src/main.rs`** — rewrote the RXSDT and kstop - acquisition: - - `Fd::open("/scheme/kernel.acpi", O_CLOEXEC, 0)` to get the - parent handle - - `.call_ro(buf, READ, &[ReadRxsdt as u64])` to fetch RXSDT bytes - - `.openat("kstop", O_CLOEXEC, 0)` to get the shutdown pipe - Also applied upstream commit `9dd6901d` (deadlock fix): moved - `setrens(0, 0)` BEFORE `daemon.ready()` so that an `openat` to - `/scheme/acpi/tables` from nsmgr doesn't deadlock. - -7. **`drivers/hwd/src/backend/acpi.rs`** — `AcpiBackend::new()` rewritten - to use the new Fd-based interface. - -**Verified by:** `CI=1 ./local/scripts/build-redbear.sh redbear-mini` -succeeded with exit 0. ISO at `build/x86_64/redbear-mini.iso` -(512 MB, timestamp 2026-06-30 04:54). - -**Gap closure:** - -- **Gap #1 RSDP validation** — closed in Phase A. -- **Gap #8 AcpiScheme fevent** — closed in Phase A. -- **acpid deadlock** (9dd6901d) — closed in Phase B. setrens now - runs before daemon.ready(), so nsmgr's openat doesn't block. - -**Open gaps (out of scope for fork-sync):** - -- Gap #2 DMAR root cause — deferred (needs hardware investigation). -- Gap #3 AML mutex stubs — depends on newer acpi crate behavior. -- Gap #4 S1-S4 sleep — needs _PTS/_WAK AML evaluation. -- Gap #5 SLP_TYPb write — still missing in acpid/acpi.rs:679. -- Gap #6 parse_lnk_irc range validation — still in hwd/backend/acpi.rs. -- Gap #7 thermal/power enumeration — still empty placeholders. - -These can be addressed in Phase C on top of the synchronized foundation. - ---- - -## Phase C outcome — 2026-06-30 - -**Status: ✅ COMPLETE. Cookbook builds; ISO produced.** - -Inner-base commit: `d844111 base: close SLP_TYPb, parse_lnk_irc, AML mutex, and S5 gaps` - -Closes 4 of the 8 critical gaps on top of the synchronized foundation: - -### Gap #5 - SLP_TYPb PM1b write (`drivers/acpid/src/acpi.rs`) - -The previous code wrote SLP_EN+SLP_TYPa to PM1a but silently dropped -SLP_TYPb. On hardware that requires both PM1a and PM1b writes -(some laptops, server boards with split power blocks), the shutdown -was incomplete. Now writes SLP_EN+SLP_TYPb to PM1b when -`pm1b_control_block` is non-zero. The FADT field is 0 when no second -block exists, in which case we skip the second write. - -### Gap #6 - parse_lnk_irc range validation (`drivers/hwd/src/backend/acpi.rs`) - -The previous code accepted any 16-bit integer as an IRQ -(`*n & 0xFFFF`), producing "Enabled at IRQ 53313" from misparsed -FieldUnit accessors on QEMU PIIX4. Now validates that the IRQ -value is ≤ 2047 (the maximum valid legacy-compatible IOAPIC IRQ). -Out-of-range values are debug-logged and skipped instead of polluting -the routing table. Also adds a 15-bit cap on the Buffer-based IRQ -bit extraction (was previously unchecked). - -### Gap #3 - AML mutex create/acquire/release (`drivers/acpid/src/aml_physmem.rs`) - -The new gitlab acpi crate (Phase B bump) added proper `Handler` -trait methods for `create_mutex`, `acquire`, and `release`. The -previous implementation was three `log::debug!()` stubs returning -fake success, which would silently corrupt AML state for any -DSDT/SSDT that uses Mutex. Now implements a real mutex table -backed by `std::sync::Mutex>`: - -- `create_mutex()` allocates a unique `u32` handle from a counter -- `acquire()` busy-waits with 1ms sleeps until the handle is free - or the AML timeout (multiplied by 1000 for ms→µs conversion) - expires; returns `AmlError::MutexAcquireTimeout` on timeout -- `release()` removes the handle from the held set - -### Gap #4a - `set_global_s_state` non-S5 explicit warning - -The previous code silently returned early when called with any state -other than 5. Now emits a `log::warn!()` with the requested state, -naming the missing dependencies (`_PTS`/`_WAK` AML evaluation, -P-state preservation, wakeup path). This converts a silent failure -into a diagnostic that's visible in the boot log. - -### Build-fix: `drivers/acpid/src/dmi.rs:158` - -Converted `e.errno` (private field) to `e.errno()` (method call). -The libredox `Error` struct changed its `errno` from a public field -to a method in a newer release; the `DmiError::Map(syscall::error::Error)` -construction was using the field-access form, which broke the -build against current libredox. This is a build-fix that the prior -dirty tree already had; included here to keep base buildable. - -**Verified by:** `CI=1 ./local/scripts/build-redbear.sh redbear-mini` -succeeded with exit 0. ISO at `build/x86_64/redbear-mini.iso` -(512 MB, timestamp 2026-06-30 05:28). - -**Remaining open gaps (out of scope for this session):** - -- Gap #2 DMAR root cause — needs investigation on real hardware. -- Gap #4b _PTS/_WAK infrastructure — needed for S1-S4 sleep. -- Gap #7 thermal/power enumeration — needs `_TZ` iteration + EC wiring. - ---- - -## Phase D outcome — 2026-06-30 - -**Status: ✅ COMPLETE. Linux 7.1 best-practices ported; consumer ported; ISO rebuilt; QEMU boot verified end-to-end.** - -Inner-base commit: `181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap` -Outer Red Bear commit: `5f1da5250 redbear-sessiond: port ACPI shutdown watcher to new Fd-based scheme` - -### Linux 7.1 cross-reference findings (the canonical reference) - -After auditing Linux 7.1's ACPI sleep infrastructure -(`drivers/acpi/acpica/hwxfsleep.c`, `drivers/acpi/acpica/hwesleep.c`, -`drivers/acpi/sleep.c`), the canonical "enter a sleep state" pattern is: - -1. `acpi_enter_sleep_state_prep(sleep_state)` (linux/drivers/acpi/acpica/hwxfsleep.c:200): - - Read SLP_TYPa/SLP_TYPb from `_Sx` package via `acpi_get_sleep_type_data` - - Save S0's SLP_TYP values for restore on wake - - **Evaluate `\_PTS(sleep_state)`** (line 222) — "Prepare To Sleep" AML method - - **Evaluate `\_SST(sst_value)`** (line 265) — "System Status" indicator - (working=0, sleeping=1, sleep-context=2, indicator-off=7) -2. Write `SLP_EN | SLP_TYPa` to PM1a, `SLP_EN | SLP_TYPb` to PM1b -3. On wake, evaluate `\_WAK(sleep_state)` and restore S0's SLP types - -There's also a separate `\_TTS` (Transition To State) for the reboot path -(`acpi_sleep_tts_switch` in `drivers/acpi/sleep.c:36`). - -### Changes applied (Phase D) - -**`drivers/acpid/src/acpi.rs` (refactored `set_global_s_state`)** - -Follows the Linux 7.1 `acpi_enter_sleep_state` pattern with 5 steps: - -1. Look up `_Sx` package in AML namespace (was hardcoded to `_S5`) -2. Evaluate `_PTS(state)` via new `aml_evaluate_simple_method` helper -3. Evaluate `_SST(sst_value)` with ACPI_SST_* constants -4. Write `SLP_EN|SLP_TYPa` to PM1a, `SLP_EN|SLP_TYPb` to PM1b -5. Spin - -Generic now — works for any Sx (not just S5). S1-S4 paths still don't -fully work (no `_WAK`, no P-state preservation, no wakeup vector), but -the new generic structure means future `_WAK` implementation only needs -to add wakeup handling after step 4. - -**`drivers/acpid/src/acpi.rs` (new `aml_evaluate_simple_method` helper)** - -Mirrors Linux 7.1's `acpi_execute_simple_method` (drivers/acpi/utils.c). -Uses `evaluate_if_present` so missing methods return `Ok(None)` cleanly -instead of `AmlError::ObjectDoesNotExist`. Takes the AML global lock -with timeout 16 (matching the existing `aml_eval` pattern). - -**`drivers/acpid/src/scheme.rs` (thermal/power enumeration)** - -`thermal_zones()` enumerates `\_TZ.` children. `power_adapters()` -enumerates `PowerResource` objects. Both populate the previously-empty -`/scheme/acpi/thermal/` and `/scheme/acpi/power/` directories. - -**`local/recipes/system/redbear-sessiond/source/src/acpi_watcher.rs` (consumer port)** - -The previous `wait_for_shutdown_edge()` tried to open -`/scheme/kernel.acpi/kstop` (a file path that no longer exists after -Phase B). Rewrote to use the new Fd-based interface: -- `Fd::open("/scheme/kernel.acpi", O_CLOEXEC, 0)` to get the parent -- `.openat("kstop", O_CLOEXEC, 0)` to get the kstop sub-handle -- `call_ro(buf, empty, &[CheckShutdown as u64])` polled at 250ms - -Uses polling instead of `RawEventQueue` to avoid pulling in `redox_event` -(currently uses the removed `llvm_asm!` macro on newer Rust nightly). - -**Verified by:** redbear-mini ISO rebuilt cleanly (2026-06-30 06:28). -QEMU boot reaches `Red Bear login:` prompt with no errors. `redbear-sessiond` -is now working correctly: -- Line 133-135: `probing 2 D-Bus socket candidate(s)... registered - org.freedesktop.login1 on the system bus` -- ACPI shutdown watcher no longer errors - -### Final gap closure status - -| Gap | Description | Status | Phase | -|-----|-------------|--------|-------| -| #1 | RSDP checksum validation | ✅ Closed | A | -| #3 | AML mutex stubs | ✅ Closed | C | -| #4a | set_global_s_state genericity + warnings | ✅ Closed | C | -| #4 | set_global_s_state refactored to Linux 7.1 pattern | ✅ Closed | D | -| #4b | `_WAK` + `_TTS` AML hooks for S1-S4 | ✅ Closed (infrastructure) | E | -| #5 | SLP_TYPb PM1b write | ✅ Closed | C | -| #6 | parse_lnk_irc range validation | ✅ Closed | C | -| #7 | thermal/power enumeration | ✅ Closed | D | -| #8 | AcpiScheme fevent | ✅ Closed | A | -| nsmgr deadlock | setrens-before-ready | ✅ Closed | B | -| **#2** | DMAR init (opt-in, hard cap) | ✅ Closed (opt-in, 32-entry cap) | E | -| **#2 kernel** | DMAR real-hardware hang root-cause | 🔴 Open (out of scope: needs QEMU/hardware trace) | — | -| **#4b kernel** | FACS wakeup vector + S3 suspend assembly | 🔴 Open (out of scope: needs kernel-side long-mode assembly + FACS parsing) | — | - -**Result: 11 of 13 issues closed-or-improved, 2 still open (both require -hardware-specific work that can't be done in a QEMU-only session).** - -The "closed" status on #4b refers to the user-space side: the `_TTS` -and `_WAK` evaluation hooks now exist on `AcpiContext`, and a future -kernel-side S3 entry point can call `enter_sleep_state(state)` then -resume by calling `wake_from_s_state(state)`. The kernel-side -suspend-to-RAM implementation (FACS parsing, wakeup vector setup, -long-mode assembly to invoke `acpi_enter_sleep_state`) is out of -scope for this session. - -The "closed" status on #2 refers to the opt-in path: DMAR init can -now be enabled with `REDBEAR_DMAR_INIT=1` on hardware known to work, -with a hard cap of 32 entries preventing any infinite-iterator hang. -The root-cause investigation of the real-hardware hang is out of -scope (would require tracing MMIO reads on the affected hardware). - ---- - -## Phase E outcome — 2026-06-30 - -**Status: ✅ COMPLETE. _TTS/_WAK API surface added; DMAR opt-in unblocked; ISO rebuilt; QEMU boot verified end-to-end.** - -Inner-base commit: `181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap` - -### Changes applied (Phase E) - -**`drivers/acpid/src/acpi.rs` (three new methods on `AcpiContext`)** - -- `transition_to_s_state(state: u8)`: evaluates `_TTS(state)` AML - method. Mirrors Linux 7.1 `acpi_sleep_tts_switch` (drivers/acpi/sleep.c:36). - Called when the system transitions between sleep states. Failure - is non-fatal: `_TTS` is optional per ACPI spec. - -- `wake_from_s_state(state: u8) -> Result`: - evaluates `_WAK(state)` AML method. Mirrors Linux 7.1 - `acpi_sleep_finish_wake` (drivers/acpi/sleep.c). Called by - userspace on resume from a sleep state. The ACPI spec requires - the OS to call `_WAK` on the same state that was passed to - `_PTS` before the sleep. - -- `enter_sleep_state(state: u8)`: top-level entry point that calls - `_TTS` (Step 0, Linux 7.1) then `set_global_s_state` (Steps 1-5, - Phase D). This is the public API that future kernel S3/S4 paths - should use. - -**`drivers/acpid/src/acpi/dmar/mod.rs` (DMAR init hard-capped and opt-in)** - -- `Dmar::init()` now calls `Dmar::init_with(acpi_ctx, false)` for - safety (no-op by default). -- New `Dmar::init_with(acpi_ctx, opt_in)` takes an explicit boolean - that callers can set to true. -- The DRHD iteration has a hard cap of 32 entries (real hardware - has 1-4 DRHDs) to prevent any infinite-iterator hang. -- Caller in `AcpiContext::init` reads `REDBEAR_DMAR_INIT=1` from - the environment and passes that to `Dmar::init_with`. - -This unblocks DMAR on QEMU and on hardware known to work, while -keeping it safe-by-default on real hardware where the hang is -reproducible. - -### Verified by - -- `CI=1 ./local/scripts/build-redbear.sh redbear-mini` succeeded with - exit 0. ISO at `build/x86_64/redbear-mini.iso` (512 MB) at - 2026-06-30 07:11. -- QEMU boot reaches `Red Bear login:` prompt cleanly with no errors. - Both `@inputd:661` and `@ps2d:96` startup logs visible - (Phase A). `redbear-sessiond` working with login1 registered - on D-Bus (Phase B). - -### Out-of-scope (requires hardware work) - -- **Gap #2 root-cause** — DMAR init opt-in is unblocked. The actual - real-hardware hang root-cause would require tracing MMIO reads on - the affected hardware. -- **Gap #4b kernel side** — `_WAK` evaluation hook is in place. - The kernel-side FACS parsing + wakeup vector + long-mode assembly - for S3 suspend is out of scope for this session. - -### Commit chain - -``` -5f1da5250 redbear-sessiond: port ACPI shutdown watcher to new Fd-based scheme -181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap -8140a2c base: refactor set_global_s_state to follow Linux 7.1 acpi_enter_sleep_state -d844111 base: close SLP_TYPb, parse_lnk_irc, AML mutex, and S5 gaps -ae57fe3 base: re-sync ACPI userspace with upstream master -4f2a043 kernel: re-sync ACPI subsystem with upstream master -de9d1f4 base: ps2d/inputd — add startup info logs for boot diagnostics -``` - -Inner kernel fork: `local/sources/kernel/` at `4f2a043`. -Inner base fork: `local/sources/base/` at `8140a2c`. \ No newline at end of file diff --git a/local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md b/local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md deleted file mode 100644 index 26d6534f01..0000000000 --- a/local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md +++ /dev/null @@ -1,356 +0,0 @@ -# Canonical Build System Improvement Proposal - -**Status:** Proposal — not yet implemented -**Context:** Identified while debugging the `redbear-mini` boot crash chain (`require_zero_offset`, `ContextHandle::Start`, `sys_statfs cbindgen.toml`, `SYS_MKNS`/`SYS_SETNS`, `mmap_min_addr` capping). The crash fixes were correct, but getting them through the build pipeline was painful and error-prone. This document proposes specific improvements to make that workflow reliable. - ---- - -## Problems Encountered (Root-Caused) - -1. **Edits in `local/sources//` are silently stashed** by `build-redbear.sh` (line 160-172), then never restored. Build runs against the clean HEAD. User loses WIP from the working tree. -2. **`--force-rebuild` bypasses the BLAKE3 content-hash cache** (cook_build.rs:462) and all 14 validation gates in `build-redbear.sh`. -3. **Manual `repo cook` outside `build-redbear.sh`** skips stale detection, prefix rebuild, pre-cook sequencing, source-fingerprint tracking, and CI=1 (TUI protection). -4. **Source invalidation is mtime-only** (cook_build.rs:425-442). A `git checkout` or `cp -a` that preserves timestamps produces a stale cache. No content hash is computed for any source file. -5. **Cookbook binary staleness** (build-redbear.sh:182-185): only checks existence, not whether `src/` has changed since last build. -6. **No failure cleanup** in `build-redbear.sh`: no `trap` handler, no signal handler, partially-built artifacts left in `repo/` after failure. -7. **No strict-by-default gate** for uncommitted edits: `verify-durable-source-edits.py` only blocks under `--strict`, which is not the default. -8. **`recipes/core/relibc/source` is stashed but the other 5 fork sources are not** — silent inconsistency. - ---- - -## Proposed Improvements (Ranked by Impact) - -### Improvement 1: Make `stash_nested_repo_if_dirty` restore on exit - -**Where:** `local/scripts/build-redbear.sh:160-172` -**Impact:** HIGH — eliminates the "where did my edits go?" confusion -**Complexity:** LOW - -The current `stash_nested_repo_if_dirty` pushes a stash with `--all` and never restores it. Replace with a `trap`-based push-and-restore pattern: - -```bash -REDBEAR_STASHED_REPOS=() - -stash_nested_repo_if_dirty() { - local target_dir="$1" - local label="$2" - if [ -d "$target_dir/.git" ]; then - if ! git -C "$target_dir" diff --quiet || \ - ! git -C "$target_dir" diff --cached --quiet || \ - [ -n "$(git -C "$target_dir" ls-files --others --exclude-standard)" ]; then - echo ">>> Stashing dirty $label checkout..." - rm -f "$target_dir/.git/index.lock" - if git -C "$target_dir" stash push --all \ - -m "build-redbear-auto-stash-$(date +%s)" >/dev/null 2>&1; then - REDBEAR_STASHED_REPOS+=("$label:$target_dir") - fi - fi - fi -} - -restore_all_stashes() { - local rc=$? - local entry - for entry in "${REDBEAR_STASHED_REPOS[@]}"; do - local label="${entry%%:*}" - local dir="${entry#*:}" - if [ -d "$dir/.git" ]; then - echo ">>> Restoring $label working tree..." - rm -f "$dir/.git/index.lock" - git -C "$dir" stash pop >/dev/null 2>&1 || \ - echo "WARN: failed to restore $label stash — check 'git -C $dir stash list'" - fi - done - exit $rc -} - -trap restore_all_stashes EXIT -``` - -Apply this to ALL fork sources, not just relibc: - -```bash -for fork in relibc kernel base bootloader installer redoxfs; do - local_fork_dir="$PROJECT_ROOT/local/sources/$fork" - if [ -d "$local_fork_dir/.git" ]; then - stash_nested_repo_if_dirty "$local_fork_dir" "$fork" - fi -done -``` - ---- - -### Improvement 2: Source-content hashing for cache invalidation - -**Where:** `src/cook/cook_build.rs:425-442` and `src/cook/fs.rs:160-168` -**Impact:** HIGH — eliminates the "stale cache after git checkout" bug class -**Complexity:** MEDIUM - -Replace the mtime-based `modified_dir_ignore_git()` with a content-hash approach that covers all meaningful inputs: - -```rust -// In src/cook/fs.rs or a new src/cook/source_hash.rs -fn compute_source_content_hash(source_dir: &Path, recipe_toml: &Path, patches: &[Path]) -> String { - let mut hasher = blake3::Hasher::new(); - - // Hash source directory contents (file-by-file, sorted by path) - if let Ok(entries) = walk_dir_files_sorted(source_dir) { - for (rel_path, abs_path) in entries { - // Skip .git, target/, *.swp, etc. - if rel_path.starts_with(".git/") || rel_path.contains("/target/") { - continue; - } - hasher.update(rel_path.as_bytes()); - hasher.update(b"\0"); - if let Ok(content) = std::fs::read(&abs_path) { - hasher.update(&content); - } - hasher.update(b"\0"); - } - } - - // Hash recipe.toml - if let Ok(content) = std::fs::read(recipe_toml) { - hasher.update(b"recipe.toml\0"); - hasher.update(&content); - } - - // Hash each patch file - for patch in patches { - if let Ok(content) = std::fs::read(patch) { - hasher.update(patch.file_name().unwrap().as_encoded_bytes()); - hasher.update(b"\0"); - hasher.update(&content); - } - } - - hasher.finalize().to_hex().to_string() -} -``` - -In `build()` (cook_build.rs:425-460): - -```rust -// Replace mtime-based source_modified with content hash -let source_hash = compute_source_content_hash(source_dir, &recipe_dir.join("recipe.toml"), &patches); -let stored_hash_file = get_sub_target_dir(target_dir, "source_hash.txt"); - -let source_changed = match std::fs::read_to_string(&stored_hash_file) { - Ok(stored) if stored.trim() == source_hash => false, - _ => true, -}; - -// ... after successful build: -std::fs::write(&stored_hash_file, &source_hash)?; -``` - -This makes the cache invalidation robust against `git checkout`, `cp -a`, and any other operation that changes content without updating mtime. It also eliminates the cbindgen.toml-specific fragility: the generated headers' source is the `*.rs` files + `cbindgen.toml`, all of which are hashed. - ---- - -### Improvement 3: Cookbook binary freshness check - -**Where:** `local/scripts/build-redbear.sh:182-185` -**Impact:** MEDIUM — prevents stale binary causing silent failures -**Complexity:** LOW - -```bash -COOKBOOK_SRC_FINGERPRINT="$PROJECT_ROOT/target/release/.cookbook-src-fingerprint" -COOKBOOK_BIN="$PROJECT_ROOT/target/release/repo" -COOKBOOK_SRC_HASH=$(find "$PROJECT_ROOT/src" -name "*.rs" -print0 | sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1) - -NEEDS_REBUILD=0 -if [ ! -f "$COOKBOOK_BIN" ]; then - NEEDS_REBUILD=1 -elif [ ! -f "$COOKBOOK_SRC_FINGERPRINT" ] || [ "$(cat "$COOKBOOK_SRC_FINGERPRINT")" != "$COOKBOOK_SRC_HASH" ]; then - NEEDS_REBUILD=1 -fi - -if [ "$NEEDS_REBUILD" = "1" ]; then - echo ">>> Rebuilding cookbook binary (source changed or missing)..." - cargo build --release - echo -n "$COOKBOOK_SRC_HASH" > "$COOKBOOK_SRC_FINGERPRINT" -fi -``` - ---- - -### Improvement 4: Strict-by-default uncommitted-edit gate - -**Where:** `local/scripts/build-redbear.sh` and `local/scripts/verify-durable-source-edits.py:48` -**Impact:** MEDIUM — prevents "what was that warning?" confusion -**Complexity:** LOW - -Add an explicit gate before any source-touching operation: - -```bash -# After .config parsing, before any fetch/cook -echo ">>> Checking for uncommitted source edits..." -DIRTY_FORKS=() -for fork in local/sources/relibc local/sources/kernel local/sources/base \ - local/sources/bootloader local/sources/installer local/sources/redoxfs; do - fork_dir="$PROJECT_ROOT/$fork" - if [ -d "$fork_dir/.git" ] && \ - (! git -C "$fork_dir" diff --quiet HEAD 2>/dev/null || \ - ! git -C "$fork_dir" diff --cached --quiet HEAD 2>/dev/null || \ - [ -n "$(git -C "$fork_dir" ls-files --others --exclude-standard 2>/dev/null)" ]); then - DIRTY_FORKS+=("$fork") - fi -done - -if [ ${#DIRTY_FORKS[@]} -gt 0 ]; then - echo "WARNING: uncommitted edits detected in:" - for f in "${DIRTY_FORKS[@]}"; do - echo " - $f" - done - echo "" - if [ "${REDBEAR_ALLOW_DIRTY:-0}" = "1" ]; then - echo ">>> REDBEAR_ALLOW_DIRTY=1 set; proceeding with WIP edits." - else - echo "ERROR: refuse to build with uncommitted edits." - echo " Either commit your changes or set REDBEAR_ALLOW_DIRTY=1 to override." - exit 1 - fi -fi -``` - -For `verify-durable-source-edits.py`, flip the default to strict and make `--no-strict` the override: - -```python -# Line 48: change default from non-strict to strict -if not args.no_strict and dirty: - print("ERROR: uncommitted edits detected in upstream-owned source trees.") - print(" Use --no-strict to allow, or commit your changes.") - sys.exit(1) -``` - ---- - -### Improvement 5: Stash-all-forks consistency - -**Where:** `local/scripts/build-redbear.sh:160-172` -**Impact:** MEDIUM — eliminates the "only relibc is special" asymmetry -**Complexity:** LOW - -Combined with Improvement 1, replace the single-relibc stash with a loop over all fork sources. The key insight: `local/sources//` is the source of truth, but the cookbook fetches into `recipes/core//source` (which is a symlink to `local/sources//`). Stashing must happen at the `local/sources//` level, not at the recipe-level symlink. - ---- - -### Improvement 6: Failure-cleanup trap - -**Where:** `local/scripts/build-redbear.sh` (top-level, after `set -euo pipefail`) -**Impact:** MEDIUM — makes failed builds debuggable -**Complexity:** LOW - -```bash -REDBEAR_BUILD_STATE_DIR=$(mktemp -d) - -cleanup_on_failure() { - local rc=$? - if [ $rc -ne 0 ]; then - echo "" - echo "========================================" - echo " BUILD FAILED (exit code: $rc)" - echo "========================================" - echo "Diagnostic info preserved at: $REDBEAR_BUILD_STATE_DIR" - echo "" - echo "Last 30 lines of each recipe log:" - for log in /tmp/build-*.log; do - [ -f "$log" ] || continue - echo "--- $log ---" - tail -30 "$log" - echo "" - done 2>/dev/null - echo "Recipe build state:" - for d in "$PROJECT_ROOT"/recipes/*/target/*/; do - [ -d "$d" ] || continue - echo " $d: $(du -sh "$d" 2>/dev/null | cut -f1)" - done - fi - rm -rf "$REDBEAR_BUILD_STATE_DIR" - # Then restore stashes (from Improvement 1) - restore_all_stashes -} -trap cleanup_on_failure EXIT -``` - ---- - -### Improvement 7: Pre-cook offline/upstream consistency - -**Where:** `local/scripts/build-redbear.sh:339-357` -**Impact:** LOW-MEDIUM — prevents pre-cook from fetching packages the main build can't use -**Complexity:** LOW - -```bash -# Replace line 349 -if [ "${REDBEAR_ALLOW_UPSTREAM:-0}" = "1" ]; then - COOKBOOK_OFFLINE=false "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1 -elif [ -n "${REDBEAR_RELEASE:-}" ]; then - COOKBOOK_OFFLINE=true "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1 -else - COOKBOOK_OFFLINE=true "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1 -fi -``` - ---- - -### Improvement 8: Cookbook protection against out-of-band invocations - -**Where:** `src/bin/repo.rs` (CLI entry point) -**Impact:** LOW — prevents accidental misuse, gentle guardrail -**Complexity:** LOW - -Add a non-blocking warning when `repo cook` is invoked outside `build-redbear.sh`: - -```rust -// In src/bin/repo.rs main_inner() -let canonical_marker = std::env::var_os("REDBEAR_CANONICAL_BUILD"); -if canonical_marker.is_none() { - eprintln!("WARNING: repo cook invoked outside build-redbear.sh."); - eprintln!(" Cache invalidation, prefix rebuild, and source-fingerprint"); - eprintln!(" tracking will not run. Set REDBEAR_CANONICAL_BUILD=1 to suppress"); - eprintln!(" this warning if you know what you're doing."); -} -``` - -In `build-redbear.sh`: -```bash -export REDBEAR_CANONICAL_BUILD=1 -``` - ---- - -## Implementation Order - -Suggested order (each step is independently useful and can be merged separately): - -1. **Improvement 1 + 5** (trap-based stash restore, all forks) — eliminates the biggest UX pain point -2. **Improvement 6** (failure-cleanup trap with diagnostics) — makes failures debuggable -3. **Improvement 3** (cookbook binary freshness) — 5-line change, high signal -4. **Improvement 4** (strict-by-default dirty gate) — prevents repeat-accumulation of uncommitted edits -5. **Improvement 7** (pre-cook offline consistency) — small correctness fix -6. **Improvement 8** (cookbook out-of-band warning) — gentle nudge -7. **Improvement 2** (source content hashing) — biggest engineering effort, defer until after the above land - ---- - -## Testing Strategy - -Each improvement needs: -1. Unit test in `src/` if it touches cookbook code (cookbook has a test directory) -2. Manual smoke test: `./local/scripts/build-redbear.sh redbear-mini` end-to-end -3. Regression test: dirty `local/sources/relibc/` + `./local/scripts/build-redbear.sh redbear-mini` + verify stash is restored on exit -4. Documentation update: this file's "Problems Encountered" should be empty after all improvements land - ---- - -## Related Documents - -- `local/AGENTS.md` — project-wide build and commit policies -- `local/docs/LOCAL-FORK-SUPREMACY-POLICY.md` — why local forks must be complete and committed -- `local/docs/BUILD-CACHE-PLAN.md` — content-hash cache design (Phase 1-3 status, sysroot deferral) -- `local/docs/BUILD-SYSTEM-INVARIANTS.md` — what the build system guarantees -- `local/docs/BUILD-SYSTEM-HARDENING-PLAN.md` — broader hardening roadmap -- `mk/repo.mk` — make-integration of cookbook -- `local/scripts/build-redbear.sh` — canonical entry point \ No newline at end of file diff --git a/local/docs/BUILD-SYSTEM-IMPROVEMENTS.md b/local/docs/BUILD-SYSTEM-IMPROVEMENTS.md index 8e0b7bcf95..7d2d5d2266 100644 --- a/local/docs/BUILD-SYSTEM-IMPROVEMENTS.md +++ b/local/docs/BUILD-SYSTEM-IMPROVEMENTS.md @@ -1,5 +1,10 @@ # Build System Improvements — v6.0 Post-Mortem (2026-06-12) +> **Status (2026-07-18): HISTORICAL POST-MORTEM.** This is a dated record of +> the v6.0 desktop bring-up cycle. Some items below were implemented, others +> abandoned; it is not a list of pending work. For current build-system work +> see `BUILD-SYSTEM-HARDENING-PLAN.md` and `local/scripts/TOOLS.md`. + This document analyzes the build system gaps that surfaced during the v6.0 KDE/Qt/Plasma desktop path bring-up (2026-04 through 2026-06) and proposes targeted, low-risk improvements. Each improvement is sized as diff --git a/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md b/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md deleted file mode 100644 index 3ce67df3c0..0000000000 --- a/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md +++ /dev/null @@ -1,313 +0,0 @@ -# Red Bear OS Build-System v6.0 Hardening — Post-Mortem - -> **Scope.** This document is the durable record of the -> 14-session v6.0 build-system hardening work arc (2026-06-08 to -> 2026-06-12). It captures the 10 build-system improvements -> (9 DONE, 1 OPEN), 32 findings addressed, the Gitea Actions CI -> pipeline, the 149-test suite covering all 17 classifier rules + -> their false-positive inverses plus the 6 new status and 7 new -> scheduler Rust unit tests, and the deferred follow-up work. -> The 7,000+ uncommitted file modifications in the user's working -> tree are not part of this post-mortem — they are ongoing WIP. -> -> **Durability caveat (added 2026-06-12 after final review).** -> The deliverables in this arc are durable **on disk** in the -> working tree, and most are now durable in `git` history. The 13 -> most recent commits on `0.2.3` (`b8c1c780d`, `975cda686`, -> `e1c2e7958`, `0f8ad8a50`, `9e5794ea7`, `827895d32`, `693e4d774`, -> `fbc32a6d8`, `5325360b4`, `ae749ffb2`, `97fa3a17a`, `bd18eefc6`, -> `03c8a38a1`) cover the first durable C-7 migration patch -> (`local/patches/kf6-karchive/01-initial-migration.patch`); the -> `make lint-build-system-all` aggregate + 11-job CI; the -> postmortem rebalance to 13-session / 9.5-DONE; the -> `make scratch-rebuild` target wiring; the improvement #10 -> scratch-rebuild skeleton + 21 tests + Makefile + Gitea CI -> integration; the migration-dry-run CI integration; the C-7 KF6 -> sed migration script v2 + 13 tests + Makefile + Gitea CI -> integration; the postmortem update to 13-session / 9.5-DONE -> / 120-Python-test state; the parallel cook pool; the cook -> status reporter; the build-system hardening arc (5 of 10 -> improvements); the BUILD-SYSTEM-IMPROVEMENTS.md doc; -> `classify-cook-failure.py`; `audit-patch-idempotency.py`; and -> the auto-link Qt sysroot dirs patch in `src/cook/script.rs`. -> The remaining v6.0 deliverable still in `git status` is this -> BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md (updated to 14-session -> / 9.5-DONE / 122-Python-test state, with the Session 14 -> entry + the b8c1c780d commit-history row). The C-1..C-6 doc -> and code fixes, `boot-logs/README.md`, and -> `migrate-kf6-seds-to-patches.sh` (v1) were committed in -> `ae749ffb2`; v2 rewrite of the script + 13 tests + Makefile + -> Gitea CI integration in `827895d32`; the #10 scratch-rebuild -> skeleton + 21 tests + Makefile + Gitea CI integration in -> `0f8ad8a50`; the migration-dry-run CI integration in -> `9e5794ea7`; the make-wrapper + postmortem rebalance in -> `e1c2e7958`; the lint-build-system-all aggregate + 11-job CI -> in `975cda686`; the first durable C-7 patch + migration -> script v3 fix in `b8c1c780d`. Going forward, any new v6.0 -> work should be committed with -> `git add ` to avoid sweeping the user's -> 7,000+ unrelated WIP modifications. - -## Timeline - -| Session | Date (2026) | Focus | -|---------|------------|-------| -| 1 | 06-08 | v6.0 policy compliance pass: 10 build-system improvements, 4 audit scripts, 7 make lint targets, 31 tests | -| 2 | 06-09 | Comprehensive doc cleanup: 12/12 docs pass review, 4 high-priority fixes (AGENTS.md, local/AGENTS.md, README.md, SCRIPT-BEHAVIOR-MATRIX.md), 3 file deletions | -| 3 | 06-10 | P0 audit-script hardening: 5 reviewer findings fixed, doc reconciliation | -| 4 | 06-11 | Audit script review + comprehensive review: 32 findings categorized CRITICAL/HIGH/MEDIUM/LOW, comprehensive fix pass | -| 5 | 06-12 | Final refinements: `local/AGENTS.md:367` reframed, KF6 migration tool created, deferred work documented | -| 6 | 06-12 (cont.) | Hidden risk fix: cub-assessment deleted (874 MB), migrate-kf6-seds-to-patches.sh +x, postmortem accuracy corrections | -| 7 | 06-12 (cont.) | Test coverage gap: added 12 missing positive rule tests + 12 false-positive tests (55/55 pass). Discovered + fixed 6 over-broad multi-pattern rules. Created `.gitea/workflows/build-system.yml` (7-job Gitea Actions pipeline, host-execution, Manjaro/Arch) and `.gitea/RUNNER-SETUP.md` (one-time host setup). Wired into `make test-lint-scripts[-quiet]`. | -| 8 | 06-12 (cont.) | **Build-system improvement #2 shipped**: `local/scripts/repair-cook.sh` (incremental-build optimizer, 134 lines) + 7 unit tests (`local/scripts/tests/test_repair_cook.py`). `make repair.` and `make clean-repair.` targets wired. Verified P0 audit-script fixes work on real upstream KF6 source (form 1 nested namespace, comment/string strip, .bak timestamp). **Test count: 62/62 pass.** | -| 9 | 06-12 (cont.) | **Build-system improvement #5 shipped**: `local/scripts/lint-recipe.py` (380 lines, 7 rules) + 24 unit tests (`local/scripts/tests/test_lint_recipe.py`). Recipe-index precomputation drops `--all` runtime from 60s+ to 1.1s. `make lint-recipe`, `make lint-recipe.`, `make lint-recipe.strict`, `make lint-recipe..strict` wired. New `lint-recipe` Gitea Actions job (job 4 of 8) added to `.gitea/workflows/build-system.yml`. First run on the live tree found: 1 broken-patch reference (`redbear-sessiond/P4-signal-implementations.patch`), 1 dangling `cookbook_apply_patches` call (`tc`), 19 sed -i calls in sddm (warning only — `cookbook_apply_patches` present), 4 sed -i calls in `qt6-wayland-smoke` (uncovers the kind of bug the libwayland fix was preventing). **Test count: 86/86 pass.** | -| 10 | 06-12 (cont.) | **Build-system hardening arc commit + improvement #4 shipped.** First durable commit: `ae749ffb2 build: ship build-system hardening arc (5 of 10 improvements)` — 22 build-system files, including the 5 prior arc deliverables (audit-kf6-deps.py + 13 tests, repair-cook.sh + 7 tests, migrate-kf6-seds-to-patches.sh, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, build-system.yml, Gitea RUNNER-SETUP.md, libdrm/02 sidecar, cache/README cleanup, Makefile lint/repair targets). Then `5325360b4 build: add cook status reporter (improvement #4)` — `src/cook/status.rs` (197 lines, 6 unit tests) + `src/bin/repo.rs` wiring. Auto-enables in `CI=1` mode when stderr is a TTY: one-line `[NN/MM] recipe: phase (Xs)` output for each cook. Verified end-to-end with 3-recipe and 5-recipe real cooks. **Test count: 86/86 Python + 20/20 Rust.** | -| 11 | 06-12 (cont.) | **Build-system improvement #1 shipped.** `src/cook/scheduler.rs` (145 lines, 7 unit tests) + `src/bin/repo.rs` `repo cook --jobs=N` flag. Dep-aware level partition via `dep_levels()` (each recipe's level = `1 + max(level of any direct dep in this vec)`, or 0 if no deps in the vec). For each level, runs all recipes in that level via `std::thread::scope` with up to `N` workers. Drain-after-spawn pattern keeps live-worker count <= jobs. Ratatui TUI path unchanged. 7 unit tests cover empty / single / linear / independent / diamond / dev_dependencies / unknown-dep. Verified end-to-end: 5-recipe batch (redbear-statusnotifierwatcher, redbear-traceroute, redbear-udisks + deps expat, dbus) cooks in level 0 (3 parallel) → level 1 (dbus) → level 2 (redbear-udisks). On clean 3-recipe rebuild: 48s serial vs 45s parallel. Speedup bounded by longest single build (17s) on this small batch — the 2-3x gain from the proposal is on 15-recipe KF6 batches with 5-10 min longest builds. Caveat: `build/qt-host-build` host toolchain not yet locked; v2 mitigation is `flock` in `src/cook/script.rs` (deferred, no current redbear-full test recipe triggers qt-host-build). **Test count: 86/86 Python + 27/27 Rust.** | -| 12 | 06-12 (cont.) | **C-7 KF6 sed migration script v2 + CI integration.** The v1 shipped in `ae749ffb2` was a stub with three structural bugs that made it unrunnable: called `repo cook ` with a path (cookbook takes bare names); created an empty pristine_dir via mktemp -d but never populated it; Step 4 was `SKIP — manual rewrite pending` so the script wrote no patch even when the inline sed chains actually edited the source. Replaced with a working v2: bare-name cookbook CLI, real pristine-source snapshot (`cp -r source/ source-pristine/`) BEFORE the cook, real diff capture, real patch save to `local/patches//01-initial-migration.patch`. Added `--dry-run` for safe CI smoke testing, `--recipe=` and `--limit=N` for targeted runs, `--help` for the script's contract. Test escape hatch via `REDBEAR_MIGRATE_RECIPES_DIR` / `REDBEAR_MIGRATE_PATCHES_DIR` env vars so the candidate discovery can be exercised on synthetic trees without touching the live project. 13 unit tests in `local/scripts/tests/test_migrate_kf6_seds.py` — 7 candidate-discovery tests (synthetic tree with `make_recipe()` helper, asserts stdout/stderr + exit code) + 6 script-structure tests (regression guards against the v1 bugs: "uses bare names not paths", "uses release/repo binary", "creates patches dir", "diff includes .git/target excludes", "unfetches after capture", "idempotent SKIP when patch exists"). Wired into `make test-migration-dry-run` and new Gitea Actions job `migration-dry-run` (job 5 of 9, every PR). **Test count: 99/99 Python + 27/27 Rust.** Verified `--dry-run --limit=5` correctly identifies `breeze`, `kde-cli-tools`, `kdecoration`, `kf6-attica`, `kf6-karchive` as the first 5 of 56 candidate recipes. The actual migration run still requires the full KF6 dep chain to be built (qtbase, qtdeclarative, kf6-extra-cmake-modules, plus per-recipe deps); the per-recipe verification + recipe-rewrite remains a manual step (the script's `Next steps:` output documents this). | -| 13 | 06-12 (cont.) | **Improvement #10 (scratch-rebuild) M-sized foundation + CI integration.** The L-sized #10 proposal is split: the M-sized foundation (autotools detection + dep closure + targeted clean + parallel rebuild) is now a runnable 190-line bash script (`local/scripts/scratch-rebuild.sh`); the L-sized remaining work (full integration with `rebuild-cascade.sh`, byte-identical rebuild check via `stage.pkgar` hash diffing, cross-host-toolchain case) is documented but deferred. The script: (1) discovers autotools-using recipes by content regex + AUTOTOOLS_CORE list, (2) computes transitive closure via BFS over the recipe TOML dep graph (both `[build].dependencies` and `[build].dev_dependencies`), (3) deletes `target//{build,sysroot,stage.tmp}/` per recipe in the closure (preserves `source/` to avoid re-fetch), (4) re-cooks in dep order via the cookbook's `--jobs=N` flag. Cook errors during step 4 do NOT abort the script — a failed cook may indicate a missing upstream dep on a fresh checkout rather than a real bug. `--dry-run`, `--jobs=N`, `--help` supported. 21 unit tests in `local/scripts/tests/test_scratch_rebuild.py`: 3 autotools-core list tests, 8 regex content-match tests (each canonical autotools command + negative cases), 4 dep-parser tests, 1 help test, 5 script-structure tests (executable bit, uses release/repo, preserves source/, uses --jobs=N, dry-run safe). The test file also surfaced a real Python regex gotcha: `^[[:space:]]*` (POSIX char class with quantifier) silently fails to match the empty string under Python's regex engine; the test uses `^[\s]*` shorthand instead. Wired into `make test-scratch-dry-run`, `make scratch-rebuild`, and new Gitea Actions job `scratch-dry-run` (job 6 of 10). **Test count: 120/120 Python + 27/27 Rust.** Verified `--dry-run` against live tree: discovers 6 autotools users (bison, diffutils, flex, grub, libtool, m4) and computes a 6-recipe closure. `make scratch-rebuild` end-to-end: deletes the 6 recipe dirs' build/sysroot/stage.tmp, runs `repo cook --jobs=4` on the closure. m4 cooks successfully; bison fails (missing host toolchain dep) — the failure is correctly captured to the log without aborting the script. | -| 14 | 06-12 (cont.) | **First durable C-7 KF6 sed migration patch (`b8c1c780d`).** Executed the v2 migration script against `local/recipes/kde/kf6-karchive` and shipped `local/patches/kf6-karchive/01-initial-migration.patch` (24 lines, 1 real hunk). The patch is the result of capturing the diff between the pristine upstream karchive-6.26.0 tarball and the post-cook state — the `ecm_install_po_files_as_qm` line is now commented out (the recipe's `sed -i 's/^ecm_install_po_files_as_qm/#ecm_install_po_files_as_qm/'`). The other 3 sed chains in the recipe (ki18n_install, arg(mode), arg(d->mode)) were no-ops against this upstream version — the migration correctly captures only the real edits. Discovered a real bug in the v2 migration script during the first run: it was producing silently empty diffs on already-cooked recipes because the cookbook's `fetch` re-uses an existing source/ tree. Fix: add explicit `unfetch` BEFORE the `fetch` (so the source/ dir is removed before re-extraction) + set `REDBEAR_ALLOW_LOCAL_UNFETCH=1` (the cookbook's default policy is to never clobber a local-overlay source, and the env var overrides that for the explicit unfetch). The patch was also filtered to drop ECM-autogenerated noise files (`.clang-format`, `.gitignore`, `target/` artifacts); 122-line raw diff → 24-line filtered patch. Added 2 regression tests: `test_sets_local_unfetch_env_var` (guards against forgetting the env var) and `test_unfetches_before_fetching` (guards against the silent-failure mode). **Test count: 122/122 Python + 27/27 Rust.** Next steps for kf6-karchive (manual, not part of this commit): edit `[build].script` to remove the 4 sed chains + add `REDBEAR_PATCHES_DIR` / `cookbook_apply_patches` invocation, re-cook, verify byte-identical `stage.pkgar`, commit recipe rewrite alongside the patch. |## Final state - -### 10 build-system improvements — 9 DONE, 1 OPEN - -| # | Title | Status | Commit | -|---|-------|--------|--------| -| 3 | Per-recipe patch idempotency auditor | **DONE** | `03c8a38a1` | -| 6 | `recipes/kf6-*` recipe dep audit | **DONE** | `ae749ffb2` | -| 8 | Auto-link Qt sysroot dirs | **DONE** | `03c8a38a1` | -| 9 | Failure classifier | **DONE** | `bd18eefc6` | -| 1 | Parallel-safe cook pool | **DONE** | `fbc32a6d8` | -| 2 | `cook --repair` mode | **DONE** | `ae749ffb2` — `local/scripts/repair-cook.sh` wrapper + `make repair.` target | -| 4 | Cook TUI status | **DONE** | `5325360b4` — `src/cook/status.rs` | -| 5 | Build-time recipe lint | **DONE** | `ae749ffb2` — `local/scripts/lint-recipe.py` + 7-rule lint + Gitea CI job | -| 7 | QML gate (4-6 weeks) | open | — (Qt6 engine fix, not a cookbook improvement) | -| 10 | Cookbook scratch-rebuild | open | — (L-sized, 1 week, M risk; separate session) | - -### 32 findings — all addressed - -**5 P0 audit-script bugs (all fixed):** -- `audit-kf6-deps.py` Form 1 regex truncation of `KF6::Some::Nested::Name` → supports `::`-chained namespaces -- `audit-kf6-deps.py` comment / string-literal false positives → `_strip_cmake_noise` helper -- `audit-kf6-deps.py` `.bak` file silent overwrite on consecutive `--fix` runs → timestamped + collision-resistant -- `classify-cook-failure.py` rule-matching loop duplicated between text and JSON branches → `_match_rules` helper extracted -- `classify-cook-failure.py` `--json` exit-code inversion → documented and tested - -**6 additional over-broad multi-pattern rules fixed (Session 7 bonus, found while writing tests):** -- Rules 4 (Qt6::GuiPrivate), 5 (PlasmaWaylandProtocols), 10 (libc.so.6), 12 (Python3), 14 (Package), 16 (fetch denied) each had 2 patterns stored as a list but the matcher uses `all()` semantics. Real cooks fired only one of the two patterns so the rules NEVER fired. Collapsed to 1 pattern each. - -### Test coverage — 17/17 classifier rules + 12 false-positive inverses - -| Test | Count | Coverage | -|------|-------|----------| -| `test_audit_patch_idempotency.py` | 7 | 3 collect tests, 2 JSON schema tests, 2 name validation tests | -| `test_audit_kf6_deps.py` | 13 | 4 regex-form tests, 5 normalize tests, 1 WIP-skip test, 1 no-fetch honesty test, 1 KF6/Qt6 test, 1 component discovery test | -| `test_classify_cook_failure.py` | 35 | 17 positive rule tests (1 per rule), 12 false-positive tests, 5 existing exit-code/JSON/explain-rule tests, 1 --no-fetch honesty test | -| `test_repair_cook.py` | 7 | synthetic recipe fixtures, fast/slow path logic, --clean-build, REPAIR_FORCE | -| `test_lint_recipe.py` | 24 | 7 rule coverage, 1 recipe-index cache, 1 clean-recipe regression test, 1 error recipe test | -| `test_migrate_kf6_seds.py` | 13 | 7 candidate-discovery tests (synthetic tree, exit-code + stdout/stderr assertions) + 6 script-structure tests (regression guards against v1 bugs) | -| `test_scratch_rebuild.py` | 21 | 3 autotools-core list tests + 8 regex content-match tests (each canonical autotools command + negatives) + 4 dep-parser tests + 1 help test + 5 script-structure tests (executable, release/repo, preserves source/, --jobs=N, dry-run safe) | -| `cook::status` (Rust) | 6 | format_elapsed boundaries, disabled no-op, phase tracking | -| `cook::scheduler::dep_levels` (Rust) | 7 | empty / single / linear / independent / diamond / dev_dependencies / unknown-dep | - -**Total: 122/122 Python + 27/27 Rust pass in <1 second (Python) / ~3 seconds (Rust).** - -**8 CRITICAL findings (all addressed):** -- C-1 libwayland `patches = [redox.patch]` line removed (was blocking the Wayland stack) -- C-2 libdrm/02 broken hunk documented with sidecar README; regen procedure in `local/patches/libdrm/02-redox-dispatch.patch.README` -- C-3 orphan `local/sources/{pipewire,wireplumber}/` removed (22 MB) -- C-4 kernel `.gitignore` fixed to recursive `/target` -- C-5 broken driver symlinks re-pointed at `local/recipes/drivers/...` -- C-6 sddm stub headers documented as known maintenance debt in `local/recipes/kde/sddm/stubs/README.md` -- C-7 56 KF6 recipes with `sed -i` chains → **FULLY COMPLETE.** Migration script at `local/scripts/migrate-kf6-seds-direct.sh` (working without `repo cook`); cleanup scripts at `local/scripts/cleanup-kf6-noop-seds.sh` (24 recipes, full removal) and `local/scripts/cleanup-kf6-noop-seds-targeted.sh` (6 recipes, targeted removal); edit script at `local/scripts/edit-kf6-recipes-for-patches.sh`. **Result: 29 migration patches in `local/patches//` (all verified `git apply --check` clean), 24 recipes' `[build].script` rewritten to call `cookbook_apply_patches`, 30 NO-OP recipes with dead sed chains removed, 164 Python tests passing (8 unit test files + 2 e2e test files).** 7 remaining NO-OP recipes (breeze, kde-cli-tools, kf6-kbookmarks, kf6-kded6, kglobalacceld, plasma-desktop, plasma-workspace) had their non-ecm sed chains preserved (they target lines that ARE in upstream) — those chains are real Red Bear edits, not migration candidates. -- C-8 2.8 GB of unzipped source cleanup → deferred until C-7 patches are durable - -**7 HIGH findings (all addressed):** -- H-1 AGENTS.md ↔ local/AGENTS.md documentation map cross-references added -- H-2 duplicate `redbear-netctl-console/` removed -- H-3 redbear-meta header: false positive (declaration order matches) -- H-4 `cub/source/cub-assessment/` and `gparted-git/` removed (874 MB + 24 KB on disk) -- H-5/H-6/H-7 KF6 source state captured in C-7 migration plan - -**8 MEDIUM findings (all addressed or documented):** -- M-2 dead `validate-patches` Makefile target removed -- M-3 legacy config rename deferred (cosmetic) -- M-4 zbus build-ordering marker deferred (user knows) -- M-5 symlink consistency deferred (cosmetic) -- M-6 `make all` → `build-redbear.sh` routing deferred (preserves advanced/unsafe escape) -- M-7 `APPLY_PATCHES` var: false positive (real use at line 158) -- M-8 .bak files removed (libwayland + ncurses) - -**9 LOW findings (all addressed):** -- L-1..L-9: doc cleanup, dead code, cosmetic — all in the doc cleanup pass - -### Build-system test infrastructure — fully deployed - -| Artifact | Status | -|----------|--------| -| `local/scripts/audit-patch-idempotency.py` | 391 lines, exit 0/1/2 contract, JSON schema doc | -| `local/scripts/audit-kf6-deps.py` | 557 lines (4 regex forms), comment/string stripping, TOML-parser-based `--fix` | -| `local/scripts/classify-cook-failure.py` | 462 lines, 17 rules, `_match_rules` helper, `--explain-rule`, inverted exit code documented | -| `local/scripts/migrate-kf6-seds-to-patches.sh` | 6300-byte migration skeleton (NEW) | -| `local/scripts/tests/test_audit_patch_idempotency.py` | 7 tests | -| `local/scripts/tests/test_audit_kf6_deps.py` | 13 tests | -| `local/scripts/tests/test_classify_cook_failure.py` | 11 tests | -| `make lint-patches`, `make lint-patches-full` | wired to audit-patch-idempotency.py | -| `make lint-kf6-deps` | wired to audit-kf6-deps.py | -| `make lint-cook-failure`, `make lint-cook-failure-explain` | wired to classify-cook-failure.py | -| `make lint-build-system`, `make lint-build-system-full` | aggregate targets | - -**Test status:** 31/31 pass in <1 second. CI-safe exit codes (0=clean, 1=failures, 2=all-skipped). - -## Deferred to future sessions - -1. **C-7**: ✅ **DONE** (this update, 2026-06-12). 29 migration patches shipped + 24 recipes' build scripts rewritten to use `cookbook_apply_patches`. 164 Python tests pass. See C-7 entry above for full status. -2. **C-8**: 2.8 GB of unzipped source cleanup → ready to ship now that C-7 is complete. Run `find local/recipes -maxdepth 4 -name 'source.tar' -size +10M -delete` (verify with `du -sh local/recipes/*/source.tar | sort -h` first). Estimated 1 hour. - -2. **C-2 regen**: Regenerate `local/patches/libdrm/02-redox-dispatch.patch` against current libdrm 2.4.125. The 8-step procedure is documented in `local/patches/libdrm/02-redox-dispatch.patch.README`. Estimated 30-60 minutes if a libdrm build host is available. - -3. **6 open build-system improvements** (parallel cook, cook --repair, cook TUI, build-time lint, QML gate, cookbook scratch-rebuild) — each is a multi-session project on its own. - -4. **User's WIP**: 7,000+ file modifications in the working tree, primarily the user's ongoing KF6 work, cub improvements, redbear-netctl development, and libpciaccess integration. Not in scope for this post-mortem. - -## Commit history (v6.0 hardening arc, durable) - -The v6.0 deliverables were committed in 3 durable chunks on -2026-06-12, after the post-mortem was first written: - -| Commit | Files | What it landed | -|--------|-------|----------------| -| `ae749ffb2` | 22 | Build-system hardening arc: audit-patch-idempotency, audit-kf6-deps, classify-cook-failure, repair-cook, migrate-kf6-seds-to-patches, the 4 Python test files, BUILD-SYSTEM-IMPROVEMENTS.md, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, libdrm/02 sidecar README, cache/README deletion, .gitea/workflows/build-system.yml (8 jobs), .gitea/RUNNER-SETUP.md, Makefile lint + repair targets. C-1..C-6 + 7 HIGH findings all addressed. | -| `5325360b4` | 4 | Cook TUI status reporter (#4): `src/cook/status.rs` (197 lines, 6 unit tests), wired into `src/bin/repo.rs`. One-line `[NN/MM] recipe: phase (Xs)` output for non-TUI cooks. | -| `fbc32a6d8` | 4 | Parallel cook pool (#1): `src/cook/scheduler.rs` (145 lines, 7 unit tests), `--jobs=N` CLI flag in `src/bin/repo.rs`, `dep_levels()` topological partition via `std::thread::scope`. | -| `827895d32` | 2 | C-7 KF6 sed migration script v2: rewrote the v1 stub to actually capture diffs (`cp -r source/ source-pristine/` BEFORE the cook, real `diff -ruN`, real patch save to `local/patches//01-initial-migration.patch`). 13 unit tests in `test_migrate_kf6_seds.py`. | -| `9e5794ea7` | 4 | CI integration for the migration script: `make test-migration-dry-run` target + Gitea Actions `migration-dry-run` job (job 5 of 9 → 10). | -| `0f8ad8a50` | 5 | Improvement #10 (scratch-rebuild) M-sized foundation: `local/scripts/scratch-rebuild.sh` (190 lines, +x) + 21 unit tests in `test_scratch_rebuild.py`. `make test-scratch-dry-run` + `make scratch-rebuild` targets. Gitea Actions `scratch-dry-run` job (job 6 of 10). | -| `e1c2e7958` | 2 | Wired `make scratch-rebuild` (no-dry-run variant). Postmortem rebalance to 13-session / 9.5-DONE / 120-Python-test state. | -| `975cda686` | 4 | New `make lint-build-system-all` aggregate (offline-safe lints + tests + smoke). New Gitea Actions job (job 7 of 11). | -| `b8c1c780d` | 3 | First durable C-7 migration patch (`local/patches/kf6-karchive/01-initial-migration.patch` — 24 lines, 1 real hunk). Discovered + fixed the v2 script's silent-failure mode: cookbook `fetch` re-uses existing source/; migration must `unfetch` first + set `REDBEAR_ALLOW_LOCAL_UNFETCH=1`. Added 2 regression tests for the env-var + unfetch-before-fetch invariants. | - -**This post-mortem itself** is still in `git status` (modified -`local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md`); the -"Durability caveat" at the top of this document tracks its -shipment status. - -## What remains uncommitted - -| Path | What it is | Why uncommitted | -|------|-----------|-----------------| -| `local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md` | This doc updated for 14-session / 9.5-DONE / 122-Python-test state (Session 14 entry + b8c1c780d commit-history row) | Next user-chosen commit; touches paths the user may have other WIP for | -| `local/docs/BUILD-SYSTEM-FINGERPRINT-HARDENING-PLAN.md` | User WIP plan (Phase 6+, "pending Oracle fingerprint architecture review") | Draft, not for committing in this arc | -| User's `AGENTS.md`, `local/AGENTS.md`, `README.md`, `config/redbear-*.toml`, `local/sources/{base,bootloader,kernel}` | 7,000+ modifications | User WIP; not in this arc | - ---- - -## v6.1 Addendum (2026-06-21) — Cleanup Safety + qtdeclarative qfeatures.h Fix - -### Finding C-7.1: Ad-hoc cleanup scripts deleted tracked sources - -**Date:** 2026-06-21 -**Severity:** HIGH (caused build failure, required manual source restoration) -**Status:** FIXED - -#### Problem - -An ad-hoc cleanup script (`/tmp/cleanup-redbear.sh`) was used to free -disk space before a full rebuild. The script used `rm -rf` on -patterns that matched **tracked source files** under -`local/recipes/*/source/` and `local/sources/*/source/`. Specifically: - -- `local/recipes/libs/libepoxy/source/` (deleted) -- `local/recipes/libs/libdisplay-info/source/` (deleted) -- `local/recipes/libs/libxcvt/source/` (deleted) -- `local/recipes/libs/lcms2/source/` (deleted) -- `local/recipes/system/diskd/source/` (deleted) -- `local/recipes/system/devfsd/source/` (deleted) -- `local/patches/tokio/vendored/src/runtime/task/core.rs` (deleted) - -These sources were committed to git history (commits `f36b6a4582`, -`8782592b9a`, `7c49c45e0e`) and had to be manually restored via -`git checkout`. - -#### Root cause - -The cleanup script did not check whether paths were tracked by git -before deletion. The mainline Redox `make clean` and `make distclean` -targets DO check this, but the ad-hoc script bypassed them. - -#### Fix - -Added `local/scripts/cleanup-build.sh` — a git-aware cleanup script -that: - -1. Uses `git ls-files --error-unmatch` to check every path before - deletion -2. Skips and warns about tracked paths instead of deleting them -3. Supports three modes: `artifacts` (safest), `--all` (also removes - recipe `target/`), `--nuclear` (also removes local recipe - `target/`) -4. Refuses to run outside a git repository - -Updated `AGENTS.md` to document the safe cleanup procedure and -explicitly warn against ad-hoc `rm -rf`. - -### Finding C-7.2: qtdeclarative missing qfeatures.h caused `division by zero in #if` - -**Date:** 2026-06-21 -**Severity:** CRITICAL (blocked SDDM and KWin builds) -**Status:** FIXED - -#### Problem - -The `local/recipes/qt/qtdeclarative/recipe.toml` manually copied -include files from the build directory to the staged sysroot. This -missed the build-time generated `qfeatures.h` (and per-module -`*config.h` files like `qtquick-config.h`). - -Without `qfeatures.h`, the preprocessor expansion -`QT_CONFIG(quick_shadereffect)` and `QT_CONFIG(quick_draganddrop)` -in `qquickitem.h` (lines 119, 460) hit a deliberate `1/0` divide-by-zero -trap, producing these errors when compiling `sddm-greeter-qt6`: - -``` -qquickitem.h:119:5: error: division by zero in #if - 119 | #if QT_CONFIG(quick_shadereffect) -qquickitem.h:460:5: error: division by zero in #if - 460 | #if QT_CONFIG(quick_draganddrop) -``` - -#### Root cause - -Two issues in the qtdeclarative recipe: - -1. **Missing `cmake --install`** — qtbase, qtsvg, and other Qt module - recipes use `cmake --install` to properly stage all generated files - (including `qfeatures.h`). The qtdeclarative recipe skipped this - step and copied files manually. - -2. **Empty `qtquick-config.h`** — Even with `cmake --install`, the - `qtquick-config.h` file was being generated empty because the - recipe did not explicitly enable `QT_FEATURE_quick_shadereffect` - or `QT_FEATURE_quick_draganddrop`. - -#### Fix - -Three changes to `local/recipes/qt/qtdeclarative/recipe.toml`: - -1. Added `cmake --install . --prefix "${COOKBOOK_STAGE}/usr"` after - the build step to properly install all generated files. - -2. Added explicit `-DQT_FEATURE_quick_shadereffect=ON`, - `-DQT_FEATURE_quick_draganddrop=ON`, `-DQT_FEATURE_draganddrop=ON`, - `-DQT_FEATURE_quick_controls2=ON` to the CMake configuration. - -3. Added a safety-net block that regenerates `qtquick-config.h` - with all required feature definitions if CMake produces an empty - file (defensive programming for future Qt version changes). - -### Commits - -| Commit | What | -|--------|------| -| `57a3ea6c9` | qtdeclarative qfeatures.h fix (cmake --install + safety net) | -| `6a7abe0b87` | qtdeclarative qtquick-config.h fix (explicit feature flags + regen safety net) | -| (pending) | Safe cleanup script + AGENTS.md + CONSOLE-TO-KDE-DESKTOP-PLAN.md v5.8 | - diff --git a/local/docs/CACHYOS-INTEGRATION.md b/local/docs/CACHYOS-INTEGRATION.md deleted file mode 100644 index b4ddc507cf..0000000000 --- a/local/docs/CACHYOS-INTEGRATION.md +++ /dev/null @@ -1,313 +0,0 @@ -# CachyOS Boot Analysis: Reference for Red Bear OS Integration - -**Source ISO:** `cachyos-desktop-linux-260628.iso` (CachyOS Desktop, 28 Jun 2026) -**Captured:** 29 Jun 2026, via QEMU/KVM (`qemu-system-x86_64 -m 8G -smp 4 -cpu host -enable-kvm -nographic -kernel ... -initrd ... -append "console=ttyS0,115200n8 earlyprintk=ttyS0,115200 loglevel=7"`) -**Raw log:** `local/docs/boot-logs/cachyos-kernel-boot.log` (441 lines of `dmesg`-grade output) -**Purpose:** Document reference boot sequences for CPU/x86 systems so that the Redox kernel, base, and pcid daemons can be evaluated against the proven Linux paths on the same QEMU target. - ---- - -## 1. Executive summary - -CachyOS is an Arch-derived distribution that ships a custom Zen-optimised kernel and an Arch base. Running its kernel under QEMU/KVM gives us a **reference dmesg** for the i440FX + PIIX machine type that Red Bear OS is also designed to run on. The captured log covers: - -* Early firmware handover (SeaBIOS → iPXE → ISOLINUX → kernel) -* ACPI table parsing (RSDP, RSDT, FACP, DSDT, FACS, APIC, HPET, WAET) -* Memory zoning (DMA / DMA32 / Normal, GB pages, KASLR) -* PCI enumeration of all 5 QEMU PIIX devices -* Storage discovery (FDC, ATA PIIX for CD-ROM) -* Network init (e1000 82540EM) -* USB stack registration -* APIC / HPET / ACPI PCI hotplug -* Initramfs → systemd-udevd hand-off - -The full log is at `local/docs/boot-logs/cachyos-kernel-boot.log`. The lines reproduced below are the -ones that map directly onto Red Bear OS subsystems. - ---- - -## 2. Hardware inventory in QEMU i440FX (and what Red Bear OS must match) - -| PCI BDF | Vendor:Device | Class | Linux driver | Red Bear OS mapping | -|---|---|---|---|---| -| 0000:00:00.0 | 8086:1237 (PIIX ISA bridge host) | 060000 bridge | host bridge | `pcid` enumerates, no driver needed | -| 0000:00:01.0 | 8086:7000 (PIIX PCI bridge) | 060100 PCI bridge | bridge | `pcid` enumerates, no driver needed | -| 0000:00:01.1 | 8086:7010 (PIIX IDE) | 010180 IDE controller | ata_piix | `ided` already in base | -| 0000:00:01.3 | 8086:7113 (PIIX ACPI/SMB) | 068000 bridge | PIIX ACPI | `hwd` (ACPI backend) consumes this | -| 0000:00:02.0 | 1234:1111 (Bochs/QEMU VGA) | 030000 VGA | bochs-drm | `vesad` already in base | -| 0000:00:03.0 | 8086:100e (e1000) | 020000 Ethernet | e1000 | `e1000d` in base | -| 0000:00:04.0 | 1af4:1001 (virtio block) | 010000 block | virtio-blk | `virtio-blkd` in base | - -Every one of these has a matching userland daemon in `local/sources/base/drivers/`. -No surface-area gap found. - ---- - -## 3. Boot phases observed - -### 3.1 SeaBIOS / iPXE (firmware) - -``` -SeaBIOS (version Arch Linux 1.17.0-2-2) -iPXE (http://ipxe.org) 00:03.0 C900 PCI2.10 PnP PMM+BEFD3CC0+BEF33CC0 C900 -Booting from DVD/CD... -ISOLINUX 6.04 6.04-pre3-3-g05ac953c* ETCD Copyright (C) 1994-2015 H. Peter Anvin -Loading /arch/boot/x86_64/vmlinuz-linux-cachyos-lts... ok -Loading /arch/boot/x86_64/initramfs-linux-cachyos-lts.img...ok -``` - -**Implication for Red Bear OS:** our `recipes/core/bootloader/` package produces `bootloader-live.efi` for the **UEFI** path (Q35 + OVMF), but the i440FX/ISOLINUX path is the one we tested here and the one the Red Bear `redbear-mini.iso` image uses. The kernel loads via `linux` (BIOS-style) GRUB entry, not EFI stub. That is fine for i440FX but is **not** the path OVMF uses; our UEFI image uses a different boot path that we have not exercised in this capture. - -### 3.2 Early ACPI / SMP bring-up - -``` -[ 0.000000] NX (Execute Disable) protection: active -[ 0.000000] APIC: Static calls initialized -[ 0.000000] SMBIOS 2.8 present. -[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014 -[ 0.000000] Hypervisor detected: KVM -[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 -[ 0.000000] tsc: Detected 3686.400 MHz processor -``` - -**Implication for Red Bear OS:** our kernel uses the same `apic` static calls, NX bit, and `kvm-clock` MSRs. The MP-table boot path (`found SMP MP-table at [mem 0x000f66b0-0x000f66bf]`) is the one we use for AP bring-up in `local/sources/kernel/src/acpi/madt/arch/x86.rs`. **No divergence** between the two kernels' expected ACPI surface on this machine. - -### 3.3 ACPI tables - -``` -[ 0.020345] ACPI: Early table checksum verification disabled -[ 0.021036] ACPI: RSDP 0x00000000000F64C0 000014 (v00 BOCHS ) -[ 0.021707] ACPI: RSDT 0x00000000BFFE2445 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.022684] ACPI: FACP 0x00000000BFFE22E1 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.023657] ACPI: DSDT 0x00000000BFFE0040 0022A1 (v01 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.024623] ACPI: FACS 0x00000000BFFE0000 000040 -[ 0.025148] ACPI: APIC 0x00000000BFFE2355 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.026113] ACPI: HPET 0x00000000BFFE23E5 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.027073] ACPI: WAET 0x00000000BFFE241D 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) -[ 0.168090] ACPI: PM-Timer IO Port: 0x608 -[ 0.168571] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) -[ 0.170036] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) -[ 0.170765] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) -[ 0.171523] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) -[ 0.172268] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) -[ 0.173034] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) -[ 0.173806] ACPI: Using ACPI (MADT) for SMP configuration information -[ 0.174537] ACPI: HPET id: 0x8086a201 base: 0xfed00000 -``` - -**Implication for Red Bear OS:** our `local/sources/base/drivers/acpid` reads the same RSDP/RSDT/MADT/HPET/WAET table set. The table-set is identical between Linux and our microkernel — there is no missing AML method we should care about for QEMU targets. The `INT_SRC_OVR` block (IRQ overrides 0,5,9,10,11) is exactly what our `hwd` scheme expects. - -### 3.4 PCI enumeration - -``` -[ 0.504336] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 conventional PCI endpoint -[ 0.505163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 conventional PCI endpoint -[ 0.507163] pci 0000:00:01.1: [8086:7010] type 01 class 0x010180 conventional PCI endpoint -[ 0.512583] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 conventional PCI endpoint -[ 0.515623] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint -[ 0.534365] pci 0000:00:03.0: [8086:100e] type 00 class 0x020000 conventional PCI endpoint -[ 0.539589] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint -``` - -**Implication for Red Bear OS:** our `pcid` driver already enumerates these (the `redbear-mini.iso` boot log from earlier in this repo confirms `8086:7010 IDE`, `1234:1111 QEMU VGA`, `8086:100e e1000`, `1b36:000d XHCI`, `1af4:1000 virtio-net` — same machine type, same PIIX). - -### 3.5 PIIX IDE quirk - -``` -[ 0.508508] pci 0000:00:01.1: BAR 4 [io 0xc0c0-0xc0cf] -[ 0.509664] pci 0000:00:01.1: BAR 0 [io 0x01f0-0x01f7]: legacy IDE quirk -[ 0.510462] pci 0000:00:01.1: BAR 1 [io 0x03f6]: legacy IDE quirk -[ 0.510643] pci 0000:00:01.1: BAR 2 [io 0x0170-0x0177]: legacy IDE quirk -[ 0.511623] pci 0000:00:01.1: BAR 3 [io 0x0376]: legacy IDE quirk -``` - -**Implication for Red Bear OS:** this is the standard QEMU PIIX "fixed BAR" emulation. Our `ided` driver must read `BAR0/1/2/3` at the legacy IDE IO ports (1F0/3F6/170/376) plus the BM-DMA port at 0xC0C0. The current `ided` driver reads `0x1F0` for the command block — that matches. - -### 3.6 PIIX ACPI/SMB quirk - -``` -[ 0.513980] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI -[ 0.514631] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB -``` - -**Implication for Red Bear OS:** IO range 0x600-0x63F is the ACPI embedded controller. Our `hwd` reads from the ACPI embedded controller via the same IO range. We should NOT register a driver for PIIX4 at this address. - -### 3.7 Video (VGA) enumeration - -``` -[ 0.532015] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] -[ 0.570799] pci 0000:00:03.0: vgaarb: setting as boot VGA device -[ 0.571424] pci 0000:00:03.0: vgaarb: bridge control possible -[ 0.571619] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none -``` - -(Note: `00:03.0` is `8086:100e` e1000 — the vgaarb message is a quirk where e1000 claims VGA class bit 3 of its PCI config. This is harmless for our `vesad` driver which only uses `00:02.0` `1234:1111` Bochs.) - -**Implication for Red Bear OS:** the `00:02.0` Bochs VGA at `fd000000` and BAR2 `febf0000` is the framebuffer MMIO target for `vesad`. Our `redbear-mini.iso` `vesad` driver reads from these BARs and reports `vesad: 1280x720 stride 1280 at 0xC0000000` (matching QEMU's `-vga std` mapping at `0xC0000000`). - -### 3.8 USB / SCSI / NIC driver registration order - -``` -[ 0.561182] usbcore: registered new interface driver usbfs -[ 0.561376] usbcore: registered new interface driver hub -[ 0.562645] usbcore: registered new device driver usb -[ 2.542810] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100 -[ 2.548031] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0c0 irq 14 lpm-pol 0 -[ 2.550428] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0c8 irq 15 lpm-pol 0 -[ 2.719342] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5 -[ 2.778176] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray -[ 2.802805] cdrom: Uniform CD-ROM driver Revision: 3.20 -[ 3.016039] e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56 -[ 3.017095] e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection -``` - -**Implication for Red Bear OS:** driver registration order is `usbcore` → `ata_piix` → `cdrom` → `e1000`. Our `ided` driver (userspace PIO mode) reads the same PATA ports. Our `e1000d` driver reads `0000:00:03.0` (the Bochs e1000). The QEMU MAC `52:54:00:12:34:56` matches what the `redbear-mini.iso` `virtio-netd` configures for the user-mode network. - -### 3.9 Input devices - -``` -[ 0.718125] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 -[ 0.769816] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 -``` - -**Implication for Red Bear OS:** we have `evdevd` (input event daemon) and the kernel exposes `/scheme/input/` events. Linux names devices like `LNXPWRBN:00` (ACPI power button) and `i8042/serio0` (PS/2 keyboard). Redox's `evdevd` should handle the same ACPI power button events. - -### 3.10 Initrd → systemd-udevd - -``` -Freeing initrd memory: 81472K -Starting systemd-udevd version 261.1-1-arch -:: running early hook [udev] -:: running hook [udev] -:: Triggering uevents... -[ 2.533807] ACPI: \_SB_.LNKC: Enabled at IRQ 10 -[ 2.542810] Floppy drive(s): fd0 is 2.88M AMI BIOS -... -:: running hook [memdisk] -:: running hook [archiso] -:: running hook [archiso_loop_mnt] -:: Mounting '' to '/run/archiso/bootmnt' -ERROR: '' device did not show up after 30 seconds... - Falling back to interactive prompt -sh: can't access tty; job control turned off -[rootfs ~]# -``` - -**Implication for Red Bear OS:** the CachyOS `archiso` hook chain is `early-udev → udev → memdisk → archiso → archiso_loop_mnt → archiso_pxe_*`. It expects the boot medium at a discoverable loop device. Without that, initrd drops to a shell. Red Bear's `initfs` (initramfs) avoids this by hard-mounting the boot device early (via `initfs.toml`). This is a **key design difference**: CachyOS does dynamic discovery, Red Bear OS does explicit configuration. Red Bear's approach is more deterministic but requires the boot device to be known at build time. - ---- - -## 4. Cross-cutting observations and integration gaps - -### 4.1 ACPI coverage match - -Red Bear OS coverage of the ACPI tables Linux reads (RSDP/RSDT/FACP/DSDT/FACS/APIC/HPET/WAET/MADT): -* `acpid` reads RSDP/RSDT via `hwd` daemon → covers table enumeration -* `hwd` opens `/scheme/acpi/tables/` and exposes them as files -* AML interpretation via `acpi` crate covers DSDT namespace walk (needed for `_SB_.LNKC`, `_INI`, etc.) -* `hwd` does NOT currently expose `\_SB_.LNKC: Enabled at IRQ 10` device linking — the IRQ 10 link callback is silently dropped. This is a **gap**. - -**Action item:** extend `hwd` to interpret IRQ link descriptors in the AML namespace so the cascade `LNK A → LNK B → LNK C` propagates. Linux does this in `acpi_irq.c`. - -### 4.2 PCI quirk coverage - -Linux applies PIIX4 legacy IDE "fixed BAR" quirks. Our `ided` driver should be aware that BAR 0..3 are at fixed IO 1F0/3F6/170/376 even if the device's config space disagrees. - -**Action item:** add a quirk handler in `ided` that pins BAR0..3 to the legacy IDE IO ports on devices matching `8086:7010/7111` (PIIX4/5 IDE). - -### 4.3 VGA arbitration - -Linux has `vgaarb`. We don't — but we only have one VGA on QEMU. On real multi-GPU systems we'd need this. - -**Action item:** add a minimal `vgaarb` in `pcid` that tracks which device owns the VGA routing registers. Currently, on the QEMU test machine, `vesad` simply claims `00:02.0`; if a second VGA class device were present, the boot would be ambiguous. - -### 4.4 CPU feature detection vs current Red Bear OS - -The CachyOS kernel detects and enables: - -``` -[ 0.000000] NX (Execute Disable) protection: active -[ 0.000000] APIC: Static calls initialized -``` - -Our local kernel fork does the same via `local/sources/kernel/src/arch/x86_shared/`. The boot speed (3 seconds from kernel to rootfs prompt) is mostly spent on `start_kernel()` → `arch_call_rest_init()` → `do_basic_setup()` → `driver_init()`. - -### 4.5 No native KVM-clock calibration visible - -`kvm-clock: Using msrs 4b564d01 and 4b564d00` is the standard. We do the same via our `kvm-clock` in `local/sources/kernel/src/clock/`. **No action needed.** - -### 4.6 USB stack - -The log shows `usbcore: registered new interface driver usb` at 0.56s. Our `xhcid` driver (XHCI) attaches on QEMU's `1b36:000d` USB3 controller (we see it in our own boot). The CachyOS log doesn't show a USB device enumeration because the `-device qemu-xhci` flag wasn't used in the kernel cmdline I used; for full USB testing we need `-device usb-ehci` and a USB device. - -### 4.7 Framebuffer / graphics - -`pci 0000:00:02.0: vgaarb: setting as boot VGA device` is the Bochs/QEMU VGA. Our `vesad` opens `fd000000` (the QEMU BDF 00:02.0 BAR0). Linux uses the same address and produces a working X11/Wayland display. Our `vesad` produces an identical `1280x720` mode and feeds it to the `vesad: 1280x720 stride 1280 at 0xC0000000` line that the `redbear-mini.iso` boot log shows. - -### 4.8 Storage stack - -``` -ata_piix for PIIX4 (00:01.1) -cdrom/scsi for the QEMU DVD-ROM at sr0 -``` - -Our `ided` driver handles the PIIX4 IDE at 00:01.1. The live ISO is mounted via `initfs` from `/usr/lib/boot/initfs`, bypassing the CD-ROM entirely. **No gap.** - -### 4.9 Network - -`e1000 0000:00:03.0 eth0: 52:54:00:12:34:56` — standard QEMU SLIRP MAC. Our `e1000d` registers the same PCI device. `virtio-netd` on QEMU `1af4:1000` also works. The user-mode SLIRP network is identical to Red Bear. - -### 4.10 Init / systemd vs initfs - -CachyOS uses `systemd` (PID 1). Red Bear OS uses `initfs` (a stripped-down init that mounts `/usr`, switches to initrd, and runs `/sbin/init` from the userland tree). The initfs approach is faster on QEMU but less flexible. We do not need to switch to systemd — initfs is correct for our use case. - -### 4.11 Kernel init ordering difference - -| Phase | Linux/CachyOS | Red Bear OS | -|---|---|---| -| Early ACPI | ACPI core reads RSDP, RSDT, FACP, DSDT, FACS, APIC, HPET, WAET | `hwd` reads same | -| SMP | APIC static calls, MP-table | MADT parse + APIC IPI sequence | -| PCI | `pci: Using configuration type 1` + `acpiphp: Slot [N] registered` | `pcid` with `acpiphp`-equivalent | -| USB | `usbcore: registered new interface driver usb` | `xhcid` on `1b36:000d` | -| ATA | `ata_piix` claims PIIX4 IDE | `ided` claims PIIX4 IDE | -| Network | `e1000` for `00:03.0` | `e1000d` for `00:03.0` | -| VGA | `pci 0000:00:02.0: vgaarb: setting as boot VGA device` | `vesad` claims `00:02.0` | -| Input | `i8042`, `AT Translated Set 2 keyboard`, `Power Button` | `evdevd` consumes `i8042` events | -| Initrd | `Freeing initrd memory: 81472K` then `systemd-udevd` | `initfs` mounts `/usr`, hands off to `init` | - -The ordering is broadly equivalent. Differences: -* Red Bear uses `initfs` (custom initramfs) instead of `systemd` for PID 1. -* Red Bear's `pcid` is a single daemon vs Linux's multi-call ACPI + PCI + platform drivers. -* Red Bear's ACPI HPET detection is identical (HPET id 0x8086a201 base 0xfed00000). - -### 4.12 Why CachyOS can't replace Red Bear OS analysis - -CachyOS is a Linux distribution with a full software stack. Red Bear OS is a microkernel with a small userspace. The boot patterns are equivalent at the *firmware and kernel* layer but diverge sharply at *userspace*. The CachyOS log is useful as a *reference* — it tells us what a correctly working x86 kernel does on QEMU, so we can verify Red Bear OS produces an equivalent boot path. - -Key equivalence points to verify in Red Bear OS: -1. ACPI tables parsed identically. -2. PCI enumeration covers the same devices. -3. USB stack registered. -4. Network initialized. -5. Framebuffer (VGA) available. -6. Input devices recognized. - -Red Bear's own `redbear-mini.iso` boot log (from previous work in this repo) shows all six pass. CachyOS log confirms that what we observe is consistent with a reference Linux boot of the same QEMU machine type. - ---- - -## 5. Files referenced - -* `local/docs/boot-logs/cachyos-kernel-boot.log` — the captured log (441 lines) -* `local/docs/boot-logs/README.md` — pointer to the log and the capture command -* `local/docs/CACHYOS-INTEGRATION.md` — this file - -## 6. Action items derived from this analysis - -1. **`hwd` PIIX4 IRQ linking** — extend `hwd` to interpret the `\_SB_.LNKC` (and related) AML objects so IRQ routing propagates instead of dropping the callback. -2. **`ided` legacy IDE quirk** — add a config-space patcher that pins BAR0..3 to legacy IDE IO ports on `8086:7010/7111`. -3. **`pcid` vgaarb** — add minimal VGA arbitration when multiple VGA devices are present. -4. **`initfs` archiso-compat path** — optionally expose an `archiso` hook so CachyOS-style live ISOs can mount via the same `loop_mnt` pattern (low priority). -5. **MSR exposure** — `cpufreqd` and other tools want `IA32_PERF_CTL` writes. The kernel currently has no MSR scheme. If we want full cpufreqd on real hardware, we need a small userspace `msrd` daemon (or kernel MSR scheme). The QEMU MSR emulation caveat is unrelated. - -These are documented as design considerations; no immediate code changes are required because Red Bear OS already boots successfully on QEMU and the integration points listed above are present in some form. diff --git a/local/docs/CARGO-PATCH-PROPAGATION.md b/local/docs/CARGO-PATCH-PROPAGATION.md deleted file mode 100644 index d07cd4809a..0000000000 --- a/local/docs/CARGO-PATCH-PROPAGATION.md +++ /dev/null @@ -1,161 +0,0 @@ -# Cargo Patch Propagation — Fork Dependency Policy - -**Created:** 2026-07-02 -**Status:** Active policy -**Scope:** All Rust recipes that depend on forked packages - -## The Rule - -Every recipe that uses a forked package **MUST** depend on the local fork — never on -the upstream/crates.io version. - -``` -Upstream package (gitlab.redox-os.org / crates.io) - │ - ▼ -Our local fork (local/sources//) - │ (patches applied, version bumps tracked) - ▼ -ALL downstream recipes depend on THIS fork -``` - -### Forked packages (as of 2026-07-02) - -| Package | Local fork | Version | Symlink | -|---------|-----------|---------|---------| -| `redox_syscall` | `local/sources/syscall/` | 0.8.1 | `recipes/core/base/syscall` | -| `libredox` | `local/sources/libredox/` | 0.1.18 | `recipes/core/base/libredox` | - -### Lifecycle: when upstream bumps version - -1. **Update fork first** — `git fetch upstream && git rebase upstream/master` in `local/sources//` -2. **Rebase all patches** — ensure Red Bear patches still apply against the new upstream -3. **Rebuild prefix** — `touch && make prefix` if the fork affects the cross-toolchain sysroot -4. **All dependents automatically use the updated fork** — no per-recipe changes needed -5. **Adapt downstream code** — if the upstream bump changed an API, fix every recipe that breaks (Golden Rule: Red Bear adapts to upstream, never the reverse) - -## Why This Matters - -### The Problem: Cargo `[patch]` doesn't propagate - -Cargo's `[patch.crates-io]` only applies from the **root package** being compiled. When -a recipe depends on a base workspace member (like `daemon`) via path dependency, the base -workspace's own `[patch]` entries are **silently ignored**. - -This means: - -1. Recipe `redox-drm` depends on `daemon` via `path = ".../base/source/daemon"` -2. `daemon` is part of the `base` workspace, which patches `redox_syscall` and `libredox` -3. But when cargo compiles `redox-drm` as root, only `redox-drm`'s own `[patch]` applies -4. `libredox` from crates.io **vendors its own copy** of `redox_syscall` internally -5. Two different `syscall::Error` types exist at compile time → E0277 type mismatch - -### The Fix: Use path dependencies, not version dependencies - -Every recipe that needs `redox_syscall` or `libredox` should use the local fork via path -dependency or `[patch.crates-io]` entry. The path is always the same relative depth from -`local/recipes///source/Cargo.toml`: - -``` -../../../../../recipes/core/base/syscall → local fork of redox_syscall -../../../../../recipes/core/base/libredox → local fork of libredox -``` - -## How to Make a Recipe Depend on the Fork - -### Option A: Direct path dependency (preferred for direct deps) - -```toml -[dependencies] -redox_syscall = { path = "../../../../../recipes/core/base/syscall", features = ["std"] } -libredox = { path = "../../../../../recipes/core/base/libredox" } -``` - -### Option B: Version dep + patch redirect (when transitive deps also need redirection) - -```toml -[dependencies] -redox_syscall = { version = "0.8", features = ["std"] } -libredox = "0.1" -daemon = { path = "../../../../../recipes/core/base/source/daemon" } - -[patch.crates-io] -redox_syscall = { path = "../../../../../recipes/core/base/syscall" } -libredox = { path = "../../../../../recipes/core/base/libredox" } -``` - -Use Option B when the recipe depends on ANY base workspace member via path (`daemon`, -`scheme-utils`, etc.). The `[patch]` entries redirect ALL transitive `redox_syscall` and -`libredox` dependencies (from `redox-scheme`, etc.) to the local fork. - -### Symlinks required - -The paths above resolve through symlinks that must exist: - -```bash -recipes/core/base/syscall → ../../../local/sources/syscall -recipes/core/base/libredox → ../../../local/sources/libredox -``` - -These symlinks bridge the base workspace's relative path resolution. Without them, the -base workspace's `path = "../syscall"` resolves to a non-existent directory. - -## Validation - -Run the validation gate to check all recipes: - -```bash -./target/release/repo validate-cargo-deps -``` - -This scans every `Cargo.toml` in `recipes/` and `local/recipes/`, detects: -- Path dependencies on base workspace members without matching `[patch]` entries -- Version dependencies on forked packages that should use the local fork -- Missing or broken symlinks - -Exit code 0 = all OK, exit code 1 = warnings found. - -## Recipes Currently Violating This Policy - -As of 2026-07-02, the following recipes pull `redox_syscall` or `libredox` from crates.io -instead of the local fork. These are policy violations that should be fixed incrementally: - -### `redox_syscall` from crates.io (25 recipes) - -Recipes using version 0.8 (straightforward conversion to fork): -- `drivers/ehcid`, `drivers/linux-kpi`, `drivers/redbear-btusb` -- `drivers/redox-driver-sys` -- `system/driver-manager`, `system/evdevd`, `system/firmware-loader` -- `system/hwrngd`, `system/iommu` -- `system/redbear-btctl`, `system/redbear-hwutils` -- `system/redbear-sessiond`, `system/redbear-traceroute` -- `system/redbear-wifictl`, `system/thermald`, `system/udev-shim` -- `tui/tlc` - -Recipes using version 0.7 (need API adaptation to 0.8): -- `drivers/virtio-inputd`, `system/devfsd`, `system/diskd` - -Recipes using version 0.4 (need significant API adaptation to 0.8): -- `system/redbear-accessibility`, `system/redbear-ime`, `system/redbear-keymapd` -- `gpu/redox-drm` (uses `syscall04` for legacy PCI config — separate concern) - -### `libredox` from crates.io (21 recipes) - -All recipes using `libredox = "0.1"` or `libredox = "0.1.x"` from crates.io should -switch to the local fork at `local/sources/libredox/` (version 0.1.18). - -## Migration Plan - -1. **Phase 1 (done):** Fix `redox-drm` — the only recipe with a path dep on `daemon` -2. **Phase 2:** Convert all 0.8.x recipes to use the local fork (straightforward) -3. **Phase 3:** Adapt 0.7.x recipes to 0.8.x API and switch to fork -4. **Phase 4:** Adapt 0.4.x recipes to 0.8.x API and switch to fork - -Each conversion is: change `version = "0.x"` to `path = "..."`, cook, fix any API breaks. - -## Related - -- `local/AGENTS.md` § "GOLDEN RULE — Red Bear adapts to upstream, never the reverse" -- `local/AGENTS.md` § "LOCAL FORK MODEL (CORE COMPONENTS)" -- `local/sources/base/Cargo.toml` lines 133-157 — the base workspace `[patch]` section - that ONLY applies within the base workspace diff --git a/local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md b/local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md index adc9f25043..d10c2d7142 100644 --- a/local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md +++ b/local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md @@ -2049,7 +2049,7 @@ This document is the authority. Subsystem plans remain for deep-dive detail: | `KERNEL-IPC-CREDENTIAL-PLAN.md` | Kernel credential syscalls, IPC, RLIMIT — Phases K1-K2,K4 complete | | `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | PCI/IRQ/MSI-X/IOMMU quality | | `ACPI-IMPROVEMENT-PLAN.md` | ACPI shutdown, power, sleep states | -| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface | +| `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface | | `DRM-MODERNIZATION-EXECUTION-PLAN.md` | DRM/KMS modernization | | `WAYLAND-IMPLEMENTATION-PLAN.md` | Wayland compositor stability | | `DBUS-INTEGRATION-PLAN.md` | D-Bus architecture | diff --git a/local/docs/CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md b/local/docs/CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md deleted file mode 100644 index 691c6953c7..0000000000 --- a/local/docs/CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md +++ /dev/null @@ -1,158 +0,0 @@ -# Red Bear OS — CPU/DMA/IRQ/MSI/Scheduler Fix Plan - -**Date**: 2026-05-04 -**Updated**: 2026-05-04 (MSI T1.1–T2.2 implemented, committed, pushed) -**Status**: Active — MSI Phase 1 complete, DMA/Scheduler pending -**Source of truth**: Linux kernel 7.1 (local/reference/linux-7.1/) - -## 1. Problem Statement - -Five critical integration gaps in the microkernel architecture: - -| Gap | Severity | Impact | Status | -|-----|----------|--------|--------| -| MSI absent from kernel | CRITICAL | All NVMe/GPU/NIC on legacy INTx | ✅ RESOLVED (P8-msi.patch) | -| DMA/IOMMU not integrated | CRITICAL | DMA buffers unprotected | ⏳ Pending | -| PIT tick (148Hz) vs LAPIC (1000Hz) | HIGH | Scheduler 6x slower than Linux | ✅ RESOLVED (P7-scheduler patch) | -| Global scheduler lock | HIGH | Serializes all context switches | ✅ RESOLVED (work-stealing) | -| Thread creation (3 IPC hops) | HIGH | 3x slower than Linux clone() | ⏳ Pending | - -## 2. Phase 1: MSI/MSI-X in Kernel (Week 1-3) ✅ COMPLETE - -### T1.1: MSI Capability Parsing ✅ DONE -- File: `kernel/src/arch/x86_shared/device/msi.rs` (61 lines) -- Commit: `678980521` in `P8-msi.patch` -- Linux ref: `arch/x86/kernel/apic/msi.c` (391 lines) -- Implements: `MsiMessage` (compose/validate), `MsiCapability` (parse 32/64-bit), `MsixCapability` (parse table/PBA), `is_valid_msi_address`, `is_valid_msi_vector` -- Bounds-safe: all `parse()` methods return `Option`, using `.get()` instead of raw indexing - -### T1.2: Vector Allocation Matrix ✅ DONE -- File: `kernel/src/arch/x86_shared/device/vector.rs` (53 lines) -- Commit: `678980521` in `P8-msi.patch` -- Linux ref: `arch/x86/kernel/apic/vector.c` (1387 lines) -- Implements: per-CPU bitmatrix (7×32-bit banks = 224 vectors 32-255), `allocate_vector`, `free_vector` -- Lock-free CAS-based allocation with `trailing_ones()` find-first-zero -- NOTE: VECTORS table is global (not yet per-CPU sharded) — sufficient for 224 vectors - -### T1.3: MSI IRQ Domain (Scheme Integration) ✅ DONE -- File: `kernel/src/scheme/irq.rs` -- Commit: `678980521` in `P8-msi.patch` -- Implements: `msi_vector_is_valid()` (32-0xEF range check), `iommu_validate_msi_irq()` hook (stub: always true), IOMMU gate at `irq_trigger()` for vectors ≥16 - -### T1.4: Userspace MSI Consumer (driver-sys) ✅ DONE -- File: `local/recipes/drivers/redox-driver-sys/source/src/irq.rs` -- Commit: `678980521` -- Implements: `MsiAllocation` with round-robin CPU allocation, `irq_set_affinity` (scheme write), `program_x86_message` with kernel-mediated address/vector validation (mask `0xFFF0_0000`) -- Quirk-aware fallback retained: FORCE_LEGACY, NO_MSI, NO_MSIX - -### T1.5: Kernel-side MSI Affinity Handler ✅ DONE -- File: `kernel/src/scheme/irq.rs` -- Commit: `678980521` in `P8-msi.patch` -- Implements: `Handle::IrqAffinity { irq, mask }` variant, path routing for `/affinity` and `cpu-XX//affinity`, kwrite validates CPU id and stores mask atomically, kfstat/kfpath/kreadoff/close all handle new variant - -## 3. Phase 2: DMA/IOMMU Integration (Week 3-5) — AUDITED 2026-05-04 - -**Status**: IOMMU daemon (1003 lines) and DmaBuffer (261 lines) already exist and are solid. Tasks re-scoped from "create" to "wire." - -### T2.1: IommuDmaAllocator (driver-sys) ⏳ P0 -- File: `local/recipes/drivers/redox-driver-sys/source/src/dma.rs` -- Add `IommuDmaAllocator` struct: holds IOMMU domain fd, wraps `DmaBuffer::allocate()` with IOMMU MAP opcode -- Uses `scheme:iommu/domain/N` write with MAP request → get IOVA -- Linux ref: `include/linux/dma-mapping.h` — `dma_alloc_coherent()` → `iommu_dma_alloc()` - -### T2.2: GPU DMA pass-through ⏳ P0 -- Wire `redox-drm` GPU drivers to open IOMMU device endpoint and use IommuDmaAllocator -- amdgpu: VRAM/GTT allocations through IOMMU domain -- Intel i915: GTT pages through IOMMU domain -- Files: `local/recipes/gpu/redox-drm/source/`, `local/recipes/gpu/amdgpu/source/` - -### T2.3: Streaming DMA (linux-kpi) ⏳ P1 -- `dma_map_single()`: allocate bounce buffer, copy data, map through IOMMU -- `dma_unmap_single()`: copy back, unmap, free bounce buffer -- Linux ref: `kernel/dma/mapping.c` — streaming API -- File: `local/recipes/drivers/linux-kpi/source/` - -### T2.4: NVMe DMA pass-through ⏳ P1 -- Wire `ahcid`/`nvmed` PRP list physical addresses through IOMMU domain -- Linux ref: `drivers/nvme/host/pci.c` — `nvme_map_data()` - -### T2.5: SWIOTLB Fallback (low priority) ⏳ P2 -- Linux ref: `kernel/dma/swiotlb.c` -- Bounce buffer for devices with <4GB DMA addressing -- Only needed for ancient hardware; x86_64 modern hardware doesn't need it - -## 4. Phase 3: Scheduler Improvements (Week 4-6) — MOSTLY DONE - -### T3.1: LAPIC Timer as Primary Tick ✅ DONE -- P7-scheduler-improvements.patch: LAPIC timer calibrated + enabled at vector 48 -- TSC-deadline mode, 1000Hz tick drives DWRR scheduler directly -- PIT fallback retained - -### T3.2: Per-CPU Scheduler Locks ✅ DONE -- Work-stealing load balancer in switch.rs -- Per-CPU nr_running counter -- Idle CPUs steal work via IPI - -### T3.3: Load Balancing ✅ DONE -- RT scheduling class (priority 0-9, skip DWRR, immediate dispatch) -- Threshold reduced: 3→1 ticks for LAPIC-driven mode -- Geometric weights in DWRR - -### T3.4: RT Scheduling Class ✅ DONE - -### T3.5: NUMA-Aware Scheduling ❌ -- Not implemented — low priority for desktop/non-NUMA systems -- Linux ref: kernel/sched/rt.c -- FIFO and Round-Robin classes -- Priority inheritance -- RT throttling: 95% CPU cap/sec - -### T3.5: TSC-Deadline Timer -- Use IA32_TSC_DEADLINE MSR for precise tick -- True tickless operation -- TSC calibration via HPET or PIT - -## 5. Phase 4: Thread Creation (Week 6-7) - -### T4.1: Batched Thread Creation -- Batch new-thread requests (reduce IPC) -- Pre-allocate stack pages during fork - -### T4.2: Kernel Thread Pool -- Pre-create idle kernel threads -- Reuse via object pool - -### T4.3: Shared Memory IPC -- Use shm for proc scheme bulk ops -- Avoid data copy through IPC channel - -## 6. Dependencies - -Phase 1 (MSI): T1.1 -> T1.2 -> T1.3 -> T1.4 -> T1.5 -Phase 2 (DMA): T2.1 -> T2.2 -> T2.3 -> T2.4 -> T2.5 -Phase 3 (Sched): T3.1 -> T3.5 -> T3.2 -> T3.3 -> T3.4 -Phase 4 (Thread): T4.1 -> T4.2 -> T4.3 - -Phase 1+2 independent (parallel). Phase 2.4 needs Phase 1.3. -Phase 3.1 partially done (start immediately). - -## 7. Timeline - -| Phase | Duration | Cumulative | -|-------|----------|------------| -| Phase 1 (MSI) | 3 weeks | Week 3 | -| Phase 2 (DMA/IOMMU) | 3 weeks | Week 5 | -| Phase 3 (Scheduler) | 3 weeks | Week 7 | -| Phase 4 (Threads) | 2 weeks | Week 7 | - -Total: 7 weeks (2 devs parallel Phase 1+2) - -## 8. Success Metrics - -| Metric | Before | After | -|--------|--------|-------| -| Scheduler tick | 148Hz (PIT) | 1000Hz (LAPIC) | -| NVMe throughput | INTx shared | MSI-X 4+ queues | -| Context switch | ~6.75ms | ~1ms | -| Thread create | 3 IPC hops | 2 IPC hops | -| DMA safety | Unprotected | IOMMU-mapped | diff --git a/local/docs/CUB-WORKFLOW-ASSESSMENT.md b/local/docs/CUB-WORKFLOW-ASSESSMENT.md deleted file mode 100644 index 288a424703..0000000000 --- a/local/docs/CUB-WORKFLOW-ASSESSMENT.md +++ /dev/null @@ -1,144 +0,0 @@ -# Cub Workflow Integration Assessment - -**Status:** Assessment + Implementation complete (2026-05-07) -**Scope:** AUR search → PKGBUILD parse → recipe.toml generation → cook with build tools - -## End-to-End Flow Assessment - -``` -User: "cub -S ripgrep-all" - │ - ├─ 1. AUR Search ✅ Works. AurClient::search() via AUR RPC v5. - │ - ├─ 2. Fetch PKGBUILD ✅ Works. Git clone from aur.archlinux.org. - │ - ├─ 3. Parse PKGBUILD ⚠️ Partial. See Gap #1-3 below. - │ - ├─ 4. Convert to recipe.toml ⚠️ Partial. See Compatibility Gaps below. - │ - └─ 5. Cook recipe ⚠️ Partial. Depends on build tool availability. -``` - -## Critical Gaps: PKGBUILD → Recipe Conversion - -### Gap 1: Install Function Silently Lost (CRITICAL) - -PKGBUILD `package()` functions with `install -Dm755` commands are not converted. -The generated recipe.toml has no install instructions. Files are never staged. - -**Impact**: Any AUR package using `package()` produces a broken recipe that -builds but installs nothing. - -### Gap 2: Multiple Source Entries → Hard Error (CRITICAL) - -`cookbook.rs` line 63-67: if a PKGBUILD has >1 source, `generate_recipe()` -returns a hard error. Many AUR packages use multiple source tarballs. - -**Impact**: `cub build` fails immediately with "Cookbook recipe generation -currently supports a single primary source." - -### Gap 3: SHA-256 Passed as BLAKE3 (HIGH) - -PKGBUILD uses SHA-256 checksums. Cookbook expects BLAKE3. The SHA-256 hex -string is copied verbatim into the `blake3` field. - -**Impact**: Cookbook hash verification will fail on packages with checksums. - -### Gap 4: Dependency Coverage ~15-20% (MEDIUM) - -`deps.rs` maps 44 Arch→Redox dependencies. The AUR ecosystem has thousands. -Unmapped deps pass through unchanged (`libxml2` → `libxml2`), which may or -may not resolve at cook time. - -### Gap 5: Split Packages Not Generated (HIGH) - -AUR packages with `pkgname=('foo' 'foo-docs' 'foo-libs')` and multiple -`package_*()` functions are detected but only the primary package is converted. -`[[optional-packages]]` is never generated. - -### Gap 6: Linuxism Detection Incomplete (MEDIUM) - -Only `systemctl`, `/usr/lib/systemd`, `systemd`, `/proc` are detected. -Missing: `dbus-daemon`, `udev`, `/sys/`, Python `systemd` imports. - -## Recipe.toml Compatibility Gaps - -| # | Gap | Severity | Impact | -|---|---|---|---| -| C1 | `dev-dependencies` missing `host:` prefix | CRITICAL | Cross-compilation filtering broken | -| C2 | `[[optional-packages]]` not generated | HIGH | Split packages impossible | -| C3 | `shallow_clone` field missing | MEDIUM | Large git repos clone slowly | -| C4 | `upstream` field missing | LOW | Fork tracking lost | -| C5 | `installs` field not populated | MEDIUM | Install manifest empty | -| C6 | `cargopath`/`cargopackages`/`cargoexamples` missing | MEDIUM | Cargo workspace builds broken | -| C7 | `script` field missing from `[source]` | LOW | Source prep scripts lost | -| C8 | `SameAs`/`Path` source variants not supported | LOW | Recipe reuse impossible | - -## Build Tool Availability for Cooking - -### ✅ Available (all templates covered) - -| Template | Tools Needed | Status | -|----------|-------------|--------| -| `cargo` | rustc + cargo | ✅ rust-native | -| `cmake` | cmake + ninja + gcc | ✅ all present | -| `meson` | meson + ninja + gcc | ✅ all present | -| `configure` | autoconf, automake, libtool, m4, gcc, make | ✅ all present | -| `custom` | whatever script declares | ✅ depends on recipe | - -### ❌ Missing / Broken - -| Tool | Status | Blocks | -|------|--------|--------| -| **texinfo** | BROKEN (compilation error) | Autotools packages with `makeinfo` | -| **intltool** | WIP (compiled, not tested) | GNOME i18n packages | -| **gobject-introspection** | WIP (not tested) | GTK/GNOME introspection packages | -| **gtk-doc** | WIP (not tested) | Development docs only (low priority) | - -### Dependency Mapping Coverage - -| Category | Count | Examples | -|----------|-------|----------| -| Explicitly mapped | 44 | glibc→relibc, openssl→openssl3, etc. | -| Dropped (unavailable) | 5 | systemd, xorg-server, linux-api-headers | -| Pass-through (unknown) | Thousands | libxml2, libpcre, etc. | - -## Build Flow Integration - -### What Works End-to-End - -1. Simple Rust packages (cargo template, single source) -2. Simple C packages (configure template, single source) -3. CMake packages (single source) -4. Meson packages (single source) - -### What Breaks - -1. **Any package with install function** → recipe missing install logic -2. **Multi-source packages** → hard error at generation -3. **Split packages** → only primary package built -4. **Packages with checksums** → BLAKE3 verification mismatch -5. **Packages needing texinfo** → build tool unavailable -6. **Cross-compilation deps** → host: prefix not added - -## Recommendations - -### Immediate (unblock basic AUR packages) - -1. Fix install function conversion — route `package()` content to `BuildKind::Custom.script` -2. Remove multi-source hard error — support multiple source entries or warn gracefully -3. Add `host:` prefix to dev-dependencies when building for target - -### Short-term (unblock common packages) - -4. Fix texinfo compilation error — unblocks many autotools packages -5. Implement `[[optional-packages]]` generation for split packages -6. Fix SHA-256 → BLAKE3 mapping — use correct hash or document the gap -7. Add `cargopath`/`cargopackages` fields to cargo template generation - -### Medium-term (broader coverage) - -8. Expand dependency mapping table to cover common AUR libraries -9. Improve linuxism detection (D-Bus, udev, sysfs patterns) -10. Add `shallow_clone`, `upstream`, `installs` fields -11. Validate intltool and gobject-introspection recipes diff --git a/local/docs/ECOSYSTEM-ADAPTION-POLICY.md b/local/docs/ECOSYSTEM-ADAPTION-POLICY.md deleted file mode 100644 index 18b537982d..0000000000 --- a/local/docs/ECOSYSTEM-ADAPTION-POLICY.md +++ /dev/null @@ -1,144 +0,0 @@ -# Red Bear OS: Ecosystem Adaption Policy - -**Rule (per user directive):** Red Bear OS internal projects MUST always update, -adapt, and go inline with Redox ecosystem changes. Red Bear adapts to Redox, -not the other way around. - -## How Ecosystem Pinning Works - -The Redox crate ecosystem has version coupling: `libredox`, `redox-scheme`, -`redox_syscall` MUST be on a consistent version set across all crates in a -build. Mixing old and new versions causes `call_ro`/`call_wo` and -`syscall::Error` vs `libredox::error::Error` type errors. - -### Canonical Versions (as of build session 2026-06-26) - -| Crate | Version | Why this version | -|-------|---------|------------------| -| `libredox` | `=0.1.18` | Current upstream version in fork-upstream-map.toml | -| `redox-scheme` | `=0.11.2` | Current upstream version in fork-upstream-map.toml | -| `redox_syscall` | `=0.9.0` | Current upstream version in fork-upstream-map.toml | - -**Note on version selection rationale (per "latest versions only" rule):** -Redox ecosystem is mid-migration (2026-05-27 → 2026-06-01 transition). The -transition broke compatibility for upstream consumers. Per research: - -- `redox-scheme 0.11.1` (2026-05-27) bumped `redox_syscall ^0.7.3 → ^0.8.0` - AND `libredox ^0.1.12 → ^0.1.17` -- `redoxfs master` (2026-06-24) still uses `redox-scheme 0.11.0` (NOT 0.11.1) - with `redox_syscall 0.7.5` and `libredox 0.1.13` -- `redox_installer master` (2026-05-27) still uses `redox_syscall 0.7` - and `redoxfs 0.9` -- Red Bear syscall fork is currently aligned with upstream `0.9.0` (the first jump past `0.7.x`) - -So the current **mutually-aligned** ecosystem is `redox-scheme 0.11.2` on `redox_syscall 0.9.0`. When upstream `redoxfs` and the rest of the fork map move together, keep this table aligned to the values in `local/fork-upstream-map.toml`: - -| Crate | New Version | -|-------|-------------| -| `libredox` | `=0.1.18` | -| `redox-scheme` | `=0.11.2` | -| `redox_syscall` | `=0.9.0` | - -Use the current map values to keep the ecosystem aligned. - -## How to Pin Ecosystem Versions in a Red Bear Recipe - -### For Red Bear local recipes (in `local/recipes/`): - -**Single-package files** (no `[workspace]`): - -```toml -[dependencies] -# These are pinned to latest ecosystem versions. -libredox = "=0.1.18" -redox-scheme = "=0.11.2" -redox_syscall = "=0.9.0" -``` - -**Workspace roots** (`[workspace]` section): - -```toml -[workspace.dependencies] -# Red Bear OS must adapt to upstream Redox ecosystem changes. -# These pins force unified versions across the workspace. -libredox = "=0.1.18" -redox-scheme = "=0.11.2" -redox_syscall = "=0.9.0" -``` - -Then in member crates: -```toml -[dependencies] -libredox = { workspace = true } -redox-scheme = { workspace = true } -redox_syscall = { workspace = true } -``` - -### For upstream Redox recipes (kernel, base, relibc, etc.): - -Upstream recipes cannot be modified directly. Use the patch system in -`local/patches//` to bump the ecosystem pins in their -`Cargo.lock` (and Cargo.toml where needed). - -Reference patch: `local/patches/base/P9-redox-scheme-latest-deps.patch`. - -## Anti-Patterns (Do Not Do) - -### ❌ `[patch.crates-io]` with `version = "..."` syntax - -```toml -[patch.crates-io] -libredox = { version = "0.1.17" } # WRONG! -``` - -This is INVALID. Cargo treats `version = "X.Y.Z"` as a patch pointing to the -same source (crates.io), which conflicts with itself. You get the error: - -``` -error: patch for `libredox` points to the same source, but patches must - point to different sources -``` - -Use `[workspace.dependencies]` with `=X.Y.Z` for version-only enforcement -instead. - -### ❌ `[patch.crates-io]` with `git = ...` for ecosystem crates - -```toml -[patch.crates-io] -redox-scheme = { git = "https://..." } # WRONG! -``` - -Use crates.io directly. Only `ring` (a Redox fork that doesn't exist on -crates.io) should use `git = ...`. - -### ❌ Loose version specifiers - -```toml -[dependencies] -libredox = "0.1" # WRONG! Allows any 0.1.x -redox-scheme = "^0.11" # WRONG! Allows any 0.11.x -``` - -Use `=X.Y.Z` to force EXACT versions. This is critical for ecosystem consistency. - -## Maintenance Scripts - -Three helper scripts in `/tmp/` (during build session, can be moved to -`local/scripts/`): - -- `apply_ecosystem_pins.py` — Adds `[patch.crates-io]` (initial step, now superseded) -- `fix_workspace_pins.py` — Detects and fixes `[patch.crates-io]` version-only bugs -- `fix_all_pins_v2.py` — Comprehensive fix: moves version-only patches to - `[workspace.dependencies]`, updates existing entries to `=X.Y.Z`, removes - empty `[patch.crates-io]` blocks -- `dedupe_deps.py` — Removes duplicate ecosystem crate entries in - `[dependencies]`/`[workspace.dependencies]` - -## When to Re-pin - -Re-check the canonical versions whenever: -- Redox releases a new `libredox`/`redox-scheme`/`redox_syscall` on crates.io -- A Redox upstream recipe bumps its ecosystem pins -- A new Red Bear local recipe is added that needs ecosystem crates -- Build fails with `call_ro`/`call_wo` or `syscall::Error` mismatch errors diff --git a/local/docs/HOOKS.md b/local/docs/HOOKS.md index 6e292ef3a8..1a6aa1dc19 100644 --- a/local/docs/HOOKS.md +++ b/local/docs/HOOKS.md @@ -5,26 +5,85 @@ Red Bear OS. Per AGENTS.md "absolutely NEVER DELETE, NEVER IGNORE" rule, hooks are always **opt-in** — operators explicitly choose to install them. No hook is auto-installed. -## Available hooks (2026-07-12, Phase 4.5) +## Available hooks (2026-07-18, Phase 4.5 + release-bump pipeline) + +### post-checkout-version-sync.sh (release-bump automation, opt-in) + +**File:** `local/scripts/post-checkout-version-sync.sh` + +**Purpose:** When an operator switches to a release branch whose name is an +anchored semver (e.g. `0.3.1` → `0.3.2`), automatically synchronise every +Cat 1 (in-house) and Cat 2 (upstream fork) crate version label to match the +new branch. This is the **label-only** half of a release bump — it rewrites +the `version = "..."` fields via `sync-versions.sh --no-regen` and does +nothing else. Source upgrades and lockfile regen remain explicit operator +steps (see `local/docs/RELEASE-BUMP-WORKFLOW.md`). + +**Exact guards** (any fail → silent exit 0, no mutations): +1. `$3 == 1` — branch checkouts only (file checkouts are skipped). +2. New branch name matches `^[0-9]+\.[0-9]+\.[0-9]+$` — silently skips + `master`, `submodule/*`, `recovered/*`, and detached HEAD. +3. No in-progress rebase / cherry-pick / merge (`.git/rebase-merge`, + `.git/rebase-apply`, `.git/CHERRY_PICK_HEAD`, `.git/MERGE_HEAD` all + absent). +4. `REDBEAR_NO_AUTO_SYNC` env var unset. +5. Working tree clean (`git status --porcelain` empty) — else warns to + stderr and skips. + +**What this hook NEVER does:** +- No network access (no `git fetch`, no upstream lookups). +- No git mutations (no commit, no branch, no push, no stash). +- No lockfile regen (never calls `sync-versions.sh --regen`). +- No source upgrades (never rebases forks or touches `local/sources/*/`). +- Does not run on non-semver branches. +- Does not run on a dirty working tree. +- Never blocks a checkout — always exits 0, even on internal failure. + +**Action when all guards pass:** reads the repo-root `Cargo.toml` +`[package] version`. If it differs from the branch version, runs +`sync-versions.sh --no-regen` (labels only) and prints the follow-up hint +pointing the operator at the explicit source-upgrade and lockfile-regen +commands. If already synced, exits silently. + +**Install (operator opt-in — NEVER auto-installed):** +```bash +# Via the installer (preferred): +./local/scripts/install-git-hooks.sh # post-checkout only +./local/scripts/install-git-hooks.sh --all # + pre-push + commit-msg + +# Or manually: +cp local/scripts/post-checkout-version-sync.sh .git/hooks/post-checkout +chmod +x .git/hooks/post-checkout +``` + +**Bypass for a single checkout:** +```bash +REDBEAR_NO_AUTO_SYNC=1 git checkout 0.3.2 +``` ### pre-push-checks.sh (recommended for serious operators) **File:** `local/scripts/pre-push-checks.sh` -**Purpose:** Run 4 critical pre-flight checks before every `git push`. -Closes the "silent drift" gap identified in AGENTS.md "Daily-upstream- -safe workflow". +**Purpose:** Run 7 pre-flight checks (4 core + 3 audit checks) before every +`git push`. Closes the "silent drift" gap identified in AGENTS.md +"Daily-upstream-safe workflow". **Checks:** 1. `sync-versions.sh --check` — Cat 0 + Cat 1 + Cat 2 versions 2. `verify-fork-versions.sh` — Cat 2 fork supremacy + content check 3. `verify-patch-content.py` — no orphan patches in `local/patches/` -4. `verify-collision-detection.py` — no config-vs-package conflicts +4. `verify-patch-content.py --report action` — orphan-patch operator decisions +5. `verify-patch-content.py --selftest` — orphan-patch detector self-test +6. `verify-collision-detection.py` — no config-vs-package conflicts +7. `verify-collision-detection.py --selftest` — collision detector self-test **Install:** ```bash cp local/scripts/pre-push-checks.sh .git/hooks/pre-push chmod +x .git/hooks/pre-push +# Or via the installer: +./local/scripts/install-git-hooks.sh --all ``` **Bypass:** @@ -73,16 +132,35 @@ in `local/scripts/` so operators can opt in by copying it to ## Audit -This directory was created in Phase 4.5 (Round 5). The current -hook inventory is: +This directory was created in Phase 4.5 (Round 5) and extended in the +release-bump pipeline (2026-07-18). The current hook inventory is: | Hook | File | Function | |------|------|----------| +| post-checkout | `local/scripts/post-checkout-version-sync.sh` | Auto-sync Cat 1+2 version labels on semver branch switch (label-only, no regen) | | pre-push | `local/scripts/pre-push-checks.sh` | Runs 7 pre-flight checks before push | | commit-msg | `local/scripts/commit-msg` | Auto-prepends `phase X.Y:` to commit messages | | (server-side) | (gitea admin only) | Pre-receive — defense in depth; not yet implemented | -## Full Tool Inventory (Round 10) +## Installer (`install-git-hooks.sh`) + +All three hooks are opt-in and can be installed individually with `cp` +(see each section above) or in one shot via the installer: + +```bash +./local/scripts/install-git-hooks.sh # post-checkout only (default) +./local/scripts/install-git-hooks.sh --all # post-checkout + pre-push + commit-msg +./local/scripts/install-git-hooks.sh --uninstall # remove default set (restores .bak if present) +./local/scripts/install-git-hooks.sh --uninstall --all +``` + +The installer is idempotent: if a hook is already up to date it reports so +and does nothing; if an existing hook differs it backs the old one up to +`.bak` (rotating older backups to `.bak.1`, `.2`, …) before +overwriting. `--uninstall` reverses an install and restores the most recent +backup if one exists. + +## Full Tool Inventory (Round 14) The build system has grown organically across 9+ rounds. Here is the complete tool inventory for reference. @@ -99,6 +177,11 @@ the complete tool inventory for reference. | `unblock-base-push.sh` | shell | Base fork deadlock resolver | `path-a`, `path-b`, `path-c` for 3 operator paths | | `commit-msg` | shell | Auto-phase-prefix hook | opt-in; install to `.git/hooks/commit-msg` | | `build-preflight.sh` | shell | Build-time validation | Called by `build-redbear.sh` | +| `bump-release.sh` | shell | Canonical release-bump orchestrator | `--with-sources`, `--with-external`, `--check`, `--dry-run` | +| `check-external-versions.sh` | shell↦py | External desktop-stack version reporter | `--strict`, `--no-fetch`, `--json` | +| `bump-graphics-recipes.sh` | shell↦py | Map-driven external source bumper | `--dry-run`, `--no-fetch` | +| `post-checkout-version-sync.sh` | shell | Auto label-sync on semver branch switch | opt-in hook; bypass via `REDBEAR_NO_AUTO_SYNC=1` | +| `install-git-hooks.sh` | shell | Idempotent opt-in hook installer | `--all`, `--uninstall` | All scripts are under `local/scripts/` and are designed to be operator-friendly. Hooks are strictly opt-in per AGENTS.md diff --git a/local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md b/local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md index 8f3d3435d0..555c28573c 100644 --- a/local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md +++ b/local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md @@ -17,7 +17,7 @@ kernel guidance in other docs. |----------|-------------| | `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Parent: CONSOLE-TO-KDE v4.0 (Kernel & Core Infrastructure) | | `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | Sibling: IRQ/PCI/MSI-X — not duplicated here | -| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Companion: relibc IPC surface — this plan covers kernel side | +| `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Companion: relibc IPC surface — this plan covers kernel side | | `ACPI-IMPROVEMENT-PLAN.md` | Sibling: ACPI power/shutdown — relevant for §4 (shutdown robustness) | | `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Consumer: desktop stack depends on kernel work here | @@ -546,7 +546,7 @@ Current state: all three are recipe-applied patches. Goal: upstream into relibc - Monitor upstream relibc for equivalent implementations - When upstream absorbs: shrink/drop Red Bear patch chain - When upstream does NOT absorb after 3+ months: promote to durable Red Bear-maintained -- See `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` Phase I5 +- See `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` Phase I5 ### Phase K4: Resource Limits and Process Management (Week 4-6) @@ -662,7 +662,7 @@ Phase K4 (limits/ptrace) ────────────────── |------|---------------|--------| | IRQ / MSI-X / IOMMU | `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | Waves 1-6 complete, hardware validation open | | ACPI shutdown / power | `ACPI-IMPROVEMENT-PLAN.md` | Waves 1-2 complete, sleep states deferred | -| relibc IPC surface | `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Phases I1-I5, message queues deferred | +| relibc IPC surface | `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Phases I1-I5, message queues deferred | | D-Bus / sessiond | `DBUS-INTEGRATION-PLAN.md` | Phase 1 complete, Phase 2 in progress | | Greeter / login | `GREETER-LOGIN-IMPLEMENTATION-PLAN.md` | Active, bounded proof passing | | Desktop path | `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Phase 1-5 model, KWin building | @@ -738,7 +738,7 @@ local/scripts/test-credential-syscalls-guest.sh # In-guest checker - `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — Canonical comprehensive plan - `docs/01-REDOX-ARCHITECTURE.md` — Architecture reference - `local/docs/IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` — IRQ/PCI plan (sibling) -- `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — IPC surface plan (companion) +- `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — IPC surface plan (companion) - `local/docs/ACPI-IMPROVEMENT-PLAN.md` — ACPI/shutdown plan (sibling) - `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — Desktop path plan (consumer) - `recipes/core/kernel/source/src/syscall/mod.rs` — Syscall dispatch (primary implementation target) diff --git a/local/docs/LOCAL-FORK-SUPREMACY-POLICY.md b/local/docs/LOCAL-FORK-SUPREMACY-POLICY.md index 2a42061b7f..1e84ad50df 100644 --- a/local/docs/LOCAL-FORK-SUPREMACY-POLICY.md +++ b/local/docs/LOCAL-FORK-SUPREMACY-POLICY.md @@ -238,7 +238,7 @@ For relibc/redox-rt specifically: ## Related Documents - `local/AGENTS.md` — full Red Bear agent guidance (branches, submodules, durability) -- `local/docs/SOURCE-ARCHIVAL-POLICY.md` — how sources are frozen and archived +- `local/docs/archived/archived/SOURCE-ARCHIVAL-POLICY.md` — how sources are frozen and archived - `local/docs/PATCH-GOVERNANCE.md` (referenced in `local/AGENTS.md`) — how patches are rebased and applied - `README.md` (project root) — high-level description including the fork model \ No newline at end of file diff --git a/local/docs/MC-EDIT-PARITY-ASSESSMENT.md b/local/docs/MC-EDIT-PARITY-ASSESSMENT.md deleted file mode 100644 index fb41394009..0000000000 --- a/local/docs/MC-EDIT-PARITY-ASSESSMENT.md +++ /dev/null @@ -1,488 +0,0 @@ -# tlcedit vs mcedit — Comprehensive Parity Assessment - -**Created:** 2026-06-20 -**Source:** MC default keymap (`misc/mc.default.keymap` `[editor]` section) vs tlcedit `handlers.rs` -**Goal:** Every keybinding in mcedit must match in tlcedit and provide the same action - ---- - -## Executive Summary - -**Overall parity: ~93%** (up from ~45% pre-P1, ~75% after P1+P2, ~85% after P1+P2+P3, ~90% after P3.5, after P1+P2+P3+P3.5+P4a+P4b implementation, 2026-06-20) - -| Category | Count | Severity | Status | -|----------|-------|----------|--------| -| **Key conflicts** (same key, different action) | 0 | ~~🔴 CRITICAL~~ | ✅ All fixed (P1) | -| **Wrong keys** (action exists, different key) | 0 | ~~🟡 HIGH~~ | ✅ All fixed (P1) | -| **Missing actions** (no key, no implementation) | ~4 | 🟡 MEDIUM | P1/P2/P3/P4 reduced from 22 | -| **Correct matches** | ~70 | ✅ | Up from ~28 | - ---- - -## CRITICAL: Key Conflicts - -These bindings use the SAME key in both editors but perform DIFFERENT actions. -These MUST be fixed first — they cause user confusion and data loss. - -### C1: Ctrl-S — Save vs SyntaxOnOff - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-S` | Toggle syntax highlighting on/off | -| tlcedit | `Ctrl-S` | **Save file** (also F2) | - -**Problem:** A user pressing Ctrl-S expecting syntax toggle will unexpectedly save. -**Fix:** tlcedit should use Ctrl-S for SyntaxOnOff. Save is already on F2 (correct). -**Impact:** Remove Ctrl-S as a save shortcut in the editor dispatcher. - -### C2: Ctrl-Y — DeleteLine vs Redo - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-Y` | Delete current line | -| tlcedit | `Ctrl-Y` | **Redo** | - -**Problem:** Redo on Ctrl-Y is a Windows/VSCode convention, not MC. -**Fix:** Move redo to `Alt-R` (MC's key). Implement DeleteLine on Ctrl-Y. -**Impact:** Add `delete_line()` to buffer; rewire handlers. - -### C3: Ctrl-P — SpellCheck vs Macro Playback - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-P` | Spell check current word | -| tlcedit | `Ctrl-P` | **Play macro** | - -**Problem:** Macro playback on Ctrl-P conflicts with MC spell check. -**Fix:** MC doesn't have a dedicated macro playback key — macros in MC use -`Ctrl-R` to start/stop recording and then numeric keys (Ctrl-0..9) to replay. -Move macro playback to a different binding. -**Impact:** Change macro system. - -### C4: Ctrl-L — Refresh vs Relative Line Numbers - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-L` | Refresh screen (redraw) | -| tlcedit | `Ctrl-L` | **Toggle relative line numbers** | - -**Problem:** Relative line numbers is a TLC addition not in MC. -**Fix:** Move relative line toggle to `Alt-N` (MC's ShowNumbers key). -Ctrl-L should trigger a screen refresh. -**Impact:** Minor — rewire two bindings. - -### C5: Ctrl-Z — Undo vs WordLeft Alias - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-Z` | Move cursor one word left (alias for Ctrl-Left) | -| tlcedit | `Ctrl-Z` | **Undo** | - -**Problem:** Undo on Ctrl-Z is standard everywhere EXCEPT MC. -**Decision needed:** Do we match MC exactly, or keep the universal Ctrl-Z=Undo? -**Recommendation:** Keep Ctrl-Z as Undo (it's a universal convention that even MC -users expect in modern contexts). Document the deviation. - ---- - -## HIGH: Wrong Keys (Action Exists, Different Key) - -These actions exist in tlcedit but are triggered by a different key than MC expects. - -### W1: Search — F7 vs Alt-F - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `F7` | Open search dialog | -| tlcedit | `Alt-F` | Open search prompt | - -**Fix:** Add F7 as primary search key. Keep Alt-F as alias. -**Also:** `F17` for search-continue (repeat last search). - -### W2: Replace — F4 vs Alt-% - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `F4` | Open replace dialog | -| tlcedit | `Alt-%` (Alt-Shift-5) | Open replace prompt | - -**Problem:** F4 in tlcedit currently has no binding in the editor. -**Fix:** Map F4 to Replace. Keep Alt-% as alias. -**Also:** `F14` for replace-continue. - -### W3: SaveAs — F12 vs Shift-F2 - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `F12` or `Ctrl-F2` | Save As | -| tlcedit | `Shift-F2` | Save As | - -**Fix:** Add F12 as primary SaveAs key. Keep Shift-F2 as alias. - -### W4: Undo — Ctrl-U vs Ctrl-Z - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Ctrl-U` | Undo | -| tlcedit | `Ctrl-Z` | Undo | - -**Fix:** Add Ctrl-U as MC-compatible undo key. Keep Ctrl-Z as modern alias. - -### W5: Redo — Alt-R vs Ctrl-Y - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Alt-R` | Redo | -| tlcedit | `Ctrl-Y` | Redo | - -**Fix:** Move redo to Alt-R (matching MC). Free up Ctrl-Y for DeleteLine. -Keep Ctrl-Y as secondary alias for transition. - -### W6: Bookmark — Alt-K vs Alt-M - -| Editor | Key | Action | -|--------|-----|--------| -| mcedit | `Alt-K` | Set bookmark at cursor | -| tlcedit | `Alt-M` | Set bookmark (BookmarkSet prompt) | - -**Fix:** Change BookmarkSet trigger from Alt-M to Alt-K. Free up Alt-M. -**Also:** Alt-O for BookmarkFlush (clear all), Alt-I for BookmarkPrev. - ---- - -## MEDIUM: Missing Actions (22 items) - -These MC actions have no tlcedit equivalent at all. - -### Navigation (4) - -| MC Action | MC Key | Description | Priority | -|-----------|--------|-------------|----------| -| ScrollUp | `Ctrl-Up` | Scroll viewport up 1 line (cursor stays) | P1 | -| ScrollDown | `Ctrl-Down` | Scroll viewport down 1 line (cursor stays) | P1 | -| TopOnScreen | `Ctrl-PgUp` | Move cursor to top visible line | P2 | -| BottomOnScreen | `Ctrl-PgDn` | Move cursor to bottom visible line | P2 | - -### Editing (6) - -| MC Action | MC Key | Description | Priority | -|-----------|--------|-------------|----------| -| DeleteLine | `Ctrl-Y` | Delete entire current line | P1 | -| DeleteToEnd | `Ctrl-K` | Delete from cursor to end of line | P1 | -| DeleteToWordBegin | `Alt-Backspace` | Delete word before cursor | P1 | -| DeleteToWordEnd | `Alt-D` | Delete word after cursor | P1 | -| Return | `Shift-Enter` / `Ctrl-Enter` | Insert newline above current | P2 | -| InsertOverwrite | `Insert` | Toggle insert/overwrite mode | P2 | - -### Selection (8) - -| MC Action | MC Key | Description | Priority | -|-----------|--------|-------------|----------| -| MarkPageUp | `Shift-PgUp` | Select one page up | P1 | -| MarkPageDown | `Shift-PgDn` | Select one page down | P1 | -| MarkToWordBegin | `Ctrl-Shift-Left` | Select to word beginning | P1 | -| MarkToWordEnd | `Ctrl-Shift-Right` | Select to word end | P1 | -| MarkToFileBegin | `Ctrl-Shift-Home` | Select to start of file | P2 | -| MarkToFileEnd | `Ctrl-Shift-End` | Select to end of file | P2 | -| MarkColumn | `F13` | Toggle column selection mode | P3 | -| Store / Cut | `Ctrl-Insert` / `Shift-Delete` | MC-style clipboard ops | P1 | - -### Editor Features (4) - -| MC Action | MC Key | Description | Priority | -|-----------|--------|-------------|----------| -| Help | `F1` | Built-in help screen | P2 | -| Shell | `Ctrl-O` | Switch to subshell (suspend editor) | P1 | -| InsertFile | `F15` | Insert file contents at cursor | P2 | -| InsertLiteral | `Ctrl-Q` | Insert next key literally | P2 | - -### File Navigation (2) - -| MC Action | MC Key | Description | Priority | -|-----------|--------|-------------|----------| -| FilePrev | `Alt-Minus` | Previous file in edit history | P3 | -| FileNext | `Alt-Plus` | Next file in edit history | P3 | - ---- - -## Viewer (mcview) Parity - -### mcview Keybindings (from `[viewer]` section) - -| MC Action | MC Key | tlcview Status | -|-----------|--------|----------------| -| Help | `F1` | ❌ Missing | -| WrapMode | `F2` | ✅ Has wrap toggle | -| Quit | `F3` / `F10` / `q` / `Esc` | ✅ Esc/F10 | -| HexMode | `F4` | ✅ Has hex toggle | -| Goto | `F5` | ⚠️ Has `g` key, missing F5 | -| Search | `F7` | ⚠️ Has `/`, missing F7 | -| SearchForward | `/` | ✅ | -| SearchBackward | `?` | ❌ Missing | -| SearchContinue | `F17` / `n` | ✅ Has `n` | -| SearchForwardContinue | `Ctrl-S` | ❌ Missing | -| SearchBackwardContinue | `Ctrl-R` | ❌ Missing | -| MagicMode | `F8` | ❌ Missing | -| NroffMode | `F9` | ✅ Has nroff | -| Home | `Ctrl-A` | ⚠️ Has Home key, missing Ctrl-A | -| End | `Ctrl-E` | ⚠️ Has End key, missing Ctrl-E | -| Left | `h` / `Left` | ✅ Left | -| Right | `l` / `Right` | ✅ Right | -| Up | `k` / `y` / `Insert` / `Up` / `Ctrl-P` | ⚠️ Has Up, missing vim keys | -| Down | `j` / `e` / `Delete` / `Down` / `Enter` / `Ctrl-N` | ⚠️ Has Down, missing vim keys | -| PageDown | `f` / `Space` / `PgDn` / `Ctrl-V` | ⚠️ Has PgDn, missing aliases | -| PageUp | `b` / `PgUp` / `Alt-V` / `Backspace` | ⚠️ Has PgUp, missing aliases | -| Top | `Home` / `Ctrl-Home` / `Ctrl-PgUp` / `g` | ⚠️ Has Home, missing aliases | -| Bottom | `End` / `Ctrl-End` / `Ctrl-PgDn` / `Shift-G` | ⚠️ Has End, missing aliases | -| BookmarkGoto | `m` | ❌ Missing | -| Bookmark | `r` | ❌ Missing | -| FileNext | `Ctrl-F` | ❌ Missing | -| FilePrev | `Ctrl-B` | ❌ Missing | -| Ruler | `Alt-R` | ❌ Missing | -| Shell | `Ctrl-O` | ❌ Missing | -| History | `Alt-Shift-E` | ❌ Missing | - ---- - -## Improvement Plan - -### Phase P1: Fix Critical Conflicts + High-Priority Missing ✅ COMPLETE (2026-06-20) - -**Objective:** Eliminate key conflicts and add the most common missing actions. - -#### Step 1: Fix Key Conflicts - -1. **Ctrl-S → SyntaxOnOff** (remove Save shortcut; F2 remains) -2. **Ctrl-Y → DeleteLine** (move Redo to Alt-R) -3. **Ctrl-P → SpellCheckCurrentWord placeholder** (move macro playback) -4. **Ctrl-L → Screen Refresh** (move relative-lines to Alt-N) -5. **Ctrl-U → Undo** (add as alias for Ctrl-Z) -6. **Macro playback → Ctrl-0..9** (MC-style: Ctrl-R records, Ctrl-0 replays) - -**Files to modify:** -- `src/editor/handlers.rs` — rewire all conflict bindings -- `src/editor/macro.rs` — new numeric-key macro system - -#### Step 2: Add Missing High-Priority Actions - -1. **DeleteLine** (`Ctrl-Y`) — `buffer.delete_line()` -2. **DeleteToEnd** (`Ctrl-K`) — delete from cursor to EOL -3. **DeleteToWordBegin** (`Alt-Backspace`) — delete word backward -4. **DeleteToWordEnd** (`Alt-D`) — delete word forward -5. **ScrollUp/ScrollDown** (`Ctrl-Up`/`Ctrl-Down`) — viewport scroll without cursor move -6. **TopOnScreen/BottomOnScreen** (`Ctrl-PgUp`/`Ctrl-PgDn`) -7. **Shell** (`Ctrl-O`) — subshell from editor (already exists in filemanager) -8. **Store/Cut** (`Ctrl-Insert`/`Shift-Delete`) — MC-style clipboard - -**Files to modify:** -- `src/editor/buffer.rs` — add `delete_line()`, `delete_to_end_of_line()` -- `src/editor/cursor.rs` — add `delete_word_backward()`, `delete_word_forward()` -- `src/editor/handlers.rs` — add new key bindings -- `src/editor/view.rs` — add `scroll_up()`/`scroll_down()` (viewport-only) - -#### Step 3: Fix Wrong Keys (add MC-primary aliases) - -1. **F7 → Search** (keep Alt-F as alias) -2. **F4 → Replace** (keep Alt-% as alias) -3. **F12 → SaveAs** (keep Shift-F2 as alias) -4. **F17 → SearchContinue** (repeat last search) -5. **F14 → ReplaceContinue** (repeat last replace) -6. **Alt-R → Redo** (keep Ctrl-Y as secondary during transition) -7. **Alt-K → Bookmark** set (keep Alt-M during transition) -8. **Alt-O → BookmarkFlush** (clear all bookmarks) -9. **Alt-I → BookmarkPrev** (previous bookmark) -10. **Alt-N → ShowNumbers** (toggle line numbers) - -**Files to modify:** -- `src/editor/handlers.rs` — add all F-key and Alt-key aliases - -#### Step 4: Add Missing Selection Keys - -1. **Shift-PgUp / Shift-PgDn** — select page up/down -2. **Ctrl-Shift-Left / Ctrl-Shift-Right** — select word -3. **Ctrl-Shift-Home / Ctrl-Shift-End** — select to file start/end - -**Files to modify:** -- `src/editor/cursor.rs` — add `select_page_up()`, `select_page_down()` -- `src/editor/handlers.rs` — add shift+page and ctrl+shift+arrow bindings - -### Phase P2: Viewer Parity ✅ COMPLETE (2026-06-20) - -1. Add F5/F7 for goto/search in viewer -2. Add `?` backward search -3. Add Ctrl-S/Ctrl-R for search direction continue -4. Add vim movement keys (h/j/k/l) in viewer -5. Add Space/`f` for page down, `b`/Backspace for page up -6. Add `g`/`Shift-G` for top/bottom -7. Add Ctrl-A/Ctrl-E for Home/End -8. Add `m`/`r` for viewer bookmarks - -**Files to modify:** -- `src/viewer/mod.rs` — expand key handler -- `src/viewer/search.rs` — add backward search - -### Phase P3: Editor Feature Gaps ✅ COMPLETE (2026-06-20) - -**Done:** -- ✅ F15 InsertFile — insert file contents at cursor -- ✅ Insert/overwrite toggle — Insert key toggles mode, [OVR] shown in status bar -- ✅ Ctrl-Q InsertLiteral — insert next key literally -- ✅ Shift-Enter — insert newline above current line -- ✅ F1 Help — built-in keybinding reference dialog (6 categories, 30+ bindings) -- ✅ F8 MagicMode in viewer — auto-detect binary vs text via magic::detect_mode - -**Deferred P3 items (require significant infrastructure):** -1. **F9 Menu** — editor command menu (needs standalone menu system, separate from filemanager) -2. **F11 UserMenu** — user-defined command menu (needs INI parser integration in editor context) -3. **Alt-Enter Find** — quick file find (needs file manager integration) -4. **Alt-Minus/Alt-Plus** — file history navigation (needs multi-file tab support) - -### Phase P4: Advanced Features (future) - -1. **Column selection mode** (F13) — block/column selection -2. **Spell check** (Ctrl-P) — integrate spell checker -3. **Block shift** — indent/outdent selected block -4. **Paragraph up/down** — move by paragraph -5. **Window management** — multi-file tabs, split, fullscreen - ---- - -## Summary Table: Complete Keybinding Comparison - -### Editor Bindings (sorted by MC key) - -| MC Key | MC Action | tlcedit Key | tlcedit Action | Status | -|--------|-----------|-------------|----------------|--------| -| `F1` | Help | `F1` | Keybinding reference | ✅ P3 | -| `F2` | Save | `F2` | Save | ✅ | -| `F3` | Mark (toggle selection) | `F3` | Toggle selection | ✅ | -| `F4` | Replace | `F4` | Replace prompt | ✅ P1 | -| `F5` | Copy block | `F5` | Copy block | ✅ | -| `F6` | Move block | `F6` | Move (cut) block | ✅ | -| `F7` | Search | `F7` | Search prompt | ✅ P1 | -| `F8` | Delete block | `F8` | Delete block | ✅ | -| `F9` | Menu | — | — | ❌ Deferred (needs menu system) | -| `F10` | Quit | `F10` / `Esc` | Close | ✅ | -| `F11` | UserMenu | — | — | ❌ Deferred (needs INI integration) | -| `F12` | SaveAs | `F12` | SaveAs prompt | ✅ P1 | -| `F13` | MarkColumn | — | — | ❌ P4 | -| `F14` | ReplaceContinue | — | — | ❌ P4 | -| `F15` | InsertFile | `F15` | Insert file at cursor | ✅ P3 | -| `F17` | SearchContinue | `F17` | Repeat last search | ✅ P1 | -| `Insert` | InsertOverwrite | `Insert` | Toggle insert/overwrite | ✅ P3 | -| `Enter` | Enter (newline) | `Enter` | Newline + auto-indent | ✅ | -| `Shift-Enter` | Return (newline above) | `Shift-Enter` | Newline above | ✅ P3 | -| `Backspace` | BackSpace | `Backspace` / `Ctrl-H` | delete_back | ✅ P1 | -| `Delete` | Delete | `Delete` / `Ctrl-D` | delete_forward | ✅ P1 | -| `Tab` | Tab | `Tab` | insert tab | ✅ | -| `Up` | Up | `Up` | move_up | ✅ | -| `Down` | Down | `Down` | move_down | ✅ | -| `Left` | Left | `Left` | move_left | ✅ | -| `Right` | Right | `Right` | move_right | ✅ | -| `Home` | Home | `Home` | move_home | ✅ | -| `End` | End | `End` | move_end | ✅ | -| `PgUp` | PageUp | `PgUp` | move_page_up | ✅ | -| `PgDn` | PageDown | `PgDn` | move_page_down | ✅ | -| `Ctrl-Home` | Top of file | `Ctrl-Home` | set_cursor(0) | ✅ | -| `Ctrl-End` | Bottom of file | `Ctrl-End` | set_cursor(end) | ✅ | -| `Ctrl-Up` | ScrollUp | `Ctrl-Up` | scroll viewport | ✅ P1 | -| `Ctrl-Down` | ScrollDown | `Ctrl-Down` | scroll viewport | ✅ P1 | -| `Ctrl-PgUp` | TopOnScreen | `Ctrl-PgUp` | Top of screen | ✅ P1 | -| `Ctrl-PgDn` | BottomOnScreen | `Ctrl-PgDn` | Bottom of screen | ✅ P1 | -| `Ctrl-Left` | WordLeft | `Ctrl-Left` | move_word_backward | ✅ | -| `Ctrl-Right` | WordRight | `Ctrl-Right` | move_word_forward | ✅ | -| `Ctrl-Z` | WordLeft (alias) | `Ctrl-Z` | Undo (universal) | ✅ (kept) | -| `Ctrl-X` | WordRight (alias) | — | — | ❌ P4 | -| `Ctrl-U` | Undo | `Ctrl-U` / `Ctrl-Z` | Undo | ✅ P1 | -| `Ctrl-Y` | DeleteLine | `Ctrl-Y` | DeleteLine | ✅ P1 | -| `Ctrl-K` | DeleteToEnd | `Ctrl-K` | Delete to end of line | ✅ P1 | -| `Ctrl-D` | Delete (alias) | `Ctrl-D` | Delete (alias) | ✅ P1 | -| `Ctrl-H` | BackSpace (alias) | `Ctrl-H` | BackSpace (alias) | ✅ P1 | -| `Ctrl-S` | SyntaxOnOff | `Ctrl-S` | Toggle syntax highlight | ✅ P1 | -| `Ctrl-L` | Refresh | `Ctrl-L` | Refresh screen | ✅ P1 | -| `Ctrl-R` | MacroStartStopRecord | `Ctrl-R` | Macro toggle | ✅ | -| `Ctrl-P` | SpellCheckCurrentWord | `Ctrl-P` | Macro play | ⚠️ P4 (spell check) | -| `Ctrl-O` | Shell | — | — | ❌ P4 | -| `Ctrl-Q` | InsertLiteral | `Ctrl-Q` | Insert next key literally | ✅ P3 | -| `Ctrl-N` | EditNew | — | — | ❌ P4 | -| `Ctrl-F` | BlockSave | — | — | ❌ P4c | -| `Ctrl-Insert` | Store (copy) | `Ctrl-Insert` | Copy selection | ✅ P4a | -| `Ctrl-N` | EditNew | `Ctrl-N` | New file (clear buffer) | ✅ P4a | -| `Shift-Insert` | Paste | `Ctrl-V` / `Shift-Insert` | Paste | ✅ P1 | -| `Shift-Delete` | Cut | `Shift-Delete` | Cut selection | ✅ P4a | -| `Shift-Tab` | Tab (alias / unindent) | `Shift-Tab` | Unindent selection | ✅ P4a | -| `Alt-R` | Redo | `Alt-R` | Redo | ✅ P1 | -| `Alt-L` | Goto line | `Alt-L` | GotoLine prompt | ✅ | -| `Alt-B` | MatchBracket | `Alt-B` | match_bracket | ✅ | -| `Alt-P` | ParagraphFormat | `Alt-P` | format_paragraph | ✅ | -| `Alt-K` | Bookmark | `Alt-K` | BookmarkSet | ✅ P1 | -| `Alt-J` | BookmarkNext | `Alt-J` | BookmarkJump | ✅ (different action name) | -| `Alt-I` | BookmarkPrev | `Alt-I` | BookmarkPrev | ✅ P1 | -| `Alt-O` | BookmarkFlush | `Alt-O` | BookmarkClear | ✅ P1 | -| `Alt-N` | ShowNumbers | `Alt-N` | Toggle relative lines | ✅ P1 | -| `Alt-D` | DeleteToWordEnd | `Alt-D` | Delete word forward | ✅ P1 | -| `Alt-Backspace` | DeleteToWordBegin | `Alt-Backspace` | Delete word backward | ✅ P1 | -| `Alt-Tab` | Complete | `Alt-Tab` | word_complete | ✅ | -| `Alt-T` | Sort | — | — | ❌ P4 | -| `Alt-M` | Mail | — | — | ❌ P4 | -| `Alt-U` | ExternalCommand | — | — | ❌ P4 | -| `Alt-Enter` | Find | — | — | ❌ Deferred (needs FM) | -| `Alt-Minus` | FilePrev | — | — | ❌ Deferred (needs multi-file) | -| `Alt-Plus` | FileNext | — | — | ❌ Deferred (needs multi-file) | -| `Alt-E` | SelectCodepage | — | — | ❌ P4 | -| `Alt-Shift-E` | History | — | — | ❌ P4 | -| `Shift-Left` | MarkLeft | `Shift-Left` | select_left | ✅ | -| `Shift-Right` | MarkRight | `Shift-Right` | select_right | ✅ | -| `Shift-Up` | MarkUp | `Shift-Up` | start_sel + move_up | ✅ | -| `Shift-Down` | MarkDown | `Shift-Down` | start_sel + move_down | ✅ | -| `Shift-Home` | MarkToHome | `Shift-Home` | select_to_home | ✅ | -| `Shift-End` | MarkToEnd | `Shift-End` | select_to_end | ✅ | -| `Shift-PgUp` | MarkPageUp | `Shift-PgUp` | Select page up | ✅ P1 | -| `Shift-PgDn` | MarkPageDown | `Shift-PgDn` | Select page down | ✅ P1 | -| `Ctrl-Shift-Left` | MarkToWordBegin | `Ctrl-Shift-Left` | Select word backward | ✅ P1 | -| `Ctrl-Shift-Right` | MarkToWordEnd | `Ctrl-Shift-Right` | Select word forward | ✅ P1 | -| `Ctrl-Shift-Home` | MarkToFileBegin | `Ctrl-Shift-Home` | Select to file begin | ✅ P1 | -| `Ctrl-Shift-End` | MarkToFileEnd | `Ctrl-Shift-End` | Select to file end | ✅ P1 | -| `Alt-Underline` | ShowTabTws | — | — | ❌ P4 | - -### TLC-only features (not in MC) - -| Key | Action | Keep? | -|-----|--------|-------| -| `Ctrl-]` | Jump to tag | ✅ Keep (TLC advantage) | -| `Ctrl-T` | Pop tag stack | ✅ Keep | -| `Ctrl-F1` | Toggle code fold | ✅ Keep | -| `Alt-W` | Toggle word wrap | ✅ Keep | -| `Shift-F2` | SaveAs (alias) | ✅ Keep as alias | -| `Ctrl-V` | Paste (alias) | ✅ Keep as alias | -| `Ctrl-Z` | Undo (universal) | ✅ Keep as alias | - ---- - -## Conflict Resolution Decision Matrix - -| Conflict | MC Key/Action | TLC Key/Action | Resolution | Status | -|----------|---------------|----------------|------------|--------| -| Ctrl-S | SyntaxOnOff | Save | Ctrl-S → SyntaxOnOff. Save = F2 only. | ✅ P1 | -| Ctrl-Y | DeleteLine | Redo | Ctrl-Y → DeleteLine. Redo → Alt-R. | ✅ P1 | -| Ctrl-P | SpellCheck | Macro play | Kept as Macro play. Spell check → P4. | ⚠️ P4 | -| Ctrl-L | Refresh | Relative lines | Ctrl-L → Refresh. Relative → Alt-N. | ✅ P1 | -| Ctrl-Z | WordLeft | Undo | Kept Ctrl-Z = Undo (universal). Ctrl-U alias. | ✅ P1 | -| Alt-K | Bookmark | BookmarkSet (was Alt-M) | Alt-K → BookmarkSet. | ✅ P1 | -| Alt-O | BookmarkFlush | BookmarkClear (was Alt-K) | Alt-O → BookmarkClear. | ✅ P1 | -| Ctrl-Q | InsertLiteral | Close (was close shortcut) | Ctrl-Q → InsertLiteral. Close = F10/Esc. | ✅ P3 | - ---- - -## Effort Estimate - -| Phase | Duration | Deliverables | Status | -|-------|----------|--------------|--------| -| P1: Conflicts + Core Missing | 1-2 weeks | 5 conflict fixes, 15+ new actions, 6 selection keys | ✅ COMPLETE | -| P2: Viewer Parity | 1 week | F-key aliases, vim keys, backward search, Ctrl-A/E | ✅ COMPLETE | -| P3: Feature Gaps | 2-3 weeks | Help, InsertFile, InsertLiteral, overwrite, Shift-Enter, MagicMode | ✅ COMPLETE | -| P3.5: Handoff + Buttonbar | 1 week | start_line plumbing, +N CLI, editor/viewer buttonbar | ✅ COMPLETE | -| P4a: More Bindings | 1 week | Ctrl-Insert (Copy), Shift-Delete (Cut), Ctrl-N (EditNew), Shift-Tab (Unindent) | ✅ COMPLETE | -| P4b: Cursor Position | 1 week | filepos save/restore on close/open (MC `~/.mc/filepos` parity) | ✅ COMPLETE | -| P4c: Advanced | Future | Column select, sort, file-nav, spell check, Mail/ExternalCommand | 📋 Planned | -| **Total P1-P4b** | **7-9 weeks** | **~93% MC parity** | **✅ DONE** | diff --git a/local/docs/MULTITHREADING-COMPREHENSIVE-ASSESSMENT-AND-PLAN.md b/local/docs/MULTITHREADING-COMPREHENSIVE-ASSESSMENT-AND-PLAN.md deleted file mode 100644 index 21853a55d7..0000000000 --- a/local/docs/MULTITHREADING-COMPREHENSIVE-ASSESSMENT-AND-PLAN.md +++ /dev/null @@ -1,763 +0,0 @@ -# Red Bear OS — Multi-Threading Comprehensive Assessment and Implementation Plan - -**Date:** 2026-07-02 (initial assessment); 2026-07-02 (Phase 0c patch recovery complete) -**Scope:** Full-stack multi-threading audit: hardware/SMP, kernel scheduler, kernel futex, kernel syscall ABI, relibc pthreads, userspace threading correctness and performance -**Status:** Authoritative — supersedes `local/docs/archived/KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` and the restored `local/docs/KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` lineage for all threading matters -**Validation levels:** `builds` → `enumerates` → `usable` → `validated` → `hardware-validated` - ---- - -## UPDATE — Phase 0c Patch Recovery (2026-07-02) - -The assessment in §1 below was accurate as of the time of writing. The Phase 0c -patch recovery was then executed in the same session, landing the following -commits on the local kernel fork (`local/sources/kernel/`): - -| Commit | Effect | -|--------|--------| -| `ed3f0e1` | **P6-futex-sharding**: replaces single global `Mutex` with 64-shard hash table | -| `5fb42fc` | **RUN_QUEUE_COUNT pre-flight**: defines `pub const RUN_QUEUE_COUNT: usize = 40;` (was missing from patch chain) | -| `cbf051e` | **P7-cache-affine-context (manual)**: surgically inserts `SchedPolicy` enum, `SCHED_PRIORITY_LEVELS`, helper functions, and 9 new Context fields (last_cpu, sched_policy, sched_rt_priority, sched_rr_ticks_consumed, sched_static_prio, sched_rr_quantum, vruntime, futex_pi_*, PhysicalAddress import) | -| `f7652fc` | **P5-context-mod-sched + P8-percpu-sched + P8-percpu-wiring**: PerCpuSched struct with SyncUnsafeCell-wrapped per-CPU run queues, get_percpu_block helper, full per-CPU scheduler wiring in switch.rs (pick_next_from_queues, pick_next_from_global_queues, select_next_context) | -| `7fc8bbf` | **P8-initial-placement + P9-numa-topology + P9-proc-lock-ordering**: least-loaded-CPU spawn, NUMA topology hints, proc scheme lock order fix | -| `327c150` | **`set_sched_policy` + `set_sched_other_prio`**: missing Context methods called by proc scheme handles | -| `e8ec916` | **fadt usize/u32 type mismatch fix**: changes FADT_MIN_SIZE constants to `u32` to match `Sdt::length()` | -| `4789d54` | **SchedPolicy/Name/Priority proc scheme handles**: adds `/proc//{name, sched-policy, priority}` paths and read/write handlers | - -**Upstream check (bg_27f3578a, 2026-07-02):** verified that `gitlab.redox-os.org/redox-os/kernel` -master (commit `aa7e7d2f44ba7cd9d1b007d37db139b345d46b8a`) has **NONE** of these features. The -local fork is the sole implementation. No upstream cherry-picks are available. - -**Plan-vs-actual state:** - -| Plan claim (Section 1) | Actual state after Phase 0c | -|---|---| -| "Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine)" | ✅ All 5 features now present. Per-CPU `PerCpuSched` with `run_queues`, `steal_work()`, `migrate_one_context()`, `maybe_balance_queues()`, `vruntime` CFS-style weighting, `last_cpu` cache-affine vruntime bonus, `SchedPolicy::Fifo`/`RoundRobin` RT scanning in `pick_next_from_queues` | -| "Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET)" | 🟡 **Sharding done** (64-shard hash). REQUEUE/PI/robust/WAKE_OP/BITSET still missing. | -| "relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent" | 🟡 relibc fork only has the `pthread_cond_signal` POSIX fix so far. `sched_*`, robust, PI still pending in relibc. | -| "❌ Missing from relibc: CPU affinity API, Thread naming" | 🟡 **Kernel side done** — `/proc//{sched-policy, name, priority}` handles. relibc pthread_setname_np / pthread_setaffinity_np / sched_setscheduler still pending. | -| "cargo check has 1 pre-existing error" | ✅ **Fixed** — `cargo check` now exits 0 with 0 errors. | - -**Phase 0c status: kernel side complete (all 8 of 8 applicable kernel P5–P9 patches -re-applied or made obsolete by the existing refactored scheduler). Remaining work -is in the relibc fork (Phase 0e) and the futex-REQUEUE/PI/robust work (Phase 1).** - -The detailed analysis in §1–§9 below is preserved as historical record. The status -column "🚧 Missing" in §1 should be re-read as "now present in the local kernel -fork, pending relibc userspace wiring." - ---- - ---- - -## 1. Executive Summary - -### The Critical Finding — Lost Threading Work - -The P5–P9 scheduler and futex enhancement work (documented as "complete" in the archived -plans) was **lost during the local fork migration** (2026-06). The local forks at -`local/sources/kernel/` and `local/sources/relibc/` were created from **upstream Redox -baselines** that did NOT include the Red Bear enhancement patches. The patches exist in -`local/patches/kernel/` and `local/patches/relibc/` but are **not wired into the recipes** -(both `recipe.toml` files use `path = "..."` with no `patches = [...]` list). - -**Impact:** The running kernel has: -- Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine) -- Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET) -- relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent - -**Recovery:** 13 of 18 kernel P5–P9 patches apply cleanly to the current fork. 5 fail due to -patch-chain dependencies (they expect earlier patches applied first). The bulk of the work is -recoverable by re-applying patches to the forks and committing them. - -### What Actually Works Today - -| Layer | Status | Detail | -|-------|--------|--------| -| **SMP boot** | ✅ Solid | INIT→SIPI sequence correct, per-CPU PCR via GS_BASE, x2APIC support | -| **Context switching** | ✅ Solid | FPU/SIMD/AVX state save via XSAVE, FSBASE/GSBASE swap (FSGSBASE or MSR), correct callee-saved register save | -| **TLB shootdown protocol** | ✅ Correct | AtomicBool flag + IPI + ack counter with `fence(SeqCst)` race prevention | -| **Basic thread lifecycle** | ✅ Functional | pthread_create/join/detach/exit through proc scheme + redox_rt clone | -| **Basic synchronization** | ✅ Functional | Futex-backed mutex, condvar, rwlock, barrier, spinlock, once | -| **TLS** | ✅ Functional | ELF PT_TLS + pthread_key_create/getspecific/setspecific | -| **Per-CPU data** | ✅ Functional | PercpuBlock via GS_BASE, all per-CPU state accessible | -| **Signal delivery** | ✅ Functional | Shared-memory Sigcontrol pages, per-thread masks, trampoline | -| **Scheduler algorithm** | 🚧 Basic DWRR | 40 priority levels, geometric weights, cooperative preemption (3-tick quantum) | -| **Futex operations** | 🚧 Basic only | WAIT/WAIT64/WAKE with single global mutex | -| **SMP load balancing** | ❌ Missing | No work stealing, no migration, contexts stuck on birth CPU | -| **RT scheduling** | ❌ Missing | No SCHED_FIFO/SCHED_RR, no kernel policy dispatch | -| **Futex REQUEUE** | ❌ Missing | Condvar broadcast causes thundering herd | -| **Robust mutexes** | ❌ Missing | Thread death while holding mutex → permanent deadlock | -| **PI futexes** | ❌ Missing | No priority inheritance → priority inversion risk | -| **CPU affinity API** | ❌ Missing from relibc | Kernel supports sched_affinity field but no userspace API | -| **Thread naming** | ❌ Missing from relibc | Kernel supports name field but no userspace API | -| **Per-page TLB flush** | ❌ Missing | `invalidate_all()` = full CR3 reload on every shootdown | -| **NUMA awareness** | ❌ Missing | No SRAT/SLIT, no proximity domains, flat memory model | -| **IRQ balancing** | ❌ Missing | All legacy IRQs hardwired to BSP | - ---- - -## 2. Layer-by-Layer Assessment - -### 2.1 Hardware / SMP Layer - -**Files:** `src/acpi/madt/arch/x86.rs`, `src/arch/x86_shared/start.rs`, -`src/arch/x86_shared/device/local_apic.rs`, `src/arch/x86_shared/device/ioapic.rs`, -`src/arch/x86_shared/ipi.rs`, `src/arch/x86_shared/interrupt/ipi.rs`, `src/percpu.rs`, -`src/arch/x86_shared/gdt.rs` - -**Verdict: Functional foundation, performance gaps.** - -| Component | Status | Detail | -|-----------|--------|--------| -| AP boot (INIT/SIPI) | ✅ validated | Correct trampoline at 0x8000, per-AP PCR/IDT/stack allocation | -| x2APIC mode | ✅ builds | Detected via CPUID, MSR-based access, APIC ID detection | -| Per-CPU PCR via GS_BASE | ✅ validated | `PercpuBlock::current()` reads from PCR, SWAPGS protocol correct | -| IPI send/receive | ✅ functional | 5 IPI kinds (Wakeup/Tlb/Switch/Pit/Profile), broadcast + unicast | -| TLB shootdown | ✅ correct | AtomicBool + IPI + ack with `fence(SeqCst)` race prevention | -| TLB granularity | ❌ coarse | Full CR3 reload (`mov cr3, cr3`) on every shootdown — no INVLPG | -| TLB broadcast | 🚧 sequential | Iterates CPUs individually, doesn't use ICR "all excluding self" shorthand | -| IRQ routing | ❌ BSP-only | Legacy I/O APIC entries hardcode `dest: bsp_apic_id` | -| NUMA | ❌ absent | No SRAT/SLIT, no proximity domains | -| SMT/HT topology | ❌ absent | No cache hierarchy, no hyperthread awareness | -| Idle loop | ✅ functional | MWAIT with deepest C-state or HLT fallback | -| W^X for trampoline | 🚧 minor | Trampoline page briefly W+X, unmapped after AP boot | - -### 2.2 Kernel Scheduler Layer - -**Files:** `src/context/switch.rs`, `src/context/mod.rs`, `src/context/context.rs`, -`src/context/timeout.rs` - -**Verdict: Correct but primitive — DWRR only, no SMP balancing, no RT classes.** - -**Algorithm:** Deficit Weighted Round Robin (DWRR) -- 40 priority levels, each a `VecDeque` -- Geometric weights: `SCHED_PRIO_TO_WEIGHT[i] ≈ 1.25^i` (88761 → 15) -- Per-CPU `balance` accumulator drives dequeue decisions -- Quantum: 3 PIT ticks (~12.2ms) per scheduling round -- Cooperative preemption: `preempt_locks > 0` disables preemption - -**Global locks:** -- `RUN_CONTEXTS: Mutex` — all 40 priority queues under one L1 lock -- `IDLE_CONTEXTS: Mutex>` — sleeping contexts -- `CONTEXT_SWITCH_LOCK: AtomicBool` — global CAS spinlock serializing all context switches - -**What's missing (all was in lost P5–P9 work):** - -| Gap | Lost Patch | Recoverable? | -|-----|-----------|-------------| -| Per-CPU run queues (eliminate global L1) | P6-percpu-runqueues, P8-percpu-sched, P8-percpu-wiring | ✅ applies cleanly | -| Work stealing | P8-work-stealing | ❌ needs rebase (depends on per-CPU wiring) | -| Initial placement (least-loaded CPU) | P8-initial-placement | ✅ applies cleanly | -| Load balancing | P8-load-balance (absorbed) | needs verification | -| Vruntime tracking + min-vruntime selection | P6-vruntime-switch | ✅ applies cleanly | -| SchedPolicy enum (FIFO/RR/Other) | P5-sched-rt-policy | ✅ applies cleanly | -| RT scheduling dispatch | P5-sched-rt-policy | ✅ applies cleanly | -| Cache-affine scheduling | P7-cache-affine-switch | ✅ applies cleanly | -| NUMA topology hints | P9-numa-topology | ✅ applies cleanly | - -### 2.3 Kernel Futex Layer - -**File:** `src/syscall/futex.rs` - -**Verdict: Baseline only — critical operations missing for desktop workloads.** - -| Operation | Status | Impact of Absence | -|-----------|--------|-------------------| -| `FUTEX_WAIT` (32-bit) | ✅ | — | -| `FUTEX_WAIT64` (64-bit) | ✅ | — | -| `FUTEX_WAKE` | ✅ | — | -| `FUTEX_REQUEUE` | ❌ returns EINVAL | `pthread_cond_broadcast` wakes ALL waiters (thundering herd) | -| `FUTEX_CMP_REQUEUE` | ❌ not defined | Same + atomicity gap | -| `FUTEX_WAKE_OP` | ❌ not defined | glibc mutex fast path unavailable | -| `FUTEX_WAIT_BITSET` | ❌ not defined | `pselect`/`ppoll` optimization unavailable | -| `FUTEX_WAKE_BITSET` | ❌ not defined | Targeted wake unavailable | -| `FUTEX_LOCK_PI` / `UNLOCK_PI` | ❌ not defined | Priority inversion unprotected | -| Robust futex list | ❌ not defined | Thread death → permanent deadlock | -| Futex sharding (per-futex lock) | ❌ single global L1 mutex | All futex ops on all CPUs contend on one lock | -| Process-private futexes | ❌ global table | Unnecessary cross-process visibility | - -**Architecture:** -``` -static FUTEXES: Mutex // single global lock -type FutexList = HashMap> -``` - -Physical address is the key (enables cross-address-space futex via MAP_SHARED). -Virtual address + Weak used for CoW disambiguation. - -**Recoverable work (lost patches):** - -| Feature | Lost Patch | Applies? | -|---------|-----------|----------| -| 64-shard hash table | P6-futex-sharding | ✅ cleanly | -| FUTEX_REQUEUE + CMP_REQUEUE | P8-futex-requeue | ❌ needs rebase | -| PI futex (LOCK_PI/UNLOCK_PI/TRYLOCK_PI) | P8-futex-pi | ❌ needs rebase | -| PI CAS fix | P9-futex-pi-cas-fix | ❌ needs rebase | -| Robust futex list | P8-futex-robust | ❌ needs rebase | - -The 4 failing patches likely fail because they depend on sharding (P6-futex-sharding) being -applied first. Apply in order: P6-sharding → P8-requeue → P8-pi → P8-robust → P9-pi-cas-fix. - -### 2.4 Kernel Syscall ABI Layer - -**Files:** `src/syscall/mod.rs`, `src/syscall/futex.rs`, `src/syscall/time.rs`, -`src/syscall/process.rs`, `local/sources/syscall/src/number.rs`, `src/scheme/proc.rs` - -**Verdict: Minimal surface — most threading done via proc scheme, not syscalls.** - -The kernel defines only ~35 syscall numbers. Threading-relevant ones: - -| Syscall | Status | Notes | -|---------|--------|-------| -| `SYS_FUTEX` (240) | ✅ partial | WAIT/WAIT64/WAKE only | -| `SYS_YIELD` (158) | ✅ | `context::switch()` + signal handler | -| `SYS_FMAP` (900) | ✅ | Anonymous + file-backed mmap | -| `SYS_FUNMAP` (92) | ✅ | munmap | -| `SYS_MPROTECT` (125) | ✅ | | -| `SYS_MREMAP` (155) | ✅ | | -| `SYS_NANOSLEEP` (162) | ✅ | EINTR-aware | -| `SYS_CLOCK_GETTIME` (265) | ✅ partial | REALTIME + MONOTONIC only | - -**Threading done via proc scheme (not syscalls):** - -| Operation | Mechanism | -|-----------|-----------| -| Thread/process creation | `proc:` scheme: open "new-context", share addr_space + files via kdup | -| waitpid | `proc:` scheme: `EVENT_READ` on context fd | -| getpid/gettid | `proc:` scheme: read "attrs" handle | -| kill/tkill | `proc:` scheme: `ForceKill` / `Interrupt` ContextVerb | -| CPU affinity | `proc:` scheme: write "sched-affinity" handle | -| Priority | `proc:` scheme: write "attrs" prio field | -| Signal setup | `proc:` scheme: write "sighandler" + shared Sigcontrol pages | -| TLS base (FSBASE) | `proc:` scheme: write "regs/env" EnvRegisters | - -**Completely missing syscalls (no number, no handler):** -`clone`, `fork`, `vfork`, `waitpid`, `wait4`, `kill`, `tkill`, `tgkill`, `arch_prctl`, -`set_thread_area`, `set_tid_address`, `set_robust_list`, `get_robust_list`, -`sched_setaffinity`, `sched_getaffinity`, `sched_setscheduler`, `sched_getparam`, -`sigaction`, `sigprocmask`, `sigpending`, `sigsuspend`, `sigtimedwait`, -`timer_create`, `timer_settime`, `timer_delete`, `timerfd_create`, -`getrusage`, `setrlimit`, `getrlimit`, `times` - -### 2.5 relibc Pthread Layer - -**Files:** `src/pthread/mod.rs`, `src/sync/*.rs`, `src/header/pthread/*.rs`, -`src/header/sched/mod.rs`, `src/ld_so/tcb.rs`, `src/platform/redox/mod.rs` - -**Verdict: Core pthreads solid, scheduling/robust/PI absent, several POSIX gaps.** - -#### Fully Working (futex-backed) - -| API Group | Backend | Notes | -|-----------|---------|-------| -| `pthread_create/join/detach/exit` | redox_rt clone + Waitval | Stack via mmap, TLS via Tcb::new() | -| `pthread_cancel/setcancelstate/testcancel` | SIGRT_RLCT_CANCEL (33) | Deferred cancellation only | -| `pthread_mutex_*` (normal/recursive/errorcheck) | AtomicU32 CAS + futex_wait/wake | 3-state: unlocked/locked/waiters | -| `pthread_cond_*` | Two-counter futex design | CLOCK_REALTIME only (monotonic = stub) | -| `pthread_rwlock_*` | AtomicU32 + futex | Reader count + WAITING_WR bit | -| `pthread_barrier_*` | Mutex + Cond | gen_id wrapping counter | -| `pthread_spin_*` | AtomicI32 CAS | No futex, pure spinning | -| `pthread_once` | 3-state futex (UNINIT→INITING→INIT) | | -| `pthread_key_create/getspecific/setspecific/delete` | BTreeMap global + thread_local values | Destructor iteration per POSIX | -| `pthread_sigmask` | Delegates to sigprocmask | | -| `pthread_kill` | redox_rt::rlct_kill | | -| `pthread_atfork` | Thread-local LinkedList hooks | | -| ELF TLS (`__thread` / `#[thread_local]`) | PT_TLS + Tcb | Static + dynamic DTV for dlopen | -| `pthread_attr_*` (getters/setters) | RlctAttr struct | | - -#### Stubs / No-ops / Missing - -| API | Status | Root Cause | -|-----|--------|------------| -| `sched_get_priority_max/min` | `todo!()` | Kernel has no scheduling policy API | -| `sched_getparam/setparam` | `todo!()` | Same | -| `sched_setscheduler` | `todo!()` | Same | -| `sched_rr_get_interval` | `todo!()` | Same | -| `pthread_setschedparam` | No-op (returns Ok) | Kernel ignores policy | -| `pthread_setschedprio` | No-op (returns Ok) | Kernel ignores priority change | -| `pthread_getschedparam` | `todo!()` | | -| `pthread_getcpuclockid` | ENOENT | No per-thread CPU clock | -| `pthread_mutex_consistent` | `todo_skip!` | Robust mutex not implemented | -| `pthread_mutex_getprioceiling` | `todo_skip!` | Priority ceiling not implemented | -| `pthread_mutex_setprioceiling` | `todo_skip!` | Same | -| `pthread_mutexattr_setprotocol` (PRIO_INHERIT) | Accepted, no-op | PI futex missing | -| `pthread_mutexattr_setrobust` (ROBUST) | Accepted, no-op | Robust futex missing | -| `pthread_cond_init` CLOCK_MONOTONIC | `todo_skip!` | | -| `pthread_cond_signal` | Calls broadcast (wakes ALL) | Missing FUTEX_REQUEUE optimization | -| `pthread_setaffinity_np` | Not defined | | -| `pthread_getaffinity_np` | Not defined | | -| `pthread_setname_np` | Not defined | | -| `pthread_getname_np` | Not defined | | -| `pthread_setcanceltype` | Always returns DEFERRED | ASYNC not tracked | -| Guard pages | Attribute stored, not mapped | No PROT_NONE page before stack | -| PTHREAD_KEYS_MAX limit | Not checked | | - ---- - -## 3. Gap Classification - -### 3.1 Correctness Gaps (Must Fix — Silent Data Corruption or Deadlock) - -| # | Gap | Impact | Fix Location | -|---|-----|--------|-------------| -| C1 | **No robust mutexes** | Thread death while holding mutex → permanent deadlock for all waiters | Kernel: robust futex list + relibc: pthread_mutex_consistent | -| C2 | **No PI futexes** | Priority inversion: low-prio thread blocks high-prio thread indefinitely | Kernel: FUTEX_LOCK_PI/UNLOCK_PI + relibc: mutexattr_setprotocol | -| C3 | **`pthread_cond_signal` wakes ALL** | Correctness: wastes CPU. Performance: thundering herd on every signal | relibc: use true wake(1) — may need FUTEX_REQUEUE | -| C4 | **`fork()` not thread-safe** | `pthread_atfork` hooks exist but child inherits locked mutexes | relibc: implement atfork child handlers properly | - -### 3.2 Performance Gaps (Must Fix for Desktop Responsiveness) - -| # | Gap | Impact | Fix Location | -|---|-----|--------|-------------| -| P1 | **No SMP load balancing** | Cores sit idle while others are overloaded | Kernel: work stealing + initial placement | -| P2 | **No futex sharding** | Single global L1 mutex for ALL futex ops on ALL CPUs | Kernel: 64-shard hash table | -| P3 | **No FUTEX_REQUEUE** | `pthread_cond_broadcast` wakes all → thundering herd | Kernel: REQUEUE + CMP_REQUEUE | -| P4 | **Full TLB flush on every shootdown** | Per-page mprotect/munmap flushes entire TLB on all cores | Kernel: INVLPG-based selective flush | -| P5 | **Global context switch lock** | Serialization bottleneck beyond ~8 cores | Kernel: per-CPU context switch (needs per-CPU run queues) | -| P6 | **All IRQs to BSP** | CPU 0 handles all interrupts, cache thrash, latency | Kernel: IRQ steering in I/O APIC + MSI/MSI-X dest field | -| P7 | **No RT scheduling** | Audio/compositor threads can't get priority | Kernel: SchedPolicy + RT dispatch + relibc: sched_setscheduler | - -### 3.3 POSIX Completeness Gaps (Must Fix for Application Compatibility) - -| # | Gap | Impact | Fix Location | -|---|-----|--------|-------------| -| X1 | `sched_*` all `todo!()` | Applications calling sched_setscheduler panic | relibc: implement via proc scheme | -| X2 | `pthread_setschedparam` no-op | Apps can't change thread priority | relibc: wire to proc scheme prio write | -| X3 | `pthread_setaffinity_np` missing | Apps can't pin threads to CPUs | relibc: implement via proc scheme affinity write | -| X4 | `pthread_setname_np` missing | Debugging harder (no thread names in /proc) | relibc: implement via proc scheme name write | -| X5 | `pthread_getcpuclockid` ENOENT | Per-thread profiling impossible | relibc + kernel: expose cpu_time via clock | -| X6 | Guard pages not mapped | Stack overflow → silent corruption, no SIGSEGV | relibc: mmap PROT_NONE guard page in pthread_create | -| X7 | `pthread_cond_init` monotonic stub | CLOCK_MONOTONIC condvars use REALTIME (affected by wall clock jumps) | relibc: implement monotonic condvar | - ---- - -## 4. Implementation Plan - -### Phase 0: Patch Recovery — Re-Apply Lost Threading Work (Week 1–2) - -**Goal:** Recover the P5–P9 work that was lost during the local fork migration. - -**This is the highest-priority phase — it restores ~6 months of work with minimal new code.** - -#### 0.1 — Re-apply kernel scheduler patches to local fork - -Apply in dependency order to `local/sources/kernel/`: - -| Order | Patch | Status | Action | -|-------|-------|--------|--------| -| 1 | P6-futex-sharding | ✅ applies | Commit directly | -| 2 | P6-percpu-runqueues | ✅ applies | Commit directly | -| 3 | P8-percpu-sched | ✅ applies | Commit directly | -| 4 | P8-percpu-wiring | ✅ applies | Commit directly | -| 5 | P8-initial-placement | ✅ applies | Commit directly | -| 6 | P5-sched-rt-policy | ✅ applies | Commit directly | -| 7 | P5-context-mod-sched | ✅ applies | Commit directly | -| 8 | P6-vruntime-switch | ✅ applies | Commit directly | -| 9 | P7-cache-affine-switch | ✅ applies | Commit directly | -| 10 | P9-numa-topology | ✅ applies | Commit directly | -| 11 | P9-proc-lock-ordering | ✅ applies | Commit directly | -| 12 | P8-work-stealing | ❌ needs rebase | Rebase against 1–11, then apply | -| 13 | P8-futex-requeue | ❌ needs rebase | Rebase against P6-sharding (#1), then apply | -| 14 | P8-futex-pi | ❌ needs rebase | Rebase against #13, then apply | -| 15 | P8-futex-robust | ❌ needs rebase | Rebase against #14, then apply | -| 16 | P9-futex-pi-cas-fix | ❌ needs rebase | Rebase against #14, then apply | -| 17 | P7-scheduler-improvements | ❌ needs rebase | Rebase against 1–11, then apply | - -**Verification after each patch:** -```bash -cd local/sources/kernel -cargo check # must pass -``` - -#### 0.2 — Re-apply relibc threading patches to local fork - -Apply to `local/sources/relibc/`: - -| Patch | Action | -|-------|--------| -| P3-threads.patch | ✅ applies — commit | -| P3-barrier-smp-futex (from absorbed/) | Verify already in fork; if not, apply | -| P3-pthread-signal-races (from absorbed/) | Verify already in fork | -| P3-pthread-yield (from absorbed/) | Verify already in fork | -| P5-robust-mutexes (from absorbed/) | Verify; re-apply if missing | -| P5-robust-mutex-enotrec-fix (from absorbed/) | Same | -| P5-sched-api (from absorbed/) | Same | -| P7-pthread-affinity (from absorbed/) | Same | -| P7-pthread-setname (from absorbed/) | Same | -| P7-setpriority (from absorbed/) | Same | -| P9-spin-and-barrier (from absorbed/) | Same | -| P9-spin-fix (from absorbed/) | Same | -| P3-semaphore-comprehensive | ✅ applies | - -**Verification:** -```bash -cd local/sources/relibc -make all # must pass -touch relibc && make prefix # rebuild prefix with new libc -``` - -#### 0.3 — Build and smoke test - -```bash -export REDBEAR_ALLOW_PROTECTED_FETCH=1 -./local/scripts/build-redbear.sh --upstream redbear-mini -make qemu # verify boot + basic operation -``` - -**Success criteria:** redbear-mini boots, multi-threaded daemons (pcid, xhcid) start, no kernel panic. - ---- - -### Phase 1: Futex Completeness (Week 2–4) - -**Goal:** Close the futex operation gaps that affect correctness and performance. - -**Depends on:** Phase 0 complete (sharding applied first). - -#### 1.1 — FUTEX_REQUEUE + FUTEX_CMP_REQUEUE - -**Kernel:** `src/syscall/futex.rs` -- Add `FUTEX_REQUEUE` and `FUTEX_CMP_REQUEUE` to the futex dispatcher -- Implement: move up to `val` waiters from addr1 → addr2, optionally compare `*addr1 == val2` -- Requires locking TWO shards (acquire both in deterministic order to avoid deadlock) - -**relibc:** `src/sync/cond.rs` -- Change `pthread_cond_broadcast` to use `FUTEX_REQUEUE` (move waiters from condvar futex to mutex futex) -- Change `pthread_cond_signal` to wake exactly 1 (not all) - -**Impact:** Eliminates thundering herd on every `pthread_cond_broadcast`. Major win for Qt event loop, KWin compositor, Mesa worker threads. - -#### 1.2 — PI Futexes (FUTEX_LOCK_PI / FUTEX_UNLOCK_PI / FUTEX_TRYLOCK_PI) - -**Kernel:** `src/syscall/futex.rs` -- Add `PiState` tracking per futex: owner context + waiter list with priorities -- On `LOCK_PI` block: boost owner's priority to waiter's priority -- On `UNLOCK_PI`: restore original priority, wake highest-priority waiter -- Requires kernel RT scheduling (Phase 0.1 #6–7: P5-sched-rt-policy) - -**relibc:** `src/sync/pthread_mutex.rs` -- Implement `PTHREAD_PRIO_INHERIT` protocol path using PI futex -- Replace `todo_skip!` in `pthread_mutex_consistent` with real implementation - -#### 1.3 — Robust Futex List - -**Kernel:** `src/syscall/futex.rs` + `src/context/context.rs` -- Add `robust_list_head: Option` to `Context` struct -- Implement `set_robust_list` / `get_robust_list` via proc scheme or syscall -- On thread exit (`exit_this_context`): walk robust list, set `FUTEX_OWNER_DIED` bit, wake one waiter with `EOWNERDEAD` - -**relibc:** `src/sync/pthread_mutex.rs` -- Implement robust list registration in `pthread_mutex_lock` -- Implement `pthread_mutex_consistent`: clear `EOWNERDEAD` state -- Replace `todo_skip!` with real implementation - -#### 1.4 — FUTEX_WAKE_OP - -**Kernel:** `src/syscall/futex.rs` -- Implement atomic op + wake: perform op on addr2, then wake up to `val` waiters on addr1 -- Operations: set, add, or, andn, xor, with comparison condition - -**Impact:** glibc mutex fast path optimization. Not critical for relibc but helps ported glibc-linked binaries. - ---- - -### Phase 2: SMP Scheduling Quality (Week 3–6) - -**Goal:** Make multi-core actually distribute work. - -**Depends on:** Phase 0 complete (per-CPU queues applied). - -#### 2.1 — Work stealing (recover + fix) - -**Kernel:** `src/context/switch.rs` -- On `select_next_context()` empty local queue: steal from victim CPU -- Pick victim by round-robin, steal highest-priority runnable context -- Limit steal batch size (1–2 contexts per steal attempt) -- Send `IpiKind::Wakeup` to target CPU if stealing woke it from idle - -**Recovery:** P8-work-stealing needs rebase against per-CPU wiring. - -#### 2.2 — Load balancing (recover + verify) - -**Kernel:** `src/context/switch.rs` -- Periodic balance trigger (every N ticks or when queue depth difference > threshold) -- Migrate contexts from overloaded CPU to most-idle CPU -- Respect `sched_affinity` mask during migration - -**Recovery:** P8-load-balance is in absorbed/ — verify it's in the fork after Phase 0. - -#### 2.3 — Reschedule IPI - -**Kernel:** `src/arch/x86_shared/ipi.rs` + `src/context/switch.rs` -- When waking a context on a different CPU, send `IpiKind::Switch` to that CPU -- Currently the Switch IPI exists but is not used by the scheduler - -#### 2.4 — Per-page TLB flush (INVLPG) - -**Kernel:** `rmm/src/arch/x86_64.rs` + `src/context/memory.rs` -- Add `invalidate_page(addr)` using `invlpg` instruction -- Modify `Flusher` to track individual pages and use INVLPG when ≤ N pages affected -- Fall back to CR3 reload only for large-scale invalidations - -**Impact:** Every `mprotect`/`mmap`/`munmap` on a multi-threaded process currently flushes the ENTIRE TLB on every core. This is one of the most impactful single fixes. - -#### 2.5 — TLB broadcast optimization - -**Kernel:** `src/percpu.rs` -- Replace per-CPU sequential `shootdown_tlb_ipi(Some(id))` loop with ICR "all excluding self" (destination shorthand 0b11) -- Single IPI + global ack counter instead of N individual IPIs + N ack counters - ---- - -### Phase 3: RT Scheduling (Week 4–6) - -**Goal:** Allow applications to request real-time scheduling for latency-sensitive threads. - -**Depends on:** Phase 0 (SchedPolicy applied) + Phase 2 (per-CPU queues). - -#### 3.1 — Kernel RT scheduling dispatch - -**Kernel:** `src/context/switch.rs` (from P5-sched-rt-policy — recovered in Phase 0) -- `select_next_context()` passes: - 1. SCHED_FIFO contexts (highest RT priority first, no preemption within same prio) - 2. SCHED_RR contexts (highest RT priority first, round-robin within same prio) - 3. SCHED_OTHER contexts (existing DWRR/vruntime) -- SCHED_RR quantum: configurable per-context (default 100ms) - -#### 3.2 — relibc sched_* API completion - -**relibc:** `src/header/sched/mod.rs` - -Replace ALL `todo!()` stubs: - -| Function | Implementation | -|----------|---------------| -| `sched_getscheduler(pid)` | Read policy from proc scheme attrs | -| `sched_setscheduler(pid, policy, param)` | Write policy + RT priority via proc scheme | -| `sched_getparam(pid, param)` | Read RT priority from proc scheme | -| `sched_setparam(pid, param)` | Write RT priority via proc scheme | -| `sched_get_priority_max(policy)` | Return 99 for FIFO/RR, 0 for OTHER | -| `sched_get_priority_min(policy)` | Return 1 for FIFO/RR, 0 for OTHER | -| `sched_rr_get_interval(pid, tp)` | Return SCHED_RR quantum (100ms default) | - -#### 3.3 — pthread_setschedparam wiring - -**relibc:** `src/pthread/mod.rs` -- Replace `set_sched_param` no-op with real proc scheme call -- Replace `set_sched_priority` no-op with real proc scheme call - ---- - -### Phase 4: POSIX Pthread Completeness (Week 5–8) - -**Goal:** Close remaining POSIX gaps that block application compatibility. - -**Depends on:** Phase 0 + Phase 3 (for sched API). - -#### 4.1 — pthread_setaffinity_np / pthread_getaffinity_np - -**relibc:** `src/header/pthread/mod.rs` + `src/header/sched/mod.rs` -- Implement using proc scheme "sched-affinity" write/read -- Define `cpu_set_t` type and `CPU_SET/CPU_CLR/CPU_ZERO/CPU_ISSET` macros - -#### 4.2 — pthread_setname_np / pthread_getname_np - -**relibc:** `src/header/pthread/mod.rs` -- Implement using proc scheme name write/read (kernel already supports 32-char name field) - -#### 4.3 — pthread_cond_init CLOCK_MONOTONIC - -**relibc:** `src/sync/cond.rs` -- Replace `todo_skip!` with real monotonic clock support -- Store clock choice in cond struct, use `CLOCK_MONOTONIC` for deadline calculations - -#### 4.4 — Guard pages - -**relibc:** `src/pthread/mod.rs` -- In `pthread_create`, when allocating stack via mmap: - - Map `[stack_base, stack_base + guard_size)` with `PROT_NONE` - - Map `[stack_base + guard_size, stack_base + guard_size + stack_size)` with `PROT_READ | PROT_WRITE` -- On thread exit, munmap both regions - -#### 4.5 — pthread_getcpuclockid - -**relibc:** `src/header/pthread/mod.rs` -- Return `CLOCK_THREAD_CPUTIME_ID` (requires kernel support — add clock to `clock_gettime`) - -**Kernel:** `src/syscall/time.rs` -- Add `CLOCK_THREAD_CPUTIME_ID` → read `context.cpu_time` - -#### 4.6 — PTHREAD_KEYS_MAX enforcement - -**relibc:** `src/header/pthread/tls.rs` -- Check `NEXTKEY` against `PTHREAD_KEYS_MAX` (1024) before allocating - ---- - -### Phase 5: IRQ Steering and NUMA (Week 8–12) - -**Goal:** Distribute interrupt load and respect memory locality. - -**Depends on:** Phase 2 (per-CPU infrastructure). - -#### 5.1 — IRQ steering - -**Kernel:** `src/arch/x86_shared/device/ioapic.rs` + `src/arch/x86_shared/idt.rs` -- Change I/O APIC redirection `dest` from `bsp_apic_id` to round-robin or RSS hash -- Add per-CPU legacy IRQ handlers in IDT (not just BSP) -- For MSI/MSI-X: set destination CPU in Message Address register - -#### 5.2 — NUMA topology discovery - -**Kernel:** `src/acpi/` (from P9-numa-topology — recovered in Phase 0) -- Parse SRAT (Static Resource Affinity Table) for proximity domains -- Parse SLIT (System Locality Distance Information Table) for inter-node distances -- Store `NumaTopology` in kernel for O(1) scheduling lookups - -#### 5.3 — NUMA-aware memory allocation - -**Kernel:** `src/memory/` + frame allocator -- Track frame NUMA node in `Frame` or `PageInfo` -- On allocation, prefer frames from requesting CPU's NUMA node -- Fallback to remote node when local node is exhausted - ---- - -## 5. Dependency Chain - -``` -Phase 0 (Patch Recovery) ← BLOCKING FOR ALL OTHERS - │ - ├──► Phase 1 (Futex Completeness) - │ │ - │ ├──► 1.1 REQUEUE ──► condvar performance - │ ├──► 1.2 PI ──► priority inversion fix (needs Phase 3.1) - │ ├──► 1.3 Robust ──► deadlock prevention - │ └──► 1.4 WAKE_OP ──► glibc compat - │ - ├──► Phase 2 (SMP Scheduling) - │ │ - │ ├──► 2.1 Work stealing ──► core utilization - │ ├──► 2.2 Load balancing ──► fair distribution - │ ├──► 2.3 Reschedule IPI ──→ cross-CPU wakeup - │ ├──► 2.4 Per-page TLB ──► mmap/mprotect performance - │ └──► 2.5 TLB broadcast ──► IPI efficiency - │ - ├──► Phase 3 (RT Scheduling) - │ │ - │ ├──► 3.1 Kernel RT dispatch (from Phase 0) - │ ├──► 3.2 relibc sched_* API ──► POSIX compat - │ └──► 3.3 pthread_setschedparam ──► app priority control - │ - ├──► Phase 4 (POSIX Pthread Completeness) - │ │ - │ ├──► 4.1 Affinity API ──► CPU pinning - │ ├──► 4.2 Thread naming ──► debuggability - │ ├──► 4.3 Monotonic condvar ──► clock correctness - │ ├──► 4.4 Guard pages ──► stack overflow detection - │ ├──► 4.5 CPU clock ──► per-thread profiling - │ └──► 4.6 Keys max ──► resource limit - │ - └──► Phase 5 (IRQ + NUMA) - │ - ├──► 5.1 IRQ steering ──► interrupt distribution - ├──► 5.2 NUMA topology ──► (from Phase 0) - └──► 5.3 NUMA allocator ──► memory locality -``` - -**Parallel work possible:** -- Phase 1 + Phase 2 + Phase 3 can run in parallel after Phase 0 -- Phase 4 items are independent of each other -- Phase 5 depends on Phase 2 but not on Phase 1/3/4 - ---- - -## 6. Validation Plan - -### 6.1 Build Evidence - -| Check | Command | -|-------|---------| -| Kernel compiles | `make r.kernel` | -| relibc compiles | `make r.relibc` | -| Prefix rebuilt | `touch relibc kernel && make prefix` | -| Full OS builds | `./local/scripts/build-redbear.sh redbear-mini` | - -### 6.2 Runtime Evidence (QEMU) - -| Test | Verification | -|------|-------------| -| Multi-threaded boot | `make qemu QEMUFLAGS="-smp 4"` — all 4 CPUs active | -| pthread smoke test | Guest: compile + run simple pthread_create/join/mutex test | -| Work stealing | Guest: spawn 8 threads on 4-CPU QEMU, verify all CPUs utilized | -| Futex REQUEUE | Guest: condvar broadcast benchmark — waiters wake in ≤2 batches, not N | -| PI futex | Guest: priority inversion test — high-prio thread unblocked within 1 tick | -| Robust mutex | Guest: kill thread holding mutex, verify EOWNERDEAD recovery | -| RT scheduling | Guest: SCHED_FIFO thread preempts SCHED_OTHER within 100μs | -| CPU affinity | Guest: pin thread to CPU 1, verify it never runs on CPU 0 | -| Thread naming | Guest: `cat /scheme/proc/*/name` shows set names | -| Guard pages | Guest: overflow stack, verify SIGSEGV (not silent corruption) | -| TLB efficiency | Guest: mprotect benchmark — compare TLB miss rate before/after | - -### 6.3 Validation Scripts (to create) - -```bash -local/scripts/test-threading-qemu.sh # Comprehensive threading smoke test -local/scripts/test-futex-requeue-qemu.sh # REQUEUE-specific test -local/scripts/test-futex-pi-qemu.sh # PI futex test -local/scripts/test-futex-robust-qemu.sh # Robust mutex test -local/scripts/test-sched-rt-qemu.sh # RT scheduling latency test -local/scripts/test-sched-balance-qemu.sh # Load balancing on multi-vCPU -local/scripts/test-threading-baremetal.sh # Bare metal multi-threaded stress -``` - ---- - -## 7. Estimated Effort - -| Phase | Duration | New Code | Recovery | Dependencies | -|-------|----------|----------|----------|-------------| -| Phase 0: Patch Recovery | 1–2 weeks | Minimal (rebase 5 patches) | 13 patches apply directly | None | -| Phase 1: Futex Completeness | 2–3 weeks | REQUEUE impl + WAKE_OP | PI/robust from P8 patches | Phase 0 | -| Phase 2: SMP Scheduling | 3–4 weeks | TLB INVLPG + broadcast opt | Work stealing from P8 | Phase 0 | -| Phase 3: RT Scheduling | 1–2 weeks | relibc sched_* API | RT dispatch from P5 | Phase 0 | -| Phase 4: POSIX Pthread | 2–3 weeks | Affinity/naming/guard/clock | Partial from P7 patches | Phase 0, 3 | -| Phase 5: IRQ + NUMA | 3–4 weeks | IRQ steering + NUMA allocator | NUMA topology from P9 | Phase 0, 2 | - -**Total:** 12–18 weeks with 1–2 developers. Phase 0 alone recovers the majority of the value in 1–2 weeks. - ---- - -## 8. Integration with Existing Plans - -| Plan | Relationship | -|------|-------------| -| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | **Consumer** — Phase 3 (KWin) needs PI futex + RT scheduling; Phase 2 (compositor) needs work stealing | -| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | **Sibling** — IRQ steering (Phase 5.1) belongs to both plans | -| `DRM-MODERNIZATION-EXECUTION-PLAN.md` | **Consumer** — GPU worker threads benefit from load balancing + affinity | -| `IMPLEMENTATION-MASTER-PLAN.md` | **Parent** — this plan covers the kernel threading substrate | -| `CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md` | **Sibling** — overlaps on scheduler/IRQ delivery | - ---- - -## 9. Bottom Line - -The Red Bear OS threading stack is **functional for basic single-threaded and lightly-threaded -workloads**. The SMP boot, context switching, TLB shootdown, and basic futex operations are -correct. - -The **critical problem** is that 6 months of threading enhancement work (P5–P9 patches) was -lost during the local fork migration. This work exists as patch files that apply cleanly to -the current fork — **Phase 0 (Patch Recovery) is the single highest-ROI action**. - -After Phase 0, the remaining gaps are: -1. **Futex REQUEUE/PI/robust** — for condvar performance and deadlock prevention -2. **SMP work stealing + load balancing** — for multi-core utilization -3. **RT scheduling** — for audio/compositor thread priority -4. **POSIX pthread completeness** — for application compatibility -5. **IRQ steering + NUMA** — for multi-socket performance - -The **desktop-critical path** (KWin responsiveness) requires Phases 0–3. The -**server-critical path** (multi-socket, NUMA) adds Phase 5. Phase 4 (POSIX completeness) -benefits all paths but is not desktop-blocking. diff --git a/local/docs/REDBEAR-BOOT-EXPERIENCE-PLAN.md b/local/docs/REDBEAR-BOOT-EXPERIENCE-PLAN.md deleted file mode 100644 index d7aa03e412..0000000000 --- a/local/docs/REDBEAR-BOOT-EXPERIENCE-PLAN.md +++ /dev/null @@ -1,483 +0,0 @@ -# Red Bear OS — CachyOS-Class Boot Experience Implementation Plan - -**Version:** 1.0 · 2026-06-11 · Branch: `0.2.3` -**Status:** Canonical plan for boot visual quality, display handoff, and boot speed -**Depends on:** existing `redox-drm`, `inputd`, `vesad`, `fbbootlogd`, `fbcond`, `bootloader` -**Supersedes:** boot-comfort fragments in `CONSOLE-TO-KDE-DESKTOP-PLAN.md` (boot pipeline layer only) - ---- - -## 0. Architecture Decision - -**The Linux model is correct: once DRM driver becomes available, it realizes handoff automatically.** - -No daemon-side config awareness. No polling. No inter-daemon handshakes. When `redox-drm` registers -`scheme:drm/card0`, the display path switches through the existing `inputd` ESTALE mechanism. Init -orchestrates the lifecycle — staging the splash, detecting DRM, withdrawing the earlyfb, forwarding -traffic to the new path. - -### Target Pipeline (Post-Plan) - -``` -UEFI GOP framebuffer (bootloader paints Red Bear logo) - → kernel boots, passes FB env vars to init - → init starts vesad (20_vesad.service) ← registers display.vesa (earlyfb) - → init starts redbear-bootanim (20_bootanim.service) ← paints splash on earlyfb - → init starts fbbootlogd (quiet mode, hidden behind splash) - → init starts fbcond (VT 2, behind splash) - → redox-drm loads (04_drivers.target), registers scheme:drm/card0 - → inputd signals ESTALE on all display.* handles - → 50_drm-handoff.service runs ← atomic swap: vesad → DRM - • bootanim re-parents onto DRM FB (memcpy, no redraw) - • fbbootlogd/fbcond reconnect to DRM - • vesad releases bootloader FB, exits - → SDDM/KWin start (08_userland.target) - → bootanim fades out as greeter paints - -Visible result: black → red bear logo + spinner → silent handoff → SDDM fade-in -No log text unless user presses Esc. No flicker. No blank screen. -``` - -### Linux Mechanism Mapping - -| CachyOS / Linux | Red Bear equivalent | -|---|---| -| `simpledrm` (kernel) | `vesad` earlyfb + bootanim mmap | -| `Plymouth` (userspace splash) | `redbear-bootanim` (Rust, per AGENTS.md "system-critical must be Rust") | -| Plymouth two-step (pre-DRM → post-DRM) | bootanim `Surface::Vesad` → `Surface::Drm` state machine | -| `drm_aperture_remove_conflicting_framebuffers()` | init-managed via `50_drm-handoff.service` + `98_release_vesad.service` | -| `CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER` | bootanim holds firmware FB visible until DRM handoff completes | -| Plymouth Esc-to-reveal | bootanim SIGUSR2 → fbbootlogd reconnects, paints log overlay | -| Plymouth fade-out on greeter ready | bootanim SIGTERM → 200ms fade → exit | - ---- - -## 1. Current State Assessment - -### What Exists - -| Component | Location | Scheme | Status | -|---|---|---|---| -| Bootloader | `local/sources/bootloader/` | UEFI GOP text menu | Text-only, no logo/splash | -| Kernel debug display | `local/sources/kernel/src/devices/graphical_debug/` | `scheme:debug` | Immediately overwrites bootloader FB | -| vesad | `local/sources/base/drivers/graphics/vesad/` | `display.vesa` | ✅ Registers earlyfb. No handoff code. Stays alive. | -| fbbootlogd | `local/sources/base/drivers/graphics/fbbootlogd/` | `fbbootlog` | ✅ Overwrites FB with log text immediately. Has handoff path. VT 1. | -| fbcond | `local/sources/base/drivers/graphics/fbcond/` | `fbcon` | ✅ Text console VTs. Handoff with 4-retry limit. VT 2+. | -| inputd | `local/sources/base/drivers/inputd/` | `scheme:input` | ✅ Display/input multiplexer. Signals ESTALE on handoff. | -| redox-drm | `local/recipes/gpu/redox-drm/source/` | `scheme:drm` | 🚧 Registers DRM. Calls inputd/handle/ to announce itself. | -| virtio-gpud | `local/sources/base/drivers/graphics/virtio-gpud/` | `display.virtio-gpu` | ⚠️ Legacy, uses old GraphicsScheme API | -| ihdgd | `local/sources/base/drivers/graphics/ihdgd/` | `display.ihdg.*` | ⚠️ Legacy Intel driver | -| Branding assets | `local/Assets/images/` | n/a | PNGs exist, NOT integrated anywhere | - -### What's Missing (Gap Analysis) - -| # | Gap | Impact | -|---|-----|--------| -| 1 | No boot splash/logo | User sees raw kernel/init log text from the first millisecond | -| 2 | fbbootlogd overwrites bootloader FB immediately | Any bootloader-painted pixels are destroyed within milliseconds | -| 3 | No smooth display handoff | vesad stays alive, doesn't release FB memory, no coordinated transition | -| 4 | No "quiet boot" mode | Kernel/init log is always shown, no way to hide it behind splash | -| 5 | Boot is slow (4 barrier syncs before SDDM) | 00→02→04→06→08 target chain; each waits for all services | -| 6 | No progress indicator | No animated spinner or progress bar during boot | -| 7 | No bootloader branding | UEFI bootloader shows text mode selection menu only | -| 8 | vesad doesn't release FB on DRM handoff | Bootloader FB stays mapped, wasting ~8MB memory | -| 9 | `29_activate_console` is a mess | Overridden to no-op in legacy-base, then overridden again in mini. 200ms sleep hack. | -| 10 | fbcond gives up after 4 handoff retries | If DRM is slow (firmware load), console silently stops | -| 11 | Legacy virtio-gpud/ihdgd may conflict | Could race with redox-drm for display scheme | - -### Init Service Order (Current) - -``` -INITFS STAGE: - 00_runtime.target → 10_inputd → 20_vesad → 20_fbbootlogd → 20_fbcond - → 40_drivers.target → 50_rootfs → 90_initfs.target → switch_root - -ROOTFS STAGE: - 00_base.target → 02_early_hw.target → 04_drivers.target → 06_services.target - → 08_userland.target → 29_activate_console → 30_console (getty 2) → login - - For redbear-full: - Same + 12_sddm → kwin_wayland → KDE Plasma -``` - ---- - -## 2. Phased Implementation Plan - -### PHASE 1 — Branding Infrastructure - -**Goal:** Single source of truth for Red Bear visual assets with deterministic conversion. - -**Effort:** 1–4 hours - -**Files:** - -| Path | Type | Purpose | -|---|---|---| -| `local/Assets/scripts/render-assets.sh` | script | PNG → BMP/RAW conversion via `imagemagick` (host-side) | -| `local/Assets/MANIFEST.sha256` | text | Deterministic checksums for all generated assets | -| `local/recipes/system/redbear-assets/recipe.toml` | recipe (Rule 1) | Stages assets to `/usr/share/redbear/assets/` | -| `local/sources/redbear-assets/` | source (Rule 1) | Trivial install crate | -| `local/docs/BOOT-BRANDING-SPEC.md` | doc | Resolution policy, color profile, animation budget | - -**Generated assets (from existing PNGs):** - -| Asset | Format | Resolution | Consumer | -|---|---|---|---| -| `bootlogo-1080p.bmp` | 32-bit BGRA BMP | 1920×1080 | Bootloader UEFI `Blt()` | -| `bootlogo-720p.bmp` | 32-bit BGRA BMP | 1280×720 | Bootloader fallback | -| `bootlogo-tiny.bmp` | 32-bit BGRA BMP | 640×480 | VESA-only firmware | -| `splash-1080p.raw` | Raw BGRA scanout | 1920×1080 | bootanim direct mmap | -| `splash-1080p.anim.json` | JSON | n/a | Animation timeline | - -**Verification:** -- `render-assets.sh` produces all assets, byte-identical across rebuilds -- `redbear-assets` recipe stages them into sysroot - ---- - -### PHASE 2 — `redbear-bootanim`: Plymouth Equivalent - -**Goal:** Rust userspace daemon that owns the framebuffer from vesad registration until greeter focus, -rendering the Red Bear brand consistently across both earlyfb and DRM. - -**Effort:** 1–2 days - -**Files:** - -| Path | Type | Purpose | -|---|---|---| -| `local/sources/redbear-bootanim/` | source (Rule 1) | Bootanim daemon source | -| `local/sources/redbear-bootanim/src/main.rs` | Rust | Daemon entry, signal handlers | -| `local/sources/redbear-bootanim/src/surface.rs` | Rust | Surface abstraction over vesad earlyfb + DRM | -| `local/sources/redbear-bootanim/src/anim.rs` | Rust | Animation loop (logo + spinner + progress) | -| `local/sources/redbear-bootanim/src/progress.rs` | Rust | Unix datagram socket for progress updates from init | -| `local/recipes/system/redbear-bootanim/recipe.toml` | recipe (Rule 1) | Depends on redbear-assets, inputd | -| `config/redbear-bootanim.toml` | config fragment | 20_bootanim.service + 50_drm-handoff + 98_release_vesad | - -**Service wiring:** - -```toml -# 20_bootanim.service — runs on earlyfb, transitions to DRM -[[files]] -path = "/etc/init.d/20_bootanim.service" -data = """ -[unit] -description = "Red Bear boot animation (splash)" -requires_weak = ["10_inputd.service", "20_vesad.service"] - -[service] -cmd = "/usr/bin/redbear-bootanim" -args = ["--surface=vesad", "--vt=1"] -type = "simple" -respawn = false -""" -``` - -**Behavior:** - -| State | Surface | Renders | Input | -|---|---|---|---| -| `Surface::Vesad` | mmap'd bootloader FB | Logo + spinner + progress | Pass-through to fbcond | -| `Surface::Drm` | `/scheme/drm/card0` | Same pixels (memcpy, no redraw) | Pass-through | -| `Reveal` (SIGUSR2/Esc) | Both | Translucent log overlay on splash | Log scrollback | -| `Exit` (SIGTERM) | n/a | 200ms fade to black, exit | n/a | - -**Key design property:** Handoff is a memcpy, not a redraw. bootanim holds a cached `Box<[u32]>` of the last frame (~8MB). On handoff, it copies this to the DRM FB. Both surfaces end up pixel-identical — zero flicker. - -**Verification:** -- `redbear-mini`: logo appears in UEFI FB, continues through init, transitions to fbbootlogd -- `redbear-full`: logo → smooth DRM handoff → SDDM fade-in (no blank gap >1 frame) -- Esc reveals log; Esc again hides it - ---- - -### PHASE 3 — Atomic DRM Handoff (Linux `drm_aperture` Equivalent) - -**Goal:** One-shot helper that orchestrates vesad → DRM transition in a single transaction. - -**Effort:** 4–8 hours - -**Files:** - -| Path | Type | Purpose | -|---|---|---| -| `local/sources/redbear-bootanim/src/bin/handoff.rs` | Rust | Handoff orchestrator binary | -| `local/sources/redbear-bootanim/src/bin/release_fb.rs` | Rust | Sends RELEASE_EARLYFB to vesad | - -**Handoff sequence (in `handoff.rs`):** - -``` -1. Send PREPARE_HANDOFF to bootanim → bootanim flushes scanout, snapshots frame, pauses animation -2. bootanim opens /scheme/drm/card0, performs ModeSetCrtc + first present -3. bootanim returns HANDOFF_READY -4. Send RELEASE_EARLYFB to vesad → vesad munmaps bootloader FB, signals ESTALE, exits -5. Send POST_HANDOFF to bootanim → bootanim resumes animation on DRM surface exclusively -6. Send REBIND_DISPLAY drm to inputd → promotes DRM to primary, ESTALE to remaining consumers -7. Exit 0 -``` - -**Why a separate binary:** Init can enforce ordering and timeout. If handoff hangs, init moves on — user still gets a working system (stuck splash, compositor paints over it). - -**Timeout/fallback:** If `redox-drm` doesn't register within 30s, handoff helper falls back to keeping splash on vesad, shows "GPU driver did not load" overlay. - -**Linux mapping:** - -| Linux | Red Bear | -|---|---| -| `drm_aperture_remove_conflicting_framebuffers()` | Init via `handoff.rs` (driver doesn't do implicit aperture management) | -| `CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER` | bootanim holds firmware FB visible until handoff step 4 | -| Plymouth `show-splash` / `hide-splash` | bootanim exit + sessiond Seat transition signal | - -> **See also:** `local/docs/boot-logs/REDBEAR-MINI-BOOT-PS2D-INPUTD-LOG-FIX.md` -> for the 2026-06-30 fix that added `inputd: scheme:input registered` and -> `ps2d: registered producer handle, listening on serio/0 (keyboard) and -> serio/1 (mouse)` startup log lines. The handoff binary's step 6 -> (`REBIND_DISPLAY drm to inputd`) now has companion observability — -> operators can confirm `inputd` is alive before the handoff completes. - -**Verification:** -- `redbear-full` QEMU: screen never black for >1 frame during handoff -- Disable redox-drm: fallback message appears, user can still log in via getty -- Kill bootanim mid-handoff: handoff helper detects and recovers - ---- - -### PHASE 4 — Quiet Boot (Log Suppression Behind Splash) - -**Goal:** Normal boot shows only splash. Kernel/init log hidden unless user presses Esc or boot fails. - -**Effort:** 1 day - -**Files to modify:** - -| Path | Change | -|---|---| -| `local/sources/base/drivers/graphics/fbbootlogd/src/main.rs` | Add `--quiet` flag (don't open display, write to logd only) | -| `local/sources/base/drivers/graphics/fbbootlogd/src/scheme.rs` | Quiet mode: no display painting until SIGUSR2 | -| `local/sources/base/drivers/inputd/src/main.rs` | Separate "log sink" consumer role from "display" consumer | -| `config/redbear-full.toml` | fbbootlogd args `["--quiet"]` | -| `config/redbear-mini.toml` | fbbootlogd args `[]` (no quiet — text target shows log) | -| `local/docs/QUIET-BOOT-SPEC.md` | Kernel cmdline `redbear_quiet=0|1`, key bindings, failure modes | - -**Reveal key:** Esc (configurable in `/etc/redbear/bootanim.toml`) → bootanim sends SIGUSR2 to fbbootlogd → fbbootlogd connects to display, paints log. Esc again → disconnects, clears overlay. - -**Force-reveal conditions (always show log, no quiet):** -- Kernel panic -- `redox-drm` register timeout -- Init restart loop > 2 times -- `redbear_quiet=0` kernel cmdline - -**Verification:** -- `redbear-full`: no log text during normal boot. Esc reveals, Esc hides. -- `redbear-mini`: log always visible (no quiet). -- Daemon crash during boot: log auto-reveals for 5s. - ---- - -### PHASE 5 — Boot Speed: Flatten the Stage Graph - -**Goal:** Parallelize display path with hardware enumeration. Remove the 200ms sleep hack. - -**Effort:** 1–2 days - -**Current chain (4 barrier syncs):** -``` -00_base → 02_early_hw → 04_drivers → 06_services → 08_userland → SDDM -``` - -**Proposed chain (parallel branches):** -``` -00_base.target (10_inputd is the ONLY hard dep) - ├─ [branch A — display] [branch B — hardware] - │ 10_bootanim 50_rootfs - │ 20_vesad 02_early_hw.target - │ 20_fbbootlogd 04_drivers.target - │ 20_fbcond redox-drm, xhcid, e1000d, ... - │ 06_services.target - │ dbus, sessiond, dhcpd - │ - └──────────────┬───────────────────┘ - │ - 08_userland.target - 12_sddm (requires 50_drm-handoff, not 04_drivers.target) - 29_activate_console (no sleep — waits on handoff FD) - 30_console (getty 2) -``` - -**Key changes:** -- Display services and driver services run in parallel -- `29_activate_console` uses FD-barrier instead of `sleep 0.2` (the FD-handoff pattern from existing pcid patches) -- SDDM requires `50_drm-handoff.service`, not `04_drivers.target` -- fbcond retry limit removed — handoff helper retries DRM internally with exponential backoff (30s budget) - -**Benchmark targets:** - -| Metric | QEMU target | Bare-metal target | -|---|---|---| -| kernel_entry → bootanim started | < 300ms | < 200ms | -| bootanim → SDDM visible | < 2.0s | < 4.0s | -| kernel_entry → SDDM painted | < 5.0s | < 7.0s | -| Regression threshold | >10% fails CI | >10% fails CI | - -**Verification:** -- `measure-boot-stages.sh` produces CSV of stage timestamps -- QEMU video recording: splash from start to SDDM, no black gap -- `redbear-mini` unchanged (speedup is redbear-full specific) - ---- - -### PHASE 6 — Bootloader Branding & Live Progress - -**Goal:** Red Bear logo visible from UEFI handoff. Branded boot menu with auto-boot countdown. - -**Effort:** 1–2 days - -**Files to add/modify:** - -| Path | Change | -|---|---| -| `local/sources/bootloader/src/os/uefi/boot_logo.rs` | New module: `Blt()` bootlogo BMP at native resolution | -| `local/sources/bootloader/src/os/uefi/display.rs` | Extend Output to support `Blt()` with 32-bit BGRA | -| `local/sources/bootloader/src/os/uefi/video_mode.rs` | Prefer largest available mode, paint bootlogo | -| `local/sources/bootloader/src/main.rs` | Add `--quiet` (default on), `--menu-timeout=3` config | -| `local/sources/bootloader/mk/uefi.mk` | Embed BMPs at compile time via `include_bytes!` | -| `recipes/core/bootloader/recipe.toml` | Add redbear-assets as dependency | -| `local/docs/BOOTLOADER-BRANDING-SPEC.md` | Menu layout, timeout, key bindings, text fallback | - -**Bootloader progress bar:** -- Logo + thin progress bar at bottom (0% at start) -- Bar fills to 10% when kernel is read from disk -- Bar fills to 100% when kernel entry is reached -- Same logo persists through kernel → init transition (no visible gap) - -**Fallback:** If UEFI GOP doesn't support `Blt()`, bootloader falls back to text mode. Splash from Phase 2 still works. - -**Verification:** -- `redbear-full` ISO in QEMU: red bear logo in UEFI FB, 3s menu, smooth transition to kernel FB -- Bare metal AMD + Intel: same behavior -- Firmware without Blt(): text fallback works - ---- - -### PHASE 7 — Early Graphical Greeter - -**Goal:** Something graphical appears before full SDDM/KWin is ready (~2s splash → ~3s minimal greeter). - -**Effort:** 1–2 days - -**Files:** - -| Path | Type | Purpose | -|---|---|---| -| `local/recipes/wayland/redbear-compositor/source/src/bin/mini.rs` | Rust | Minimal Wayland greeter (user selector on black bg) | -| `config/redbear-greeter-services.toml` | config | `11_mini-greeter.service` between handoff and SDDM | - -**The mini greeter:** -- Tiny Wayland compositor (few hundred lines Rust) -- Shows single user selector per configured user -- Owns the `wl_display` before KWin -- On user selection: calls `org.freedesktop.login1.Manager.SwitchToUser(uid)`, exits -- Init then starts `12_sddm` which inherits the Wayland display - -**Verification:** -- `redbear-full`: splash → mini greeter (~500ms) → user selection → KWin/Plasma -- Total time < 7s on QEMU -- `redbear-mini`: unchanged - ---- - -### PHASE 8 — Clean FB Resource Management - -**Goal:** vesad releases bootloader FB on handoff. Memory accounting is auditable. - -**Effort:** 4–8 hours - -**Files to modify:** - -| Path | Change | -|---|---| -| `local/sources/base/drivers/graphics/vesad/src/main.rs` | On RELEASE_EARLYFB: munmap FB, close FD, log freed bytes, exit 0 | -| `local/sources/base/drivers/graphics/vesad/src/scheme.rs` | Track FB lifetime in `Resource` struct | -| `local/sources/base/drivers/inputd/src/main.rs` | On handoff: query vesad resource, log freed bytes, 30s kill watchdog | -| `config/redbear-bootanim.toml` | Add vesad-release-timeout watchdog service | -| `local/docs/FB-RESOURCE-LIFECYCLE.md` | Full lifecycle diagram with byte counts | - -**FB lifecycle:** - -``` -Bootloader → vesad mmap (8MB) → redox-drm allocates DRM FB (8MB) -→ handoff: both mapped briefly (16MB) → release vesad → only DRM (8MB) -``` - -**Verification:** -- `/var/log/logd` shows FB byte counts through lifecycle -- Watchdog kills vesad if release hangs >30s -- `redbear-mini`: vesad stays alive (no DRM, no release) - ---- - -## 3. Dependency Graph - -``` -Phase 1 (branding assets) ← everything downstream - │ -Phase 2 (bootanim daemon) ← needs Phase 1 assets - │ -Phase 3 (atomic handoff) ← needs Phase 2 state machine - │ -Phase 4 (quiet boot) ← independent, parallelizable - │ -Phase 5 (boot speed graph) ← needs Phase 3 (handoff is the barrier) - │ -Phase 6 (bootloader branding) ← independent, parallelizable - │ -Phase 7 (mini greeter) ← needs Phase 3 + Phase 5 - │ -Phase 8 (FB resource mgmt) ← needs Phase 3 (release step) - -Critical path: 1 → 2 → 3 → 5 → 7 -Parallelizable: 4, 6, 8 -``` - ---- - -## 4. Effort Summary - -| Phase | Effort | Risk | Rollback | -|---|---|---|---| -| 1. Branding assets | 1–4 h | Trivial (host-side imagemagick) | Delete recipe + config | -| 2. bootanim daemon | 1–2 d | Handoff correctness is subtle | Disable service; log/console still works | -| 3. Atomic handoff | 4–8 h | Low (thin orchestrator) | Fallback to vesad if handoff fails | -| 4. Quiet boot | 1 d | Reveal key must work pre-fbcond | Per-config opt-in; mini unchanged | -| 5. Boot speed | 1–2 d | Invasive stage graph restructure | Revert config; one git checkout | -| 6. Bootloader branding | 1–2 d | UEFI Blt() varies by firmware | Text mode fallback preserved | -| 7. Mini greeter | 1–2 d | New UI; keyboard handling | Opt-in per config; SDDM still works | -| 8. FB resource mgmt | 4–8 h | Force-killing vesad could break consumers | Disable watchdog service | - -**Total: ~7–10 working days** for a single engineer to land all 8 phases. -**First visible improvement:** Phase 1 + Phase 2 (~2 days) → bootloader logo + splash on earlyfb. -**Full CachyOS-class experience:** All 8 phases. - ---- - -## 5. Watch-Outs - -1. **Bootloader `Blt()` is firmware-dependent.** Test on ≥2 bare-metal firmwares + QEMU OVMF. If GOP doesn't support `Blt()`, text fallback kicks in. -2. **Resolution mismatch on handoff.** If DRM mode differs from vesad earlyfb, bootanim resamples the cached frame (Lanczos). Worst case: Intel i915 at 1366×768 panel + 1920×1080 DRM mode. -3. **Init FD-handoff semantics** assumed by Phase 5 (`pass_fds = [3]`) must be verified in init source before restructuring the boot graph. -4. **No patches in `local/patches/`.** All changes are direct edits in `local/sources//` (Rule 1) or tracked config fragments. -5. **Actual source paths:** `local/sources/base/drivers/graphics//`, not `local/sources/base/src/daemon/`. Verify before editing. -6. **KWin QML gate:** If full Plasma can't boot, Phase 7's mini greeter is the graceful degradation. Working graphical session without Plasma is better than stuck boot. -7. **Legacy virtio-gpud/ihdgd conflict:** Verify `config/redbear-full.toml` excludes these. If they ship alongside redox-drm, they'll race for the display scheme. - ---- - -## 6. Immediate Next Steps (Blocking Issues) - -Before starting Phase 1, fix these existing issues that block a clean boot: - -1. **Init stops at thermald** — why console services (29-31) never start. Need runtime debug output from init. -2. **`29_activate_console.service` no-op** — redbear-legacy-base.toml overrides to `cmd = "true"`. VT 2 never activated. -3. **Remove temporary debug code** from init main.rs (INIT_LOG_LEVEL=DEBUG, debug_log function). -4. **Fix `00_acpid.service` reference** — `00_driver-manager.service` references non-existent `00_acpid.service` (should be `30_acpid.service`). diff --git a/local/docs/RELEASE-BUMP-WORKFLOW.md b/local/docs/RELEASE-BUMP-WORKFLOW.md new file mode 100644 index 0000000000..179171aac6 --- /dev/null +++ b/local/docs/RELEASE-BUMP-WORKFLOW.md @@ -0,0 +1,368 @@ +# Red Bear OS — Release-Bump Workflow + +**Status:** canonical operator runbook (2026-07-18) +**Audience:** operators performing a Red Bear OS release bump +**Scope:** moving version labels, fork sources, and the external desktop +stack forward across a release-branch switch. + +This document is the **runbook** for the release-bump pipeline. It is the +companion to two contracts: + +- `local/AGENTS.md` § "Version conventions — two categories" and + § "No-fake-version-label rule" — the *policies* this pipeline enforces. +- `local/scripts/TOOLS.md` — the *tool inventory* (modes, flags, where each + tool runs). + +If the docs disagree, AGENTS.md is the policy authority and TOOLS.md is the +behavioural contract of each script. This document is the workflow that +sits on top of both. + +--- + +## TL;DR — the 30-second version + +A Red Bear OS release bump is four steps. The first is human; the rest are +mechanical or operator-judgement. + +| Step | What | Who/What | Touches source? | +|------|------|----------|-----------------| +| **1. Cut the branch** | Create the release branch on gitea (`0.3.2`) | Operator only | No | +| **2. Switch + label-sync** | `git checkout 0.3.2` → hook rewrites all version *labels* to match the branch (or run `./local/scripts/bump-release.sh` by hand) | post-checkout hook (opt-in) or operator | **No** — labels only | +| **3. Source upgrades** | `./local/scripts/bump-release.sh --with-sources --with-external` rebases eligible forks to newer upstream tags and bumps the Qt6/KF6/Plasma recipes | Operator | **Yes** — real rebases + recipe bumps | +| **4. Rebuild + stabilize** | Rebuild, fix patch fallout, regenerate lockfiles | Operator | Yes | + +Nothing in steps 2–4 ever auto-commits. Every step that changes the tree +prints the exact `git add` / `git commit` sequence the operator should run. + +--- + +## The four-step release model in detail + +### Step 1 — Cut the release branch (operator only) + +Per `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY", release branches are +created **by the operator only** — one per Red Bear OS release cycle. Agents +must never create branches. The branch name MUST be an anchored semver: +`0.3.1`, `0.3.2`, `0.4.0`. Non-semver names (`master`, `submodule/*`, +`recovered/*`) are never release branches and are ignored by every tool in +this pipeline. + +The branch is created on the canonical gitea repo +(`https://gitea.redbearos.org/vasilito/RedBear-OS.git`) from the prior +release tip, then fetched locally: + +```bash +git fetch origin +git checkout 0.3.2 # tracks origin/0.3.2 +``` + +Nothing else happens at branch creation time. The bump machinery fires on +the *checkout*, not on the branch creation. + +### Step 2 — Switch and label-sync + +When the operator checks out the new release branch, two things can happen: + +**A. If the post-checkout hook is installed** (opt-in; see +`local/docs/HOOKS.md`): + +```bash +git checkout 0.3.2 +# post-checkout-version-sync.sh fires automatically: +# - guards pass (semver branch, clean tree, no rebase in progress, …) +# - detects root Cargo.toml is at 0.3.1, not 0.3.2 +# - runs: sync-versions.sh --no-regen (labels only) +# - prints the follow-up hint pointing at step 3 +``` + +The hook does the **label-only** half of the bump: + +- Cat 0 (cookbook root `Cargo.toml`): `version = "0.3.1"` → `"0.3.2"`. +- Cat 1 (in-house crates under `local/recipes/*/source/`): same. +- Cat 2 (upstream forks under `local/sources/*/`): the *suffix* moves — + `0.9.0+rb0.3.1` → `0.9.0+rb0.3.2`. The upstream base (`0.9.0`) is + preserved because no source rebase has happened yet. + +The hook explicitly does NOT: + +- regenerate lockfiles (`--regen` is never passed — it is a separate + explicit step in step 4), +- touch fork source content, +- commit anything, +- run on non-semver branches or dirty trees, +- block the checkout under any circumstance (always exits 0). + +**B. If the hook is not installed**, run the orchestrator by hand: + +```bash +./local/scripts/bump-release.sh # label sync + reports, no mutations beyond labels +# or equivalently for labels only: +./local/scripts/sync-versions.sh --no-regen +``` + +`bump-release.sh` (no args) delegates the label rewrite to +`sync-versions.sh --no-regen` and additionally prints a fork-upstream +report and an external-version report so the operator can see what step 3 +will need to touch. + +After step 2 the tree has correct labels but possibly stale sources and +lockfiles. The operator should commit the label bump: + +```bash +git add -A +git commit -m "release: sync version labels to 0.3.2" +``` + +### Step 3 — Source upgrades + +This is the expensive half. It is always operator-initiated via explicit +flags and never triggered by the hook. + +#### 3a. Cat 2 fork source bumps — `--with-sources` + +```bash +./local/scripts/bump-release.sh --with-sources +``` + +For each fork in `local/fork-upstream-map.toml`, `bump-release.sh`: + +1. **Classifies** the fork by its map `mode` column (decision tree below). +2. For eligible forks (mode `snapshot` or `tracked`), `git ls-remote --tags` + the upstream URL, find the latest anchored-semver tag, and compare to + the fork's current base (the `` before `+rb`). +3. If upstream moved: invoke `upgrade-forks.sh --to= `, which + fetches + pins `upstream_ref` to `refs/tags/`, creates a backup + branch (`rb-backup/`), resets, re-applies the Red Bear net-diff, + falls back to cherry-pick if needed, regenerates the fork lockfile, and + runs `verify-fork-functions.sh`. +4. On success: set the fork `Cargo.toml` `version = "+rb"`, + update map column 3 in place, and print the exact commit commands for + the parent repo and the `submodule/` branch. +5. On failure: report, leave map/label untouched. No partial state. + +Per the **no-fake-label rule** (`local/AGENTS.md` § "No-fake-version-label +rule (STRICT)"), labels never move without matching source content. When +upstream is unchanged, only the `+rb` suffix moves (already done in +step 2). When upstream moved, the fork is really rebased first, then the +label moves to match. + +#### 3b. External desktop stack — `--with-external` + +```bash +./local/scripts/bump-release.sh --with-external +# or just the version report first: +./local/scripts/check-external-versions.sh +``` + +This drives the rewritten, map-driven `bump-graphics-recipes.sh` over the +Qt6/KF6/Plasma desktop stack and singletons (mesa, libdrm, libwayland, +sddm, …) declared in `local/external-upstream-map.toml`. Groups resolve +their version **once** (e.g. all KF6 frameworks move to the same +`frameworks/X.Y/`), symlinked recipes are deduped by realpath so qtbase is +bumped exactly once, and each recipe is validated with +`repo validate-patches ` after the bump. Results land in +`.redbear-recipe-bump/last-report.txt` with the line format: + +``` +recipe= old= new= patches_total= patches_pass= patches_fail= fail_details= +``` + +Failing patch validation is **reported, not auto-fixed** and never causes a +recipe to be removed (per the AGENTS.md ABSOLUTE RULE). Failures become +work items for step 4 (stabilization). + +The two flags compose: + +```bash +./local/scripts/bump-release.sh --with-sources --with-external +``` + +### Step 4 — Rebuild and stabilize + +After sources move, the build invalidation machinery fires automatically: + +- **Source fingerprints** — each recipe's content-hash cache (BLAKE3 of + build deps) detects that the fork tarball changed and forces a rebuild. +- **relibc ABI wipe** — when relibc itself moved, every downstream recipe + is rebuilt against the new `libc.a`. +- **Prefix auto-rebuild** — `build-redbear.sh` detects that relibc/kernel/ + base have commits newer than `prefix/.../libc.a` and rebuilds the prefix + before any recipe build begins. +- **Preflight gates** — `build-preflight.sh` re-runs `sync-versions.sh + --check`, `verify-fork-versions.sh`, `verify-patch-content.py`, and + `verify-collision-detection.py` before the build proceeds. + +Stabilization work then proceeds in the normal build-fix loop: patch +failures surfaced by step 3's report are fixed at the root cause (real +implementation, no stubs — see `local/AGENTS.md` § "STUB AND WORKAROUND +POLICY"). Lockfile regeneration is a separate explicit operator step: + +```bash +./local/scripts/sync-versions.sh --regen # after labels + sources settle +``` + +--- + +## Fork decision tree (for `--with-sources`) + +The fork's `mode` column in `local/fork-upstream-map.toml` decides what +`bump-release.sh --with-sources` is allowed to do. Modes currently in use: + +| Mode | Meaning | Source-bump eligible? | +|------|---------|-----------------------| +| `snapshot` | Imported from an archived upstream snapshot; git history tracks a single upstream tag. (syscall, libredox, redoxfs, redox-scheme, relibc, userutils) | **Yes** — auto source-bump when upstream semver moves. | +| `tracked` | Fork follows an upstream ref (e.g. `main`); content checked but not byte-pinned. (base) | **Yes** — but `base`'s map tag is `main`, so it always resolves to suffix-only (see below). | +| `diverged` | Fork has diverged materially from upstream; no clean rebase path without operator work. (kernel, bootloader, installer) | **No** — report-only. Escape hatch: `--force-diverged`. | + +Decision logic, per fork: + +``` +mode == snapshot or tracked: + latest_upstream = git ls-remote --tags | latest anchored semver + if latest_upstream > current_base: + action = source-bump (upgrade-forks.sh --to=) + else: + action = suffix-only (already done in step 2) + +mode == diverged: + action = report-only + # operator may pass --force-diverged to attempt a rebase anyway + +tag == main (base special case): + action = suffix-only, always + # base member crates carry no +rb suffix and the verifier accepts that + # (see AGENTS.md § "Version conventions — two categories") +``` + +### Bootloader note (special case) + +`bootloader` is `diverged` and additionally has **genuinely unrelated git +history** — there is no merge-base between the fork HEAD and upstream +`master`. `git merge-base HEAD upstream/master` fails. This means +`upgrade-forks.sh` cannot compute a net-diff to reapply and any attempt to +source-bump it would destroy committed work. + +`bump-release.sh` NEVER attempts a bootloader source bump, even with +`--force-diverged`. It reports bootloader as `action=report-only` with the +guidance: + +> Bootloader fork has no merge-base with upstream. A one-time +> `git replace --graft ` intervention is required +> to synthesize a common ancestor before any rebase can be attempted. This +> is future work; the orchestrator never performs it. + +The graft path (documented for the future, not run by any script today): + +```bash +cd local/sources/bootloader +git replace --graft $(git rev-parse HEAD) +# now upgrade-forks.sh can compute a merge-base and reapply the Red Bear delta +``` + +Until that graft lands, bootloader stays at its current upstream base and +only its `+rb` suffix moves with each release bump. + +--- + +## Notes and invariants + +### `Cargo.toml.orig` files are never touched + +Every Cat 2 fork may carry a `Cargo.toml.orig` that mirrors the upstream +`Cargo.toml` byte-for-byte (Cargo leaves it behind when the fork edits the +real `Cargo.toml`). `sync-versions.sh`, `bump-release.sh`, and the +post-checkout hook all edit ONLY the live `Cargo.toml`. `Cargo.toml.orig` +is a read-only witness of upstream state and MUST stay byte-identical to +upstream so that `verify-fork-versions.sh` can compute the Red Bear delta. + +### Lockfile regeneration is a separate explicit step + +`sync-versions.sh --no-regen` (the default, and what the hook calls) rewrites +version labels but does not regenerate `Cargo.lock`. Lockfile regen is +opt-in via `sync-versions.sh --regen` and should be run after *both* labels +and sources have settled, because a source rebase can pull new transitive +deps that an earlier label-only regen would miss. + +### Nothing auto-commits + +Every step that changes the tree prints the exact commit sequence: + +- **Parent repo** (`RedBear-OS` on the release branch): label bumps, map + column-3 updates, recipe tarball/blake3/rev updates. +- **`submodule/` branches**: fork source rebases. + +The operator runs `git add` / `git commit` / `git push` themselves. This is +non-negotiable per `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY" and +the project's general "agents never commit unless asked" rule. + +### Role of `provision-release.sh` and `REDBEAR_RELEASE` + +`provision-release.sh` is the **freeze-time** tool: it provisions a new +immutable release archive from a Redox ref into `sources/redbear-/`. +It is invoked explicitly and human-initiated, never by the bump pipeline. + +`REDBEAR_RELEASE=` is the env var that switches the build into +**sealed/release mode**: sources are extracted from the immutable archive, +online fetching is completely disabled, and local forks are ignored. In +development mode `REDBEAR_RELEASE` MUST be unset (see `.config` rules in +`AGENTS.md` § "BUILD COMMANDS"). The bump pipeline runs in development mode; +it has no behaviour in sealed mode. + +--- + +## Cross-references + +| Topic | Where | +|-------|-------| +| Version categories (Cat 1 / Cat 2), `+rb` suffix rules | `local/AGENTS.md` § "Version conventions — two categories" | +| No-fake-version-label rule (labels match source content) | `local/AGENTS.md` § "No-fake-version-label rule" | +| Latest-upstream-before-freeze rule | `local/AGENTS.md` § "Local fork dependency rule" | +| Local fork dependency rule (path deps, no version strings) | `local/AGENTS.md` § "Local fork dependency rule" | +| Branch and submodule policy (operator-only branches) | `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY" | +| Tool inventory + modes | `local/scripts/TOOLS.md` | +| Hook install / bypass / inventory | `local/docs/HOOKS.md` | +| Fork → upstream tag/mode map | `local/fork-upstream-map.toml` | +| External desktop-stack version map | `local/external-upstream-map.toml` | +| Post-checkout hook source | `local/scripts/post-checkout-version-sync.sh` | +| Installer source | `local/scripts/install-git-hooks.sh` | +| Orchestrator source | `local/scripts/bump-release.sh` | + +--- + +## Appendix — full worked example + +Bumping from `0.3.1` to `0.3.2` with the hook installed: + +```bash +# Step 1 — operator creates 0.3.2 on gitea (out of band), then: +git fetch origin +git checkout 0.3.2 +# ⟶ post-checkout hook fires: +# post-checkout-version-sync: branch '0.3.2' root Cargo.toml at '0.3.1' -> '0.3.2' +# post-checkout-version-sync: running sync-versions.sh --no-regen (labels only, no lockfile regen) +# ... sync output ... +# post-checkout-version-sync: label sync complete for branch '0.3.2'. +# follow-up steps (NOT run automatically): ... + +git add -A && git commit -m "release: sync version labels to 0.3.2" + +# Step 3 — source upgrades (operator decides) +./local/scripts/bump-release.sh --with-sources --with-external +# ⟶ per-fork decision lines, per-recipe validation report at +# .redbear-recipe-bump/last-report.txt. Prints commit commands. + +# Step 4 — rebuild + stabilize +./local/scripts/sync-versions.sh --regen # lockfiles +./local/scripts/build-redbear.sh redbear-mini # build + preflight gates +# fix any patch fallout at the root cause; commit on 0.3.2 / submodule/ +``` + +Without the hook, step 2 becomes: + +```bash +git checkout 0.3.2 +./local/scripts/bump-release.sh # labels + reports +git add -A && git commit -m "release: sync version labels to 0.3.2" +``` + +The rest is identical. diff --git a/local/docs/RELIBC-AGAINST-GLIBC-ASSESSMENT.md b/local/docs/RELIBC-AGAINST-GLIBC-ASSESSMENT.md deleted file mode 100644 index bf73e88da7..0000000000 --- a/local/docs/RELIBC-AGAINST-GLIBC-ASSESSMENT.md +++ /dev/null @@ -1,207 +0,0 @@ -# Relibc vs GNU libc — Cross-Reference Assessment - -**Date:** 2026-05-05 -**Reference:** glibc 2.41 (2026-05-05 clone from sourceware.org) -**Relibc pinned:** commit 861bbb0 with Red Bear patch chain (26 patches) - ---- - -## 1. eventfd - -### glibc reference - -```c -// sysdeps/unix/sysv/linux/eventfd.c (not cloned yet — syscall wrapper) -// bits/eventfd.h: -EFD_SEMAPHORE = 00000001 // octal 1 -EFD_CLOEXEC = 02000000 // octal 0x80000 -EFD_NONBLOCK = 00004000 // octal 0x800 -``` - -glibc calls `INLINE_SYSCALL(eventfd2, 2, initval, flags)` — a kernel syscall. The kernel creates an anonymous file descriptor for event notification. Supports `EFD_SEMAPHORE` (semaphore-like counting), `EFD_CLOEXEC`, `EFD_NONBLOCK`. - -### relibc current state (updated 2026-05-05 — S1-S4 implemented) - -```rust -// src/header/sys_eventfd/mod.rs — 30 lines -// Full eventfd() implementation with EFD_SEMAPHORE/CLOEXEC/NONBLOCK. -// Opens scheme:event/eventfd/{initval}/{sem} via Sys::open. -``` - -**Implementation shipped** ✅ -**Kernel support**: `P0-eventfd-kernel.patch` extends event scheme with eventfd path parsing ✅ - -### Gaps - -| Gap | Severity | Detail | -|-----|----------|--------| -| No `eventfd()` in relibc | Medium | libwayland has its own inline, but relibc should be canonical | -| No `eventfd_read()`/`eventfd_write()` | Low | POSIX-adjacent convenience wrappers (glibc provides them) | - ---- - -## 2. signalfd - -### glibc reference - -```c -// sysdeps/unix/sysv/linux/signalfd.c -int signalfd(int fd, const sigset_t *mask, int flags) { - return INLINE_SYSCALL(signalfd4, 4, fd, mask, __NSIG_BYTES, flags); -} - -// bits/signalfd.h: -SFD_CLOEXEC = 02000000 // octal 0x80000 -SFD_NONBLOCK = 00004000 // octal 0x800 -``` - -glibc is a thin syscall wrapper. Kernel handles signal mask, fd management, and non-blocking reads returning `struct signalfd_siginfo`. - -### relibc current state - -```rust -// src/header/signal/signalfd.rs — 103 lines -// Full implementation: opens /scheme/event, applies CLOEXEC/NONBLOCK via fcntl, -// calls sigprocmask(SIG_BLOCK, mask), returns fd. -// signalfd4 supports modifying existing fd's flags. -``` - -**Flags match glibc** ✅ -**signalfd_siginfo struct matches** ✅ -**signalfd4 with existing fd** ✅ (fcntl-based flag modification) - -### Prowess vs glibc - -| Aspect | glibc | relibc | -|--------|-------|--------| -| Implementation | Syscall wrapper (5 lines) | Userspace via `/scheme/event` (100 lines) | -| Existing fd support | Kernel handles | fcntl O_CLOEXEC/O_NONBLOCK modification ✅ | -| Errno mapping | Kernel errno | Wraps in Errno, proper EINVAL/EFAULT | -| Signal blocking | Kernel auto-blocks on read | `sigprocmask(SIG_BLOCK, mask)` called explicitly ✅ | - -### Gaps - -| Gap | Severity | Detail | -|-----|----------|--------| -| No read path | High | Nothing reads `signalfd_siginfo` from the fd — the `/scheme/event` fd is opened but signals aren't delivered through it | -| Signal delivery unverified | High | The `sigprocmask(SIG_BLOCK)` blocks signals but there's no evidence the kernel delivers them via the event fd | -| `signalfd_siginfo` read not implemented | Critical | `struct signalfd_siginfo` is defined but never populated via read(2) | - ---- - -## 3. Semaphores - -### glibc reference - -```c -// sysdeps/pthread/sem_open.c — 216 lines -// Uses: -// __shm_get_name(name) → canonical path in /dev/shm -// O_CREAT+O_EXCL path: creates temp file, writes semaphore header, ftruncate to sizeof(sem_t), mmap -// Non-create path: open existing, __sem_check_add_mapping(name, fd) → reuse or mmap -// pthread_setcancelstate(PTHREAD_CANCEL_DISABLE) — cancellation-safe -// va_arg for mode_t when O_CREAT -``` - -glibc uses a sophisticated named semaphore implementation: -1. **Name canonicalization**: `__shm_get_name` transforms `/name` → `/dev/shm/sem.name` -2. **Existing mapping reuse**: `__sem_check_add_mapping` checks global list of already-mapped semaphores -3. **Atomic creation**: O_CREAT+O_EXCL with temp file, then writes header, ftruncate, mmap -4. **Cancellation safety**: `pthread_setcancelstate(PTHREAD_CANCEL_DISABLE)` around file operations -5. **Proper mode_t**: va_arg for mode when O_CREAT -6. **Reference counting**: `__sem_check_add_mapping` increments refcount, `sem_close` decrements - -### relibc current state - -```rust -// src/header/semaphore/mod.rs — 176 lines -// Uses: shm_open(name, O_CREAT|O_EXCL|O_RDWR, mode) → ftruncate → mmap → init -// NamedSemaphore struct with RlctSempahore (futex-based) -``` - -**Core mechanism works** ✅ (shm_open + mmap) -**sem_init/destroy/post/wait/trywait/timedwait/clockwait** ✅ -**sem_open/close/unlink** ✅ (implemented in P3-semaphore-comprehensive.patch) - -### Gaps vs glibc - -| Gap | Severity | Detail | -|-----|----------|--------| -| **No name canonicalization** | ~~Medium~~ ✅ FIXED | Names now prefixed with `sem.` before `shm_open`. glibc uses `/dev/shm/sem.NAME` equivalent. | -| **No existing mapping reuse** | ~~High~~ ✅ FIXED | Global `BTreeMap` with `AtomicUsize` refcount. `sem_open` reuses existing mappings, increments refcount. | -| **No refcounting** | ~~High~~ ✅ FIXED | `sem_close` decrements `AtomicUsize`, munmaps only when zero. | -| **No cancellation safety** | Low | No `pthread_setcancelstate` around file ops | -| **va_list not parsed** | Medium | `sem_open` hardcodes `value=0` when O_CREAT, ignoring mode and initial value from varargs | -| **No `__sem_check_add_mapping` equivalent** | High | Opens named sem every time instead of reusing existing mapping | -| **No O_NOFOLLOW** | Low | glibc uses `O_NOFOLLOW` for security | - ---- - -## 4. Cross-Cutting Gaps - -### Error Handling - -| Area | glibc | relibc | -|------|-------|--------| -| errno thread-safety | TLS errno via kernel | `Cell` per platform ✅ | -| errno after close | Preserved (close may overwrite) | `let _ = Sys::close(fd)` — ignores errors ✅ | -| EINTR | Handled in syscall wrappers | `Semaphore::wait` returns `Result<(), c_int>`. sem_wait/timedwait loop on EINTR ✅ | -| `sem_wait` | AS-safe (futex wait, EINTR) | EINTR retry loop ✅ | -| sem_open refcount | Mutex-protected global list | `BTreeMap` with `AtomicUsize` ✅ | -| sem_close/sem_unlink | Mutex-protected | `Mutex>>` protects registry ✅ | -| signalfd mask | Per-process (kernel) | Per-call sigprocmask ✅ | - ---- - -## 5. Priority Improvement Plan - -### Phase S1: Critical Correctness (1-2 weeks) - -1. **sem_open refcounting** — Add global `HashMap, AtomicUsize)>` to reuse existing mappings. `sem_close` decrements refcount, munmaps only when zero. -2. **Eventfd implementation** — Implement `eventfd()` via `/scheme/event/eventfd/` using the existing scheme mechanism. Remove libwayland's inline copy. - -### Phase S2: Completeness (2-3 weeks) - -3. **signalfd read path** — Implement read(2) → `signalfd_siginfo` struct population. The `/scheme/event` fd must deliver signal info formatted as `signalfd_siginfo`. -4. **sem_open va_list** — Parse `mode_t` and `value` from varargs when O_CREAT. Requires `crate::header::stdarg` or manual stack walking. -5. **sem_open name canonicalization** — Prefix names with `/scheme/shm/sem.` for namespace isolation. - -### Phase S3: Robustness (3-4 weeks) - -6. **EINTR handling** — Wrap futex waits to retry on `EINTR`. -7. **sem_open cancellation safety** — Add `pthread_setcancelstate` around file ops (if pthread cancellation is supported). -8. **eventfd semaphore mode** — Implement `EFD_SEMAPHORE` counting semantics (decrements on read, blocks at 0). - -### Phase S4: POSIX Conformance (2-3 weeks) - -9. **eventfd_read/eventfd_write** — Convenience wrappers. -10. **sem_open existing-semaphore reopening** — Handle `(oflag & O_CREAT) == 0` path (open existing without O_EXCL). -11. **Signalfd signal delivery verification** — Runtime tests proving signals arrive via signalfd. - ---- - -## 6. Eventfd Kernel Requirement - -Unlike signalfd and sem_open which can be implemented in userspace via existing schemes (`/scheme/event`, `shm_open`), eventfd currently relies on libwayland's inline implementation that opens `/scheme/event/eventfd/`. A canonical relibc implementation should use the same path. - -The `/scheme/event` kernel scheme needs: -- Support for `eventfd` sub-path -- EFD_SEMAPHORE counting semantics in kernel -- Non-blocking reads returning `u64` count - -This is a **kernel change** and should be tracked separately from relibc. - ---- - -## 7. Summary - -| Component | relibc Status | Matches glibc | Critical Gaps | -|-----------|--------------|---------------|---------------| -| eventfd | Full implementation | Constants ✅ | eventfd() implemented via /scheme/event/eventfd/ ✅ | -| signalfd | 103-line impl | Flags ✅, struct ✅, signalfd4 ✅ | Read path needs kernel signal delivery ⚠️ | -| sem_open | 226-line impl with refcount | Core works ✅, shm+mmap ✅ | va_list ✅, refcounting ✅, mapping reuse ✅ | -| sem_close | Refcounted munmap | Semantics correct ✅ | Atomic refcount decrement ✅ | -| sem_wait/post | Futex-based, EINTR retry | Works ✅ | EINTR loop ✅, errno returned on other errors | - -**Total estimated effort: 8-12 weeks for all gaps.** -**Critical path: eventfd kernal + signalfd read + sem_open refcounting (Phase S1).** diff --git a/local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md b/local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md deleted file mode 100644 index ef5c1131cd..0000000000 --- a/local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md +++ /dev/null @@ -1,456 +0,0 @@ -# Red Bear OS /scheme/ Namespace Population Plan - -**Version**: 1.0 (2026-06-12) -**Status**: Draft — pending review -**Canonical**: `local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md` -**Blocks**: Writable rootfs on live ISO, `redoxfs` disk discovery, `ls /scheme/` in shell -**Cross-references**: Linux kobject/uevent, Fuchsia Zircon/Component Manager, seL4 CSpace, Plan 9 per-process namespace, Genode capability routing, MINIX 3 driver model - -## 1. Problem Statement - -`ls /scheme/` hangs or returns empty in Red Bear OS. Three root causes: - -1. **initnsmgr `getdents` depends on daemons registering** — but boot ordering means some schemes - haven't registered yet when `redoxfs` calls `fs::read_dir("/scheme")` to find disk devices. -2. **No aggregator for block devices** — `redoxfs` must enumerate all `disk.*` schemes individually, - but `/scheme/disk.live` may not exist yet when the rootfs mount runs at priority 50. -3. **driver-block `getdents` returns `EOPNOTSUPP`** — individual disk schemes use legacy text-based - listing, not proper `getdents`. - -The result: `redoxfs` can't discover disks, rootfs fails to mount read-write, and `/scheme/` -listing is incomplete. - -## 2. Design Principles (Informed by Cross-Reference) - -### 2.1 Microkernel Principle (seL4, Red Bear OS) - -The kernel tracks scheme IDs (integers), not names. All name→ID mapping happens in userspace -(`initnsmgr`). This is correct per the user's explicit correction: - -> "Kernel does not have to track id-name mapping! Kernel only knows about IDs. It's a microkernel -> and stuff like this must be done in userspace" - -**Implication**: We never modify the kernel to "export" scheme names. The namespace is purely -a userspace construct managed by `initnsmgr`. - -### 2.2 Aggregator Pattern (Linux devtmpfs + Fuchsia devcoordinator) - -Linux populates `/dev` via two mechanisms: -- **devtmpfs** — kernel auto-creates basic `/dev/null`, `/dev/sda1` etc. at boot -- **udev** — userspace daemon receives uevents via netlink, applies rules, creates additional nodes - -Fuchsia uses **devcoordinator** (now driver-index + device-finder): -- Drivers register devices with the driver manager -- devcoordinator exposes them via `devfs` (listable, browsable) -- Component Manager routes specific devices to components via capability declarations - -Red Bear OS should follow the **aggregator** pattern: userspace daemons that discover, -enumerate, and expose device categories through listable scheme namespaces. - -### 2.3 Bootstrap Ordering (Plan 9, Fuchsia) - -Plan 9 bootstraps namespace incrementally: -1. Kernel boots with `#` device drivers (kernel-resident, like Red Bear's `GlobalSchemes`) -2. `boot(8)` script binds drivers into the namespace -3. `init(8)` builds the per-process namespace from `/lib/namespace` - -Fuchsia bootstraps similarly: -1. Zircon boots, creates root job + resource handles -2. component_manager starts, receives boot info (device handles from ZBI) -3. driver_index enumerates drivers, binds them to devices -4. devfs provides the listable namespace - -Red Bear OS boot sequence (current): -``` -bootstrap → initnsmgr (initial schemes: 10 kernel globals + "proc" + "initfs") - → init starts service targets - → 10_lived.service (priority 10): registers "disk.live" - → 40_drivers.target: pcid, graphics, etc. - → 45_diskd.service (NEW): scans disk.* schemes, registers "diskd" - → 50_rootfs.service: redoxfs uses diskd to find root device -``` - -### 2.4 Separation of Discovery and Access (Genode, seL4) - -Genode separates: -- **Platform session** — device discovery (what hardware exists) -- **I/O session** — device access (read/write/mmio) - -seL4 separates: -- **Device Untyped caps** — raw hardware access -- **Platform description** — structured description of what devices exist - -In Red Bear OS terms: `diskd` provides discovery (listing), but actual block I/O goes through -the original `disk.live`/`disk.sata0` schemes directly. `diskd` returns `OpenResult::OtherScheme` -so the kernel hands the caller a raw fd to the underlying scheme — zero overhead. - -## 3. Current Architecture - -### 3.1 Kernel Global Schemes (10) - -Registered by bootstrap in `exec.rs` → `initnsmgr::run()`: - -| Scheme | GlobalSchemes Variant | Kernel Source | -|--------|-----------------------|---------------| -| debug | Debug | `scheme/debug.rs` | -| event | Event | `scheme/event.rs` | -| memory | Memory | `scheme/memory.rs` | -| pipe | Pipe | `scheme/pipe.rs` | -| serio | Serio | `scheme/serio.rs` | -| irq | Irq | `scheme/irq.rs` | -| time | Time | `scheme/time.rs` | -| sys | Sys | `scheme/sys/mod.rs` | -| proc | Proc | `scheme/proc/mod.rs` | -| acpi | Acpi | `scheme/acpi.rs` | -| dtb | Dtb | `scheme/dtb.rs` | - -These are registered in the `KernelSchemes` enum (kernel/src/scheme/mod.rs:438) and -exposed to initnsmgr during bootstrap. - -### 3.2 initnsmgr Namespace Manager - -Located at `local/sources/base/bootstrap/src/initnsmgr.rs`. - -Key structures: -```rust -struct Namespace { - schemes: HashMap>, // name → fd -} -``` - -- `open("")` → `Handle::List` (directory listing handle) -- `getdents(Handle::List)` → iterates `schemes` HashMap, returns `DirEntry` for each name -- Daemons register via `NsDup::IssueRegister` + sendfd mechanism -- Bootstrap passes initial set: kernel globals + "proc" + "initfs" - -### 3.3 Userspace Scheme Registration - -Daemons register via: -1. `Socket::create()` → creates scheme socket -2. `NsDup::IssueRegister` → tells initnsmgr the scheme name -3. `sendfd` → sends the scheme socket fd to initnsmgr -4. initnsmgr stores in `schemes: HashMap>` - -### 3.4 Current Userspace Schemes (at boot) - -| Scheme | Daemon | Priority | Source | -|--------|--------|----------|--------| -| initfs | bootstrap | 0 | bootstrap exec.rs | -| proc | kernel | 0 | GlobalSchemes | -| disk.live | lived | 10 | init.initfs.d/10_lived.service | -| disk.sata0 | ahcid | 40 | pcid-spawner | -| disk.virtio0 | virtio-blkd | 40 | pcid-spawner | -| display | vesad | 20 | init.initfs.d/20_vesad.service | -| drm | redox-drm | 30 | init.initfs.d/30_graphics.service | -| net | e1000d / virtio-netd | 40 | pcid-spawner | -| orbital | orbital | rootfs | (legacy, not used in redbear-full) | - -### 3.5 The Root Cause Chain - -``` -redoxfs mount (priority 50) - → fs::read_dir("/scheme") → initnsmgr getdents - → iterates schemes HashMap → finds "disk.live" (registered at priority 10) - → is_scheme_category("disk") → true - → Fd::open("/scheme/disk.live") → reads text listing - → finds block device → opens /scheme/disk.live/0 → reads UUID - → UUID matches → mounts as rootfs -``` - -**The bug**: `redoxfs` retries 20×200ms = 4 seconds. If disk discovery takes longer than -4 seconds (e.g., AHCI probe on real hardware), rootfs mount fails → read-only fallback. - -**The fix**: `diskd` aggregator + longer timeout + event-driven notification. - -## 4. Solution Architecture - -### 4.1 Component Overview - -``` -┌─────────────────────────────────────────────────────────┐ -│ /scheme/ namespace │ -│ (initnsmgr) │ -│ │ -│ Kernel globals: │ -│ debug, event, memory, pipe, serio, irq, │ -│ time, sys, proc, acpi, dtb │ -│ │ -│ Boot schemes (initfs): │ -│ initfs, disk.live, display │ -│ │ -│ Aggregators: │ -│ diskd ← /scheme/diskd lists ALL block devices │ -│ │ -│ Hardware daemons (post-drivers.target): │ -│ disk.sata0..7 (ahcid) │ -│ disk.virtio0..7 (virtio-blkd) │ -│ disk.nvme0..7 (nvmed) │ -│ disk.usb0..7 (usbscsid) │ -│ disk.ide0..3 (ideid) │ -│ net (e1000d, virtio-netd, ixgbed, rt8169d) │ -│ drm (redox-drm) │ -│ │ -│ System daemons (post-rootfs): │ -│ audio (audiod) │ -│ firmware (firmware-loader) │ -│ input (evdevd) │ -│ udev (udev-shim) │ -│ ... │ -└─────────────────────────────────────────────────────────┘ -``` - -### 4.2 diskd — Disk Aggregator Daemon (IMPLEMENTED) - -**Location**: `local/recipes/system/diskd/` -**Scheme name**: `diskd` -**Binary**: `/usr/bin/diskd` -**Status**: Code complete, cargo check/clippy/fmt clean - -**How it works**: - -1. At boot (priority 45), diskd starts -2. Probes `/scheme/disk.live`, `/scheme/disk.sata0`..7, `/scheme/disk.virtio0`..7, etc. -3. For each found scheme, reads its text listing to discover devices and partitions -4. Registers scheme `diskd` with initnsmgr -5. `getdents` on `diskd:` returns real `DirEntry` with `DirentKind::BlockDev` -6. `open("0")` or `open("0p1")` opens the underlying scheme and returns `OtherScheme` - (zero-copy — caller talks directly to the block device) - -**Why this solves the root cause**: - -- `redoxfs` currently must enumerate ALL `/scheme/disk.*` individually — 50+ `Fd::open` calls -- With `diskd`, `redoxfs` does ONE `read_dir("/scheme/diskd")` to get all block devices -- diskd already did the probing and enumeration -- Even if AHCI hasn't registered yet, diskd's retry logic handles late registration -- `redoxfs` timeout only needs to wait for `diskd` to be ready, not all individual schemes - -### 4.3 Changes Required to Existing Components - -#### 4.3.1 redoxfs — Use diskd for disk discovery - -**File**: `local/sources/redoxfs/src/bin/mount.rs` (function `filesystem_by_uuid`) - -**Current behavior**: -```rust -// Line 224: fs::read_dir("/scheme") → filter is_scheme_category("disk") -// For each disk.* scheme: open, read listing, find block devices, check UUID -// Retry 20×200ms = 4 seconds total -``` - -**New behavior** (two-path approach): - -```rust -fn filesystem_by_uuid(uuid: &[u8; 16]) -> Option { - // Path A: Try diskd aggregator first (fast, single enumeration) - if let Some(f) = try_diskd_uuid(uuid) { - return Some(f); - } - // Path B: Fall back to legacy per-scheme enumeration - // (for backwards compat and environments without diskd) - try_legacy_uuid_search(uuid) -} - -fn try_diskd_uuid(uuid: &[u8; 16]) -> Option { - // Wait for diskd scheme to appear - for _ in 0..50 { // 50 × 200ms = 10 seconds - if let Ok(dir) = fs::read_dir("/scheme/diskd") { - for entry in dir { - let entry = entry.ok()?; - let name = entry.file_name().to_string_lossy().into_owned(); - // Open the block device via diskd (which proxies to underlying scheme) - let path = format!("/scheme/diskd/{name}"); - if let Ok(mut f) = File::open(&path) { - if check_uuid(&mut f, uuid) { - return Some(f); - } - } - } - } - thread::sleep(Duration::from_millis(200)); - } - None -} -``` - -#### 4.3.2 init.initfs.d — Add diskd service - -**New file**: `local/sources/base/init.initfs.d/45_diskd.service` - -```ini -[[service]] -name = "diskd" -command = "/usr/bin/diskd" -priority = 45 -requires = ["lived"] -``` - -This ensures diskd starts after lived (which provides disk.live at priority 10) and before -rootfs mount (priority 50). - -#### 4.3.3 config/redbear-mini.toml — Add diskd package - -Add `diskd` to the `[packages]` section so it's included in the image. - -### 4.4 /scheme/ Namespace Completeness Matrix - -After all changes, `/scheme/` will expose: - -| Category | Scheme Name | Provider | getdents | Notes | -|----------|-------------|----------|----------|-------| -| **Kernel globals** | | | | | -| Debug | `debug` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/debug.rs | -| Event | `event` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/event.rs | -| Memory | `memory` | kernel GlobalSchemes | EOPNOTSUPP | No sub-entries expected | -| Pipe | `pipe` | kernel GlobalSchemes | EOPNOTSUPP | Anonymous, no listing | -| Serio | `serio` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/serio.rs | -| IRQ | `irq` | kernel GlobalSchemes | ✅ real DirEntry | cpu-XX entries | -| Time | `time` | kernel GlobalSchemes | ✅ real DirEntry | CLOCK_* entries | -| Sys | `sys` | kernel GlobalSchemes | ✅ real DirEntry | scheme:/scp/ sub-entries | -| Proc | `proc` | kernel GlobalSchemes | ✅ real DirEntry | pid entries | -| ACPI | `acpi` | kernel GlobalSchemes | ✅ real DirEntry | rxsdt, kstop | -| DTB | `dtb` | kernel GlobalSchemes | EOPNOTSUPP | Single blob | -| **Bootstrap** | | | | | -| InitFS | `initfs` | bootstrap | ✅ real DirEntry | initramfs contents | -| **Storage** | | | | | -| Live disk | `disk.live` | lived | ✅ text listing | virtio/ahci backend | -| SATA disk | `disk.sata0..7` | ahcid | ✅ text listing | per-disk scheme | -| VirtIO disk | `disk.virtio0..7` | virtio-blkd | ✅ text listing | per-disk scheme | -| NVMe disk | `disk.nvme0..7` | nvmed | ✅ text listing | per-disk scheme | -| USB disk | `disk.usb0..7` | usbscsid | ✅ text listing | per-disk scheme | -| IDE disk | `disk.ide0..3` | ideid | ✅ text listing | per-disk scheme | -| **Aggregators** | | | | | -| Disk aggregator | `diskd` | diskd | ✅ real DirEntry BlockDev | THIS PLAN | -| **Display** | | | | | -| Framebuffer | `display` | vesad | EOPNOTSUPP | Legacy text listing | -| DRM/KMS | `drm` | redox-drm | ✅ real DirEntry | card0, card0-*, connectors | -| **Network** | | | | | -| Ethernet | `net` | e1000d/virtio-netd | ✅ real DirEntry | interface entries | -| **Input** | | | | | -| Input events | `input` | evdevd | ✅ real DirEntry | event0, event1, ... | -| **Audio** | | | | | -| Audio | `audio` | audiod | ✅ text listing | Audio streams | -| **System** | | | | | -| Firmware | `firmware` | firmware-loader | ✅ real DirEntry | GPU/device blobs | -| Udev | `udev` | udev-shim | ✅ real DirEntry | Linux-compatible device nodes | - -### 4.5 initnsmgr getdents — Already Correct - -The `initnsmgr` `getdents` implementation at line 402-439 of `initnsmgr.rs` iterates -`schemes: HashMap>` and emits a `DirEntry` for each registered scheme. -This is already correct — it will list any scheme that has been registered, including `diskd`. - -**The `/scheme/` listing issue was NOT a getdents bug** — it was a timing issue: -- Daemons hadn't registered yet when `fs::read_dir("/scheme")` was called -- The fix is proper boot ordering (diskd at priority 45) and the diskd aggregator - -## 5. Future Enhancements (Beyond Current Scope) - -### 5.1 Event-Driven Discovery (uevent Equivalent) - -Currently `diskd` probes statically at startup. For hotplug (USB drives, PCIe hot-add): - -- **pcid** sends a `uevent`-like notification when a new PCI device appears -- **diskd** listens for these notifications and re-scans -- Alternative: inotify-like watch on `/scheme/` (would need kernel support) - -This mirrors Linux's `uevent` netlink broadcast → `udev` listener pattern. - -### 5.2 devfs-Style Aggregation - -A future `devfsd` could provide Linux-compatible `/dev` paths: -``` -/scheme/devfs/sda → /scheme/diskd/0 -/scheme/devfs/sda1 → /scheme/diskd/0p1 -/scheme/devfs/null → /scheme/debug (write sink) -/scheme/devfs/zero → /scheme/memory (zero-filled read) -/scheme/devfs/random → /scheme/entropy -/scheme/devfs/tty0 → /scheme/display.0 -/scheme/devfs/input/event0 → /scheme/input/event0 -``` - -This would be the Fuchsia devcoordinator equivalent — a unified, Linux-compatible -device namespace. The `udev-shim` already provides parts of this. - -### 5.3 Per-Process Namespace (Plan 9 Style) - -Plan 9's `bind` and `mount` allow per-process namespace customization. Red Bear OS's -`setrens` syscall provides a basic version (switch namespace fd). Future enhancement: -- Per-container namespaces (for `contain` and future container runtime) -- Namespace inheritance rules (like Fuchsia's `.cml` capability routing) -- `chroot`-like namespace restriction for sandboxed applications - -### 5.4 Capability-Based Access (seL4 Style) - -seL4 uses CSpace (capability spaces) for device access. Each process has a CSpace that -contains only the capabilities it should have access to. Red Bear OS could evolve toward -this model: - -- `initnsmgr` tracks which schemes each process can access -- `open("/scheme/net")` checks the caller's capability set -- `setrens` evolves from "switch namespace" to "restrict to capability subset" - -This would require kernel changes (per-process scheme allowlists), which is beyond current -scope but worth keeping in mind for security hardening. - -## 6. Implementation Plan - -### Phase 1 — Immediate Fix (This Session) - -| Step | Action | Files | Status | -|------|--------|-------|--------| -| 1 | diskd daemon implementation | `local/recipes/system/diskd/` | ✅ Done | -| 2 | Add diskd init service | `local/sources/base/init.initfs.d/45_diskd.service` | Pending | -| 3 | Add diskd to config | `config/redbear-mini.toml` | Pending | -| 4 | Modify redoxfs to use diskd | `local/sources/redoxfs/src/bin/mount.rs` | Pending | -| 5 | Commit uncommitted changes | driver-manager, config | Pending | -| 6 | Remove pcid debug logging | `local/sources/base/drivers/pcid/src/cfg_access/fallback.rs` | Pending | -| 7 | Make C++ header fix durable | `mk/prefix.mk` | Pending | -| 8 | Build and test ISO | `./local/scripts/build-redbear.sh redbear-mini` | Pending | -| 9 | Boot test in QEMU | `scripts/run_mini1.sh` | Pending | - -### Phase 2 — Hotplug Support (Future) - -| Step | Action | Dependencies | -|------|--------|--------------| -| 1 | pcid uevent notification | pcid-spawner enhancement | -| 2 | diskd dynamic re-scan | uevent listener | -| 3 | devfsd Linux-compatible /dev | udev-shim + diskd integration | - -### Phase 3 — Namespace Security (Future) - -| Step | Action | Dependencies | -|------|--------|--------------| -| 1 | Per-process scheme allowlist | kernel scheme access control | -| 2 | Container namespace isolation | contain enhancement | -| 3 | Capability routing | initnsmgr capability model | - -## 7. Cross-Reference Summary - -| System | Mechanism | Red Bear Equivalent | Status | -|--------|-----------|---------------------|--------| -| **Linux** | kobject/uevent → udev → /dev | pcid → diskd → /scheme/diskd | Phase 1 | -| **Fuchsia** | devcoordinator → devfs | initnsmgr → diskd | Phase 1 | -| **seL4** | CSpace capabilities | setrens (basic) | Phase 3 | -| **Plan 9** | bind/mount per-process | setrens (basic) | Phase 3 | -| **Genode** | Platform session | redox-driver-sys | Existing | -| **MINIX 3** | driver announce → devfs | daemon register → initnsmgr | Existing | - -## 8. Risk Assessment - -| Risk | Mitigation | -|------|------------| -| diskd probe takes too long on real hardware | Increase retry count (50×200ms = 10s), add event-driven re-scan | -| diskd crashes and disk namespace disappears | init service auto-restart (`restart = true` in service file) | -| redoxfs legacy path broken by diskd changes | Two-path approach: try diskd first, fall back to legacy | -| Boot ordering regression (diskd starts before lived) | Explicit `requires = ["lived"]` in service file | -| diskd returns stale device list after hotplug | Phase 2: event-driven re-scan; Phase 1: manual re-trigger via signal | - -## 9. Acceptance Criteria - -1. `ls /scheme/` in shell shows all registered schemes (no hang, no empty) -2. `ls /scheme/diskd/` shows all block devices discovered by diskd -3. `redoxfs` mounts rootfs read-write via diskd path -4. `/tmp` is writable by non-root users -5. Boot completes to login prompt with zero warnings -6. QEMU boot test passes: `scripts/run_mini1.sh` reaches login prompt -7. `./local/scripts/build-redbear.sh redbear-mini` produces working ISO diff --git a/local/docs/SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md b/local/docs/SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md index 3c59ed4eb7..5a11c06313 100644 --- a/local/docs/SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md +++ b/local/docs/SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md @@ -12,11 +12,11 @@ versioning, upstream sync). It delegates subsystem-specific detail to specialize | Plan Document | Covers | When to Consult | |---|---|---| -| `IMPROVEMENT-PLAN.md` | USB/Wi-Fi/Bluetooth code quality audit findings (P0–P3) | USB, Wi-Fi, BT quality remediation | -| `IMPLEMENTATION-MASTER-PLAN.md` | Driver/subsystem feature gaps (storage, audio, input, CPU, virtio) | Driver feature implementation | +| `archived/IMPROVEMENT-PLAN.md` | USB/Wi-Fi/Bluetooth code quality audit findings (P0–P3) | USB, Wi-Fi, BT quality remediation | +| `archived/IMPLEMENTATION-MASTER-PLAN.md` | Driver/subsystem feature gaps (storage, audio, input, CPU, virtio) | Driver feature implementation | | `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Desktop/KDE path from console to hardware-accelerated Plasma | Wayland, Mesa, KWin, SDDM | | `UPSTREAM-SYNC-PROCEDURE.md` | Per-component sync procedure for local forks | Executing individual fork syncs | -| `STUBS-FIX-PROGRESS.md` | Stub→real-code rewrite tracking | Replacing stubs with real implementations | +| `archived/STUBS-FIX-PROGRESS.md` | Stub→real-code rewrite tracking | Replacing stubs with real implementations | | `BUILD-SYSTEM-IMPROVEMENTS.md` | Build system hardening, collision detection, manifests | Build system changes | | `ACPI-IMPROVEMENT-PLAN.md` | ACPI sleep, thermal, EC, power | ACPI improvements | | `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | PCI IRQ, MSI-X, IOMMU, controllers | IRQ/PCI quality | @@ -446,11 +446,11 @@ git commit -m "submodule: sync to upstream HEAD" ### 3.4 Integrate USB Quirk Enforcement -**Context:** The existing `IMPROVEMENT-PLAN.md` Section 3.3 identifies 49/50 xHCI quirks declared but not enforced at runtime. This is a critical gap for supporting real hardware. +**Context:** The existing `archived/IMPROVEMENT-PLAN.md` Section 3.3 identifies 49/50 xHCI quirks declared but not enforced at runtime. This is a critical gap for supporting real hardware. **File:** `local/sources/base/drivers/usb/xhcid/src/xhci/quirks.rs` -**Priority enforcement gaps (from IMPROVEMENT-PLAN.md):** +**Priority enforcement gaps (from archived/IMPROVEMENT-PLAN.md):** | Quirk | Affected HW | Action | |-------|-------------|--------| @@ -466,7 +466,7 @@ if (pdev->vendor == PCI_VENDOR_ID_ASMEDIA && pdev->device == 0x1042) xhci->quirks |= XHCI_ASMEDIA_MODIFY_FLOWCONTROL; ``` -**See also:** `IMPROVEMENT-PLAN.md` § 3.3 for the complete quirk enforcement gap analysis. +**See also:** `archived/IMPROVEMENT-PLAN.md` § 3.3 for the complete quirk enforcement gap analysis. **Estimated time:** 4–8 hours **Dependencies:** Phase 1.5, Phase 3.1. @@ -544,7 +544,7 @@ syscalls needed by modern software. Input device handling is comprehensive. ### 4.3 Stub Replacement: Identify and Replace Remaining Stubs -**Context:** The project has a zero-tolerance stub policy (`local/AGENTS.md` § STUB AND WORKAROUND POLICY). The `STUBS-FIX-PROGRESS.md` tracks the stub→real-code rewrite campaign. Audit the remaining stubs and prioritize fixes. +**Context:** The project has a zero-tolerance stub policy (`local/AGENTS.md` § STUB AND WORKAROUND POLICY). The `archived/STUBS-FIX-PROGRESS.md` tracks the stub→real-code rewrite campaign. Audit the remaining stubs and prioritize fixes. **Action:** 1. Run a workspace-wide grep for stub patterns: @@ -560,15 +560,15 @@ syscalls needed by modern software. Input device handling is comprehensive. | Stub Pattern | Correct Fix | |---|---| -| `unimplemented!("event ring growth")` | Implement `grow_event_ring()` per `IMPROVEMENT-PLAN.md` § 3.1 | -| `todo!("BOS descriptor")` | Un-comment `fetch_bos_desc()` per `IMPROVEMENT-PLAN.md` § 3.2 | +| `unimplemented!("event ring growth")` | Implement `grow_event_ring()` per `archived/IMPROVEMENT-PLAN.md` § 3.1 | +| `todo!("BOS descriptor")` | Un-comment `fetch_bos_desc()` per `archived/IMPROVEMENT-PLAN.md` § 3.2 | | `return Err(ENOSYS)` for a known syscall | Implement the syscall in kernel or relibc | | `#![allow(warnings)]` in xhcid | Fix the underlying warnings, then remove | -| `rate_idx = 0` hardcoded | Implement Minstrel rate scaling per `IMPROVEMENT-PLAN.md` § 6.3 | +| `rate_idx = 0` hardcoded | Implement Minstrel rate scaling per `archived/IMPROVEMENT-PLAN.md` § 6.3 | **See also:** -- `IMPROVEMENT-PLAN.md` — 35 items covering USB/WiFi/BT stubs -- `STUBS-FIX-PROGRESS.md` — ~517 TODO/FIXME items tracked +- `archived/IMPROVEMENT-PLAN.md` — 35 items covering USB/WiFi/BT stubs +- `archived/STUBS-FIX-PROGRESS.md` — ~517 TODO/FIXME items tracked **Estimated time:** 16–32 hours (ongoing across Phase 4) **Dependencies:** Phase 3.1 (synced forks provide the correct APIs to implement against). @@ -671,10 +671,10 @@ baseline established. **Linux reference:** `drivers/usb/storage/usb.c:1050-1100` — `usb_stor_control_thread()` with proper error recovery on device disconnect. Red Bear's usbscsid must handle the same scenario: device removed mid-transfer → error, not panic. -**See also:** `IMPROVEMENT-PLAN.md` § 2.1 (usbscsid `.unwrap()` removal) — this must be done before hardware testing. +**See also:** `archived/IMPROVEMENT-PLAN.md` § 2.1 (usbscsid `.unwrap()` removal) — this must be done before hardware testing. **Estimated time:** 4–8 hours -**Dependencies:** Phase 5.1, `IMPROVEMENT-PLAN.md` P0 items complete. +**Dependencies:** Phase 5.1, `archived/IMPROVEMENT-PLAN.md` P0 items complete. ### 5.4 Wi-Fi and Bluetooth Maturity Assessment @@ -695,11 +695,11 @@ baseline established. - `drivers/net/wireless/intel/iwlwifi/mvm/fw.c:300-500` — firmware init sequence **See also:** -- `IMPROVEMENT-PLAN.md` § 6 — Wi-Fi subsystem improvements +- `archived/IMPROVEMENT-PLAN.md` § 6 — Wi-Fi subsystem improvements - `WIFI-IMPLEMENTATION-PLAN.md` — Wi-Fi architecture plan **Estimated time:** 4–8 hours -**Dependencies:** Phase 5.1, `IMPROVEMENT-PLAN.md` P1-P2 Wi-Fi items. +**Dependencies:** Phase 5.1, `archived/IMPROVEMENT-PLAN.md` P1-P2 Wi-Fi items. --- @@ -795,12 +795,12 @@ preferable to inventing heuristics. 1. **Run `sync-versions.sh --check`** — identify version drift immediately 2. **`cd local/sources/base && git status --short`** — catalog WIP changes 3. **`cargo check` on xhcid** — confirm orphan `}` issue exists -4. **Read `IMPROVEMENT-PLAN.md` P0 items** — these are the USB safety fixes that block hardware testing +4. **Read `archived/IMPROVEMENT-PLAN.md` P0 items** — these are the USB safety fixes that block hardware testing 5. **Read `UPSTREAM-SYNC-PROCEDURE.md`** — prepare for Phase 3 fork syncs ### Ongoing Discipline - Every commit to a local fork MUST also update the parent repo's submodule pointer - Every upstream cherry-pick MUST be tested with `cargo check` + `repo cook` before push -- Every stub fixed MUST update `STUBS-FIX-PROGRESS.md` +- Every stub fixed MUST update `archived/STUBS-FIX-PROGRESS.md` - Every new Linux algorithm ported MUST include a comment referencing the specific Linux 7.1 file and line range diff --git a/local/docs/IMPLEMENTATION-MASTER-PLAN.md b/local/docs/archived/IMPLEMENTATION-MASTER-PLAN.md similarity index 100% rename from local/docs/IMPLEMENTATION-MASTER-PLAN.md rename to local/docs/archived/IMPLEMENTATION-MASTER-PLAN.md diff --git a/local/docs/IMPROVEMENT-PLAN.md b/local/docs/archived/IMPROVEMENT-PLAN.md similarity index 100% rename from local/docs/IMPROVEMENT-PLAN.md rename to local/docs/archived/IMPROVEMENT-PLAN.md diff --git a/local/docs/archived/README.md b/local/docs/archived/README.md index d87f997652..ec8a54b59c 100644 --- a/local/docs/archived/README.md +++ b/local/docs/archived/README.md @@ -1,6 +1,6 @@ # Archived Documentation -Last synced: 2026-07-13 — this archive inventory is dynamically maintained and +Last synced: 2026-07-18 — this archive inventory is dynamically maintained and tracks only files that currently exist in this directory. These documents were written during earlier phases of Red Bear OS development. @@ -12,10 +12,15 @@ current plans. They are kept for reference only. | Archived | Status | |----------|--------| | `ACPI-I2C-HID-IMPLEMENTATION-PLAN.md` | Deferred; USB HID remains the primary input path. | +| `IMPLEMENTATION-MASTER-PLAN.md` | Completion report of the 2026-07 audit sprint; superseded by `../CONSOLE-TO-KDE-DESKTOP-PLAN.md`. | +| `IMPROVEMENT-PLAN.md` | RESOLVED quality-audit plan (38/38); still referenced by `../SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md` § xHCI quirks (§3.3). | | `INTEL-HDA-IMPLEMENTATION-PLAN.md` | Deferred; audio remains a later priority. | | `KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` | Historical scheduler analysis kept for reference. | | `README.md` | Archive metadata only. | +| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface plan; credential work resolved in `../KERNEL-IPC-CREDENTIAL-PLAN.md`. | | `repo-governance.md` | Historical governance note kept for reference. | +| `SOURCE-ARCHIVAL-POLICY.md` | Absorbed into `../../AGENTS.md` § RELEASE MODEL / SOURCE-OF-TRUTH RULE. | +| `STUBS-FIX-PROGRESS.md` | Final-state log of the completed stub→real-code campaign. | | `USB-BOOT-INPUT-PLAN.md` | Superseded by the current USB implementation planning. | | `USB-VALIDATION-RUNBOOK-2026-07.md` | Historical validation runbook for the earlier USB phase. | | `XHCID-DEVICE-IMPROVEMENT-PLAN.md` | Superseded by the current USB implementation planning. | @@ -28,5 +33,10 @@ current plans. They are kept for reference only. remain as deferred historical references. - `KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` and `repo-governance.md` are retained as archive-only context. +- 2026-07-18 archival (release-bump cleanup): `IMPROVEMENT-PLAN.md`, + `IMPLEMENTATION-MASTER-PLAN.md`, `STUBS-FIX-PROGRESS.md`, + `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md`, `SOURCE-ARCHIVAL-POLICY.md` + moved here from `local/docs/`. All inbound references were updated to the + `archived/` paths in the same pass. -## Date archived: 2026-05-03 +## Date archived: 2026-05-03 (first batch); 2026-07-18 (second batch) diff --git a/local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md b/local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md similarity index 100% rename from local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md rename to local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md diff --git a/local/docs/SOURCE-ARCHIVAL-POLICY.md b/local/docs/archived/SOURCE-ARCHIVAL-POLICY.md similarity index 100% rename from local/docs/SOURCE-ARCHIVAL-POLICY.md rename to local/docs/archived/SOURCE-ARCHIVAL-POLICY.md diff --git a/local/docs/STUBS-FIX-PROGRESS.md b/local/docs/archived/STUBS-FIX-PROGRESS.md similarity index 100% rename from local/docs/STUBS-FIX-PROGRESS.md rename to local/docs/archived/STUBS-FIX-PROGRESS.md diff --git a/local/docs/cub-assessment-and-improvement-plan.md b/local/docs/cub-assessment-and-improvement-plan.md deleted file mode 100644 index a7d9949e15..0000000000 --- a/local/docs/cub-assessment-and-improvement-plan.md +++ /dev/null @@ -1,508 +0,0 @@ -# cub Assessment and Improvement Plan (v6.0 2026) - -**Date:** 2026-06 -**Status:** Assessment complete, critical fixes in progress -**Scope:** Comprehensive empirical review of cub's AUR → RBPKGBUILD → Redox recipe.toml conversion pipeline, with concrete bug fixes and a forward improvement plan. - ---- - -## Executive Summary - -cub (Red Bear OS package manager, located at `local/recipes/system/cub/source/`) is a substantial 17-module Rust workspace with a real, working AUR → RBPKGBUILD → recipe.toml conversion pipeline. The architecture is sound: a 937-line `pkgbuild` parser, a 408-line `rbpkgbuild` serializer, a 465-line `cookbook` recipe generator, plus AUR fetching, dependency mapping, sandbox, cook, storage, and resolver modules. All 70 unit tests pass and the source compiles cleanly. - -However, the conversion pipeline has **8 known bugs** that would prevent any real-world Arch package from converting to a working Red Bear recipe. This assessment is based on running the pipeline against 12 representative real-world PKGBUILDs and observing the output. - -**Verdict:** cub is the right tool for the job. The architecture is correct, the code is mostly right, and the bugs are surgical fixes — not architectural rewrites. The first 7 bugs are being fixed in commit `` (this PR). The 8th is deferred as a cookbook-integration concern. - ---- - -## What cub does well - -The 12-PKGBUILD assessment found that cub's conversion pipeline handles these cases correctly: - -| Case | Status | -|---|---| -| Simple meson package (libevdev) | ✅ clean conversion | -| Cargo package (fd-find) | ✅ clean conversion | -| CMake package (fmt) | ✅ clean conversion | -| Configure package (libpciaccess 0.19) | ✅ clean conversion | -| Git source with `git+url#tag=` (gzip, mesa 24.3 single-source) | ✅ clean conversion | -| Split package with `pkgbase` + `package_()` (openssl) | ⚠️ converts primary only, warns about split | -| Custom build template (raw shell script) | ✅ extracts build/package bodies | -| `pkgver()` function (dynamic version) | ⚠️ ignored with warning | -| `sha256sums=('SKIP')` for git sources | ✅ correct handling | -| TOML validity of generated recipe | ✅ always valid | -| Detection of Linux-only deps (libx11, libxcb, libinput, alsa) | ✅ correctly flags as unmapped | -| Linuxism detection (glibc, x11, gtk, etc.) | ✅ warns + actions_required | - -The unit tests (70 in total) cover the rbpkgbuild, rbsrcinfo, sandbox, storage, recipe, resolver modules. The test coverage is solid for the serializer/parser side. - ---- - -## The 8 bugs - -### Bug 1 — `glibc` mapped to `relibc` (CRITICAL) - -**Location:** `cub-lib/src/deps.rs:13` -**Severity:** Critical — affects every Linux C library consumer -**Symptom:** `depends=('glibc')` produces `[package] dependencies = ["relibc"]` in the generated recipe -**Root cause:** The dependency mapping hard-codes: -```rust -"glibc" => ("relibc".to_string(), false), -``` -**Why wrong:** glibc is the Linux C library; relibc is the Redox C library, which is part of the OS itself (not a package). Putting relibc as a runtime dependency in a Redox recipe is a tautology — every Redox program already links against relibc implicitly. Worse, it suggests relibc is a separately-installable package, which it isn't. - -**Fix:** Drop the mapping. Use the standard "empty string" pattern that cub already uses for other Linux-only dependencies (systemd, xorgproto, libcap, libpulse, etc.): -```rust -"glibc" => (String::new(), false), -``` -The `map_dep_list` function already filters out empty mappings. - -**Status:** Fixed in this commit. - -### Bug 2 — `gcc-libs` mapped to `gcc` (CRITICAL) - -**Location:** `cub-lib/src/deps.rs:15` -**Severity:** Critical — affects every Arch package that links libstdc++/libgcc -**Symptom:** `depends=('gcc-libs')` produces `[package] dependencies = ["gcc"]` in the generated recipe -**Root cause:** The mapping hard-codes: -```rust -"gcc-libs" => ("gcc".to_string(), false), -``` -**Why wrong:** gcc-libs is the **runtime** libgcc + libstdc++ (analogous to Arch's `gcc-libs`). gcc is the **compiler**. They're entirely different artifacts. Mapping runtime C++ libraries to a compiler creates broken cross-compilation graphs (the cookbook would try to build gcc as a runtime dep). - -**Fix:** Drop the mapping, same as Bug 1. relibc provides the runtime; the compiler gcc is a host-only build tool (covered by Bug 3): -```rust -"gcc-libs" => (String::new(), false), -``` -**Status:** Fixed in this commit. - -### Bug 3 — Build tools not prefixed with `host:` (CRITICAL) - -**Location:** `cub-lib/src/deps.rs` (entire `map_dependency` function) -**Severity:** Critical — affects every package that uses standard build tools -**Symptom:** Generated recipes list `meson`, `ninja`, `cmake`, `make`, `pkg-config`, `git`, `perl`, `python`, `rust`, `cargo`, `autoconf`, `automake`, `libtool`, etc. as bare `[build] dependencies`, without the `host:` prefix -**Root cause:** The hard-coded mapping table returns bare names. The Redox cookbook convention is to mark build tools as `host:` so the cookbook knows they're host-only and not part of the cross-compiled Redox target. - -The current `prefix_host_deps` function in `cookbook.rs` already handles `host:` prefixing for `dev-dependencies` (the `check` deps) but not for `dependencies` (the `makedepends`). - -**Why wrong:** Without `host:`, the Redox cookbook would try to *cook* meson, ninja, and cmake as Redox packages, which is wrong — they're host tools that run on the build host, not target tools that get cross-compiled. - -**Fix:** Add a `BUILD_TOOLS: &[&str]` constant containing all known build tools. In `map_dependency`, when the name matches a build tool, return `MappedDep { mapped: "host:", is_exact: true }` instead of the bare name. - -The set of build tools (~30 entries): `make`, `cmake`, `ninja`, `meson`, `pkg-config`, `pkgconf`, `pkgconfig`, `autoconf`, `automake`, `libtool`, `m4`, `git`, `svn`, `mercurial`, `perl`, `python`, `python2`, `python3`, `rust`, `cargo`, `rustc`, `go`, `golang`, `bison`, `flex`, `yacc`, `gperf`, `gettext`, `intltool`, `help2man`, `gengetopt`, `xmlto`, `asciidoc`, `doxygen`, `graphviz`, `swig`. - -**Status:** Fixed in this commit. - -### Bug 4 — AUR `git+url#tag=branch` source syntax not parsed (CRITICAL) - -**Location:** `cub-lib/src/pkgbuild.rs` (the `source_from_arch` function or equivalent) -**Severity:** Critical — affects every AUR git package -**Symptom:** Generated recipes contain literal `git+https://...git#tag=mesa-24.3.0` as the source URL, which is invalid Redox cookbook format -**Root cause:** AUR PKGBUILDs use Arch-style source notation. `source_from_arch` does not strip the `git+` prefix or extract the `#tag=branch` fragment. - -**Real example:** `mesa 24.3` PKGBUILD has: -```bash -source=("git+https://gitlab.freedesktop.org/mesa/mesa.git#tag=mesa-${pkgver}") -``` -cub produces: `git = "git+https://gitlab.freedesktop.org/mesa/mesa.git#tag=mesa-${pkgver}"` in the recipe. - -**Fix:** Modify `source_from_arch` (or wherever source URLs are normalized) to: -1. Check if the source starts with `git+` — if so, set `source_type = SourceType::Git` and strip the prefix -2. Check for a `#` fragment — if present, split off and parse the fragment. Common fragment keys: - - `tag=...` → `SourceEntry::branch` (since the Redox cookbook uses `branch` for tags) - - `branch=...` → `SourceEntry::branch` - - `commit=...` or `revision=...` → `SourceEntry::rev` -3. For git sources without a fragment, set `branch = "HEAD"` as a reasonable default - -The Redox cookbook's git source format is: -```toml -[source] -git = "https://..." -branch = "v1.0" # or rev = "..." -``` - -**Status:** Fixed in this commit. - -### Bug 5 — Multi-source PKGBUILDs not supported (HIGH) - -**Location:** `cub-lib/src/cookbook.rs:80-100` -**Severity:** High — blocks mesa, ffmpeg, and many real packages -**Symptom:** Multi-source PKGBUILDs (e.g., mesa with 2 sources: git repo + llvmpipe_generated.h.tar.xz) error with: -``` -Error: multiple sources not yet supported (found 2: https://..., https://...). -Use single-source packages or manually create recipe.toml -``` -**Root cause:** The `generate_recipe` function explicitly errors on multi-source. The `convert_pkgbuild` function preserves all sources in the `RbPkgBuild` but `generate_recipe` only handles 1. - -**Why wrong:** Many real packages have multiple sources (a primary git repo + an auxiliary file). Forcing the user to manually create the recipe for these defeats the purpose of automated conversion. - -**Fix:** In `convert_pkgbuild` (in `pkgbuild.rs`), after the full source array is parsed, **truncate to the first source only** and add a warning to the `ConversionReport`: -```rust -if sources.len() > 1 { - warnings.push(format!( - "PKGBUILD has {} sources; using the first as primary ({}). Auxiliary sources dropped: {}", - sources.len(), - sources[0], - sources[1..].join(", ") - )); - actions_required.push( - "review multi-source PKGBUILD and re-add auxiliary sources manually".to_string() - ); - sources.truncate(1); -} -``` -This way `generate_recipe` doesn't need to know about multi-source, and the user gets a clear warning + action item. - -**Status:** Fixed in this commit. - -### Bug 6 — `options=()` flags silently dropped (MEDIUM) - -**Location:** `cub-lib/src/pkgbuild.rs` (no extraction at all) -**Severity:** Medium — silent data loss -**Symptom:** PKGBUILDs with `options=('!lto' '!emptydirs' '!strip')` lose all option flags. The generated recipe has no trace of them. -**Root cause:** The current `convert_pkgbuild` doesn't extract `options=()`. - -**Why wrong:** Options like `!lto` (disable link-time optimization) and `!strip` (don't strip binaries) can be required for a package to build correctly. Silently dropping them creates recipes that fail in ways the user can't predict. - -**Fix:** Extend the `CompatSection` struct in `rbpkgbuild.rs` to include an `options: Vec` field. Extract `options=()` in `convert_pkgbuild` and store it. The recipe generator doesn't need to consume it — the user reads the `compat.options` field in the RBPKGBUILD and re-adds the relevant flags to their Redox recipe. - -**Status:** Fixed in this commit. - -### Bug 7 — `${pkgver}` literal in source URL (HIGH) - -**Location:** `cub-lib/src/pkgbuild.rs` (source array not passed through `resolve_shell_vars`) -**Severity:** High — affects every AUR package that uses `${pkgver}` in its source URL -**Symptom:** Generated recipes have `tar = ".../libevdev-${pkgver}.tar.xz"` with the literal `${pkgver}` string. The Redox cookbook doesn't do shell variable substitution at source-URL time, so the resulting tar URL is invalid. -**Root cause:** The `convert_pkgbuild` function calls `resolve_shell_vars` on `pkgname`, `pkgver`, `pkgdesc`, `url` but NOT on the `source` array. - -**Why wrong:** The recipe is broken as soon as it's generated. The user would have to manually fix every source URL. - -**Fix:** After extracting the `source` array, apply `resolve_shell_vars` to each entry. Since `pkgver` is already resolved (it's a top-level scalar in the PKGBUILD), the substitution should be straightforward. The existing `resolve_shell_vars` function is the right tool. - -**Status:** Fixed in this commit. - -### Bug 8 — Recipes lack cookbook boilerplate (DEFERRED) - -**Location:** `cub-lib/src/cookbook.rs` (the `custom_script` function) -**Severity:** Medium — affects every Custom-template package -**Symptom:** Custom-template recipes generate raw shell scripts without the Redox cookbook's `DYNAMIC_INIT` prelude or the `cookbook_apply_patches` helper that the Rule 2 migration recipes use. -**Root cause:** `custom_script` (in `cookbook.rs:244-282`) joins `prepare`, `build_script`, `check`, `install_script`, and install lines into a single script. It does not prepend `DYNAMIC_INIT` (which is required for any Redox cookbook shell script to find the cookbook helpers like `cookbook_meson`, `cookbook_apply_patches`, etc.). - -**Why wrong:** A real Redox recipe that doesn't start with `DYNAMIC_INIT` will fail at build time because the shell helpers aren't loaded. - -**Fix:** Prepend `DYNAMIC_INIT\n` to the generated `custom_script`. Add an optional `redox_compat/` shim dir or `cookbook_apply_patches "${REDBEAR_PATCHES_DIR}"` call if the package has patches (this requires the user to declare patches in the RBPKGBUILD). - -**Status:** Deferred. This is a cookbook-integration concern that intersects with the project's Rule 2 patch policy. A separate task should: -1. Decide whether the cub-generated recipe's `patches` field is treated as the source-of-truth (per Rule 2) or as auxiliary patches -2. Add a `local/patches//` directory generation step to cub -3. Update the `custom_script` to include `DYNAMIC_INIT` and `cookbook_apply_patches` - ---- - -## Test methodology - -The `cub-assessment` test binary at `local/recipes/system/cub/source/cub-assessment/src/main.rs` is the integration test that exercises 12 representative real-world PKGBUILDs: - -1. `libevdev` — simple meson (reference case) -2. `fd-find` — cargo -3. `libpciaccess 0.18.1` — meson -4. `fmt` — cmake -5. `wlroots-git` — git source, complex deps -6. `libpciaccess 0.19` (extra/-style) — meson + ninja -7. `ffmpeg` — configure + options -8. `mesa 24.3` — git+url + multi-source + pkgver() (used to fail pre-fix) -9. `gzip` — configure + git source + check -10. `zlib` — simple C, configure -11. `openssl` — pkgbase split package -12. `glib2` — complex deps, real-world - -**Pre-fix results (before this commit):** -- 11 of 12 converted successfully -- 1 of 12 errored (mesa 24.3, multi-source) -- All recipes valid TOML -- Several recipes had wrong deps (glibc→relibc, gcc-libs→gcc) -- All git+url sources kept literal prefix -- All multi-source attempts failed - -**Post-fix results (after this commit):** -- All 12 convert successfully -- mesa 24.3 produces a valid recipe with a multi-source warning -- glibc, gcc-libs, x11, alsa, libinput are correctly dropped/flagged -- Build tools are correctly `host:`-prefixed -- git+url sources parse correctly into branch/rev fields -- 70 existing unit tests still pass - ---- - -## Test cases by category - -### A. Conversion path success (verifies the core pipeline) -- ✅ `convert_pkgbuild` returns Ok for all 12 cases -- ✅ `generate_recipe` returns Ok for all 12 cases post-fix -- ✅ All generated recipes parse as valid TOML - -### B. Dependency mapping correctness -- ✅ `glibc` → dropped (empty string) -- ✅ `gcc-libs` → dropped (empty string) -- ✅ `glib2` → `glib` (exact) -- ✅ `openssl` → `openssl3` (inexact, warning) -- ✅ `zlib` → `zlib` (exact) -- ✅ `meson`, `ninja`, `cmake` → `host:meson`, etc. (post-fix) -- ✅ `libinput` → dropped with action_required -- ✅ `libx11`, `libxcb`, `libxrandr` → dropped (no Redox mapping) - -### C. Source URL handling -- ✅ Tar source with `${pkgver}` → resolved to actual version (post-fix) -- ✅ Git source with `git+url#tag=v1.0` → git+ stripped, branch=v1.0 (post-fix) -- ✅ Git source without fragment → branch=HEAD (post-fix) -- ✅ Multi-source → first kept, others warned (post-fix) -- ✅ `sha256sums=('SKIP')` → ignored, source_type=Git - -### D. Build template detection -- ✅ meson template (cookbook_meson path) -- ✅ cmake template (cookbook_cmake path) -- ✅ cargo template (cookbook_cargo path) -- ✅ configure template (cookbook_configure path) -- ✅ Custom template (raw script path) - -### E. Edge cases -- ✅ Split package (pkgbase + pkgname array + package_* functions) -- ✅ pkgver() function (ignored with warning) -- ✅ options=() (preserved in compat section, post-fix) -- ✅ check() function (extracted when allow_tests=true) - -### F. Validation -- ✅ RbPkgBuild.validate() passes for all -- ✅ Generated RBSRCINFO is valid -- ✅ All arch values are "x86_64-unknown-redox" -- ✅ All licenses are preserved - ---- - -## Forward improvement plan (post-fix) - -After the 7 critical bug fixes, the conversion pipeline produces recipes that are *valid TOML with correct deps* but still need work to be *buildable in the Redox cookbook*. Here's the prioritized follow-up list: - -### Tier 1: Cookbook integration (Bugs 8 + others) -1. **Custom template boilerplate** — prepend `DYNAMIC_INIT` to all `custom_script` outputs. Without this, the recipe will fail at runtime because cookbook helpers like `cookbook_meson` aren't loaded. -2. **Patch application hook** — for recipes with `patches.files` non-empty, automatically emit `cookbook_apply_patches "${REDBEAR_PATCHES_DIR}"` in the custom script (matching the Rule 2 migration pattern established in commits `1a291fbb9` and `b0f440c47`). -3. **Systematic dep audit** — the 80-entry hard-coded dep map in `deps.rs` is a maintenance burden. Migrate to a data-driven table that lives in `local/docs/dependency-mapping.md` (or a similar canonical source) and is loaded at runtime. This makes the mapping auditable and easy to extend. -4. **Per-package overrides** — sometimes a single package needs a non-standard dep. For example, `mesa` needs `llvm21` not `llvm`. Add a `compat.dependency_overrides: HashMap` field in `RBPKGBUILD` that overrides the hard-coded mapping. - -### Tier 2: Conversion quality -5. **Better Linuxism detection** — currently flags `glibc`, `x11`, `alsa`, `libpulse` as Linux-only. Many AUR packages have implicit Linux-only paths (e.g., `/proc/cpuinfo`, `/sys/class/...`, `/dev/...`) that the heuristic doesn't catch. -6. **Multi-source preservation** — when truncating to 1 source, the user loses access to auxiliary files (e.g., a gitignore or systemd unit). Generate a `redox/aux-sources.txt` next to the recipe listing the dropped sources, so the user can manually re-add them. -7. **`pkgver()` execution** — for git sources, the Arch convention is to have a `pkgver()` function that calls `git describe` to compute the version. This is impossible to execute in cub (we don't have a git clone at conversion time). Generate a static `version = "GIT"` and add a `compat.notes = "git source — version detected dynamically"` annotation. -8. **License mapping** — Arch licenses like `('GPL3' 'Apache-2.0')` are SPDX-like but not exact. Map to SPDX identifiers and warn when an Arch license doesn't have a Red Bear SPDX equivalent. - -### Tier 3: UX -9. **`cub -S ` should run end-to-end** — currently the AUR → recipe → cook flow is fragmented (separate `get-aur`, `build` subcommands). The user should be able to say `cub -S mesa` and get a built Redox package. -10. **Conversion report** — the `ConversionReport` already exists but is only printed by `get-aur` and `import-aur`. The `cook` and `install` subcommands should print the report after the recipe is generated so the user can see warnings and action items. -11. **TUI integration** — the cub TUI at `cub-tui/` should show a "Convert from AUR" action that drives the full conversion and shows the report. -12. **BUR (Redox User Repo) sync** — `cub -Sy` syncs from BUR at `https://gitlab.redox-os.org/redox-os/bur.git`. This works but the BUR is sparse. A "publish" subcommand would let users contribute their converted recipes back to BUR. - -### Tier 4: Infrastructure -13. **Network access policy** — AUR fetching requires network access. The project's `REPO_OFFLINE=1` default blocks this. Add a `cub config set aur-fetch off` knob to disable AUR fetches in offline mode while keeping BUR sync (which uses git's local cache) working. -14. **Cache management** — cub caches AUR metadata in `/tmp/cub_cache/`. On Redox this maps to ramfs and is volatile. A persistent cache in `~/.cub/cache/` (or in the `var/lib/packages` directory) would survive reboots. -15. **Test coverage** — 70 unit tests is good, but the integration tests (the 12-PKGBUILD assessment) should be moved into `cargo test` so they run in CI. Currently they're in a separate `cub-assessment` sub-crate that nobody runs automatically. -16. **Sparse AUR metadata** — instead of cloning the full AUR git repo for every package, use the AUR RPC API (`https://aur.archlinux.org/rpc/?v=5&type=info&arg=`) to get just the metadata, then clone only the source. This is already partially implemented in `aur.rs` but the full flow isn't exercised in tests. - ---- - -## Files involved - -- `local/recipes/system/cub/source/cub-lib/src/deps.rs` — bug 1, 2, 3 -- `local/recipes/system/cub/source/cub-lib/src/pkgbuild.rs` — bugs 4, 5, 6, 7 -- `local/recipes/system/cub/source/cub-lib/src/cookbook.rs` — bug 5, 8 -- `local/recipes/system/cub/source/cub-lib/src/rbpkgbuild.rs` — bug 6 (CompatSection extension) -- `local/recipes/system/cub/source/cub-lib/src/converter.rs` — bug 4 (also has its own `convert_pkgbuild`) -- `local/recipes/system/cub/source/cub-assessment/src/main.rs` — integration test (no changes) - ---- - -## Conclusion - -cub is the right tool for the job. The architecture is correct (PKGBUILD → RbPkgBuild → recipe.toml is a clean three-stage pipeline), the existing code is mostly right (70 unit tests pass, all 12 real-world PKGBUILDs convert to valid TOML), and the bugs are surgical fixes that don't require architectural changes. - -After the 7 critical fixes in this commit, cub will be a useful Red Bear package tool. The remaining 16 follow-up items in the forward plan address cookbook integration, conversion quality, UX, and infrastructure — they're substantial but well-scoped. - -The recommendation is to ship the 7 fixes, then use cub to bootstrap the next 5-10 missing recipes (zlib, libffi, pcre2, freetype, libpng, etc.) as a real-world validation. If the generated recipes cook successfully, we proceed to scale up cub-driven recipe generation. If they fail, the next round of fixes will be more targeted because we'll have concrete error messages. - ---- - -## Appendix A: cub architecture map - -``` -cub-cli (binary) -├── main.rs -│ ├── Cli (clap derive) -│ ├── Commands: Install, Search, Info, Sync, SystemUpgrade, Build, -│ │ Get, GetAur, GetPkgbuild, Inspect, ImportAur, UpdateAll, -│ │ Remove, QueryLocal, QueryInfo, QueryList, CleanCache, CleanAll -│ ├── Aliases: -S, -Ss, -Si, -Sy, -Syu, -B, -G, -R, -Q, -Qi, -Ql, -│ │ -Pi, --import-aur, -Sua, -Sc, -Scc, -Gp -│ └── launch_tui_or_help() / run_command() -│ -├── CookbookAdapter → calls cub::cookbook::generate_recipe() -├── PkgbuildConverter → calls pkgbuild::convert_pkgbuild() -└── PackageCreator → calls cub::package (gated on `full` feature) - -cub-lib (library) -├── lib.rs → re-exports all modules -├── aur.rs → AUR RPC client + AurPackage -├── converter.rs → simpler convert_pkgbuild (used by tests) -├── cook.rs → invokes `repo cook` on the generated recipe -├── cookbook.rs → RbPkgBuild → cookbook recipe.toml -│ (THE PIPELINE ENDPOINT) -├── deps.rs → Arch → Red Bear dep mapping -│ (Bugs 1, 2, 3 live here) -├── depresolve.rs → dependency resolution across packages -├── error.rs → CubError enum -├── package.rs → pkgar creation (gated on `full` feature) -├── pkgbuild.rs → PKGBUILD → RbPkgBuild (937 lines) -│ (Bugs 4, 5, 6, 7 live here) -├── rbpkgbuild.rs → RbPkgBuild TOML serialization (408 lines) -├── rbpkgbuild tests -├── rbsrcinfo.rs → .SRCINFO format reader/writer -├── recipe.rs → save_recipe_to_store, etc. -├── resolver.rs → check_missing_header, library, pkgconfig -├── sandbox.rs → build sandboxing for repo cook -└── storage.rs → CubStore: ~/.cub/ directory management - -cub-tui (Ratatui TUI) -├── app.rs → TUI state machine -├── lib.rs → public API: cub_tui::run() -└── theme.rs → colors and styles -``` - -## Appendix B: RBPKGBUILD format - -cub's intermediate format (between PKGBUILD and recipe.toml): - -```toml -format = 1 - -[package] -name = "mesa" -version = "24.3.0" -release = 1 -description = "Open-source implementation of the OpenGL specification" -homepage = "https://mesa.freedesktop.org" -license = ["MIT"] -architectures = ["x86_64-unknown-redox"] -maintainers = [] - -[[source]] -type = "git" -url = "https://gitlab.freedesktop.org/mesa/mesa.git" -sha256 = "" -rev = "" -branch = "mesa-24.3.0" - -[dependencies] -build = ["host:meson", "host:ninja", "llvm"] -runtime = ["libdrm", "wayland"] -check = [] -optional = [] -provides = [] -conflicts = [] - -[build] -template = "meson" -release = true -features = [] -args = [] -build_dir = "" -prepare = [] -build_script = [] -check = [] -install_script = [] - -[install] -bins = [] -libs = [] -headers = [] -docs = [] -man = [] - -[compat] -imported_from = "aur" -original_pkgbuild = "..." # full PKGBUILD text -conversion_status = "Full" # or "Partial" -target = "x86_64-unknown-redox" -split_packages = [] -options = [] # added in this commit - -[policy] -allow_network = false -allow_tests = false -review_required = false -``` - -## Appendix C: Generated recipe.toml format - -What cub produces (post-fix): - -```toml -[source] -git = "https://gitlab.freedesktop.org/mesa/mesa.git" -shallow_clone = true -branch = "mesa-24.3.0" - -[build] -template = "meson" -dependencies = [ - "host:meson", - "host:ninja", - "llvm", -] - -[package] -dependencies = [ - "libdrm", - "wayland", -] -version = "24.3.0-1" -description = "Open-source implementation of the OpenGL specification" -``` - -What a real Red Bear recipe looks like (e.g., `local/recipes/libs/mesa/recipe.toml`): - -```toml -# Comprehensive header comment block explaining Rule 2 + migration rationale -[source] -git = "https://gitlab.redox-os.org/redox-os/mesa" -branch = "redox-24.0" - -[build] -template = "custom" -dependencies = [ - "expat", - "libdrm", - "libwayland", - "llvm21", - "wayland-protocols", - "zlib", -] -dev-dependencies = [ - "llvm21.dev", -] -script = """ -DYNAMIC_INIT - -# Apply Red Bear OS external patches (per local/AGENTS.md Rule 2) -cookbook_apply_patches "${REDBEAR_PATCHES_DIR}" - -# TODO: Should be CPPFLAGS but cookbook_meson isn't reading it -export CFLAGS+=" -DHAVE_PTHREAD=1 -I${COOKBOOK_SYSROOT}/include/libdrm" -... -cookbook_meson -Dplatforms=wayland -Dvirgl=enabled ... -""" - -[package] -version = "0.2.3" -description = "mesa 0.2.3 (v6.0 2026) — Mesa 3-D graphics library, Red Bear OS fork..." -``` - -The cub-generated recipe is **structurally correct** but **stylistically and contextually different** from a real Red Bear recipe. The forward plan addresses this gap. diff --git a/local/external-upstream-map.toml b/local/external-upstream-map.toml new file mode 100644 index 0000000000..72c0e0dc65 --- /dev/null +++ b/local/external-upstream-map.toml @@ -0,0 +1,355 @@ +# ============================================================================ +# external-upstream-map.toml — Red Bear OS graphical/desktop external sources +# ============================================================================ +# +# Purpose +# Declarative description of every external (non-fork, non-path) upstream +# source consumed by the graphical / desktop build surface. Drives two +# tools: +# - local/scripts/check-external-versions.sh (read-only status report) +# - local/scripts/bump-graphics-recipes.sh (rewrites tar=/rev=/blake3=) +# +# Schema +# [groups.] a version-coherent family (one version resolved, applied +# to every member). Qt6 / KF6 / Plasma each resolve their +# upstream version ONCE per group. +# version_index URL listing available versions (HTML or JSON). +# version_regex Python regex; capture group 1 holds the version token. +# resolve "version" (default; index lists full X.Y.Z) or +# "majmin" (index lists X.Y series dirs; full version +# is constructed as ".0"). +# recipe_root directory holding every member recipe (e.g. +# "local/recipes/kde"). Members are "/". +# tar_template default tar URL; supports {ver}, {majmin}, {name}. +# url_name_strip_prefix strip this prefix from a member's directory name +# to get its URL name (e.g. "kf6-" → "attica"). Members +# not starting with the prefix are used verbatim. +# members explicit recipe directory-name list. +# member_url_names overrides: { "" = "" } for irregulars +# (kf6-kded6 → kded, qt6-sensors → qtsensors, ...). +# member_templates overrides: { "" = { tar_template = "..." } } +# for members whose URL family differs from the group +# default (kirigami on invent.kde.org, plasma members +# on invent.kde.org, ...). +# +# [packages.] a singleton (one recipe, independently versioned). +# kind "tar" — version_index + tar_template bump model. +# "git-rev" — git ls-remote --tags + tag_regex bump model. +# recipe canonical recipe directory ("local/recipes/..."). +# For kind = "tar": version_index, version_regex, resolve, tar_template. +# For kind = "git-rev": git_url, tag_regex. +# +# Update policy +# - Operator-edited. Agents MUST NOT edit this file without operator approval. +# - Every entry MUST be verifiable against the real recipe.toml before commit +# (run check-external-versions.sh and confirm CURRENT/OUTDATED is accurate). +# - When a series moves (e.g. Qt 6.11 → 6.12, KF6 6.28 → 6.29, GNOME series +# bump), update the affected version_index URL. The bumper then resolves +# the new series automatically. +# - series-locked indexes (glib/pango point at a specific X.Y subdir of +# download.gnome.org) are intentional: GNOME ships many patch releases +# within a series; locking the index avoids cross-series noise. Bump the +# series in this map when GNOME moves. +# +# Exclusions (deliberately NOT in this map) +# - Git branch-followers (curl, openssl, sdl2, seatd, polkit-qt6, icon themes): +# they track a branch tip, not a versioned release. +# - path = recipes (libepoxy, libdisplay-info, libpciaccess, libudev, +# libxcvt, zbus, pam-redbear, qt6-wayland-smoke): in-tree sources, no +# upstream tarball. +# - recipes/core/*: fork territory, managed by the fork pipeline. +# - icu: non-standard tag/tarball separator transform (release-75-1 tag vs +# icu4c-75_1-src.tgz); cannot be expressed by {ver}/{majmin} cleanly. +# Manually managed until the templating supports separator transforms. +# - fontconfig, gdk-pixbuf: no local/recipes/ entry exists (only mainline +# recipes/dev/*). Added to the map when a local fork/recipe is created. +# +# Verified 2026-07-18 against actual recipe.toml tar=/rev= lines. +# ============================================================================ + +# --------------------------------------------------------------------------- +# Groups +# --------------------------------------------------------------------------- + +# ---- KF6 Frameworks ------------------------------------------------------- +# 45 members: 44 tar from download.kde.org/stable/frameworks// + +# kirigami (same 6.28.0 version, different host/template). +# Irregulars verified against recipe.toml: +# kf6-extra-cmake-modules → extra-cmake-modules +# kf6-kded6 → kded +# kf6-pty → kpty +# kf6-notifyconfig → knotifyconfig +# kf6-parts → kparts +# kf6-syntaxhighlighting → syntax-highlighting +# NOTE: kf6-kwayland and kf6-plasma-activities are Plasma-group members +# (they pull from stable/plasma/) and are listed under [groups.plasma]. +[groups.kf6] +version_index = "https://download.kde.org/stable/frameworks/" +version_regex = 'href="([0-9]+\.[0-9]+)/"' +resolve = "majmin" +recipe_root = "local/recipes/kde" +url_name_strip_prefix = "kf6-" +tar_template = "https://download.kde.org/stable/frameworks/{majmin}/{name}-{ver}.tar.xz" +members = [ + "kf6-attica", "kf6-extra-cmake-modules", "kf6-karchive", "kf6-kauth", + "kf6-kbookmarks", "kf6-kcmutils", "kf6-kcodecs", "kf6-kcolorscheme", + "kf6-kcompletion", "kf6-kconfig", "kf6-kconfigwidgets", "kf6-kcoreaddons", + "kf6-kcrash", "kf6-kdbusaddons", "kf6-kdeclarative", "kf6-kded6", + "kf6-kglobalaccel", "kf6-kguiaddons", "kf6-ki18n", "kf6-kiconthemes", + "kf6-kidletime", "kf6-kimageformats", "kf6-kio", "kf6-kitemmodels", + "kf6-kitemviews", "kf6-kjobwidgets", "kf6-knewstuff", "kf6-knotifications", + "kf6-kpackage", "kf6-kservice", "kf6-ksvg", "kf6-ktexteditor", + "kf6-ktextwidgets", "kf6-kwallet", "kf6-kwidgetsaddons", + "kf6-kwindowsystem", "kf6-kxmlgui", "kf6-notifyconfig", "kf6-parts", + "kf6-prison", "kf6-pty", "kf6-solid", "kf6-sonnet", + "kf6-syntaxhighlighting", "kirigami", +] + +[groups.kf6.member_url_names] +"kf6-extra-cmake-modules" = "extra-cmake-modules" +"kf6-kded6" = "kded" +"kf6-notifyconfig" = "knotifyconfig" +"kf6-parts" = "kparts" +"kf6-pty" = "kpty" +"kf6-syntaxhighlighting" = "syntax-highlighting" + +# kirigami ships the same framework version but via invent.kde.org archive. +[groups.kf6.member_templates.kirigami] +tar_template = "https://invent.kde.org/frameworks/kirigami/-/archive/v{ver}/kirigami-v{ver}.tar.gz" + + +# ---- Qt6 ------------------------------------------------------------------ +# 6 members. qt6-sensors recipe → "qtsensors" URL submodule name. +[groups.qt6] +version_index = "https://download.qt.io/official_releases/qt/6.11/" +version_regex = 'href="([0-9]+\.[0-9]+\.[0-9]+)/"' +resolve = "version" +recipe_root = "local/recipes/qt" +url_name_strip_prefix = "" +tar_template = "https://download.qt.io/official_releases/qt/{majmin}/{ver}/submodules/{name}-everywhere-src-{ver}.tar.xz" +members = ["qtbase", "qtdeclarative", "qtsvg", "qtwayland", "qtshadertools", "qt6-sensors"] + +[groups.qt6.member_url_names] +"qt6-sensors" = "qtsensors" + + +# ---- Plasma --------------------------------------------------------------- +# Version resolved ONCE from download.kde.org/stable/plasma/, applied to: +# - download.kde.org/stable/plasma/ family (default template): +# breeze, kf6-kwayland(→kwayland), kf6-plasma-activities(→plasma-activities), +# plasma-workspace +# - invent.kde.org/plasma// family (per-member overrides): +# kwin, kdecoration, kglobalacceld, kde-cli-tools, plasma-desktop +# - invent.kde.org/frameworks/plasma-framework/ (different group path): +# plasma-framework +# plasma-wayland-protocols is intentionally OUT of this group (independent +# 1.21.0 versioning); it is a singleton below. +[groups.plasma] +version_index = "https://download.kde.org/stable/plasma/" +version_regex = 'href="([0-9]+\.[0-9]+\.[0-9]+)/"' +resolve = "version" +recipe_root = "local/recipes/kde" +url_name_strip_prefix = "kf6-" +tar_template = "https://download.kde.org/stable/plasma/{ver}/{name}-{ver}.tar.xz" +members = [ + "breeze", "kf6-kwayland", "kf6-plasma-activities", "plasma-workspace", + "kwin", "kdecoration", "kglobalacceld", "kde-cli-tools", + "plasma-desktop", "plasma-framework", +] + +[groups.plasma.member_url_names] +"kf6-kwayland" = "kwayland" +"kf6-plasma-activities" = "plasma-activities" + +[groups.plasma.member_templates.kwin] +tar_template = "https://invent.kde.org/plasma/kwin/-/archive/v{ver}/kwin-v{ver}.tar.gz" +[groups.plasma.member_templates.kdecoration] +tar_template = "https://invent.kde.org/plasma/kdecoration/-/archive/v{ver}/kdecoration-v{ver}.tar.gz" +[groups.plasma.member_templates.kglobalacceld] +tar_template = "https://invent.kde.org/plasma/kglobalacceld/-/archive/v{ver}/kglobalacceld-v{ver}.tar.gz" +[groups.plasma.member_templates.kde-cli-tools] +tar_template = "https://invent.kde.org/plasma/kde-cli-tools/-/archive/v{ver}/kde-cli-tools-v{ver}.tar.gz" +[groups.plasma.member_templates.plasma-desktop] +tar_template = "https://invent.kde.org/plasma/plasma-desktop/-/archive/v{ver}/plasma-desktop-v{ver}.tar.gz" +[groups.plasma.member_templates.plasma-framework] +tar_template = "https://invent.kde.org/frameworks/plasma-framework/-/archive/v{ver}/plasma-framework-v{ver}.tar.gz" + + +# --------------------------------------------------------------------------- +# Singletons — tar +# --------------------------------------------------------------------------- + +# mesa — software 3D / LLVMpipe / virgl. GitLab archive URL. +[packages.mesa] +kind = "tar" +recipe = "local/recipes/libs/mesa" +version_index = "https://archive.mesa3d.org/" +version_regex = 'href="mesa-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://gitlab.freedesktop.org/mesa/mesa/-/archive/mesa-{ver}/mesa-mesa-{ver}.tar.gz" + +# libdrm — DRM user-space library. Static dri.freedesktop.org listing for +# version discovery; tarball from GitLab archive. +[packages.libdrm] +kind = "tar" +recipe = "local/recipes/libs/libdrm" +version_index = "https://dri.freedesktop.org/libdrm/" +version_regex = 'href="libdrm-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://gitlab.freedesktop.org/mesa/libdrm/-/archive/libdrm-{ver}/libdrm-libdrm-{ver}.tar.gz" + +# libwayland — Wayland protocol C library. +[packages.libwayland] +kind = "tar" +recipe = "local/recipes/wayland/libwayland" +version_index = "https://gitlab.freedesktop.org/wayland/wayland/-/tags" +version_regex = '/wayland/wayland/-/tags/([0-9]+\.[0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://gitlab.freedesktop.org/wayland/wayland/-/releases/{ver}/downloads/wayland-{ver}.tar.xz" + +# wayland-protocols — Wayland protocol XML. +[packages.wayland-protocols] +kind = "tar" +recipe = "local/recipes/wayland/wayland-protocols" +version_index = "https://gitlab.freedesktop.org/wayland/wayland-protocols/-/tags" +version_regex = '/wayland/wayland-protocols/-/tags/([0-9]+\.[0-9]+(?:\.[0-9]+)?)' +resolve = "version" +tar_template = "https://gitlab.freedesktop.org/wayland/wayland-protocols/-/releases/{ver}/downloads/wayland-protocols-{ver}.tar.xz" + +# libevdev — evdev wrapper. Static freedesktop.org listing. +[packages.libevdev] +kind = "tar" +recipe = "local/recipes/libs/libevdev" +version_index = "https://www.freedesktop.org/software/libevdev/" +version_regex = 'libevdev-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://www.freedesktop.org/software/libevdev/libevdev-{ver}.tar.xz" + +# libinput — input device management. GitLab tags page. +[packages.libinput] +kind = "tar" +recipe = "local/recipes/libs/libinput" +version_index = "https://gitlab.freedesktop.org/libinput/libinput/-/tags" +version_regex = '/libinput/libinput/-/tags/([0-9]+\.[0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://gitlab.freedesktop.org/libinput/libinput/-/archive/{ver}/libinput-{ver}.tar.bz2" + +# libxkbcommon — keyboard description library. GitHub archive tag. +[packages.libxkbcommon] +kind = "tar" +recipe = "local/recipes/libs/libxkbcommon" +version_index = "https://github.com/xkbcommon/libxkbcommon/tags" +version_regex = 'xkbcommon-([0-9]+\.[0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://github.com/xkbcommon/libxkbcommon/archive/refs/tags/xkbcommon-{ver}.tar.gz" + +# dbus — system + session message bus. Static freedesktop.org releases listing. +[packages.dbus] +kind = "tar" +recipe = "local/recipes/system/dbus" +version_index = "https://dbus.freedesktop.org/releases/dbus/" +version_regex = 'dbus-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://dbus.freedesktop.org/releases/dbus/dbus-{ver}.tar.xz" + +# glib — GNOME core library. Series-locked index (operator bumps series here). +[packages.glib] +kind = "tar" +recipe = "local/recipes/libs/glib" +version_index = "https://download.gnome.org/sources/glib/2.89/" +version_regex = 'glib-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://download.gnome.org/sources/glib/{majmin}/glib-{ver}.tar.xz" + +# cairo — 2D graphics. Static cairographics.org releases listing. +[packages.cairo] +kind = "tar" +recipe = "local/recipes/libs/cairo" +version_index = "https://www.cairographics.org/releases/" +version_regex = 'cairo-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://www.cairographics.org/releases/cairo-{ver}.tar.xz" + +# pango — text layout. Series-locked GNOME index. +[packages.pango] +kind = "tar" +recipe = "local/recipes/libs/pango" +version_index = "https://download.gnome.org/sources/pango/1.56/" +version_regex = 'pango-([0-9]+\.[0-9]+\.[0-9]+)\.tar' +resolve = "version" +tar_template = "https://download.gnome.org/sources/pango/{majmin}/pango-{ver}.tar.xz" + +# harfbuzz — text shaping. GitHub tags HTML is JS-only; use the API (JSON). +[packages.harfbuzz] +kind = "tar" +recipe = "local/recipes/libs/harfbuzz" +version_index = "https://api.github.com/repos/harfbuzz/harfbuzz/tags" +version_regex = '"name":\s*"([0-9]+\.[0-9]+\.[0-9]+)"' +resolve = "version" +tar_template = "https://github.com/harfbuzz/harfbuzz/releases/download/{ver}/harfbuzz-{ver}.tar.xz" + +# freetype2 — font rendering. GitLab tags use VER-X-Y-Z format; ver_transform +# converts the captured dashes to dots. Tarball via sourceforge (canonical URL +# used by the recipe). +[packages.freetype2] +kind = "tar" +recipe = "local/recipes/libs/freetype2" +version_index = "https://gitlab.freedesktop.org/freetype/freetype/-/tags" +version_regex = '/freetype/freetype/-/tags/VER-([0-9]+-[0-9]+-[0-9]+)' +ver_transform = "s/-/./g" +resolve = "version" +tar_template = "https://sourceforge.net/projects/freetype/files/freetype2/{ver}/freetype-{ver}.tar.xz/download" + +# lcms2 — Little CMS. GitHub archive tag (tag "lcms2.19" → version "2.19"). +[packages.lcms2] +kind = "tar" +recipe = "local/recipes/libs/lcms2" +version_index = "https://github.com/mm2/Little-CMS/tags" +version_regex = 'lcms([0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://github.com/mm2/Little-CMS/archive/refs/tags/lcms{ver}.tar.gz" + +# konsole — KDE terminal. Independent release cycle (KDE Gear, not Frameworks). +# invent.kde.org tags page. +[packages.konsole] +kind = "tar" +recipe = "local/recipes/kde/konsole" +version_index = "https://invent.kde.org/utilities/konsole/-/tags" +version_regex = '/utilities/konsole/-/tags/v([0-9]+\.[0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://invent.kde.org/utilities/konsole/-/archive/v{ver}/konsole-v{ver}.tar.gz" + +# plasma-wayland-protocols — INDEPENDENT versioning (1.21.0, NOT plasma 6.7.x). +[packages.plasma-wayland-protocols] +kind = "tar" +recipe = "local/recipes/kde/plasma-wayland-protocols" +version_index = "https://invent.kde.org/libraries/plasma-wayland-protocols/-/tags" +version_regex = '/libraries/plasma-wayland-protocols/-/tags/v([0-9]+\.[0-9]+\.[0-9]+)' +resolve = "version" +tar_template = "https://invent.kde.org/libraries/plasma-wayland-protocols/-/archive/v{ver}/plasma-wayland-protocols-v{ver}.tar.gz" + + +# --------------------------------------------------------------------------- +# Singletons — git-rev +# --------------------------------------------------------------------------- + +# sddm — display manager. rev is a commit sha; resolve latest tag's sha. +[packages.sddm] +kind = "git-rev" +recipe = "local/recipes/kde/sddm" +git_url = "https://github.com/sddm/sddm.git" +tag_regex = '^v?[0-9]+\.[0-9]+\.[0-9]+$' + +# pipewire — audio/video server. rev is a tag name (e.g. "0.3.85"). +[packages.pipewire] +kind = "git-rev" +recipe = "local/recipes/libs/pipewire" +git_url = "https://gitlab.freedesktop.org/pipewire/pipewire.git" +tag_regex = '^[0-9]+\.[0-9]+\.[0-9]+$' + +# wireplumber — PipeWire session manager. rev is a tag name (e.g. "0.4.14"). +[packages.wireplumber] +kind = "git-rev" +recipe = "local/recipes/libs/wireplumber" +git_url = "https://gitlab.freedesktop.org/pipewire/wireplumber.git" +tag_regex = '^[0-9]+\.[0-9]+\.[0-9]+$' diff --git a/local/scripts/bump-fork.sh b/local/scripts/bump-fork.sh index 4b9b2de576..8ec4124944 100755 --- a/local/scripts/bump-fork.sh +++ b/local/scripts/bump-fork.sh @@ -1,4 +1,35 @@ #!/usr/bin/env bash +# +# ============================================================================ +# DEPRECATED — DO NOT USE (2026-07-18, release-bump-pipeline.md Deliverable 4) +# ============================================================================ +# This script is superseded by: +# - local/scripts/bump-release.sh (canonical orchestrator) +# - local/scripts/upgrade-forks.sh --to=... (per-fork source bump) +# +# WHY IT IS DEPRECATED: +# bump-fork.sh uses a WHOLESALE-REPLACEMENT model: it clones a fresh +# upstream tag, applies patch files from local/patches//, and then +# SWAPS the entire local/sources// directory with the new tree. +# This DESTROYS any committed fork work that is not also present as a +# patch file under local/patches// — including commits made +# directly to the submodule/ branch (the path-fork model that +# replaced overlay patches in 2026-06). +# +# Concretely: kernel, relibc, base, bootloader, installer, redoxfs, and +# userutils now carry Red Bear changes as direct commits on their +# `submodule/` branches, NOT as patch files. Running +# bump-fork.sh on any of them would silently discard that work. +# +# RETENTION POLICY: +# This file is KEPT for historical reference only, per the project's +# never-delete rule. The runtime guard below prevents accidental use. +# Its logic is otherwise unchanged. +# +# ESCAPE HATCH (operators only, with eyes open): +# REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1 ./local/scripts/bump-fork.sh ... +# ============================================================================ +# # bump-fork.sh — Auto-bump a local fork to match upstream version # # Usage: bump-fork.sh [--version=X.Y.Z] @@ -11,6 +42,35 @@ set -euo pipefail +# ---- Runtime deprecation guard ------------------------------------------- +# Refuse to run unless the operator explicitly acknowledges the +# deprecation by exporting REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1. +# Without the env var, print the deprecation explanation to stderr and +# exit 1 BEFORE any potentially destructive operation runs. +if [[ "${REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED:-0}" != "1" ]]; then + cat >&2 <<'EOF' +============================================================================ +bump-fork.sh is DEPRECATED (release-bump-pipeline.md Deliverable 4). + +Its wholesale-replacement model destroys committed fork work that is not +also present as a patch file under local/patches//. Under the current +path-fork model, kernel/relibc/base/bootloader/installer/redoxfs/userutils +carry Red Bear changes as direct commits on submodule/ branches +— running this script on any of them would silently discard that work. + +Use instead: + ./local/scripts/bump-release.sh # canonical orchestrator + ./local/scripts/upgrade-forks.sh --to= # per-fork source bump + +This file is retained for historical reference per the never-delete rule. + +To override (operators only, with full understanding of the risk): + REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1 ./local/scripts/bump-fork.sh ... +============================================================================ +EOF + exit 1 +fi + # Derive Red Bear OS version from the current git branch name. # Branch "0.3.1" → BRANCH_VERSION="0.3.1". Same pattern as # local/scripts/build-redbear.sh:158-165. Falls back to "0.0.0" if diff --git a/local/scripts/bump-graphics-recipes.sh b/local/scripts/bump-graphics-recipes.sh index 4804c2914e..121a85bb55 100755 --- a/local/scripts/bump-graphics-recipes.sh +++ b/local/scripts/bump-graphics-recipes.sh @@ -1,121 +1,298 @@ #!/usr/bin/env bash -# bump-graphics-recipes.sh — Update tarball URLs/BLAKE3 for graphics recipes. -# Scope: recipes/libs, recipes/gpu, recipes/kde, recipes/qt, recipes/wip/{wayland,gpu,kde,qt}. -# Excludes: recipes/core/* (mini-ISO path), recipes/system/* (services). +# bump-graphics-recipes.sh — Map-driven external source-bump engine. # -# Usage: bump-graphics-recipes.sh [--dry-run] [--no-fetch] +# Rewritten 2026-07-18 from the legacy hardcoded UPSTREAM_INDEX approach. +# The single source of truth is local/external-upstream-map.toml; this script +# never scans directories (symlinked duplicates like recipes/qt/qtbase ↔ +# recipes/wip/qt/qtbase are handled by iterating the MAP, which lists each +# canonical local/recipes/ path exactly once). +# +# What it does: +# - Resolves each group's latest version ONCE, then applies to all members. +# - Resolves each singleton's latest version individually. +# - tar bumps: download new tarball → b3sum → sed tar=/blake3= in recipe.toml +# (+ sync [package] version = "..." if present, semver, and differs). +# - git-rev bumps: git ls-remote --tags → latest matching tag → resolve its +# commit sha (annotated tags dereferenced via ^{}) → sed rev = "...". +# - After each non-dry-run bump: runs ./target/release/repo validate-patches +# and captures [PASS]/[FAIL] counts. +# - Writes a machine-parseable stabilization report to stdout AND +# .redbear-recipe-bump/last-report.txt. +# +# Usage: +# bump-graphics-recipes.sh [--dry-run] [--no-fetch] +# +# --dry-run resolve and print intended changes; perform no downloads, +# edits, or validate-patches runs. +# --no-fetch reuse cached version indexes under .redbear-recipe-bump/; +# error on entries whose cache is missing. +# +# Never commits. Never deletes recipes. Never creates branches. +# See local/external-upstream-map.toml for the package inventory and schema. set -euo pipefail ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +MAP="$ROOT/local/external-upstream-map.toml" +CACHE_DIR="$ROOT/.redbear-recipe-bump" +REPORT="$CACHE_DIR/last-report.txt" + DRY_RUN=0 NO_FETCH=0 - for arg in "$@"; do case "$arg" in --dry-run) DRY_RUN=1 ;; --no-fetch) NO_FETCH=1 ;; - --help|-h) echo "Usage: $0 [--dry-run] [--no-fetch]"; exit 0 ;; + --help|-h) grep -E '^#' "$0" | head -30; exit 0 ;; *) echo "Unknown option: $arg" >&2; exit 1 ;; esac done -declare -A UPSTREAM_INDEX=( - [cairo]=https://cairographics.org/releases/ - [freetype2]=https://download.savannah.gnu.org/freetype/ - [glib]=https://download.gnome.org/sources/glib/ - [harfbuzz]=https://github.com/harfbuzz/harfbuzz/releases/ - [libepoxy]=https://github.com/anholt/libepoxy/releases/ - [libxkbcommon]=https://github.com/xkbcommon/libxkbcommon/releases/ - [pango]=https://download.gnome.org/sources/pango/ - [libjpeg]=https://github.com/libjpeg-turbo/libjpeg-turbo/releases/ - [shared-mime-info]=https://gitlab.gnome.org/GNOME/shared-mime-info/-/tags/ - [gdk-pixbuf]=https://download.gnome.org/sources/gdk-pixbuf/ - [libxml2]=https://download.gnome.org/sources/libxml2/ - [kf6-karchive]=https://download.kde.org/stable/frameworks/ - [kf6-attica]=https://download.kde.org/stable/frameworks/ - [kf6-kcodecs]=https://download.kde.org/stable/frameworks/ +if [ ! -f "$MAP" ]; then + echo "ERROR: map not found: $MAP" >&2 + exit 2 +fi + +REPO_BIN="$ROOT/target/release/repo" +NO_FETCH_FLAG="" +[ "$NO_FETCH" = "1" ] && NO_FETCH_FLAG="--no-fetch" + +mkdir -p "$CACHE_DIR" + +# --------------------------------------------------------------------------- +# Phase 1: Python resolves every entry and emits a TSV action plan. +# Fields (tab-separated): +# ACTION RECIPE_NAME RECIPE_DIR KIND OLD NEW NEW_VERSION NEW_PKG_VERSION NOTE +# ACTION ∈ {bump, current, failed} +# --------------------------------------------------------------------------- +ACTIONS_FILE="$(mktemp)" +trap 'rm -f "$ACTIONS_FILE"' EXIT + +python3 - "$ROOT" "$NO_FETCH_FLAG" <<'PYEOF' > "$ACTIONS_FILE" +import re +import sys +import tomllib +from pathlib import Path + +ROOT = Path(sys.argv[1]) +NO_FETCH = "--no-fetch" in sys.argv[2:] +MAP_PATH = ROOT / "local/external-upstream-map.toml" +CACHE_DIR = ROOT / ".redbear-recipe-bump" + +with open(MAP_PATH, "rb") as f: + MAP = tomllib.load(f) + +sys.path.insert(0, str(ROOT / "local/scripts/lib")) +import external_version_lib +external_version_lib.configure(CACHE_DIR, NO_FETCH) +from external_version_lib import ( + fetch, version_sort_key, apply_ver_transform, resolve_index, + render_template, member_url_name, member_template, read_recipe_text, + current_tar_url, current_rev, resolve_git_rev, ) -GRAPHICS_DIRS=( - "$ROOT/recipes/libs" "$ROOT/recipes/gpu" "$ROOT/recipes/kde" "$ROOT/recipes/qt" - "$ROOT/recipes/wip/wayland" "$ROOT/recipes/wip/gpu" "$ROOT/recipes/wip/kde" "$ROOT/recipes/wip/qt" -) -VER_RE='-v?[0-9]+(\.[0-9]+){1,3}(-rc[0-9]+)?(-alpha[0-9]+)?(-beta[0-9]+)?' -resolve_latest() { - local cached="$1" prefix="$2" - grep -oE "${prefix}${VER_RE}\.tar\.(gz|xz|bz2|zst)" "$cached" | sort -uV | tail -1 -} + +def current_pkg_version(text): + m = re.search(r'^\[package\]\s*\n(?:[^\[]*?\n)*?\s*version\s*=\s*"([^"]+)"', + text, re.MULTILINE) + return m.group(1) if m else None + + +def is_semver(v): + return bool(re.match(r'^\d+\.\d+(\.\d+)?$', v or "")) + + +def emit(action, name, rdir, kind, old, new, new_ver, pkg_ver, note): + print("\t".join([action, name, str(rdir), kind, + old or "", new or "", new_ver or "", + pkg_ver or "", note or ""])) + + +# Groups +for gid, g in MAP.get("groups", {}).items(): + v, _src, err = resolve_index(g["version_index"], g["version_regex"], + g.get("resolve", "version")) + for member in g["members"]: + rdir = ROOT / g["recipe_root"] / member + text = read_recipe_text(rdir) + cur = current_tar_url(text) + tmpl = member_template(g, member) + uname = member_url_name(g, member) + if cur is None: + emit("failed", f"{gid}:{member}", rdir, "tar", "", "", "", "", + "no-tar-in-recipe") + continue + if v is None: + emit("failed", f"{gid}:{member}", rdir, "tar", cur, "", "", "", + f"index:{err}") + continue + new_url = render_template(tmpl, {**v, "name": uname}) + if new_url == cur: + emit("current", f"{gid}:{member}", rdir, "tar", cur, new_url, + v["ver"], "", "") + else: + pkg = current_pkg_version(text) + new_pkg = v["ver"] if (pkg and is_semver(pkg) and pkg != v["ver"]) else "" + emit("bump", f"{gid}:{member}", rdir, "tar", cur, new_url, + v["ver"], new_pkg, "") + +# Singletons +for pid, p in MAP.get("packages", {}).items(): + rdir = ROOT / p["recipe"] + text = read_recipe_text(rdir) + if p["kind"] == "tar": + cur = current_tar_url(text) + if cur is None: + emit("failed", pid, rdir, "tar", "", "", "", "", "no-tar-in-recipe") + continue + v, _src, err = resolve_index(p["version_index"], p["version_regex"], + p.get("resolve", "version"), + p.get("ver_transform")) + if v is None: + emit("failed", pid, rdir, "tar", cur, "", "", "", f"index:{err}") + continue + new_url = render_template(p["tar_template"], {**v, "name": pid}) + if new_url == cur: + emit("current", pid, rdir, "tar", cur, new_url, v["ver"], "", "") + else: + pkg = current_pkg_version(text) + new_pkg = v["ver"] if (pkg and is_semver(pkg) and pkg != v["ver"]) else "" + emit("bump", pid, rdir, "tar", cur, new_url, v["ver"], new_pkg, "") + elif p["kind"] == "git-rev": + cur = current_rev(text) + if cur is None: + emit("failed", pid, rdir, "git-rev", "", "", "", "", "no-rev-in-recipe") + continue + res, err = resolve_git_rev(p["git_url"], p["tag_regex"]) + if res is None: + emit("failed", pid, rdir, "git-rev", cur, "", "", "", f"git:{err}") + continue + new_sha = res["sha"] + if cur == new_sha or cur.startswith(new_sha[:12]) or cur == res["tag"]: + emit("current", pid, rdir, "git-rev", cur, new_sha, res["tag"], "", "") + else: + emit("bump", pid, rdir, "git-rev", cur, new_sha, res["tag"], "", "") +PYEOF + +# --------------------------------------------------------------------------- +# Phase 2: bash applies the action plan. +# --------------------------------------------------------------------------- UPDATED=0 -SKIPPED=0 +CURRENT=0 FAILED=0 +REPORT_LINES=() -for dir in "${GRAPHICS_DIRS[@]}"; do - [ -d "$dir" ] || continue - for rdir in "$dir"/*/; do - [ -f "$rdir/recipe.toml" ] || continue - name=$(basename "$rdir") +run_validate_patches() { + # $1 = recipe name (basename). Outputs patches_total/pass/fail on stdout + # as "total=N pass=N fail=N". Returns non-zero only if the binary is + # missing (caller treats that as a warning, not a failure). + local recipe="$1" + if [ ! -x "$REPO_BIN" ]; then + echo "total=0 pass=0 fail=0" + echo "WARN: $REPO_BIN not found; skipped validate-patches for $recipe" >&2 + return 1 + fi + local out + out="$(REDBEAR_CANONICAL_BUILD=1 "$REPO_BIN" validate-patches "$recipe" 2>&1)" || true + local pass fail + pass="$(printf '%s\n' "$out" | grep -c '\[PASS\]' || true)" + fail="$(printf '%s\n' "$out" | grep -c '\[FAIL\]' || true)" + local total=$((pass + fail)) + echo "total=${total} pass=${pass} fail=${fail}" +} - if [ -z "${UPSTREAM_INDEX[$name]:-}" ]; then - SKIPPED=$((SKIPPED+1)) - continue - fi +# Read the TSV action plan line by line. +while IFS=$'\t' read -r ACTION RNAME RDIR KIND OLD NEW NVER NPKG NOTE; do + [ -n "$ACTION" ] || continue + RTOML="$RDIR/recipe.toml" - current=$(grep '^tar = ' "$rdir/recipe.toml" | head -1 | sed -E 's/^tar = "//;s/"$//') - [ -z "$current" ] && { SKIPPED=$((SKIPPED+1)); continue; } + case "$ACTION" in + current) + CURRENT=$((CURRENT + 1)) + REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=") + if [ "$DRY_RUN" = "1" ]; then + printf '[current] %s (already at %s)\n' "$RNAME" "$NVER" + fi + ;; - case "$name" in - glib|pango|shared-mime-info|gdk-pixbuf|libxml2) prefix="$name" ;; - freetype2) prefix="freetype" ;; - libepoxy) prefix="libepoxy" ;; - libxkbcommon) prefix="libxkbcommon" ;; - libjpeg) prefix="libjpeg-turbo" ;; - cairo) prefix="cairo" ;; - harfbuzz) prefix="harfbuzz" ;; - kf6-*) prefix="$(echo $name | sed 's/^kf6-//')" ;; - *) prefix="$name" ;; - esac + failed) + FAILED=$((FAILED + 1)) + REPORT_LINES+=("recipe=$RNAME old=$OLD new= patches_total= patches_pass= patches_fail= fail_details=$NOTE") + printf '[FAILED] %s: %s\n' "$RNAME" "$NOTE" >&2 + ;; - cached="$ROOT/.redbear-recipe-bump/$name.html" - mkdir -p "$(dirname "$cached")" - if [ "$NO_FETCH" = "0" ]; then - if ! curl -sLf "${UPSTREAM_INDEX[$name]}" -o "$cached"; then - echo "WARN: $name: failed to fetch ${UPSTREAM_INDEX[$name]}" >&2 - FAILED=$((FAILED+1)) + bump) + if [ "$DRY_RUN" = "1" ]; then + if [ "$KIND" = "tar" ]; then + printf '[would-bump] %s\n %s\n -> %s\n (would download + b3sum + sed tar=/blake3=)\n' \ + "$RNAME" "$OLD" "$NEW" + else + printf '[would-bump] %s\n rev=%s\n -> rev=%s (%s)\n' \ + "$RNAME" "$OLD" "$NEW" "$NVER" + fi + UPDATED=$((UPDATED + 1)) + REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=(dry-run)") continue fi - fi - [ -f "$cached" ] || continue - latest=$(resolve_latest "$cached" "$prefix") - [ -z "$latest" ] && { SKIPPED=$((SKIPPED+1)); continue; } + # --- tar bump --- + if [ "$KIND" = "tar" ]; then + tmp="$(mktemp)" + if ! curl -sLf --retry 3 --retry-delay 2 --connect-timeout 20 --max-time 300 "$NEW" -o "$tmp" 2>/dev/null; then + FAILED=$((FAILED + 1)) + REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=download-failed") + printf '[FAILED] %s: download failed for %s\n' "$RNAME" "$NEW" >&2 + rm -f "$tmp" + continue + fi + new_hash="$(b3sum "$tmp" | awk '{print $1}')" + rm -f "$tmp" - latest_url=$(grep -m1 -oE "${UPSTREAM_INDEX[$name]}${latest}" "$cached" | head -1) - [ -z "$latest_url" ] && latest_url="${UPSTREAM_INDEX[$name]}${latest}" + # Escape sed-special replacement chars (& \ |) so URLs containing + # query strings or other metacharacters do not corrupt the rewrite. + esc_new=$(printf '%s' "$NEW" | sed 's/[&\\|]/\\&/g') + sed -i -E "s|^\\s*tar\\s*=\\s*\"[^\"]+\"|tar = \"$esc_new\"|" "$RTOML" + sed -i -E "s|^\\s*blake3\\s*=\\s*\"[a-f0-9]+\"|blake3 = \"$new_hash\"|" "$RTOML" - if [ "$current" = "$latest_url" ]; then - SKIPPED=$((SKIPPED+1)) - continue - fi + if [ -n "$NPKG" ]; then + sed -i -E "s|^(\\s*version\\s*=\\s*)\"[^\"]+\"|\\1\"$NPKG\"|" "$RTOML" + fi - echo "[$name] $current -> $latest_url" - if [ "$DRY_RUN" = "0" ]; then - tmp=$(mktemp) - curl -sLf "$latest_url" -o "$tmp" - new_hash=$(b3sum "$tmp" | awk '{print $1}') - sed -i -E "s|^tar = \"[^\"]+\"|tar = \"$latest_url\"|" "$rdir/recipe.toml" - sed -i -E "s|^blake3 = \"[a-f0-9]+\"|blake3 = \"$new_hash\"|" "$rdir/recipe.toml" - rm -f "$tmp" - fi - UPDATED=$((UPDATED+1)) + UPDATED=$((UPDATED + 1)) + + # --- git-rev bump --- + elif [ "$KIND" = "git-rev" ]; then + sed -i -E "s|^\\s*rev\\s*=\\s*\"[^\"]+\"|rev = \"$NEW\"|" "$RTOML" + UPDATED=$((UPDATED + 1)) + fi + + # --- validate-patches --- + validate_out="$(run_validate_patches "$(basename "$RDIR")" 2>/dev/null || true)" + vt="$(printf '%s' "$validate_out" | grep -oE 'total=[0-9]*' | head -1 | cut -d= -f2)" + vp="$(printf '%s' "$validate_out" | grep -oE 'pass=[0-9]*' | head -1 | cut -d= -f2)" + vf="$(printf '%s' "$validate_out" | grep -oE 'fail=[0-9]*' | head -1 | cut -d= -f2)" + vt="${vt:-0}"; vp="${vp:-0}"; vf="${vf:-0}" + REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total=$vt patches_pass=$vp patches_fail=$vf fail_details=") + ;; + esac +done < "$ACTIONS_FILE" + +# --------------------------------------------------------------------------- +# Phase 3: report + summary. +# --------------------------------------------------------------------------- + +{ + for line in "${REPORT_LINES[@]}"; do + printf '%s\n' "$line" done -done +} | tee "$REPORT" echo "" echo "=== Bump summary ===" echo "Updated (or would-update): $UPDATED" -echo "Skipped (no index / already-current / non-tar): $SKIPPED" -echo "Failed (network/parse): $FAILED" -[ "$DRY_RUN" = "1" ] && echo "Dry-run only. No files changed." +echo "Already current: $CURRENT" +echo "Failed (network/parse): $FAILED" +[ "$DRY_RUN" = "1" ] && echo "(dry-run; no files changed)" +echo "Report written to: $REPORT" diff --git a/local/scripts/bump-release.sh b/local/scripts/bump-release.sh new file mode 100755 index 0000000000..403581704d --- /dev/null +++ b/local/scripts/bump-release.sh @@ -0,0 +1,579 @@ +#!/usr/bin/env bash +# bump-release.sh — Canonical release-bump orchestrator for Red Bear OS. +# +# Policy (release-bump-pipeline.md): +# - bump-release.sh DECIDES per fork. +# - sync-versions.sh EXECUTES suffix-only label rewrites (single code path). +# - refresh-fork-upstream-map.sh maintains the map's column-3 tag. +# - upgrade-forks.sh --to= performs actual fork source bumps. +# +# Fork routing by map mode (release-bump-pipeline.md design constraint #2): +# - snapshot/tracked → eligible for source bump via upgrade-forks.sh --to +# - diverged → report-only; operator-manual rebase (Phase 2.4+). +# Override with --force-diverged. +# - base (tag=main) → always suffix-only; member crates untouched. +# - bootloader → NEVER source-bumped (no merge-base with upstream; +# one-time 'git replace --graft' is the documented path). +# Upgrade candidates are resolved by latest upstream semver tag EXCLUDING any +# tag already reachable from the fork's HEAD (ancestor tags are history, not +# upgrades — see latest_tag_for_url). +# +# Modes: +# (no args) Fork upstream discovery REPORT + label sync (delegates to +# sync-versions.sh --no-regen) + hint to rerun with +# --with-sources / --with-external when upgrades exist. +# --with-sources For each fork whose map mode is snapshot/tracked AND whose +# latest upstream semver tag is newer than the current base +# (current base = fork Cargo.toml version stripped of +rb*): +# upgrade-forks.sh --to= +# On success: set fork Cargo.toml version to +# "+rb", update the map row's column 3 in +# place, regen that fork's Cargo.lock (best-effort), report +# the rb-backup/* branch. On failure: report, leave label +# and map untouched, continue with remaining forks. After +# all source bumps: re-run sync-versions.sh --no-regen. +# --with-external Invoke local/scripts/bump-graphics-recipes.sh (pass +# through --dry-run / --no-fetch). Tolerate its absence +# with a warning (a sibling unit is rewriting it +# concurrently). +# --check Fully read-only; print the same report; exit 1 if any +# label drift (via sync-versions.sh --check) OR any fork +# has an available source bump; exit 0 when fully synced. +# --dry-run Print decisions, mutate nothing (implies no delegation +# that mutates). +# --no-fetch Skip all network (ls-remote); use the map's current tags +# as "latest" (actions become report-only). +# --regen Pass --regen instead of --no-regen to sync-versions.sh. +# --force-diverged Pass --force-diverged to upgrade-forks.sh for diverged +# forks (still reports them first). +# --fork= Scope all fork operations to one fork. +# +# GUARDS: +# - Current git branch must match ^[0-9]+\.[0-9]+\.[0-9]+$ (anchored semver). +# Otherwise refuse — release bumps only happen on release branches. +# - REDBEAR_CI=1: only --check / --dry-run are allowed. --with-sources and +# --with-external are refused. +# +# NEVER AUTO-COMMITS. The script ends with an exact copy-paste commit guide +# listing parent-repo files changed (map, Cargo.toml of cookbook + Cat 1 +# crates) and per-fork git -C local/sources/ commit + push to +# submodule/ reminders. + +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +SCRIPT_DIR="$ROOT/local/scripts" +MAP="$ROOT/local/fork-upstream-map.toml" +cd "$ROOT" + +# ---- argument parsing ------------------------------------------------------ +WITH_SOURCES=0 +WITH_EXTERNAL=0 +CHECK_ONLY=0 +DRY_RUN=0 +NO_FETCH=0 +REGEN=0 +FORCE_DIVERGED=0 +FORK_FILTER="" + +for arg in "$@"; do + case "$arg" in + --with-sources) WITH_SOURCES=1 ;; + --with-external) WITH_EXTERNAL=1 ;; + --check) CHECK_ONLY=1 ;; + --dry-run) DRY_RUN=1 ;; + --no-fetch) NO_FETCH=1 ;; + --regen) REGEN=1 ;; + --force-diverged) FORCE_DIVERGED=1 ;; + --fork=*) FORK_FILTER="${arg#*=}" ;; + -h|--help) + sed -n '2,55p' "$0" + exit 0 + ;; + *) echo "Unknown option: $arg" >&2; exit 1 ;; + esac +done + +# ---- branch guard ---------------------------------------------------------- +BRANCH="$(git branch --show-current 2>/dev/null || echo "")" +if [[ -z "$BRANCH" ]]; then + echo "ERROR: cannot determine current git branch" >&2 + exit 1 +fi +if [[ ! "$BRANCH" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "ERROR: branch '$BRANCH' is not anchored semver (X.Y.Z)" >&2 + echo " Release bumps only happen on release branches." >&2 + exit 1 +fi + +# ---- CI guard -------------------------------------------------------------- +if [[ "${REDBEAR_CI:-0}" == "1" ]]; then + if [[ $WITH_SOURCES -eq 1 || $WITH_EXTERNAL -eq 1 ]]; then + echo "ERROR: REDBEAR_CI=1 forbids --with-sources / --with-external" >&2 + echo " Use --check or --dry-run only." >&2 + exit 1 + fi +fi + +# Aggregate read-only flag: anything that must NOT mutate the tree. +READ_ONLY=0 +[[ $CHECK_ONLY -eq 1 || $DRY_RUN -eq 1 ]] && READ_ONLY=1 + +# ---- helpers --------------------------------------------------------------- +SEMVER_RE='^[0-9]+\.[0-9]+\.[0-9]+$' + +# Resolve the newest GENUINE upgrade tag for a remote via `git ls-remote`. +# Output: "|" — tag empty when nothing qualifies. +# Two filters: (a) tags already reachable from the fork's HEAD are history, +# not upgrades (relibc's 0.6.0/2020 outranks its newer 0.2.x API line +# lexically); (b) tags not version-greater than the current base are not +# upgrades either (the base tag itself is usually an ancestor of HEAD). +latest_tag_for_url() { + local url="$1" fork="${2:-}" base="${3:-}" + [[ $NO_FETCH -eq 0 ]] || { echo "|0"; return 0; } + local tag forkdir="" raw=0 + [[ -n "$fork" && -d "$ROOT/local/sources/$fork/.git" ]] \ + && forkdir="$ROOT/local/sources/$fork" + while IFS= read -r tag; do + [[ -z "$tag" ]] && continue + raw=$((raw + 1)) + if [[ -n "$forkdir" ]] \ + && git -C "$forkdir" rev-parse --verify --quiet "refs/tags/$tag" >/dev/null 2>&1 \ + && git -C "$forkdir" merge-base --is-ancestor "refs/tags/$tag" HEAD 2>/dev/null; then + continue + fi + if [[ -n "$base" ]] && ! version_greater "$tag" "$base"; then + continue + fi + echo "${tag}|${raw}" + return 0 + done < <(git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \ + | awk '{print $2}' \ + | sed 's|refs/tags/||' \ + | sed 's|\^{}$||' \ + | grep -E "$SEMVER_RE" || true) + echo "|${raw}" + return 0 +} + +# version_greater → true iff a is version-greater than b (sort -V). +version_greater() { + [[ "$1" != "$2" ]] \ + && [[ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | tail -1)" == "$1" ]] +} + +# Compare two semvers. Echo -1 / 0 / 1. Echo "ERR" if either is non-semver. +semver_cmp() { + local a="$1" b="$2" + if [[ ! "$a" =~ $SEMVER_RE || ! "$b" =~ $SEMVER_RE ]]; then + echo "ERR"; return + fi + local a1 a2 a3 b1 b2 b3 + IFS=. read -r a1 a2 a3 <<<"$a" + IFS=. read -r b1 b2 b3 <<<"$b" + if (( a1 < b1 )); then echo "-1"; return; fi + if (( a1 > b1 )); then echo "1"; return; fi + if (( a2 < b2 )); then echo "-1"; return; fi + if (( a2 > b2 )); then echo "1"; return; fi + if (( a3 < b3 )); then echo "-1"; return; fi + if (( a3 > b3 )); then echo "1"; return; fi + echo "0" +} + +# Strip any +rb build-metadata suffix from a version string. +strip_rb_suffix() { + echo "$1" | sed -E 's/\+rb[0-9]+(\.[0-9]+\.[0-9]+)?$//' +} + +# Read a fork's current base version from its Cargo.toml (stripped of +rb*). +fork_current_base_version() { + local fork="$1" + local ct="$ROOT/local/sources/$fork/Cargo.toml" + [[ -f "$ct" ]] || { echo ""; return; } + local ver + ver=$(grep -E '^version[[:space:]]*=' "$ct" | head -1 | sed 's/.*"\(.*\)".*/\1/' || true) + strip_rb_suffix "$ver" +} + +# ---- parse the map into 4 parallel arrays ---------------------------------- +declare -a MAP_FORKS=() +declare -a MAP_URLS=() +declare -a MAP_TAGS=() +declare -a MAP_MODES=() + +parse_map() { + MAP_FORKS=() + MAP_URLS=() + MAP_TAGS=() + MAP_MODES=() + [[ -f "$MAP" ]] || return 0 + while IFS=$'\t' read -r fork url tag mode; do + [[ -z "$fork" ]] && continue + MAP_FORKS+=("$fork") + MAP_URLS+=("$url") + MAP_TAGS+=("$tag") + MAP_MODES+=("${mode:-}") + done < <(awk ' + /^[^#[:space:]]/ { + fork = $1; url = $2; tag = $3; mode = $4 + sub(/[ \t]*#.*/, "", mode) + print fork "\t" url "\t" tag "\t" mode + } + ' "$MAP") +} + +# In-place column-3 update for a single fork (byte-exact awk rewrite). +# A sidecar tmp file is consumed by `mv`; no trap needed (if mv fails the +# tmp file lingers harmlessly under $MAP.tmp.). +update_map_tag_inplace() { + local fork="$1" new_tag="$2" + [[ -f "$MAP" ]] || return 0 + local tmp="${MAP}.tmp.$$" + awk -v fork="$fork" -v new_tag="$new_tag" ' + $1 == fork && /^[^#[:space:]]/ { + prefix_re = fork "[ \t]+[^ \t]+[ \t]+" + if (match($0, prefix_re)) { + prefix = substr($0, 1, RLENGTH) + rest = substr($0, RLENGTH + 1) + if (match(rest, /^[^ \t]+/)) { + suffix = substr(rest, RLENGTH + 1) + $0 = prefix new_tag suffix + } + } + } + { print } + ' "$MAP" > "$tmp" && mv "$tmp" "$MAP" + # Clean up a straggler if mv failed for some reason. + [[ -e "$tmp" ]] && rm -f "$tmp" || true +} + +# ---- counters -------------------------------------------------------------- +SOURCE_BUMPS_AVAILABLE=0 +SOURCE_BUMPS_APPLIED=0 +SOURCE_BUMPS_FAILED=0 +LABEL_DRIFT=0 + +# Latest-tag cache filled by phase_report; phase_source_bumps reuses it so +# each fork costs exactly one ls-remote per run. +declare -A LATEST_FOR=() + +# ---- phase: report --------------------------------------------------------- +phase_report() { + echo "=== Fork upstream discovery report ===" + echo "branch=$BRANCH read_only=$READ_ONLY no_fetch=$NO_FETCH" + echo "" + local idx fork url current mode latest cargo_base basis cmp action + for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do + fork="${MAP_FORKS[$idx]}" + url="${MAP_URLS[$idx]}" + current="${MAP_TAGS[$idx]}" + mode="${MAP_MODES[$idx]}" + + if [[ -n "$FORK_FILTER" && "$fork" != "$FORK_FILTER" ]]; then + continue + fi + + if [[ "$current" == "main" ]]; then + echo "fork=$fork mode=$mode current=$current latest=(skip) action=skip-main" + continue + fi + + if [[ "$fork" == "bootloader" ]]; then + echo "fork=$fork mode=$mode current=$current latest=(skip) action=report-only note=no-merge-base-needs-git-replace-graft" + continue + fi + + if [[ "$mode" == "diverged" ]]; then + echo "fork=$fork mode=diverged current=$current latest=(skip) action=report-only" + continue + fi + + # Decision basis: the fork's Cargo.toml base is ground truth; the map + # tag is the fallback when the fork has no Cargo.toml version. + cargo_base=$(fork_current_base_version "$fork") + basis="$current" + [[ -n "$cargo_base" ]] && basis="$cargo_base" + + local result raw + latest="" + if [[ $NO_FETCH -eq 0 ]]; then + result=$(latest_tag_for_url "$url" "$fork" "$basis") + latest="${result%%|*}" + raw="${result##*|}" + # Tags existed upstream but none qualified as newer → already current. + [[ -z "$latest" && "$raw" -gt 0 ]] && latest="$basis" + else + latest="$basis" + fi + LATEST_FOR[$fork]="$latest" + + if [[ -z "$latest" ]]; then + echo "fork=$fork mode=$mode current=$current latest=(unknown) action=skip-no-tag" + continue + fi + + cmp=$(semver_cmp "$basis" "$latest") + + case "$cmp" in + -1) + action="upgrade" + SOURCE_BUMPS_AVAILABLE=$((SOURCE_BUMPS_AVAILABLE + 1)) + ;; + 0) action="ok" ;; + 1) action="map-ahead-of-upstream" ;; + *) action="skip-non-semver" ;; + esac + + echo "fork=$fork mode=$mode current=$current latest=$latest cargo=$cargo_base basis=$basis action=$action" + done + echo "" + echo "source_bumps_available=$SOURCE_BUMPS_AVAILABLE" +} + +# ---- phase: label sync ----------------------------------------------------- +phase_label_sync() { + local sync_flag="--no-regen" + [[ $REGEN -eq 1 ]] && sync_flag="--regen" + + if [[ $READ_ONLY -eq 1 ]]; then + echo "" + echo "=== Label sync (read-only: sync-versions.sh --check) ===" + if "$SCRIPT_DIR/sync-versions.sh" --check >/tmp/br-label-sync.$$ 2>&1; then + LABEL_DRIFT=0 + echo " OK: no label drift" + else + LABEL_DRIFT=1 + echo " DRIFT detected (sync-versions.sh --check exit non-zero)" + grep -E '^(DRIFT|FAIL)' /tmp/br-label-sync.$$ | head -20 | sed 's/^/ /' || true + fi + rm -f /tmp/br-label-sync.$$ + return + fi + + echo "" + echo "=== Label sync: sync-versions.sh $sync_flag ===" + "$SCRIPT_DIR/sync-versions.sh" "$sync_flag" || true +} + +# ---- phase: source bumps --------------------------------------------------- +phase_source_bumps() { + if [[ $WITH_SOURCES -eq 0 ]]; then + return + fi + + if [[ $READ_ONLY -eq 1 ]]; then + echo "" + echo "(--dry-run / --check: source bumps NOT executed)" + if [[ $SOURCE_BUMPS_AVAILABLE -gt 0 ]]; then + echo "(rerun with --with-sources to perform $SOURCE_BUMPS_AVAILABLE bump(s))" + fi + return + fi + + echo "" + echo "=== Fork source bumps (--with-sources) ===" + + local idx fork url current mode latest cargo_base basis cmp ct old_ver new_ver backup_branch + local -a upgrade_args=() + for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do + fork="${MAP_FORKS[$idx]}" + url="${MAP_URLS[$idx]}" + current="${MAP_TAGS[$idx]}" + mode="${MAP_MODES[$idx]}" + + if [[ -n "$FORK_FILTER" && "$fork" != "$FORK_FILTER" ]]; then continue; fi + if [[ "$current" == "main" ]]; then continue; fi + + # Bootloader is never source-bumped, even with --force-diverged: its + # git history is genuinely unrelated to upstream (no merge-base), so + # the rebase machinery cannot work. The documented path is a one-time + # operator 'git replace --graft' intervention (RELEASE-BUMP-WORKFLOW.md). + if [[ "$fork" == "bootloader" ]]; then + echo " (bootloader: never auto-bumped — no merge-base with upstream;" + echo " one-time operator 'git replace --graft' intervention required;" + echo " see local/docs/RELEASE-BUMP-WORKFLOW.md)" + continue + fi + + case "$mode" in + snapshot|tracked) ;; + diverged) + # phase_report already showed action=report-only; with + # --force-diverged we could attempt, but the map's current + # tag is the manual base. Skip unless --force-diverged. + if [[ $FORCE_DIVERGED -eq 1 ]]; then + echo " ($fork: diverged + --force-diverged; attempting)" + else + continue + fi + ;; + *) continue ;; + esac + + cargo_base=$(fork_current_base_version "$fork") + basis="$current" + [[ -n "$cargo_base" ]] && basis="$cargo_base" + + # Reuse the report phase's ls-remote result; only fetch when this fork + # was not covered (e.g. report filtered it out). + if [[ -v "LATEST_FOR[$fork]" ]]; then + latest="${LATEST_FOR[$fork]}" + elif [[ $NO_FETCH -eq 0 ]]; then + local result raw + result=$(latest_tag_for_url "$url" "$fork" "$basis") + latest="${result%%|*}" + raw="${result##*|}" + [[ -z "$latest" && "$raw" -gt 0 ]] && latest="$basis" + else + latest="$basis" + fi + [[ -z "$latest" ]] && continue + + cmp=$(semver_cmp "$basis" "$latest") + [[ "$cmp" == "-1" ]] || continue + + echo "" + echo "--- Bumping $fork: $basis -> $latest ---" + upgrade_args=(--to="$latest") + [[ $FORCE_DIVERGED -eq 1 ]] && upgrade_args+=(--force-diverged) + upgrade_args+=("$fork") + + if ! "$SCRIPT_DIR/upgrade-forks.sh" "${upgrade_args[@]}"; then + echo " FAIL: upgrade-forks.sh ${upgrade_args[*]} failed" >&2 + echo " Leaving label + map untouched for $fork" >&2 + SOURCE_BUMPS_FAILED=$((SOURCE_BUMPS_FAILED + 1)) + continue + fi + + # On success: set fork Cargo.toml version to +rb. + # Quote-splice via awk with literal index() — sed regexes break on the + # '.'/'+' metacharacters present in versions like 0.9.0+rb0.3.1. + ct="$ROOT/local/sources/$fork/Cargo.toml" + if [[ -f "$ct" ]]; then + old_ver=$(grep -E '^version[[:space:]]*=' "$ct" | head -1 | sed 's/.*"\(.*\)".*/\1/' || true) + new_ver="${latest}+rb${BRANCH}" + if [[ -n "$old_ver" ]]; then + awk -v old="$old_ver" -v new="$new_ver" ' + $0 ~ "^version[[:space:]]*=" { + q1 = index($0, "\"") + q2 = index(substr($0, q1 + 1), "\"") + q1 + if (q1 > 0 && q2 > q1 && substr($0, q1 + 1, q2 - q1 - 1) == old) { + $0 = substr($0, 1, q1) new substr($0, q2) + } + print; next + } + { print } + ' "$ct" > "$ct.tmp.$$" && mv "$ct.tmp.$$" "$ct" + [[ -e "$ct.tmp.$$" ]] && rm -f "$ct.tmp.$$" || true + echo " Cargo.toml: $old_ver -> $new_ver" + fi + fi + + # Regen the fork's Cargo.lock (best-effort: warn on failure, continue) + if [[ -f "$ROOT/local/sources/$fork/Cargo.toml" ]]; then + if (cd "$ROOT/local/sources/$fork" && cargo generate-lockfile >/dev/null 2>&1); then + echo " Cargo.lock: regenerated" + else + echo " WARN: cargo generate-lockfile failed for $fork (continuing)" >&2 + fi + fi + + # Update the map row's column 3 in place + update_map_tag_inplace "$fork" "$latest" + echo " Map: column 3 ($current -> $latest)" + + # Report the rb-backup/* branch created by upgrade-forks.sh + backup_branch=$(git -C "$ROOT/local/sources/$fork" branch --list 'rb-backup/*' 2>/dev/null | tail -1 | sed 's/^[* ]*//' || true) + if [[ -n "$backup_branch" ]]; then + echo " Backup branch: $backup_branch" + fi + + SOURCE_BUMPS_APPLIED=$((SOURCE_BUMPS_APPLIED + 1)) + done + + # Re-sync labels after all source bumps land + echo "" + echo "=== Post-bump label sync (sync-versions.sh --no-regen) ===" + "$SCRIPT_DIR/sync-versions.sh" --no-regen || true +} + +# ---- phase: external bumps ------------------------------------------------- +phase_external() { + [[ $WITH_EXTERNAL -eq 0 ]] && return + local script="$SCRIPT_DIR/bump-graphics-recipes.sh" + if [[ ! -e "$script" ]]; then + echo "" + echo "WARN: bump-graphics-recipes.sh not present; skipping external bump" >&2 + echo " (a sibling unit is rewriting it concurrently)" >&2 + return + fi + local -a args=() + [[ $DRY_RUN -eq 1 ]] && args+=(--dry-run) + [[ $NO_FETCH -eq 1 ]] && args+=(--no-fetch) + echo "" + echo "=== External bumps (bump-graphics-recipes.sh ${args[*]:-}) ===" + "$script" "${args[@]}" || true +} + +# ---- phase: commit guide --------------------------------------------------- +print_commit_guide() { + echo "" + echo "=== Commit guide (NEVER executed automatically by this script) ===" + echo "" + echo "Parent-repo files potentially changed:" + echo " local/fork-upstream-map.toml (column-3 tag updates)" + echo " Cargo.toml (cookbook root, on label drift)" + echo " local/recipes/*/source/Cargo.toml (Cat 1 crates, on label drift)" + echo " local/sources//Cargo.toml (fork base version, on source bump)" + echo " local/sources//Cargo.lock (fork lockfile, on source bump)" + echo "" + echo "Commit them on branch '$BRANCH':" + echo " git -C \"$ROOT\" add local/fork-upstream-map.toml Cargo.toml \\" + echo " local/recipes/ local/sources/" + echo " git -C \"$ROOT\" commit -m \"release: bump fork tags + labels to $BRANCH\"" + echo "" + echo "Per-fork commits + pushes (for source bumps; replace per fork):" + if [[ -n "$FORK_FILTER" ]]; then + echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" add -A" + echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" commit -m \"bump: $FORK_FILTER to +rb$BRANCH\"" + echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" push origin submodule/$FORK_FILTER" + else + local idx fork mode current + for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do + fork="${MAP_FORKS[$idx]}" + mode="${MAP_MODES[$idx]}" + current="${MAP_TAGS[$idx]}" + [[ "$current" == "main" ]] && continue + [[ "$mode" == "diverged" ]] && continue + echo " # $fork (current tag: $current)" + echo " git -C \"$ROOT/local/sources/$fork\" add -A" + echo " git -C \"$ROOT/local/sources/$fork\" commit -m \"bump: $fork to +rb$BRANCH\"" + echo " git -C \"$ROOT/local/sources/$fork\" push origin submodule/$fork" + done + fi +} + +# ---- main ------------------------------------------------------------------ +parse_map + +phase_report +phase_label_sync +phase_source_bumps +phase_external +print_commit_guide + +if [[ $CHECK_ONLY -eq 1 ]]; then + echo "" + if [[ $LABEL_DRIFT -gt 0 ]]; then + echo "CHECK FAILED: label drift detected" + exit 1 + fi + if [[ $SOURCE_BUMPS_AVAILABLE -gt 0 ]]; then + echo "CHECK FAILED: $SOURCE_BUMPS_AVAILABLE fork(s) have available source bumps" + exit 1 + fi + echo "CHECK PASSED: no drift, no available source bumps" + exit 0 +fi + +exit 0 diff --git a/local/scripts/check-external-versions.sh b/local/scripts/check-external-versions.sh new file mode 100755 index 0000000000..fda702475a --- /dev/null +++ b/local/scripts/check-external-versions.sh @@ -0,0 +1,268 @@ +#!/usr/bin/env bash +# check-external-versions.sh — Report current vs latest upstream versions for +# every entry in local/external-upstream-map.toml. +# +# Read-only. Network is used to resolve latest versions unless --no-fetch is +# passed (in which case the on-disk cache under .redbear-recipe-bump/ is used +# and the script errors on a missing cache entry). +# +# Usage: +# check-external-versions.sh [--no-fetch] [--strict] [--json] [--help] +# +# Output (default): a fixed-width table +# name current latest STATUS +# qtbase 6.11.1 6.11.1 CURRENT +# ... +# Groups expand to one row per member; the group's resolved version is shown +# for every member, so any member whose embedded URL version differs from the +# group's is flagged OUTDATED. +# +# Exit codes: +# 0 always (default); --strict exits 1 if any entry is OUTDATED or failed +# to resolve. +# +# Map-driven: the TOML at local/external-upstream-map.toml is the single source +# of truth. See that file's header for schema and update policy. + +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +MAP="$ROOT/local/external-upstream-map.toml" + +if [ ! -f "$MAP" ]; then + echo "ERROR: map not found: $MAP" >&2 + exit 2 +fi + +# Delegate everything to an embedded Python3 script. Python (stdlib tomllib) +# gives correct TOML parsing; curl (called via subprocess) is used for the +# actual network fetches per the plan's requirement. +exec python3 - "$ROOT" "$@" <<'PYEOF' +import json +import re +import sys +import tomllib +from pathlib import Path + +ROOT = Path(sys.argv[1]) +MAP_PATH = ROOT / "local/external-upstream-map.toml" +CACHE_DIR = ROOT / ".redbear-recipe-bump" + +args = sys.argv[2:] +NO_FETCH = "--no-fetch" in args +STRICT = "--strict" in args +AS_JSON = "--json" in args +if "--help" in args or "-h" in args: + print(__doc__.replace("check-external-versions.sh", Path(sys.argv[0]).name)) + sys.exit(0) + +with open(MAP_PATH, "rb") as f: + MAP = tomllib.load(f) + + +sys.path.insert(0, str(ROOT / "local/scripts/lib")) +import external_version_lib +external_version_lib.configure(CACHE_DIR, NO_FETCH) +from external_version_lib import ( + cache_key_for, fetch, version_sort_key, apply_ver_transform, + resolve_index, render_template, member_url_name, member_template, + read_recipe_text, current_tar_url, current_rev, resolve_git_rev, +) + +def template_to_regex(template: str) -> re.Pattern: + """Convert a tar_template into a regex that captures {ver}/{majmin}/{name}. + + Handles templates where the same variable appears more than once: the first + occurrence becomes a named capture group, subsequent ones become + backreferences. {name} is greedy; {ver}/{majmin} are anchored to a leading + digit so '{name}-{ver}' doesn't split 'extra-cmake-modules' wrong. + """ + escaped = re.escape(template) + # re.escape emits '\{ver\}' for '{ver}' on Python 3.7+ + for var in ("ver", "majmin", "name"): + token = "\\{" + var + "\\}" # literal \{var\} as it appears post-escape + if token not in escaped: + continue + if var == "name": + capture = r'(?P[^/]+)' + else: + capture = f'(?P<{var}>\\d[^/]*?)' + escaped = escaped.replace(token, capture, 1) + # remaining occurrences → backreference + escaped = escaped.replace(token, f"(?P={var})") + return re.compile(escaped) + + +def extract_version_vars(url: str, template: str): + """Reverse-match url against template to recover {ver}/{majmin}.""" + try: + rx = template_to_regex(template) + except re.error: + return {} + m = rx.search(url) + if not m: + return {} + d = m.groupdict() + out = {} + if d.get("ver"): + out["ver"] = d["ver"] + if d.get("majmin"): + out["majmin"] = d["majmin"] + return out + + +# --- main ----------------------------------------------------------------- + +rows = [] # list of dicts: name, current, latest, status, note +summary = {"current": 0, "outdated": 0, "failed": 0} + +# Groups: resolve the group version ONCE, then expand to members. +for gid, g in MAP.get("groups", {}).items(): + vars_dict, src, err = resolve_index( + g["version_index"], g["version_regex"], g.get("resolve", "version")) + group_latest = vars_dict["ver"] if vars_dict else "?" + + for member in g["members"]: + recipe_path = ROOT / g["recipe_root"] / member + tmpl = member_template(g, member) + uname = member_url_name(g, member) + text = read_recipe_text(recipe_path) + cur_url = current_tar_url(text) + if cur_url is None: + rows.append({ + "name": f"{gid}:{member}", "current": "?", + "latest": group_latest, "status": "FAILED", + "note": "no-tar-line", + }) + summary["failed"] += 1 + continue + # construct what the latest tar URL would be + if vars_dict is None: + status = "FAILED" + note = f"index:{err}" + else: + render_vars = dict(vars_dict) + render_vars["name"] = uname + latest_url = render_template(tmpl, render_vars) + cur_vars = extract_version_vars(cur_url, tmpl) + cur_ver = cur_vars.get("ver", "?") + if latest_url == cur_url: + status = "CURRENT" + note = "" + else: + status = "OUTDATED" + note = "" + rows.append({ + "name": f"{gid}:{member}", "current": cur_ver, + "latest": group_latest, "status": status, "note": note, + }) + if status == "CURRENT": + summary["current"] += 1 + else: + summary["outdated"] += 1 + continue + # failed resolution path + cur_vars = extract_version_vars(cur_url, tmpl) + rows.append({ + "name": f"{gid}:{member}", "current": cur_vars.get("ver", "?"), + "latest": "?", "status": status, "note": note, + }) + summary["failed"] += 1 + +# Packages (singletons) +for pid, p in MAP.get("packages", {}).items(): + recipe_path = ROOT / p["recipe"] + text = read_recipe_text(recipe_path) + kind = p["kind"] + if kind == "tar": + cur_url = current_tar_url(text) + tmpl = p["tar_template"] + cur_vars = extract_version_vars(cur_url or "", tmpl) + cur_ver = cur_vars.get("ver", "?") + vars_dict, src, err = resolve_index( + p["version_index"], p["version_regex"], p.get("resolve", "version"), + p.get("ver_transform")) + if vars_dict is None: + rows.append({ + "name": pid, "current": cur_ver, "latest": "?", + "status": "FAILED", "note": f"index:{err}", + }) + summary["failed"] += 1 + continue + latest_url = render_template(tmpl, {**vars_dict, "name": pid}) + status = "CURRENT" if latest_url == cur_url else "OUTDATED" + rows.append({ + "name": pid, "current": cur_ver, "latest": vars_dict["ver"], + "status": status, "note": "", + }) + if status == "CURRENT": + summary["current"] += 1 + else: + summary["outdated"] += 1 + elif kind == "git-rev": + cur = current_rev(text) or "?" + resolved, err = resolve_git_rev(p["git_url"], p["tag_regex"]) + if resolved is None: + rows.append({ + "name": pid, "current": cur[:12], "latest": "?", + "status": "FAILED", "note": f"git:{err}", + }) + summary["failed"] += 1 + continue + latest_sha = resolved["sha"] + latest_tag = resolved["tag"] + # current may be either a sha or a tag name + is_current = ( + cur == latest_sha or cur.startswith(latest_sha[:12]) or + cur == latest_tag + ) + status = "CURRENT" if is_current else "OUTDATED" + rows.append({ + "name": pid, "current": cur[:12] if len(cur) >= 40 else cur, + "latest": f"{latest_tag}@{latest_sha[:12]}", + "status": status, "note": "", + }) + if status == "CURRENT": + summary["current"] += 1 + else: + summary["outdated"] += 1 + else: + rows.append({ + "name": pid, "current": "?", "latest": "?", + "status": "FAILED", "note": f"unknown-kind:{kind}", + }) + summary["failed"] += 1 + + +# --- output --------------------------------------------------------------- + +if AS_JSON: + payload = { + "summary": summary, + "rows": rows, + } + print(json.dumps(payload, indent=2)) +else: + # fixed-width table + name_w = max(len("name"), max((len(r["name"]) for r in rows), default=0)) + cur_w = max(len("current"), max((len(str(r["current"])) for r in rows), default=0)) + lat_w = max(len("latest"), max((len(str(r["latest"])) for r in rows), default=0)) + hdr = (f"{'name':<{name_w}} {'current':<{cur_w}} " + f"{'latest':<{lat_w}} STATUS") + print(hdr) + print("-" * len(hdr)) + for r in rows: + note = f" ({r['note']})" if r.get("note") else "" + print(f"{r['name']:<{name_w}} {str(r['current']):<{cur_w}} " + f"{str(r['latest']):<{lat_w}} {r['status']}{note}") + print() + print(f"summary: current={summary['current']} " + f"outdated={summary['outdated']} failed={summary['failed']}") + if NO_FETCH: + print("(from cache; --no-fetch)") + +exit_code = 0 +if STRICT and (summary["outdated"] > 0 or summary["failed"] > 0): + exit_code = 1 +sys.exit(exit_code) +PYEOF diff --git a/local/scripts/install-git-hooks.sh b/local/scripts/install-git-hooks.sh new file mode 100755 index 0000000000..86d7746664 --- /dev/null +++ b/local/scripts/install-git-hooks.sh @@ -0,0 +1,200 @@ +#!/usr/bin/env bash +# install-git-hooks.sh — Idempotent installer for Red Bear OS optional git hooks. +# +# Hooks in Red Bear OS are ALWAYS opt-in (per AGENTS.md "absolutely NEVER +# DELETE, NEVER IGNORE" rule — operators explicitly choose to install them). +# This script is a convenience wrapper around the manual `cp` instructions +# documented in local/docs/HOOKS.md. It never silently clobbers an existing +# hook: if a hook file already exists and differs from the canonical copy, +# it is backed up to `.bak` before the new one is written, and a +# warning is printed. +# +# Usage: +# ./local/scripts/install-git-hooks.sh # install post-checkout only (default) +# ./local/scripts/install-git-hooks.sh --all # post-checkout + pre-push + commit-msg +# ./local/scripts/install-git-hooks.sh --uninstall # remove hooks installed by this script +# ./local/scripts/install-git-hooks.sh --uninstall --all +# +# Exit codes: +# 0 success (or already-installed, or nothing to uninstall) +# 1 fatal misconfiguration (not a git repo, scripts missing, etc.) +# +# See: local/docs/HOOKS.md (hook inventory + bypass instructions), +# local/docs/RELEASE-BUMP-WORKFLOW.md (release-bump pipeline). + +set -euo pipefail + +# --------------------------------------------------------------------------- +# Resolve repo root. Must be a Red Bear OS checkout (has local/scripts/). +# --------------------------------------------------------------------------- + +ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)" +if [[ -z "$ROOT" ]]; then + echo "install-git-hooks: not inside a git work tree" >&2 + exit 1 +fi + +SCRIPTS_DIR="$ROOT/local/scripts" +HOOKS_DIR="$(git rev-parse --git-path hooks 2>/dev/null)" +# `git rev-parse --git-path hooks` resolves correctly for linked worktrees +# and custom $GIT_DIR. Fall back to the conventional path if it fails. +if [[ -z "$HOOKS_DIR" ]]; then + GIT_DIR="$(git rev-parse --git-dir 2>/dev/null || echo .git)" + [[ "$GIT_DIR" = /* ]] || GIT_DIR="$ROOT/$GIT_DIR" + HOOKS_DIR="$GIT_DIR/hooks" +fi + +mkdir -p "$HOOKS_DIR" + +# --------------------------------------------------------------------------- +# Parse flags. +# --------------------------------------------------------------------------- + +INSTALL_ALL=0 +UNINSTALL=0 +for arg in "$@"; do + case "$arg" in + --all) INSTALL_ALL=1 ;; + --uninstall) UNINSTALL=1 ;; + -h|--help) + sed -n '1,30p' "$0" + exit 0 + ;; + *) + echo "install-git-hooks: unknown option: $arg" >&2 + echo " usage: $0 [--all] [--uninstall]" >&2 + exit 1 + ;; + esac +done + +# --------------------------------------------------------------------------- +# Hook manifest. The default install set is intentionally minimal (only +# post-checkout) so that a bare `install-git-hooks.sh` does the least +# surprising thing. `--all` extends to the pre-push and commit-msg hooks. +# --------------------------------------------------------------------------- + +# Each entry: ; +HOOKS_DEFAULT=( + "post-checkout;post-checkout-version-sync.sh" +) + +HOOKS_EXTRA=( + "pre-push;pre-push-checks.sh" + "commit-msg;commit-msg" +) + +# Build the active set from the flags. +ACTIVE_HOOKS=() +if [[ $UNINSTALL -eq 1 ]]; then + # Uninstall needs to know the full set that --all would have installed, + # so a default-install-then---all-uninstall cleans everything up. + for entry in "${HOOKS_DEFAULT[@]}" "${HOOKS_EXTRA[@]}"; do + ACTIVE_HOOKS+=("$entry") + done + if [[ $INSTALL_ALL -eq 0 ]]; then + # Without --all, only uninstall the default set (the ones a default + # install would have placed). Operators who used --all should pass + # --uninstall --all. + ACTIVE_HOOKS=("${HOOKS_DEFAULT[@]}") + fi +else + ACTIVE_HOOKS=("${HOOKS_DEFAULT[@]}") + [[ $INSTALL_ALL -eq 1 ]] && ACTIVE_HOOKS+=("${HOOKS_EXTRA[@]}") +fi + +# --------------------------------------------------------------------------- +# Install / uninstall loop. +# --------------------------------------------------------------------------- + +install_one() { + local hook_name="$1" + local src_rel="$2" + local src="$SCRIPTS_DIR/$src_rel" + local dst="$HOOKS_DIR/$hook_name" + + if [[ ! -f "$src" ]]; then + echo "install-git-hooks: WARN source script not found: $src_rel — skipping $hook_name" >&2 + return 0 + fi + + if [[ -e "$dst" || -L "$dst" ]]; then + # Content-based, not path-based: a copied hook has a different path + # than the source, so only byte comparison detects "already current". + if cmp -s "$dst" "$src" 2>/dev/null; then + echo "install-git-hooks: $hook_name already up to date ($dst)" + chmod +x "$dst" 2>/dev/null || true + return 0 + fi + # Existing file differs — back it up before overwriting. + local bak="${dst}.bak" + # Rotate any pre-existing .bak so we don't destroy the first backup. + if [[ -e "$bak" ]]; then + local i=1 + while [[ -e "${dst}.bak.${i}" && $i -le 100 ]]; do i=$((i+1)); done + if [[ $i -gt 100 ]]; then + bak="${dst}.bak.$(date +%Y%m%d%H%M%S)-$$" + else + bak="${dst}.bak.${i}" + fi + fi + echo "install-git-hooks: WARN existing $hook_name differs — backing up to $bak" >&2 + mv "$dst" "$bak" + fi + + cp "$src" "$dst" + chmod +x "$dst" + echo "install-git-hooks: installed $hook_name -> $dst" +} + +uninstall_one() { + local hook_name="$1" + local dst="$HOOKS_DIR/$hook_name" + + if [[ ! -e "$dst" && ! -L "$dst" ]]; then + echo "install-git-hooks: $hook_name not present ($dst) — nothing to remove" + return 0 + fi + rm -f "$dst" + echo "install-git-hooks: removed $hook_name ($dst)" + # Restore the most recent backup if one exists, so uninstall is reversible. + if [[ -e "${dst}.bak" ]]; then + mv "${dst}.bak" "$dst" + chmod +x "$dst" 2>/dev/null || true + echo "install-git-hooks: restored previous $hook_name from ${dst}.bak" + fi +} + +echo "=== Red Bear OS git hook installer ===" +echo " repo root: $ROOT" +echo " hooks dir: $HOOKS_DIR" +echo " mode: $([[ $UNINSTALL -eq 1 ]] && echo 'uninstall' || echo 'install') ($([[ $INSTALL_ALL -eq 1 ]] && echo 'all hooks' || echo 'default: post-checkout only'))" +echo "" + +if [[ $UNINSTALL -eq 1 ]]; then + for entry in "${ACTIVE_HOOKS[@]}"; do + hook_name="${entry%%;*}" + uninstall_one "$hook_name" + done + echo "" + echo "Uninstall complete. Remaining hooks (if any):" + ls -1 "$HOOKS_DIR" 2>/dev/null | grep -vE '\.bak(\.[0-9]+)?$' || echo " (none)" + exit 0 +fi + +for entry in "${ACTIVE_HOOKS[@]}"; do + hook_name="${entry%%;*}" + src_rel="${entry##*;}" + install_one "$hook_name" "$src_rel" +done + +echo "" +echo "=== Install complete ===" +echo "" +echo "Installed hooks are OPT-IN and never auto-commit, auto-push, or block" +echo "checkouts. Bypass / configuration:" +echo " post-checkout: REDBEAR_NO_AUTO_SYNC=1 git checkout " +echo " pre-push: REDBEAR_SKIP_PRE_PUSH=1 git push (or git push --no-verify)" +echo "" +echo "Full documentation: local/docs/HOOKS.md" +echo "Release-bump runbook: local/docs/RELEASE-BUMP-WORKFLOW.md" diff --git a/local/scripts/lib/external_version_lib.py b/local/scripts/lib/external_version_lib.py new file mode 100644 index 0000000000..d966e75a19 --- /dev/null +++ b/local/scripts/lib/external_version_lib.py @@ -0,0 +1,213 @@ +"""Shared upstream-version resolution helpers for Red Bear external-package tooling. + +Consumers (add local/scripts/lib to sys.path, then import): + - local/scripts/check-external-versions.sh (embedded Python) + - local/scripts/bump-graphics-recipes.sh (embedded Python) + +Single source of truth for: index fetching (cache + fallback), version sort +keys, sed-style version transforms, index resolution, tar-template rendering, +group member helpers, recipe.toml source pins, and git ls-remote tag +resolution. Behavior changes belong here, not in the consumers. +""" + +import hashlib +import os +import re +import subprocess +from pathlib import Path +from typing import Optional + +_CACHE_DIR: Optional[Path] = None +_NO_FETCH = False + + +def configure(cache_dir: Path, no_fetch: bool) -> None: + global _CACHE_DIR, _NO_FETCH + _CACHE_DIR = Path(cache_dir) + _NO_FETCH = bool(no_fetch) + + +def cache_key_for(url: str) -> Path: + assert _CACHE_DIR is not None, "configure() must be called before fetch use" + safe = re.sub(r'[^a-zA-Z0-9_.-]', '_', url) + if len(safe) > 180: + # Stable hash (blake2b) — NOT Python's hash(), which is randomized + # per-process via PYTHONHASHSEED and would orphan cache entries. + digest = hashlib.blake2b(url.encode(), digest_size=8).hexdigest() + safe = safe[:80] + "___" + digest + "___" + safe[-80:] + return _CACHE_DIR / f"{safe}.cache" + + +def fetch(url: str): + """Return (text, source); source is fetch/cache/cache-fallback*/error.""" + assert _CACHE_DIR is not None, "configure() must be called before fetch use" + cache = cache_key_for(url) + if _NO_FETCH: + if cache.exists(): + return cache.read_text(errors="replace"), "cache" + return None, "no-cache" + try: + result = subprocess.run( + ["curl", "-sLf", "--retry", "3", "--retry-delay", "2", + "--connect-timeout", "20", "--max-time", "90", url], + capture_output=True, text=True, timeout=120, + ) + if result.returncode == 0 and result.stdout: + _CACHE_DIR.mkdir(parents=True, exist_ok=True) + tmp = cache.with_suffix(cache.suffix + ".tmp") + tmp.write_text(result.stdout) + os.replace(str(tmp), str(cache)) + return result.stdout, "fetch" + if cache.exists(): + return cache.read_text(errors="replace"), "cache-fallback" + err = (result.stderr or "").strip() + return None, err or f"curl-exit-{result.returncode}" + except subprocess.TimeoutExpired: + if cache.exists(): + return cache.read_text(errors="replace"), "cache-fallback-timeout" + return None, "timeout" + except Exception as e: + if cache.exists(): + return cache.read_text(errors="replace"), "cache-fallback-error" + return None, str(e) + + +def version_sort_key(v: str): + """Sort key that orders pre-releases BEFORE the corresponding final. + + Handles forms like 1.0.0, 1.0.0-rc1, 1.0.0-alpha1, 1.0.0-beta2. + A final release sorts after any pre-release of the same X.Y.Z base. + """ + m = re.match(r'^(\d+(?:[._]\d+)*)(?:[-.]([0-9A-Za-z]+))?$', v) + if not m: + parts = re.split(r'[._+-]', v) + return [(1, 0, p) if not p.isdigit() else (0, int(p), "") for p in parts] + main_str, pre = m.group(1), m.group(2) + main_key = [(0, int(p), "") if p.isdigit() else (1, 0, p) + for p in re.split(r'[._]', main_str)] + if pre is None or pre == "": + pre_key = [(1, 0, "zzzzzz_final")] + else: + pre_key = [(-1, 0, pre)] + return main_key + pre_key + + +def apply_ver_transform(ver: str, transform: str) -> str: + """Apply a sed-style transform like 's/-/./g' to a version token.""" + m = re.match(r'^s/(.*)/(.*)/([gi]*)$', transform) + if not m: + return ver + pat, repl, flags = m.group(1), m.group(2), m.group(3) + count = 0 if 'g' in flags else 1 + try: + return re.sub(pat, repl, ver, count=count) + except re.error: + return ver + + +def resolve_index(index_url: str, regex_str: str, resolve_kind: str, + ver_transform: Optional[str] = None): + """Resolve latest version vars from an index page -> (vars, source, error).""" + text, src = fetch(index_url) + if text is None: + return None, src, "fetch-failed" + try: + pattern = re.compile(regex_str) + except re.error as e: + return None, src, f"bad-regex: {e}" + matches = pattern.findall(text) + if not matches: + return None, src, "no-match" + norm = [] + for mm in matches: + if isinstance(mm, tuple): + mm = "-".join(str(x) for x in mm) + norm.append(mm) + unique = list(dict.fromkeys(norm)) + unique.sort(key=version_sort_key, reverse=True) + best = unique[0] + if ver_transform: + best = apply_ver_transform(best, ver_transform) + if resolve_kind == "majmin": + return {"majmin": best, "ver": f"{best}.0"}, src, None + parts = best.split(".") + majmin = ".".join(parts[:2]) if len(parts) >= 2 else best + return {"majmin": majmin, "ver": best}, src, None + + +def render_template(template: str, variables: dict) -> str: + out = template + for k in ("ver", "majmin", "name"): + out = out.replace("{" + k + "}", variables.get(k, "")) + return out + + +def member_url_name(group: dict, member: str) -> str: + overrides = group.get("member_url_names", {}) or {} + if member in overrides: + return overrides[member] + strip = group.get("url_name_strip_prefix", "") or "" + if strip and member.startswith(strip): + return member[len(strip):] + return member + + +def member_template(group: dict, member: str) -> str: + mts = group.get("member_templates", {}) or {} + if member in mts and "tar_template" in mts[member]: + return mts[member]["tar_template"] + return group["tar_template"] + + +def read_recipe_text(recipe_path: Path) -> str: + toml = recipe_path / "recipe.toml" + if not toml.exists(): + return "" + return toml.read_text(errors="replace") + + +def current_tar_url(text: str): + m = re.search(r'^\s*tar\s*=\s*"([^"]+)"', text, re.MULTILINE) + return m.group(1) if m else None + + +def current_rev(text: str): + m = re.search(r'^\s*rev\s*=\s*"([^"]+)"', text, re.MULTILINE) + return m.group(1) if m else None + + +def resolve_git_rev(git_url: str, tag_regex_str: str): + """Resolve latest tag + commit sha via git ls-remote --tags.""" + try: + result = subprocess.run( + ["git", "ls-remote", "--tags", git_url], + capture_output=True, text=True, timeout=90, + ) + except subprocess.TimeoutExpired: + return None, "timeout" + except Exception as e: + return None, str(e) + if result.returncode != 0: + return None, (result.stderr or "git-error").strip() + try: + pattern = re.compile(tag_regex_str) + except re.error as e: + return None, f"bad-regex: {e}" + shas = {} + for line in result.stdout.splitlines(): + parts = line.split("\t") + if len(parts) != 2: + continue + sha, ref = parts + if not ref.startswith("refs/tags/"): + continue + tag = ref[len("refs/tags/"):] + deref = tag.endswith("^{}") + clean = tag[:-3] if deref else tag + if pattern.fullmatch(clean): + if clean not in shas or deref: + shas[clean] = sha.strip() + if not shas: + return None, "no-tags-matched" + best = max(shas.keys(), key=version_sort_key) + return {"tag": best, "sha": shas[best]}, None diff --git a/local/scripts/post-checkout-version-sync.sh b/local/scripts/post-checkout-version-sync.sh new file mode 100755 index 0000000000..62f028f8d4 --- /dev/null +++ b/local/scripts/post-checkout-version-sync.sh @@ -0,0 +1,214 @@ +#!/usr/bin/env bash +# post-checkout-version-sync.sh — Optional post-checkout hook for Red Bear OS. +# +# Purpose: +# When an operator switches to a release branch whose name is an anchored +# semver (e.g. `0.3.1`, `0.3.2`), automatically synchronise every Cat 1 +# (in-house) and Cat 2 (upstream fork) crate version label to match that +# branch. This is the label-only half of a release bump — it does NOT +# touch fork source content, regenerate lockfiles, or commit anything. +# +# Why label-only: +# Labels are cheap and mechanical (sed rewrites performed by +# sync-versions.sh). Source upgrades (rebasing a fork onto a newer upstream +# tag) and lockfile regen are expensive and operator-judgement steps, so +# they remain explicit (`bump-release.sh --with-sources --with-external` +# and `sync-versions.sh --regen`). See +# local/docs/RELEASE-BUMP-WORKFLOW.md for the full release model. +# +# Git post-checkout contract (git invokes this hook with three arguments): +# $1 = previous HEAD sha +# $2 = new HEAD sha +# $3 = 1 for a branch checkout, 0 for a file checkout +# +# Install (operator opt-in — NEVER auto-installed): +# cp local/scripts/post-checkout-version-sync.sh .git/hooks/post-checkout +# chmod +x .git/hooks/post-checkout +# +# Or via the installer (preferred): +# ./local/scripts/install-git-hooks.sh # post-checkout only +# ./local/scripts/install-git-hooks.sh --all # + pre-push + commit-msg +# +# Bypass for a single checkout: +# REDBEAR_NO_AUTO_SYNC=1 git checkout 0.3.2 +# +# Hard guarantees (a failing hook MUST NEVER block a checkout): +# - Always exits 0. Any internal failure degrades to a stderr note + exit 0. +# - No network access. No git mutations (no commit / branch / push / stash). +# - No lockfile regeneration (NEVER calls `sync-versions.sh --regen`). +# - Does not run on non-semver branches (master, submodule/*, recovered/*, +# detached HEAD). +# - Does not run inside an in-progress rebase / cherry-pick / merge. +# - Does not run when the working tree is dirty (warns + skips). +# - Total runtime budget: < ~5 seconds on a warm tree. +# +# See: local/docs/HOOKS.md, local/docs/RELEASE-BUMP-WORKFLOW.md, +# local/AGENTS.md § "Version conventions — two categories". + +# NOTE: deliberately `set -uo pipefail` and NOT `-e`. A post-checkout hook +# must never abort mid-way and leave the operator staring at a non-zero exit +# that blocks the checkout. Every fallible step is either guarded or wrapped +# so that failure degrades to an exit 0 with a stderr note. +set -uo pipefail + +# --------------------------------------------------------------------------- +# Locate the repo root. We prefer `git rev-parse --show-toplevel` over a +# dirname-relative derivation because git places the hook under `.git/hooks/` +# which may itself be a gitdir pointer (worktrees, linked checkouts). +# --------------------------------------------------------------------------- + +ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)" +if [[ -z "$ROOT" ]]; then + # Not inside a git work tree — nothing to do. + exit 0 +fi + +# Git passes exactly three arguments to post-checkout. If invoked manually +# without them, treat $3 as unknown (skip the branch-flag guard). +BRANCH_FLAG="${3:-}" + +# --------------------------------------------------------------------------- +# Guard (a): branch checkouts only. $3 == 1 means a branch checkout; +# $3 == 0 means a file checkout (e.g. `git checkout -- path`). Silently +# skip file checkouts — there is no branch context to sync against. +# --------------------------------------------------------------------------- + +if [[ "$BRANCH_FLAG" != "1" ]]; then + exit 0 +fi + +# --------------------------------------------------------------------------- +# Guard (b): the new branch name MUST be an anchored semver +# `^[0-9]+\.[0-9]+\.[0-9]+$`. This silently excludes master, main, +# submodule/* branches, recovered/* branches, and detached HEAD. +# --------------------------------------------------------------------------- + +BRANCH="$(git branch --show-current 2>/dev/null || true)" +if [[ -z "$BRANCH" ]]; then + # Detached HEAD — no branch name to anchor a version against. + exit 0 +fi + +if ! printf '%s' "$BRANCH" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then + # Non-semver branch (master, submodule/*, recovered/*, feature/*, …). + # Silently skip — version labels are only meaningful on release branches. + exit 0 +fi + +# --------------------------------------------------------------------------- +# Guard (c): never run inside an in-progress rebase / cherry-pick / merge. +# Touching version labels mid-sequence would corrupt the operator's +# in-flight history rewrite. Silently skip in all such states. +# --------------------------------------------------------------------------- + +GIT_DIR="$(git rev-parse --git-dir 2>/dev/null || true)" +if [[ -n "$GIT_DIR" ]]; then + # Resolve relative to the worktree root for linked worktrees. + [[ "$GIT_DIR" = /* ]] || GIT_DIR="$ROOT/$GIT_DIR" + for marker in rebase-merge rebase-apply CHERRY_PICK_HEAD MERGE_HEAD; do + if [[ -e "$GIT_DIR/$marker" ]]; then + # An interactive operation is in progress — stay out of the way. + exit 0 + fi + done +fi + +# --------------------------------------------------------------------------- +# Guard (d): honour the explicit opt-out env var. +# --------------------------------------------------------------------------- + +if [[ -n "${REDBEAR_NO_AUTO_SYNC:-}" ]]; then + exit 0 +fi + +# --------------------------------------------------------------------------- +# Guard (e): working tree MUST be clean. A bump rewrites Cargo.toml files in +# place; running it over uncommitted operator edits would silently swallow +# those edits into the bump. Warn + skip instead. +# --------------------------------------------------------------------------- + +if ! git -C "$ROOT" diff --quiet --ignore-submodules HEAD 2>/dev/null \ + || [[ -n "$(git -C "$ROOT" status --porcelain 2>/dev/null)" ]]; then + echo "post-checkout-version-sync: skipping version auto-sync: uncommitted changes present" >&2 + echo " (commit or stash, then run: ./local/scripts/sync-versions.sh --no-regen)" >&2 + exit 0 +fi + +# --------------------------------------------------------------------------- +# All guards passed. Compare the root cookbook Cargo.toml `[package] version` +# against the branch version. They are the canonical indicator of whether a +# label sync is needed: sync-versions.sh Phase 0 keeps them in lock-step, +# so if the root is already at the branch version, every Cat 1/Cat 2 label +# is already correct too. +# --------------------------------------------------------------------------- + +ROOT_CARGO="$ROOT/Cargo.toml" +if [[ ! -f "$ROOT_CARGO" ]]; then + # Not a Red Bear OS checkout (no root Cargo.toml). Stay silent. + exit 0 +fi + +CURRENT_VERSION="" +if grep -qE '^\[package\]' "$ROOT_CARGO" 2>/dev/null; then + # Must match sync-versions.sh Phase 0's extraction so both scripts agree. + CURRENT_VERSION="$(awk ' + /^\[package\]/ { in_pkg=1; next } + /^\[/ { in_pkg=0 } + in_pkg && /^version[[:space:]]*=/ { print; exit } + ' "$ROOT_CARGO" 2>/dev/null | sed 's/.*"\(.*\)".*/\1/' || true)" +fi + +if [[ "$CURRENT_VERSION" == "$BRANCH" ]]; then + # Labels already synced for this branch — nothing to do. + exit 0 +fi + +# --------------------------------------------------------------------------- +# Drift detected. Run the label-only sync (NEVER --regen — lockfile regen is +# an explicit operator step). sync-versions.sh is the single executor of +# suffix-only label rewrites; this hook never duplicates that sed logic. +# --------------------------------------------------------------------------- + +SYNC="$ROOT/local/scripts/sync-versions.sh" +if [[ ! -x "$SYNC" ]]; then + echo "post-checkout-version-sync: $SYNC not found or not executable — skipping label sync" >&2 + echo " run manually: ./local/scripts/sync-versions.sh --no-regen" >&2 + exit 0 +fi + +echo "post-checkout-version-sync: branch '$BRANCH' root Cargo.toml at '$CURRENT_VERSION' -> '$BRANCH'" +echo "post-checkout-version-sync: running sync-versions.sh --no-regen (labels only, no lockfile regen)" + +# Run label sync. Capture output so the operator sees a concise summary. +# We do NOT propagate a non-zero exit: even if sync fails, the checkout +# itself must not be blocked. +SYNC_OUTPUT="$("$SYNC" --no-regen 2>&1)" || true +if [[ -n "$SYNC_OUTPUT" ]]; then + printf '%s\n' "$SYNC_OUTPUT" +fi + +echo "post-checkout-version-sync: label sync complete for branch '$BRANCH'." + +# --------------------------------------------------------------------------- +# Follow-up hint. The hook handles ONLY mechanical label rewrites. The +# expensive half of a release bump — fork source upgrades to newer upstream +# tags and external desktop-stack version refresh — is the operator's +# explicit decision and is never triggered automatically. +# --------------------------------------------------------------------------- + +cat >&2 < # TODO: classify mode` +# at the end of the file. +# +# Usage: +# ./local/scripts/refresh-fork-upstream-map.sh # Update col 3 in place +# ./local/scripts/refresh-fork-upstream-map.sh --check # Read-only report, exit 1 on drift +# ./local/scripts/refresh-fork-upstream-map.sh --no-fetch # Use map's current tags (no network) +# ./local/scripts/refresh-fork-upstream-map.sh --help # This header +# +# Acceptance: +# - `git diff local/fork-upstream-map.toml` shows only column-3 token changes +# (if any) plus any newly appended rows. +# - `grep -c '^base\b' local/fork-upstream-map.toml` = 1. +# - `grep -c diverged local/fork-upstream-map.toml` unchanged vs before. set -euo pipefail @@ -11,45 +36,269 @@ cd "$ROOT" MAP="$ROOT/local/fork-upstream-map.toml" -# fork_name upstream_url latest_tag -declare -A FORKS=( - [syscall]="https://gitlab.redox-os.org/redox-os/syscall.git" - [libredox]="https://gitlab.redox-os.org/redox-os/libredox.git" - [redoxfs]="https://gitlab.redox-os.org/redox-os/redoxfs.git" - [redox-scheme]="https://gitlab.redox-os.org/redox-os/redox-scheme.git" - [relibc]="https://gitlab.redox-os.org/redox-os/relibc.git" - [kernel]="https://gitlab.redox-os.org/redox-os/kernel.git" - [bootloader]="https://gitlab.redox-os.org/redox-os/bootloader.git" - [installer]="https://gitlab.redox-os.org/redox-os/installer.git" - [userutils]="https://gitlab.redox-os.org/redox-os/userutils.git" -) +CHECK_ONLY=0 +NO_FETCH=0 -{ - echo "# fork-upstream-map.toml — Auto-generated by" - echo "# local/scripts/refresh-fork-upstream-map.sh. Do not edit by hand." - echo "# Format: " +for arg in "$@"; do + case "$arg" in + --check) CHECK_ONLY=1 ;; + --no-fetch) NO_FETCH=1 ;; + -h|--help) sed -n '2,35p' "$0"; exit 0 ;; + *) echo "Unknown option: $arg" >&2; exit 1 ;; + esac +done + +if [[ ! -f "$MAP" ]]; then + echo "ERROR: map file not found: $MAP" >&2 + exit 1 +fi + +# Anchored semver regex (release-bump-pipeline.md design constraint #2). +SEMVER_RE='^[0-9]+\.[0-9]+\.[0-9]+$' + +# ---- latest_tag_for_url [fork] [base] -------------------------------- +# Resolve the newest GENUINE upgrade tag for a remote via `git ls-remote`. +# Output: "|" — tag empty when nothing qualifies. +# Two filters: (a) tags already reachable from the fork's HEAD are history, +# not newer releases (relibc's 0.6.0/2020 outranks its newer 0.2.x API line +# lexically); (b) tags not version-greater than the current base are not +# newer either (the base tag itself is usually an ancestor of HEAD). +latest_tag_for_url() { + local url="$1" fork="${2:-}" base="${3:-}" + local tag forkdir="" raw=0 + [[ -n "$fork" && -d "$ROOT/local/sources/$fork/.git" ]] \ + && forkdir="$ROOT/local/sources/$fork" + while IFS= read -r tag; do + [[ -z "$tag" ]] && continue + raw=$((raw + 1)) + if [[ -n "$forkdir" ]] \ + && git -C "$forkdir" rev-parse --verify --quiet "refs/tags/$tag" >/dev/null 2>&1 \ + && git -C "$forkdir" merge-base --is-ancestor "refs/tags/$tag" HEAD 2>/dev/null; then + continue + fi + if [[ -n "$base" ]] && ! version_greater "$tag" "$base"; then + continue + fi + echo "${tag}|${raw}" + return 0 + done < <(git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \ + | awk '{print $2}' \ + | sed 's|refs/tags/||' \ + | sed 's|\^{}$||' \ + | grep -E "$SEMVER_RE" || true) + echo "|${raw}" + return 0 +} + +# version_greater → true iff a is version-greater than b (sort -V). +version_greater() { + [[ "$1" != "$2" ]] \ + && [[ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | tail -1)" == "$1" ]] +} + +# ---- parse existing map into associative arrays ---------------------------- +# Sets MAP_FORK_URL[fork]=url, MAP_FORK_TAG[fork]=tag, MAP_FORK_MODE[fork]=mode. +declare -A MAP_FORK_URL=() +declare -A MAP_FORK_TAG=() +declare -A MAP_FORK_MODE=() + +# Match a data row: first non-whitespace, non-# token = fork; followed by +# whitespace-separated url, tag, optional mode + trailing comment. +# We use awk to emit TAB-separated columns; bash reads them in. +while IFS=$'\t' read -r fork url tag mode; do + [[ -z "$fork" ]] && continue + MAP_FORK_URL["$fork"]="$url" + MAP_FORK_TAG["$fork"]="$tag" + MAP_FORK_MODE["$fork"]="$mode" +done < <(awk ' + /^[^#[:space:]]/ { + fork = $1 + url = $2 + tag = $3 + mode = $4 + sub(/[ \t]*#.*/, "", mode) + print fork "\t" url "\t" tag "\t" mode + } +' "$MAP") + +# ---- collect on-disk forks with upstream remote ---------------------------- +declare -A DISK_FORK_URL=() +for d in "$ROOT"/local/sources/*/; do + [[ -d "$d/.git" ]] || continue + name=$(basename "$d") + url=$(git -C "$d" remote get-url upstream 2>/dev/null || true) + [[ -n "$url" ]] && DISK_FORK_URL["$name"]="$url" +done + +# ---- plan column-3 updates + appends --------------------------------------- +declare -A NEW_TAG_FOR=() +declare -a APPEND_ROWS=() +DRIFT_COUNT=0 +UPDATE_COUNT=0 + +echo "=== refresh-fork-upstream-map ===" +if [[ "$NO_FETCH" -eq 1 ]]; then + echo "(--no-fetch: skipping git ls-remote; using current map tags)" +fi +echo "" + +# Iterate in map-order (preserve stable output for --check reports). +MAP_ORDER=() +while IFS= read -r line; do + if [[ "$line" =~ ^[^#[:space:]] ]]; then + fork=$(awk '{print $1}' <<<"$line") + MAP_ORDER+=("$fork") + fi +done < <(awk 1 "$MAP") + +for fork in "${MAP_ORDER[@]}"; do + url="${MAP_FORK_URL[$fork]:-}" + current="${MAP_FORK_TAG[$fork]:-}" + mode="${MAP_FORK_MODE[$fork]:-}" + + # Skip the base row (tag=main). + if [[ "$current" == "main" ]]; then + echo "fork=$fork mode=$mode current=$current latest=(skip) action=skip-main" + continue + fi + + # Skip diverged forks: tag documents CURRENT base. + if [[ "$mode" == "diverged" ]]; then + echo "fork=$fork mode=diverged current=$current latest=(skip) action=skip-diverged" + continue + fi + + latest="" + if [[ "$NO_FETCH" -eq 0 ]]; then + result=$(latest_tag_for_url "$url" "$fork" "$current") + latest="${result%%|*}" + raw="${result##*|}" + # Tags existed upstream but none qualified as newer → already current. + [[ -z "$latest" && "$raw" -gt 0 ]] && latest="$current" + fi + + if [[ -z "$latest" ]]; then + echo "fork=$fork mode=$mode current=$current latest=(unknown) action=skip-no-tag" + continue + fi + + if [[ "$latest" != "$current" ]]; then + NEW_TAG_FOR["$fork"]="$latest" + if [[ $CHECK_ONLY -eq 1 ]]; then + echo "fork=$fork mode=$mode current=$current latest=$latest action=DRIFT" + DRIFT_COUNT=$((DRIFT_COUNT + 1)) + else + echo "fork=$fork mode=$mode current=$current latest=$latest action=UPDATE" + UPDATE_COUNT=$((UPDATE_COUNT + 1)) + fi + else + echo "fork=$fork mode=$mode current=$current latest=$latest action=ok" + fi +done + +# Forks on disk but missing from the map → append. +# Iterate in sorted order for stable output. +for fork in $(printf '%s\n' "${!DISK_FORK_URL[@]}" | sort); do + if [[ -z "${MAP_FORK_URL[$fork]:-}" ]]; then + url="${DISK_FORK_URL[$fork]}" + latest="" + if [[ "$NO_FETCH" -eq 0 ]]; then + result=$(latest_tag_for_url "$url" "$fork") + latest="${result%%|*}" + fi + latest="${latest:-unknown}" + row="$fork $url $latest # TODO: classify mode" + if [[ $CHECK_ONLY -eq 1 ]]; then + echo "fork=$fork mode=(missing) current=(none) latest=$latest action=DRIFT-MISSING" + DRIFT_COUNT=$((DRIFT_COUNT + 1)) + else + echo "fork=$fork mode=(missing) current=(none) latest=$latest action=APPEND" + APPEND_ROWS+=("$row") + fi + fi +done + +# ---- --check exit ---------------------------------------------------------- +if [[ $CHECK_ONLY -eq 1 ]]; then echo "" - for fork in syscall libredox redoxfs redox-scheme relibc kernel bootloader installer userutils; do - url="${FORKS[$fork]:-}" - if [ -z "$url" ]; then - echo "WARN: no upstream URL for $fork, skipping" >&2 - continue - fi - # Get latest tag matching the project's version scheme - # (e.g. 0.9.0, 0.1.18, 0.5.12, 1.0.0). Exclude pre-release and - # non-stable tags. - latest=$(cd /tmp && git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \ - | awk '{print $2}' \ - | sed 's|refs/tags/||' \ - | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' \ - | head -1) - if [ -z "$latest" ]; then - echo "WARN: couldn't determine latest tag for $fork from $url" >&2 - continue - fi - printf "%-13s %-55s %s\n" "$fork" "$url" "$latest" - done -} > "$MAP" + echo "Summary: drift=$DRIFT_COUNT" + if [[ $DRIFT_COUNT -gt 0 ]]; then + echo "FAIL: $DRIFT_COUNT drift(s) detected" >&2 + exit 1 + fi + echo "OK: no drift" + exit 0 +fi -echo "Updated $MAP" -cat "$MAP" +# ---- apply column-3 rewrites in place (byte-exact awk) --------------------- +if [[ ${#NEW_TAG_FOR[@]} -eq 0 && ${#APPEND_ROWS[@]} -eq 0 ]]; then + echo "" + echo "No changes needed ($MAP already current)" + exit 0 +fi + +TMP="${MAP}.tmp.$$" +APPEND_FILE="" +trap 'rm -f "$TMP" "$APPEND_FILE"' EXIT + +# Sidecar file with append rows (avoids awk quoting issues). +if [[ ${#APPEND_ROWS[@]} -gt 0 ]]; then + APPEND_FILE="${TMP}.append" + : > "$APPEND_FILE" + for row in "${APPEND_ROWS[@]}"; do + printf '%s\n' "$row" >> "$APPEND_FILE" + done +fi + +# Two-pass awk: +# pass 1 (control): UPDATES[fork] = newtag +# pass 2 (map): rewrite column 3 in matching rows; everything else +# is printed verbatim. Append sidecar rows at END. +# +# The column-3 rewrite uses match() to locate the prefix +# `^forkurl` and the next non-whitespace run (the tag), so all +# leading whitespace, the URL, and everything after the tag (mode, +# comments, EOL) are preserved byte-for-byte. +# +# A sentinel record guarantees the control stream always has at least one +# line. Without it, when CONTROL is empty (only appends, no col-3 +# rewrites), awk's `NR == FNR` test would misidentify every map line as a +# control line and corrupt the output. +CONTROL=$'#__rbum_sentinel__\t0.0.0\n' +for fork in "${!NEW_TAG_FOR[@]}"; do + CONTROL+="${fork}"$'\t'"${NEW_TAG_FOR[$fork]}"$'\n' +done + +awk -v APPEND_FILE="$APPEND_FILE" ' + NR == FNR { + UPDATES[$1] = $2 + next + } + { + if ($0 ~ /^[^#[:space:]]/) { + fork = $1 + if (fork in UPDATES) { + prefix_re = fork "[ \t]+[^ \t]+[ \t]+" + if (match($0, prefix_re)) { + prefix = substr($0, 1, RLENGTH) + rest = substr($0, RLENGTH + 1) + if (match(rest, /^[^ \t]+/)) { + suffix = substr(rest, RLENGTH + 1) + $0 = prefix UPDATES[fork] suffix + } + } + } + } + print + } + END { + if (APPEND_FILE != "") { + while ((getline line < APPEND_FILE) > 0) print line + close(APPEND_FILE) + } + } +' <(printf '%s' "$CONTROL") "$MAP" > "$TMP" + +mv "$TMP" "$MAP" +echo "" +echo "Updated $MAP ($UPDATE_COUNT col-3 rewrite(s), ${#APPEND_ROWS[@]} append(s))" diff --git a/local/scripts/upgrade-forks.sh b/local/scripts/upgrade-forks.sh index 08c2b28a94..68207058ac 100755 --- a/local/scripts/upgrade-forks.sh +++ b/local/scripts/upgrade-forks.sh @@ -5,7 +5,7 @@ # 1. Fetch latest upstream (full depth for accurate merge base) # 2. Identify Red Bear commits (ahead of upstream merge-base) # 3. Save RB commits to a patch file -# 4. Reset to upstream/master (or upstream/main) +# 4. Reset to upstream/master (or upstream/main, or refs/tags/ with --to) # 5. Reapply RB commits via cherry-pick (or patch fallback) # 6. Verify no upstream functions were dropped (verify-fork-functions.sh) # 7. Report success/failure per fork @@ -13,14 +13,29 @@ # Usage: # ./local/scripts/upgrade-forks.sh # Upgrade all forks # ./local/scripts/upgrade-forks.sh kernel redoxfs # Upgrade specific forks +# ./local/scripts/upgrade-forks.sh --to=0.5.12 kernel # Pin to a specific tag # ./local/scripts/upgrade-forks.sh --dry-run # Show plan, don't execute # ./local/scripts/upgrade-forks.sh --force # Skip safety checks # ./local/scripts/upgrade-forks.sh --no-fetch # Use cached upstream refs (no network) # ./local/scripts/upgrade-forks.sh --force-reapply # Reapply even when 0 commits behind +# ./local/scripts/upgrade-forks.sh --force-diverged # Allow running on diverged-mode forks +# +# --to= (release-bump-pipeline.md Deliverable B): +# After fetching upstream, pin upstream_ref to refs/tags/. The script +# tries refs/tags/ first, then refs/tags/v, resolving via +# `git rev-parse`. When --to is given, the master/main default-branch +# logic is skipped entirely. +# +# Diverged-mode guard (release-bump-pipeline.md Deliverable B): +# At the top of each fork's iteration, the script looks up the fork's mode +# (4th column of local/fork-upstream-map.toml). If mode is `diverged` and +# neither --force nor --force-diverged is given, the script REFUSES with +# exit 1 and a message explaining that diverged rebases are operator-manual +# per policy (Phase 2.4+). A missing map entry is a warning, not a refusal. # # Safety: # - Requires clean working tree in each fork (no uncommitted changes) -# - Creates backup branch before reset +# - Creates backup branch (rb-backup/--) before reset # - Stops on first conflict (interactive resolution or --abort) # - Post-upgrade: verifies all upstream functions are present # @@ -28,6 +43,7 @@ # REDBEAR_UPGRADE_FORCE=1 Same as --force # REDBEAR_UPGRADE_NO_FETCH=1 Same as --no-fetch # REDBEAR_UPGRADE_FORCE_REAPPLY=1 Same as --force-reapply +# REDBEAR_UPGRADE_FORCE_DIVERGED=1 Same as --force-diverged set -euo pipefail @@ -38,6 +54,8 @@ DRY_RUN=0 FORCE=0 NO_FETCH=0 FORCE_REAPPLY=0 +FORCE_DIVERGED=0 +PIN_TO_TAG="" declare -a TARGET_FORKS=() while [[ $# -gt 0 ]]; do @@ -46,8 +64,17 @@ while [[ $# -gt 0 ]]; do --force) FORCE=1 ;; --no-fetch) NO_FETCH=1 ;; --force-reapply) FORCE_REAPPLY=1 ;; + --force-diverged) FORCE_DIVERGED=1 ;; + --to=*) + PIN_TO_TAG="${1#*=}" + if [[ ! "$PIN_TO_TAG" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "ERROR: --to requires an anchored semver X.Y.Z (got: $PIN_TO_TAG)" >&2 + exit 1 + fi + ;; -h|--help) - echo "Usage: $0 [--dry-run] [--force] [--no-fetch] [--force-reapply] [fork1 fork2 ...]" + echo "Usage: $0 [--dry-run] [--force] [--no-fetch] [--force-reapply]" + echo " [--to=] [--force-diverged] [fork1 fork2 ...]" echo "" echo "Upgrades local forks to latest upstream and reapplies Red Bear patches." echo "" @@ -56,6 +83,12 @@ while [[ $# -gt 0 ]]; do echo " --force Skip safety checks (dirty tree, etc.)" echo " --no-fetch Use cached upstream refs (no network required)" echo " --force-reapply Reapply even when 0 commits behind upstream" + echo " --to= Pin upstream_ref to refs/tags/ (or refs/tags/v)" + echo " instead of upstream/master or upstream/main." + echo " Resolved via git rev-parse. Skips default-branch logic." + echo " --force-diverged Allow running on forks whose map mode is 'diverged'." + echo " Without this (or --force), diverged forks are refused" + echo " because their rebase is operator-manual (Phase 2.4+)." echo "" echo "Forks (default: all in local/sources/ with upstream remotes):" for d in "$PROJECT_ROOT"/local/sources/*/; do @@ -72,6 +105,7 @@ done [[ "${REDBEAR_UPGRADE_FORCE:-0}" == "1" ]] && FORCE=1 [[ "${REDBEAR_UPGRADE_NO_FETCH:-0}" == "1" ]] && NO_FETCH=1 [[ "${REDBEAR_UPGRADE_FORCE_REAPPLY:-0}" == "1" ]] && FORCE_REAPPLY=1 +[[ "${REDBEAR_UPGRADE_FORCE_DIVERGED:-0}" == "1" ]] && FORCE_DIVERGED=1 cd "$PROJECT_ROOT" @@ -131,9 +165,48 @@ for fork in "${TARGET_FORKS[@]}"; do continue fi + # ---- Diverged-mode guard (release-bump-pipeline.md Deliverable B) ---- + # Look up the fork's mode (4th column of local/fork-upstream-map.toml). + # If mode is `diverged` and neither --force nor --force-diverged is given, + # REFUSE: diverged rebases are operator-manual per policy (Phase 2.4+). + # A missing map entry is a warning, not a refusal. + fork_mode="" + if [[ -f "$PROJECT_ROOT/local/fork-upstream-map.toml" ]]; then + fork_mode=$(awk -v want="$fork" ' + $1 == want && /^[^#[:space:]]/ { + mode = $4 + sub(/[ \t]*#.*/, "", mode) + print mode + exit + } + ' "$PROJECT_ROOT/local/fork-upstream-map.toml" 2>/dev/null || true) + fi + if [[ -z "$fork_mode" ]]; then + echo -e " ${YELLOW}WARN: '$fork' not found in local/fork-upstream-map.toml${NC}" + echo -e " ${YELLOW} (proceeding without diverged-mode check)${NC}" + elif [[ "$fork_mode" == "diverged" && "$FORCE" -eq 0 && "$FORCE_DIVERGED" -eq 0 ]]; then + echo -e " ${RED}REFUSE: '$fork' is mode=diverged in the upstream map${NC}" + echo -e " ${YELLOW}Diverged forks track upstream via merge commits and cannot be${NC}" + echo -e " ${YELLOW}auto-rebased by this script. Their rebase is operator-manual${NC}" + echo -e " ${YELLOW}(Phase 2.4+ work). Override with --force-diverged (or --force).${NC}" + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_FORKS+=("$fork") + echo "" + continue + elif [[ "$fork_mode" == "diverged" ]]; then + echo -e " ${YELLOW}WARN: '$fork' is mode=diverged; proceeding due to --force-diverged${NC}" + fi + # Safety: check working tree is clean if [[ "$FORCE" -eq 0 ]]; then - dirty=$(cd "$fork_dir" && git status --porcelain 2>/dev/null | wc -l) + if ! cd "$fork_dir" 2>/dev/null; then + echo -e " ${RED}FAIL: cannot enter fork dir $fork_dir${NC}" + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_FORKS+=("$fork") + echo "" + continue + fi + dirty=$(git status --porcelain 2>/dev/null | wc -l) if [[ "$dirty" -gt 0 ]]; then echo -e " ${RED}FAIL: working tree has $dirty uncommitted change(s)${NC}" echo -e " Commit or stash first, or use --force" @@ -146,7 +219,38 @@ for fork in "${TARGET_FORKS[@]}"; do # Determine upstream branch and ref upstream_branch="master" - if [[ "$NO_FETCH" -eq 1 ]]; then + if [[ -n "$PIN_TO_TAG" ]]; then + # --to=: pin upstream_ref to refs/tags/, skipping the + # default master/main logic entirely. Try refs/tags/ first, + # then refs/tags/v (some upstreams prefix tags with `v`). + # Resolve via `git rev-parse` so we end up with a concrete SHA-bearing ref. + if [[ "$NO_FETCH" -eq 0 ]]; then + if ! (cd "$fork_dir" && git fetch upstream --quiet --tags 2>&1); then + echo -e " ${RED}FAIL: git fetch upstream --tags failed (network error?)${NC}" + echo -e " ${YELLOW}Use --no-fetch if upstream refs are already cached${NC}" + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_FORKS+=("$fork") + echo "" + continue + fi + fi + pinned_ref="" + for candidate in "refs/tags/${PIN_TO_TAG}" "refs/tags/v${PIN_TO_TAG}"; do + if (cd "$fork_dir" && git rev-parse --verify "$candidate" >/dev/null 2>&1); then + pinned_ref="$candidate" + break + fi + done + if [[ -z "$pinned_ref" ]]; then + echo -e " ${RED}FAIL: --to=${PIN_TO_TAG}: neither refs/tags/${PIN_TO_TAG}${NC}" + echo -e " ${RED} nor refs/tags/v${PIN_TO_TAG} exists after fetch${NC}" + FAIL_COUNT=$((FAIL_COUNT + 1)) + FAILED_FORKS+=("$fork") + echo "" + continue + fi + upstream_ref="$pinned_ref" + elif [[ "$NO_FETCH" -eq 1 ]]; then # Use cached upstream refs — verify they exist if ! (cd "$fork_dir" && git rev-parse --verify upstream/master >/dev/null 2>&1); then if (cd "$fork_dir" && git rev-parse --verify upstream/main >/dev/null 2>&1); then @@ -159,6 +263,7 @@ for fork in "${TARGET_FORKS[@]}"; do continue fi fi + upstream_ref="upstream/$upstream_branch" else if ! (cd "$fork_dir" && git fetch upstream --quiet 2>&1); then echo -e " ${RED}FAIL: git fetch upstream failed (network error?)${NC}" @@ -179,9 +284,8 @@ for fork in "${TARGET_FORKS[@]}"; do continue fi fi + upstream_ref="upstream/$upstream_branch" fi - - upstream_ref="upstream/$upstream_branch" old_sha=$(cd "$fork_dir" && git rev-parse --short HEAD) upstream_sha=$(cd "$fork_dir" && git rev-parse --short "$upstream_ref") diff --git a/recipes/wip/AGENTS.md b/recipes/wip/AGENTS.md index 89435dc02c..747cddc6e4 100644 --- a/recipes/wip/AGENTS.md +++ b/recipes/wip/AGENTS.md @@ -51,8 +51,8 @@ recipes/wip/ │ └── xwayland/ # XWayland (partially patched) ├── services/ │ └── seatd/ # Seat daemon recipe (service category, runtime trust still open) -├── kde/ # 9 KDE app recipes -│ ├── kde-dolphin/ # File manager (needs kio) +├── kde/ # HISTORICAL: 9 pre-KF6 KDE app recipes (not the active KDE surface) +│ ├── kde-dolphin/ # File manager (needs kio — kf6-kio now built under local/recipes/kde/) │ ├── kdenlive/ # Video editor (needs MLT) │ ├── krita/ # Painting (needs Qt + OpenGL) │ ├── kdevelop/ # IDE (needs Qt + kio) @@ -64,6 +64,12 @@ recipes/wip/ └── drivers/ # WIP driver ports (planned) ``` +**Note:** the recipes in `wip/kde/` are pre-KF6-era experiments. The ACTIVE KDE +surface is `local/recipes/kde/` (40+ KF6 frameworks, KWin, Plasma, SDDM, +konsole) plus `config/redbear-full.toml`. Do not treat `wip/kde/` recipes as +the KDE path; they remain only as reference material until upstream promotes +them (see "OWNERSHIP RULE FOR UPSTREAM WIP" above). + ## WHERE TO LOOK | Task | Location | diff --git a/scripts/run_mini1.sh b/scripts/run_mini1.sh index 753b03751b..6ced18ebca 100755 --- a/scripts/run_mini1.sh +++ b/scripts/run_mini1.sh @@ -11,7 +11,7 @@ esac # virtio-gl, native CPU, net boost -qemu-system-x86_64 -m 8G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-mini.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio +qemu-system-x86_64 -machine type=q35,accel=kvm -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -m 8G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-mini.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio #qemu-system-x86_64 -m 12G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-full.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio