vasilito/RedBear-OS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: release-bump pipeline + external graphics version sync

Browse Source 
Pipeline (3 operator asks):
- bump-release.sh: canonical orchestrator (forks + sources + external)
- upgrade-forks.sh --to=<tag>: rebase forks with diverged-mode guard
- bump-graphics-recipes.sh: map-driven group-aware graphics bumps
- check-external-versions.sh: drift checker for Qt6/KF6/Plasma/Mesa/Wayland
- refresh-fork-upstream-map.sh: append-only map updater with --check
- post-checkout-version-sync.sh + install-git-hooks.sh: opt-in branch hook
- external_version_lib.py: shared version-parsing/bumping library
- external-upstream-map.toml: ~80 external package entries
- bump-fork.sh: deprecated (REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1)
- RELEASE-BUMP-WORKFLOW.md: operator runbook

Quality fixes (8 defects from two independent audits):
- blake2b stable cache keys (was hash(), non-portable)
- atomic cache writes via os.replace
- version_sort_key pre-release demotion (was sorting after finals)
- apply_ver_transform re.error tolerance
- grep || true (pipefail abort)
- cd failure detection in upgrade-forks
- sed URL escape (injection hardening)
- refresh-fork-upstream-map last-row drop fix

Doc cleanup:
- Archive 5 obsolete plans to local/docs/archived/
- Remove 14 stale/superseded docs
- Update 18 docs to reference bump-release.sh and fix inbound links
- TOOLS.md drift fixes
This commit is contained in:
vasilito
2026-07-18 14:45:41 +09:00
parent 7b004272a3
commit 99e5641127
44 changed files with 3143 additions and 5226 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+29 -17
View File
@@ -458,34 +458,46 @@ On every build, the cookbook MUST:
b. Emit a clear actionable error: `LOCAL FORK OUTDATED: local/sources/<component> is at vX.Y.Z but vA.B.C is required by <dep>`
c. Do NOT silently proceed with a broken build.
**Remediation (automatic, when invoked)**: The `local/scripts/bump-fork.sh`
script (or equivalent `repo bump-fork <component>` command) MUST:
1. Fetch the upstream source at the required version.
2. Apply all Red Bear patches from `local/patches/<component>/` using the
atomic patch application mechanism (see "Atomic Patch Application" below).
3. Update the version field in the local fork's `Cargo.toml`.
4. Commit the result to the `submodule/<component>` branch in the RedBear-OS repo.
5. Rebuild the affected packages.
**Remediation (canonical: `bump-release.sh`)**: The
`local/scripts/bump-release.sh` orchestrator is the canonical fork-bump path
(see `local/docs/RELEASE-BUMP-WORKFLOW.md`). For each fork it resolves the
latest upstream release tag (excluding tags already reachable from the fork's
HEAD — ancestor tags are history, not upgrades) and:
1. Rebases the fork onto the new upstream tag via
`upgrade-forks.sh --to=<tag>` (net-diff reapply with cherry-pick fallback,
then verifies no upstream functions were dropped).
2. Updates the version field in the local fork's `Cargo.toml` to
`<new-tag>+rb<branch>`.
3. Updates `local/fork-upstream-map.toml` column 3 in place.
4. Regenerates the fork's `Cargo.lock`.
5. Prints the exact commit sequence for the operator (fork commits to
`submodule/<component>` + the parent map commit). It never commits itself.
6. The stale-prefix and source-fingerprint machinery then rebuilds affected
packages on the next `build-redbear.sh` run.
`local/scripts/bump-fork.sh` is **DEPRECATED** (guarded by
`REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1`): its wholesale-replacement model
discards committed fork work not present as patch files. Kept for historical
reference only — do not use it.
**Symlink aliasing**: `recipes/<component>/` MUST be a symlink to
`../../../local/sources/<component>/` when a local fork exists. This ensures
backward compatibility for recipes that reference the `recipes/` path while
still preferring the local fork as the source of truth.
**Implementation status**: Detection is fully implemented across 5 tools:
**Implementation status**: Detection is fully implemented across the
release-bump toolchain:
- `bump-release.sh`: canonical release-bump orchestrator (fork source+label coherence)
- `sync-versions.sh`: Cat 0 (cookbook) + Cat 1 (in-house) + Cat 2 (fork +rb suffix) versions
- `verify-fork-versions.sh`: Cat 2 fork supremacy + content check vs upstream tag
- `verify-patch-content.py`: orphan-patch detection + operator-decision automation
- `bump-fork.sh` + `upgrade-forks.sh`: fork rebase + patch reapplication
- `upgrade-forks.sh`: fork rebase + patch reapplication (`--to=<tag>`, diverged guard)
- `push-fork-branches.sh`: operator-reviewed fork push helper
- `refresh-fork-upstream-map.sh`: append-only upstream-tag maintenance for the fork map
- `check-external-versions.sh` + `bump-graphics-recipes.sh`: external desktop-stack version check + source bumps
See `local/docs/HOOKS.md` for the full tool inventory (10 tools as of Phase 9.2).
process is:
1. `cd local/sources/<component>`
2. Edit `Cargo.toml` version field to match upstream
3. `git pull` or fetch upstream at the new tag
4. Reapply all `local/patches/<component>/*.patch` (see AGENTS.md "Atomic Patch Application")
5. Commit to `submodule/<component>` branch
See `local/scripts/TOOLS.md` for the full tool inventory (15 tools as of
Round 14) and `local/docs/HOOKS.md` for the opt-in git hooks.
### Offline-First By Default
README.md
+1 -1
View File
@@ -207,7 +207,7 @@ Red Bear OS operates under strict discipline. Full policies: [`local/AGENTS.md`]
- [Patch Preservation Audit](local/docs/PATCH-PRESERVATION-AUDIT-2026-07-12.md) — orphan-patch governance, Rounds 1-6 audit results, SUPERSEDED.md log
- [Collision Detection Status](local/docs/COLLISION-DETECTION-STATUS.md) — runtime collision-detection current state
- [Hooks](local/docs/HOOKS.md) — opt-in git hooks (pre-push safety net, etc.)
- [Build Tools](local/scripts/TOOLS.md) — 10-tool reference (patch-status, sync, verify, collision, etc.)
- [Build Tools](local/scripts/TOOLS.md) — 15-tool reference (patch-status, sync, verify, collision, release-bump, etc.)
- [Fork Push Status](local/docs/fork-push-status/2026-07-12-Round-9-phase-8.3.md) — fork-branch push results + base deadlock (updated Round 9)
- [Wi‑Fi Plan](local/docs/WIFI-IMPLEMENTATION-PLAN.md) — Wireless architecture and driver plan
- [Bluetooth Plan](local/docs/BLUETOOTH-IMPLEMENTATION-PLAN.md) — Bluetooth stack design
docs/07-RED-BEAR-OS-IMPLEMENTATION-PLAN.md
+1 -1
View File
@@ -411,7 +411,7 @@ The current subsystem plans to treat as first-class are:
- `local/docs/WIFI-IMPLEMENTATION-PLAN.md`
- `local/docs/BLUETOOTH-IMPLEMENTATION-PLAN.md`
- `local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md` — implemented credential syscalls + kernel robustness
- `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md`
- `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md`
- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — **canonical comprehensive plan**
The older architecture/roadmap docs under `docs/01`–`docs/05` remain useful, but they should be
docs/README.md
+1 -1
View File
@@ -66,7 +66,7 @@ console-to-KDE plan.
- `../local/docs/IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` — PCI/IRQ quality, MSI/MSI-X
- `../local/docs/DRM-MODERNIZATION-EXECUTION-PLAN.md` — DRM-focused execution (subsystem detail)
- `../local/docs/WAYLAND-IMPLEMENTATION-PLAN.md` — Wayland compositor (subsystem detail)
- `../local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — relibc IPC surface
- `../local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — relibc IPC surface
- `../local/docs/GREETER-LOGIN-IMPLEMENTATION-PLAN.md` — greeter/login design
- `../local/docs/DBUS-INTEGRATION-PLAN.md` — D-Bus architecture
- `../local/docs/SCRIPT-BEHAVIOR-MATRIX.md` — script guarantees and non-guarantees
local/AGENTS.md
+50 -6
View File
@@ -718,6 +718,44 @@ newer upstream release.
The `--check` mode is suitable for CI gates and preflight checks.
### Release-bump orchestrator (canonical)
`./local/scripts/bump-release.sh` is the canonical entry point for a Red Bear
OS release bump. It DECIDES what moves and DELEGATES execution to existing
single-purpose tools:
- **no args** — label sync (`sync-versions.sh --no-regen`) + fork upstream
report + external version report. Fast, mostly local.
- `--with-sources` — additionally rebase eligible Cat 2 forks onto newer
upstream tags via `upgrade-forks.sh --to=<tag>`, update the fork
`Cargo.toml` label + `local/fork-upstream-map.toml` column 3 in place.
- `--with-external` — additionally run the rewritten map-driven
`bump-graphics-recipes.sh` for the Qt6/KF6/Plasma desktop stack.
- `--check`, `--dry-run`, `--no-fetch`, `--regen`, `--force-diverged`,
`--fork=<name>` — see `local/scripts/TOOLS.md` for the full flag matrix.
Fork routing is by map mode: `snapshot`/`tracked` → source-bump eligible;
`diverged` (kernel/bootloader/installer) → report-only (`--force-diverged`
escape hatch); `base` (tag=`main`) → always suffix-only. Nothing ever
auto-commits; the orchestrator prints the exact `git add`/`git commit`
sequence for the parent repo and each affected `submodule/<fork>` branch.
### Branch-switch automation (opt-in post-checkout hook)
`./local/scripts/post-checkout-version-sync.sh` is an **opt-in** git hook
that runs the label-only half of a release bump automatically when an
operator checks out a semver release branch. Guards: branch checkout only,
anchored semver branch name, no rebase/cherry-pick/merge in progress,
`REDBEAR_NO_AUTO_SYNC` unset, clean working tree. It NEVER does network,
lockfile regen, source upgrades, or commits, and never blocks a checkout.
Bypass: `REDBEAR_NO_AUTO_SYNC=1 git checkout <branch>`. Install via
`./local/scripts/install-git-hooks.sh`.
**Full runbook** (4-step release model, fork decision tree, bootloader
`git replace --graft` note, lockfile-regen sequencing, commit-sequence
conventions): `local/docs/RELEASE-BUMP-WORKFLOW.md`. **Tool inventory:**
`local/scripts/TOOLS.md`. **Hook details:** `local/docs/HOOKS.md`.
### Fork authorship attribution
Every Cat 2 fork (`local/sources/<component>/Cargo.toml`) that carries Red Bear
@@ -1022,14 +1060,20 @@ As of 2026-06, the following core components are built from **local forks** in
| Component | Local fork path | Recipe |
|-------------|------------------------------------------|-------------------------------------------------------|
| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (git URL) |
| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (git URL) |
| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` |
| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (+ Cargo.toml dep) |
| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` |
| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` |
| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (path = ...) |
| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (path = ...) |
| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` (path = ...) |
| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (path = ..., + Cargo.toml dep) |
| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` (path = ...) |
| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` (path = ...) |
| **base** | **`local/sources/base/`** (path source) | **`recipes/core/base/recipe.toml` (path = ...) ** |
All seven recipe'd components use `path = "../../../local/sources/<comp>"`;
the cookbook symlinks `recipes/core/<comp>/source` to the fork worktree. The
remaining forks — `syscall`, `libredox`, `redox-scheme` — have **no recipe**:
they are consumed purely as Cargo `path` dependencies (plus `[patch.crates-io]`)
by the sibling forks (base, relibc, …). See "Local fork dependency rule".
### Why local forks for these components?
1. **Critical bug fixes** — these components had multiple broken patches that drifted
local/docs/ACPI-FORK-SYNC-STRATEGY-2026-06-30.md
-536
View File
@@ -1,536 +0,0 @@
# ACPI Fork-Sync Strategy — 2026-06-30
This document captures the upstream-fork survey for the ACPI fixes.
Based on the user's direction ("our kernel must bump too"), the strategy
is to **synchronize the local Red Bear kernel+base forks with upstream
Redox main, then implement the remaining gaps on top of the synchronized
foundation**.
## Upstream commits worth pulling in
### Kernel (`gitlab.redox-os.org/redox-os/kernel`)
| SHA | Title | Author | Date | Gap it addresses | Verdict |
|---|---|---|---|---|---|
| `f49c7d99` | RSDP validation and fixing a few clippy lints | Speedy_Lex | 2026-05-05 | **Gap #1: RSDP checksum validation** | ✅ Direct fix |
| `678a8c56` | Fail softly on bad RSDP | Speedy_Lex | 2026-05-05 | Complements #1 | ✅ Robustness |
| `6b55ffea` | Make rsdp pointer NonNull | Speedy_Lex | 2026-05-05 | Type-safety refactor | ✅ Trivial merge |
| `8011f3f6` | Simplify acpi scheme (MR #613) | 4lDO2 | 2026-06-02 | **Gap #8: AcpiScheme fevent + restructures kernel-side ACPI** | 🚧 Major refactor |
| `a47b3a8d` | Add timeout for events | Wildan M | 2026-05-30 | Shutdown timeout support | ✅ Direct |
| `50c68cd7` | Remove ACPI search from the kernel | (recent) | (recent) | Architectural | 🚧 Couples with #613 |
| `f3394673` | Remove dead_code lint allow for ACPI | (recent) | (recent) | Cleanup | ✅ Trivial |
| `62f220b1` | Use cfg! rather than #[cfg] for ACPI enabling | (recent) | (recent) | Cleanup | ✅ Trivial |
### Base (`gitlab.redox-os.org/redox-os/base`)
| SHA | Title | Author | Date | Gap it addresses | Verdict |
|---|---|---|---|---|---|
| `9dd6901d` | acpid: setrens before ready(), fixing deadlock | 4lDO2 | 2026-06-05 | **acpid startup correctness** | ✅ 4-line drop-in fix |
| `f2f834d4` | Update ACPI crate | Wildan M | 2026-04-27 | **Bumps jackpot51/acpi to redox-6.x fork** | 🚧 Requires amlserde enum update |
| `2045364b` | Merge `simplify_acpi` (MR #275) | Jeremy Soller | 2026-06-02 | **Pairs with kernel MR #613** | 🚧 Major refactor |
### Kernel MR #613: "Simplify acpi scheme"
**Description** (from gitlab): "Complexity can be reduced somewhat by not providing a
filesystem-like interface for RXSDT+shutdown. If file-based interfaces should exist
for the rsdt/xsdt, it can be done by userspace. Reduces kernel size by 4%,
presumably due to the removed hashmap."
**What it does:**
- Removes `/scheme/kernel.acpi/rxsdt` and `/scheme/kernel.acpi/kstop` files
- Replaces with a single FD-based `call()` interface using `AcpiVerb` enum
- `AcpiVerb::ReadRxsdt` returns rxsdt bytes
- `AcpiVerb::RegisterKstop` registers a kstop listener
- Uses `bitflags!` for handle permission tracking
- Uses new `Mutex<L4>` from `crate::sync::ordered`
- Adds `EXISTS_KSTOP_HANDLE` atomic to track handler presence
- Simplifies `kfpath` requirement
**Required dependency bumps:**
- `redox_syscall 0.7.4 → 0.8.1` (we're 2 versions behind upstream's 0.8.1)
- Pulls in new `AcpiVerb` enum, `CallFlags` enum
- Uses `crate::sync::ordered::Mutex<L4>` (new in upstream)
### Base MR #275: "Use simplified kernel ACPI interface"
**Description** (from gitlab): "Switches to the `SYS_CALL`-based kernel ACPI interface.
Needs https://gitlab.redox-os.org/redox-os/kernel/-/merge_requests/613"
**Required user-space changes** (in acpid, hwd):
- Replace `fs::read("/scheme/kernel.acpi/rxsdt")` with `Fd::open("/scheme/kernel.acpi").call_ro(..., AcpiVerb::ReadRxsdt)`
- Replace `File::open("/scheme/kernel.acpi/kstop")` with `Fd::open(...).call_ro(..., AcpiVerb::RegisterKstop)`
- Adds `[patch.crates-io] redox_syscall = { git = ".../syscall.git" }` in workspace Cargo.toml
- Adds `redox_syscall = "0.8.1"` to hwd's Cargo.toml
- Splits `AmlSerdeReferenceKind::LocalOrArg` into 4 separate variants (`Local`, `Arg`, `Index`, `Named`)
## Coupling matrix
Pulling in the upstream changes requires coordinated bumps:
```
Kernel Cargo.toml: redox_syscall 0.7.4 → 0.8.1
Kernel src/scheme/acpi.rs: rewrite using AcpiVerb + bitflags + Mutex<L4>
Kernel src/syscall/debug.rs: add _ => Err(EINVAL) catch-all (per MR #613 src/scheme/proc.rs diff)
Base Cargo.toml: redox_syscall 0.7.4 → 0.8.1 + patch.crates-io
Base drivers/acpid: rewrite using Fd::open + call_ro(AcpiVerb::ReadRxsdt/RegisterKstop)
Base drivers/hwd: rewrite using Fd::open + call_ro(AcpiVerb::ReadRxsdt); add redox_syscall dep
Base drivers/amlserde: enum variant split (LocalOrArg → Local/Arg/Index/Named)
Base drivers/acpid/src/scheme.rs: may need Fd-based kstop register instead of open("/scheme/kernel.acpi/kstop")
All other base drivers: bump redox_syscall transitive dep
```
## Recommended execution order
This is **NOT** a 1-day fix. It's a multi-day fork-sync. Recommended sequence:
### Phase A — Kernel fork-sync (1 session)
1. **Pull MR #613** into `local/sources/kernel/`:
- `git fetch` (if remote configured) or apply patches manually
- Update `Cargo.toml`: `redox_syscall 0.7.4 → 0.8.1`
- Port `src/scheme/acpi.rs` to the new call() interface
- Add `_ => Err(EINVAL)` catch-all in `src/scheme/proc.rs` (per upstream diff)
- Build kernel with `cargo build --release --target x86_64-unknown-none`
- Verify kernel still compiles
2. **Pull `f49c7d99` + `678a8c56` + `6b55ffea`** — these are trivial RSDP improvements that ride on the same kernel bump.
### Phase B — Base fork-sync (1-2 sessions)
1. **Pull `9dd6901d`** — 4-line deadlock fix, drop-in.
2. **Pull MR #275 / `2045364b`**:
- Update workspace `Cargo.toml`: `redox_syscall 0.8.0 → 0.8.1` + `[patch.crates-io]`
- Update `acpid/src/main.rs`: Fd-based rxsdt + kstop
- Update `hwd/src/backend/acpi.rs`: Fd-based rxsdt
- Update `hwd/Cargo.toml`: add `redox_syscall = "0.8.1"`
3. **Pull `f2f834d4`** — ACPI crate update:
- Workspace `Cargo.toml`: add `acpi = { git = "...redox-os/acpi.git", branch = "redox-6.x" }`
- `drivers/acpid/Cargo.toml`: `acpi.workspace = true`
- `drivers/amlserde/Cargo.toml`: `acpi.workspace = true`
- `drivers/amlserde/src/lib.rs`: split `AmlSerdeReferenceKind::LocalOrArg` into 4 variants
- Verify `acpid` builds against new acpi crate.
### Phase C — Apply remaining gaps (1 session each)
After Phase A+B, the new infrastructure (Fd-based call interface, updated acpi crate) makes these gaps easier:
- **Gap #1 RSDP validation**: already pulled from upstream `f49c7d99`.
- **Gap #5 SLP_TYPb write**: now in `acpid/acpi.rs:679` — straightforward 3-line addition.
- **Gap #3 AML Mutex stubs**: the new acpi crate may have fixed these. Check after pulling.
- **Gap #4 S1-S4 sleep**: requires `_PTS`/`_WAK` AML method evaluation — non-trivial.
- **Gap #6 parse_lnk_irc range validation**: independent of upstream.
- **Gap #7 thermal/power enumeration**: independent of upstream.
## Risk register
- **Phase A kernel bump risk: HIGH.** `redox_syscall 0.7.4 → 0.8.1` is a 2-version jump. The new `Mutex<L4>` and `ordered` module may not exist in our kernel. Build failures likely.
- **Phase B base bump risk: MEDIUM.** MR #275 already has working code in acpid and hwd; just needs porting.
- **amlserde enum breakage risk: MEDIUM.** Every consumer of `AmlSerdeReferenceKind` must be updated. Need to find all callers.
- **Sync conflict risk with Red Bear's own changes**: Our `local/sources/base` has Red Bear commits (`76e0928` etc.). Pulling upstream may require rebasing.
## Out of scope for this session
- Implementing the full `_PTS`/`_WAK` AML semantics for S1-S4 (gap #4)
- DMAR root-cause investigation (gap #2)
- 35 minor TODO/FIXME/panic markers
- LegacyBackend implementation
- AML region handlers for SMBus/IPMI/GeneralPurposeBus
## Decision
User said "our kernel must bump too." This is the right call — the upstream
changes are coupled and pulling them in one piece is much cleaner than
backporting individual fixes that would otherwise be incompatible with the
new infrastructure.
**Recommendation:** Start with Phase A (kernel fork-sync) in this session.
If that succeeds (kernel builds), proceed to Phase B (base fork-sync).
If anything in Phase A breaks irrecoverably, document and defer.
---
## Phase A outcome — 2026-06-30
**Status: ✅ COMPLETE. Kernel compiles and links.**
Inner-kernel commit: `4f2a043 kernel: re-sync ACPI subsystem with upstream master`
Changes applied (11 files, +136/-278):
1. **`Cargo.toml`** — `redox_syscall` from `version = "0.7.4"` to
`git = "https://gitlab.redox-os.org/redox-os/syscall.git"`. The crates.io
0.8.1 release predates the `AcpiVerb` enum that MR #613 introduced,
so a git ref is required to get the latest syscall interface.
2. **`src/acpi/rsdp.rs`** — full rewrite with RSDP validation:
- signature check `"RSD PTR "`
- 20-byte base checksum
- extended checksum for `revision >= 2`
- `NonNull<u8>` pointer type instead of `*const u8`
3. **`src/startup/mod.rs`** — `acpi_rsdp()` returns `Option<NonNull<u8>>`.
4. **`src/acpi/mod.rs`** — `init()` takes `Option<NonNull<u8>>`.
5. **`src/scheme/acpi.rs`** — full rewrite to upstream MR #613. Drops the
`/scheme/kernel.acpi/` filesystem in favor of a single `Fd::open` +
`call()` interface with `AcpiVerb` verbs (`ReadRxsdt`, `CheckShutdown`).
Uses `HandleBits` bitflags, atomic `EXISTS_KSTOP_HANDLE`, `Mutex<L4>`.
6. **`src/scheme/mod.rs`** — `KernelScheme::kcall` signature changed from
`id: usize` to `fds: &[usize]`. `kfpath` now defaults to `EOPNOTSUPP`.
7. **`src/scheme/{memory,proc,user}.rs`** — `kcall` impls updated for new
trait signature.
8. **`src/scheme/proc.rs`** — added `_ => Err(EINVAL)` catch-all in
`kcall` dispatch to handle new `ProcSchemeVerb` variants
(`RegsInt`, `RegsFloat`, `RegsEnv`, `SchedAffinity`, `Start`) that
the gitlab syscall crate adds.
9. **`src/syscall/fs.rs`** — `SYS_CALL` dispatcher passes `&[number]` to
`scheme.kcall()` to match new signature.
10. **`Makefile`** — removed `-Z json-target-spec` flag (no longer
needed; was added by commit 4cb9d80 to compensate for our nightly
being too old, but our nightly is now too NEW for that flag).
**Verified by:** `make` in `local/sources/kernel/` with cross-toolchain
in PATH. Kernel binary `kernel` produced at 1.2 MiB.
**Not pushed to remote** — the inner kernel fork's remote is
`gitlab.redox-os.org/redox-os/kernel.git` (upstream Redox, not Red
Bear's gitea). This is the same footgun documented as build-system
issue #11. The change is durable in the inner fork's commit
`4f2a043` and will be reflected in the outer Red Bear repo's
`local/sources/kernel/` submodule pointer when the outer repo
is updated.
**Gap #1 closed:** RSDP checksum validation now active in the kernel.
**Gap #8 closed:** `AcpiScheme` now uses an atomic + bitflags approach
that wakes waiters immediately (the new `kcall` interface is event-driven
by design).
**Next step:** Phase B (base fork-sync) — port acpid + hwd + amlserde to
the new Fd::open + call() interface, and bump `redox_syscall 0.7.4 → 0.8.1`
in the workspace `Cargo.toml`.
---
## Phase B outcome — 2026-06-30
**Status: ✅ COMPLETE. Cookbook builds; ISO produced.**
Inner-base commit: `ae57fe3 base: re-sync ACPI userspace with upstream master`
Changes applied (8 files, +54/-23):
1. **`Cargo.toml` (workspace)** — added `acpi` workspace dep pointing to
the gitlab redox-os/acpi fork at branch `redox-6.x`. Switched
`redox_syscall` from crates.io `0.8.1` to a git ref of
gitlab.redox-os.org/redox-os/syscall.git, plus
`[patch.crates-io] redox_syscall = { git = ".../syscall.git" }` to
redirect crates.io consumers.
2. **`drivers/acpid/Cargo.toml`** — `acpi.workspace = true`.
3. **`drivers/amlserde/Cargo.toml`** — `acpi.workspace = true`.
4. **`drivers/hwd/Cargo.toml`** — added `redox_syscall.workspace = true`.
5. **`drivers/amlserde/src/lib.rs`** — split `AmlSerdeReferenceKind::LocalOrArg`
into `Local`, `Arg`, `Index`, `Named` to match the new gitlab acpi
crate's `ReferenceKind` enum (which already split the old combined
variant). Both the `to_serde` and `from_serde` match arms updated.
6. **`drivers/acpid/src/main.rs`** — rewrote the RXSDT and kstop
acquisition:
- `Fd::open("/scheme/kernel.acpi", O_CLOEXEC, 0)` to get the
parent handle
- `.call_ro(buf, READ, &[ReadRxsdt as u64])` to fetch RXSDT bytes
- `.openat("kstop", O_CLOEXEC, 0)` to get the shutdown pipe
Also applied upstream commit `9dd6901d` (deadlock fix): moved
`setrens(0, 0)` BEFORE `daemon.ready()` so that an `openat` to
`/scheme/acpi/tables` from nsmgr doesn't deadlock.
7. **`drivers/hwd/src/backend/acpi.rs`** — `AcpiBackend::new()` rewritten
to use the new Fd-based interface.
**Verified by:** `CI=1 ./local/scripts/build-redbear.sh redbear-mini`
succeeded with exit 0. ISO at `build/x86_64/redbear-mini.iso`
(512 MB, timestamp 2026-06-30 04:54).
**Gap closure:**
- **Gap #1 RSDP validation** — closed in Phase A.
- **Gap #8 AcpiScheme fevent** — closed in Phase A.
- **acpid deadlock** (9dd6901d) — closed in Phase B. setrens now
runs before daemon.ready(), so nsmgr's openat doesn't block.
**Open gaps (out of scope for fork-sync):**
- Gap #2 DMAR root cause — deferred (needs hardware investigation).
- Gap #3 AML mutex stubs — depends on newer acpi crate behavior.
- Gap #4 S1-S4 sleep — needs _PTS/_WAK AML evaluation.
- Gap #5 SLP_TYPb write — still missing in acpid/acpi.rs:679.
- Gap #6 parse_lnk_irc range validation — still in hwd/backend/acpi.rs.
- Gap #7 thermal/power enumeration — still empty placeholders.
These can be addressed in Phase C on top of the synchronized foundation.
---
## Phase C outcome — 2026-06-30
**Status: ✅ COMPLETE. Cookbook builds; ISO produced.**
Inner-base commit: `d844111 base: close SLP_TYPb, parse_lnk_irc, AML mutex, and S5 gaps`
Closes 4 of the 8 critical gaps on top of the synchronized foundation:
### Gap #5 - SLP_TYPb PM1b write (`drivers/acpid/src/acpi.rs`)
The previous code wrote SLP_EN+SLP_TYPa to PM1a but silently dropped
SLP_TYPb. On hardware that requires both PM1a and PM1b writes
(some laptops, server boards with split power blocks), the shutdown
was incomplete. Now writes SLP_EN+SLP_TYPb to PM1b when
`pm1b_control_block` is non-zero. The FADT field is 0 when no second
block exists, in which case we skip the second write.
### Gap #6 - parse_lnk_irc range validation (`drivers/hwd/src/backend/acpi.rs`)
The previous code accepted any 16-bit integer as an IRQ
(`*n & 0xFFFF`), producing "Enabled at IRQ 53313" from misparsed
FieldUnit accessors on QEMU PIIX4. Now validates that the IRQ
value is ≤ 2047 (the maximum valid legacy-compatible IOAPIC IRQ).
Out-of-range values are debug-logged and skipped instead of polluting
the routing table. Also adds a 15-bit cap on the Buffer-based IRQ
bit extraction (was previously unchecked).
### Gap #3 - AML mutex create/acquire/release (`drivers/acpid/src/aml_physmem.rs`)
The new gitlab acpi crate (Phase B bump) added proper `Handler`
trait methods for `create_mutex`, `acquire`, and `release`. The
previous implementation was three `log::debug!()` stubs returning
fake success, which would silently corrupt AML state for any
DSDT/SSDT that uses Mutex. Now implements a real mutex table
backed by `std::sync::Mutex<FxHashSet<u32>>`:
- `create_mutex()` allocates a unique `u32` handle from a counter
- `acquire()` busy-waits with 1ms sleeps until the handle is free
or the AML timeout (multiplied by 1000 for ms→µs conversion)
expires; returns `AmlError::MutexAcquireTimeout` on timeout
- `release()` removes the handle from the held set
### Gap #4a - `set_global_s_state` non-S5 explicit warning
The previous code silently returned early when called with any state
other than 5. Now emits a `log::warn!()` with the requested state,
naming the missing dependencies (`_PTS`/`_WAK` AML evaluation,
P-state preservation, wakeup path). This converts a silent failure
into a diagnostic that's visible in the boot log.
### Build-fix: `drivers/acpid/src/dmi.rs:158`
Converted `e.errno` (private field) to `e.errno()` (method call).
The libredox `Error` struct changed its `errno` from a public field
to a method in a newer release; the `DmiError::Map(syscall::error::Error)`
construction was using the field-access form, which broke the
build against current libredox. This is a build-fix that the prior
dirty tree already had; included here to keep base buildable.
**Verified by:** `CI=1 ./local/scripts/build-redbear.sh redbear-mini`
succeeded with exit 0. ISO at `build/x86_64/redbear-mini.iso`
(512 MB, timestamp 2026-06-30 05:28).
**Remaining open gaps (out of scope for this session):**
- Gap #2 DMAR root cause — needs investigation on real hardware.
- Gap #4b _PTS/_WAK infrastructure — needed for S1-S4 sleep.
- Gap #7 thermal/power enumeration — needs `_TZ` iteration + EC wiring.
---
## Phase D outcome — 2026-06-30
**Status: ✅ COMPLETE. Linux 7.1 best-practices ported; consumer ported; ISO rebuilt; QEMU boot verified end-to-end.**
Inner-base commit: `181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap`
Outer Red Bear commit: `5f1da5250 redbear-sessiond: port ACPI shutdown watcher to new Fd-based scheme`
### Linux 7.1 cross-reference findings (the canonical reference)
After auditing Linux 7.1's ACPI sleep infrastructure
(`drivers/acpi/acpica/hwxfsleep.c`, `drivers/acpi/acpica/hwesleep.c`,
`drivers/acpi/sleep.c`), the canonical "enter a sleep state" pattern is:
1. `acpi_enter_sleep_state_prep(sleep_state)` (linux/drivers/acpi/acpica/hwxfsleep.c:200):
- Read SLP_TYPa/SLP_TYPb from `_Sx` package via `acpi_get_sleep_type_data`
- Save S0's SLP_TYP values for restore on wake
- **Evaluate `\_PTS(sleep_state)`** (line 222) — "Prepare To Sleep" AML method
- **Evaluate `\_SST(sst_value)`** (line 265) — "System Status" indicator
(working=0, sleeping=1, sleep-context=2, indicator-off=7)
2. Write `SLP_EN | SLP_TYPa` to PM1a, `SLP_EN | SLP_TYPb` to PM1b
3. On wake, evaluate `\_WAK(sleep_state)` and restore S0's SLP types
There's also a separate `\_TTS` (Transition To State) for the reboot path
(`acpi_sleep_tts_switch` in `drivers/acpi/sleep.c:36`).
### Changes applied (Phase D)
**`drivers/acpid/src/acpi.rs` (refactored `set_global_s_state`)**
Follows the Linux 7.1 `acpi_enter_sleep_state` pattern with 5 steps:
1. Look up `_Sx` package in AML namespace (was hardcoded to `_S5`)
2. Evaluate `_PTS(state)` via new `aml_evaluate_simple_method` helper
3. Evaluate `_SST(sst_value)` with ACPI_SST_* constants
4. Write `SLP_EN|SLP_TYPa` to PM1a, `SLP_EN|SLP_TYPb` to PM1b
5. Spin
Generic now — works for any Sx (not just S5). S1-S4 paths still don't
fully work (no `_WAK`, no P-state preservation, no wakeup vector), but
the new generic structure means future `_WAK` implementation only needs
to add wakeup handling after step 4.
**`drivers/acpid/src/acpi.rs` (new `aml_evaluate_simple_method` helper)**
Mirrors Linux 7.1's `acpi_execute_simple_method` (drivers/acpi/utils.c).
Uses `evaluate_if_present` so missing methods return `Ok(None)` cleanly
instead of `AmlError::ObjectDoesNotExist`. Takes the AML global lock
with timeout 16 (matching the existing `aml_eval` pattern).
**`drivers/acpid/src/scheme.rs` (thermal/power enumeration)**
`thermal_zones()` enumerates `\_TZ.<zone>` children. `power_adapters()`
enumerates `PowerResource` objects. Both populate the previously-empty
`/scheme/acpi/thermal/` and `/scheme/acpi/power/` directories.
**`local/recipes/system/redbear-sessiond/source/src/acpi_watcher.rs` (consumer port)**
The previous `wait_for_shutdown_edge()` tried to open
`/scheme/kernel.acpi/kstop` (a file path that no longer exists after
Phase B). Rewrote to use the new Fd-based interface:
- `Fd::open("/scheme/kernel.acpi", O_CLOEXEC, 0)` to get the parent
- `.openat("kstop", O_CLOEXEC, 0)` to get the kstop sub-handle
- `call_ro(buf, empty, &[CheckShutdown as u64])` polled at 250ms
Uses polling instead of `RawEventQueue` to avoid pulling in `redox_event`
(currently uses the removed `llvm_asm!` macro on newer Rust nightly).
**Verified by:** redbear-mini ISO rebuilt cleanly (2026-06-30 06:28).
QEMU boot reaches `Red Bear login:` prompt with no errors. `redbear-sessiond`
is now working correctly:
- Line 133-135: `probing 2 D-Bus socket candidate(s)... registered
org.freedesktop.login1 on the system bus`
- ACPI shutdown watcher no longer errors
### Final gap closure status
| Gap | Description | Status | Phase |
|-----|-------------|--------|-------|
| #1 | RSDP checksum validation | ✅ Closed | A |
| #3 | AML mutex stubs | ✅ Closed | C |
| #4a | set_global_s_state genericity + warnings | ✅ Closed | C |
| #4 | set_global_s_state refactored to Linux 7.1 pattern | ✅ Closed | D |
| #4b | `_WAK` + `_TTS` AML hooks for S1-S4 | ✅ Closed (infrastructure) | E |
| #5 | SLP_TYPb PM1b write | ✅ Closed | C |
| #6 | parse_lnk_irc range validation | ✅ Closed | C |
| #7 | thermal/power enumeration | ✅ Closed | D |
| #8 | AcpiScheme fevent | ✅ Closed | A |
| nsmgr deadlock | setrens-before-ready | ✅ Closed | B |
| **#2** | DMAR init (opt-in, hard cap) | ✅ Closed (opt-in, 32-entry cap) | E |
| **#2 kernel** | DMAR real-hardware hang root-cause | 🔴 Open (out of scope: needs QEMU/hardware trace) | — |
| **#4b kernel** | FACS wakeup vector + S3 suspend assembly | 🔴 Open (out of scope: needs kernel-side long-mode assembly + FACS parsing) | — |
**Result: 11 of 13 issues closed-or-improved, 2 still open (both require
hardware-specific work that can't be done in a QEMU-only session).**
The "closed" status on #4b refers to the user-space side: the `_TTS`
and `_WAK` evaluation hooks now exist on `AcpiContext`, and a future
kernel-side S3 entry point can call `enter_sleep_state(state)` then
resume by calling `wake_from_s_state(state)`. The kernel-side
suspend-to-RAM implementation (FACS parsing, wakeup vector setup,
long-mode assembly to invoke `acpi_enter_sleep_state`) is out of
scope for this session.
The "closed" status on #2 refers to the opt-in path: DMAR init can
now be enabled with `REDBEAR_DMAR_INIT=1` on hardware known to work,
with a hard cap of 32 entries preventing any infinite-iterator hang.
The root-cause investigation of the real-hardware hang is out of
scope (would require tracing MMIO reads on the affected hardware).
---
## Phase E outcome — 2026-06-30
**Status: ✅ COMPLETE. _TTS/_WAK API surface added; DMAR opt-in unblocked; ISO rebuilt; QEMU boot verified end-to-end.**
Inner-base commit: `181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap`
### Changes applied (Phase E)
**`drivers/acpid/src/acpi.rs` (three new methods on `AcpiContext`)**
- `transition_to_s_state(state: u8)`: evaluates `_TTS(state)` AML
method. Mirrors Linux 7.1 `acpi_sleep_tts_switch` (drivers/acpi/sleep.c:36).
Called when the system transitions between sleep states. Failure
is non-fatal: `_TTS` is optional per ACPI spec.
- `wake_from_s_state(state: u8) -> Result<u64, AmlEvalError>`:
evaluates `_WAK(state)` AML method. Mirrors Linux 7.1
`acpi_sleep_finish_wake` (drivers/acpi/sleep.c). Called by
userspace on resume from a sleep state. The ACPI spec requires
the OS to call `_WAK` on the same state that was passed to
`_PTS` before the sleep.
- `enter_sleep_state(state: u8)`: top-level entry point that calls
`_TTS` (Step 0, Linux 7.1) then `set_global_s_state` (Steps 1-5,
Phase D). This is the public API that future kernel S3/S4 paths
should use.
**`drivers/acpid/src/acpi/dmar/mod.rs` (DMAR init hard-capped and opt-in)**
- `Dmar::init()` now calls `Dmar::init_with(acpi_ctx, false)` for
safety (no-op by default).
- New `Dmar::init_with(acpi_ctx, opt_in)` takes an explicit boolean
that callers can set to true.
- The DRHD iteration has a hard cap of 32 entries (real hardware
has 1-4 DRHDs) to prevent any infinite-iterator hang.
- Caller in `AcpiContext::init` reads `REDBEAR_DMAR_INIT=1` from
the environment and passes that to `Dmar::init_with`.
This unblocks DMAR on QEMU and on hardware known to work, while
keeping it safe-by-default on real hardware where the hang is
reproducible.
### Verified by
- `CI=1 ./local/scripts/build-redbear.sh redbear-mini` succeeded with
exit 0. ISO at `build/x86_64/redbear-mini.iso` (512 MB) at
2026-06-30 07:11.
- QEMU boot reaches `Red Bear login:` prompt cleanly with no errors.
Both `@inputd:661` and `@ps2d:96` startup logs visible
(Phase A). `redbear-sessiond` working with login1 registered
on D-Bus (Phase B).
### Out-of-scope (requires hardware work)
- **Gap #2 root-cause** — DMAR init opt-in is unblocked. The actual
real-hardware hang root-cause would require tracing MMIO reads on
the affected hardware.
- **Gap #4b kernel side** — `_WAK` evaluation hook is in place.
The kernel-side FACS parsing + wakeup vector + long-mode assembly
for S3 suspend is out of scope for this session.
### Commit chain
```
5f1da5250 redbear-sessiond: port ACPI shutdown watcher to new Fd-based scheme
181a36a base: add _TTS/_WAK AML hooks + opt-in DMAR init with hard cap
8140a2c base: refactor set_global_s_state to follow Linux 7.1 acpi_enter_sleep_state
d844111 base: close SLP_TYPb, parse_lnk_irc, AML mutex, and S5 gaps
ae57fe3 base: re-sync ACPI userspace with upstream master
4f2a043 kernel: re-sync ACPI subsystem with upstream master
de9d1f4 base: ps2d/inputd — add startup info logs for boot diagnostics
```
Inner kernel fork: `local/sources/kernel/` at `4f2a043`.
Inner base fork: `local/sources/base/` at `8140a2c`.
local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md
-356
View File
@@ -1,356 +0,0 @@
# Canonical Build System Improvement Proposal
**Status:** Proposal — not yet implemented
**Context:** Identified while debugging the `redbear-mini` boot crash chain (`require_zero_offset`, `ContextHandle::Start`, `sys_statfs cbindgen.toml`, `SYS_MKNS`/`SYS_SETNS`, `mmap_min_addr` capping). The crash fixes were correct, but getting them through the build pipeline was painful and error-prone. This document proposes specific improvements to make that workflow reliable.
---
## Problems Encountered (Root-Caused)
1. **Edits in `local/sources/<fork>/` are silently stashed** by `build-redbear.sh` (line 160-172), then never restored. Build runs against the clean HEAD. User loses WIP from the working tree.
2. **`--force-rebuild` bypasses the BLAKE3 content-hash cache** (cook_build.rs:462) and all 14 validation gates in `build-redbear.sh`.
3. **Manual `repo cook` outside `build-redbear.sh`** skips stale detection, prefix rebuild, pre-cook sequencing, source-fingerprint tracking, and CI=1 (TUI protection).
4. **Source invalidation is mtime-only** (cook_build.rs:425-442). A `git checkout` or `cp -a` that preserves timestamps produces a stale cache. No content hash is computed for any source file.
5. **Cookbook binary staleness** (build-redbear.sh:182-185): only checks existence, not whether `src/` has changed since last build.
6. **No failure cleanup** in `build-redbear.sh`: no `trap` handler, no signal handler, partially-built artifacts left in `repo/` after failure.
7. **No strict-by-default gate** for uncommitted edits: `verify-durable-source-edits.py` only blocks under `--strict`, which is not the default.
8. **`recipes/core/relibc/source` is stashed but the other 5 fork sources are not** — silent inconsistency.
---
## Proposed Improvements (Ranked by Impact)
### Improvement 1: Make `stash_nested_repo_if_dirty` restore on exit
**Where:** `local/scripts/build-redbear.sh:160-172`
**Impact:** HIGH — eliminates the "where did my edits go?" confusion
**Complexity:** LOW
The current `stash_nested_repo_if_dirty` pushes a stash with `--all` and never restores it. Replace with a `trap`-based push-and-restore pattern:
```bash
REDBEAR_STASHED_REPOS=()
stash_nested_repo_if_dirty() {
local target_dir="$1"
local label="$2"
if [ -d "$target_dir/.git" ]; then
if ! git -C "$target_dir" diff --quiet || \
! git -C "$target_dir" diff --cached --quiet || \
[ -n "$(git -C "$target_dir" ls-files --others --exclude-standard)" ]; then
echo ">>> Stashing dirty $label checkout..."
rm -f "$target_dir/.git/index.lock"
if git -C "$target_dir" stash push --all \
-m "build-redbear-auto-stash-$(date +%s)" >/dev/null 2>&1; then
REDBEAR_STASHED_REPOS+=("$label:$target_dir")
fi
fi
fi
}
restore_all_stashes() {
local rc=$?
local entry
for entry in "${REDBEAR_STASHED_REPOS[@]}"; do
local label="${entry%%:*}"
local dir="${entry#*:}"
if [ -d "$dir/.git" ]; then
echo ">>> Restoring $label working tree..."
rm -f "$dir/.git/index.lock"
git -C "$dir" stash pop >/dev/null 2>&1 || \
echo "WARN: failed to restore $label stash — check 'git -C $dir stash list'"
fi
done
exit $rc
}
trap restore_all_stashes EXIT
```
Apply this to ALL fork sources, not just relibc:
```bash
for fork in relibc kernel base bootloader installer redoxfs; do
local_fork_dir="$PROJECT_ROOT/local/sources/$fork"
if [ -d "$local_fork_dir/.git" ]; then
stash_nested_repo_if_dirty "$local_fork_dir" "$fork"
fi
done
```
---
### Improvement 2: Source-content hashing for cache invalidation
**Where:** `src/cook/cook_build.rs:425-442` and `src/cook/fs.rs:160-168`
**Impact:** HIGH — eliminates the "stale cache after git checkout" bug class
**Complexity:** MEDIUM
Replace the mtime-based `modified_dir_ignore_git()` with a content-hash approach that covers all meaningful inputs:
```rust
// In src/cook/fs.rs or a new src/cook/source_hash.rs
fn compute_source_content_hash(source_dir: &Path, recipe_toml: &Path, patches: &[Path]) -> String {
let mut hasher = blake3::Hasher::new();
// Hash source directory contents (file-by-file, sorted by path)
if let Ok(entries) = walk_dir_files_sorted(source_dir) {
for (rel_path, abs_path) in entries {
// Skip .git, target/, *.swp, etc.
if rel_path.starts_with(".git/") || rel_path.contains("/target/") {
continue;
}
hasher.update(rel_path.as_bytes());
hasher.update(b"\0");
if let Ok(content) = std::fs::read(&abs_path) {
hasher.update(&content);
}
hasher.update(b"\0");
}
}
// Hash recipe.toml
if let Ok(content) = std::fs::read(recipe_toml) {
hasher.update(b"recipe.toml\0");
hasher.update(&content);
}
// Hash each patch file
for patch in patches {
if let Ok(content) = std::fs::read(patch) {
hasher.update(patch.file_name().unwrap().as_encoded_bytes());
hasher.update(b"\0");
hasher.update(&content);
}
}
hasher.finalize().to_hex().to_string()
}
```
In `build()` (cook_build.rs:425-460):
```rust
// Replace mtime-based source_modified with content hash
let source_hash = compute_source_content_hash(source_dir, &recipe_dir.join("recipe.toml"), &patches);
let stored_hash_file = get_sub_target_dir(target_dir, "source_hash.txt");
let source_changed = match std::fs::read_to_string(&stored_hash_file) {
Ok(stored) if stored.trim() == source_hash => false,
_ => true,
};
// ... after successful build:
std::fs::write(&stored_hash_file, &source_hash)?;
```
This makes the cache invalidation robust against `git checkout`, `cp -a`, and any other operation that changes content without updating mtime. It also eliminates the cbindgen.toml-specific fragility: the generated headers' source is the `*.rs` files + `cbindgen.toml`, all of which are hashed.
---
### Improvement 3: Cookbook binary freshness check
**Where:** `local/scripts/build-redbear.sh:182-185`
**Impact:** MEDIUM — prevents stale binary causing silent failures
**Complexity:** LOW
```bash
COOKBOOK_SRC_FINGERPRINT="$PROJECT_ROOT/target/release/.cookbook-src-fingerprint"
COOKBOOK_BIN="$PROJECT_ROOT/target/release/repo"
COOKBOOK_SRC_HASH=$(find "$PROJECT_ROOT/src" -name "*.rs" -print0 | sort -z | xargs -0 sha256sum | sha256sum | cut -d' ' -f1)
NEEDS_REBUILD=0
if [ ! -f "$COOKBOOK_BIN" ]; then
NEEDS_REBUILD=1
elif [ ! -f "$COOKBOOK_SRC_FINGERPRINT" ] || [ "$(cat "$COOKBOOK_SRC_FINGERPRINT")" != "$COOKBOOK_SRC_HASH" ]; then
NEEDS_REBUILD=1
fi
if [ "$NEEDS_REBUILD" = "1" ]; then
echo ">>> Rebuilding cookbook binary (source changed or missing)..."
cargo build --release
echo -n "$COOKBOOK_SRC_HASH" > "$COOKBOOK_SRC_FINGERPRINT"
fi
```
---
### Improvement 4: Strict-by-default uncommitted-edit gate
**Where:** `local/scripts/build-redbear.sh` and `local/scripts/verify-durable-source-edits.py:48`
**Impact:** MEDIUM — prevents "what was that warning?" confusion
**Complexity:** LOW
Add an explicit gate before any source-touching operation:
```bash
# After .config parsing, before any fetch/cook
echo ">>> Checking for uncommitted source edits..."
DIRTY_FORKS=()
for fork in local/sources/relibc local/sources/kernel local/sources/base \
local/sources/bootloader local/sources/installer local/sources/redoxfs; do
fork_dir="$PROJECT_ROOT/$fork"
if [ -d "$fork_dir/.git" ] && \
(! git -C "$fork_dir" diff --quiet HEAD 2>/dev/null || \
! git -C "$fork_dir" diff --cached --quiet HEAD 2>/dev/null || \
[ -n "$(git -C "$fork_dir" ls-files --others --exclude-standard 2>/dev/null)" ]); then
DIRTY_FORKS+=("$fork")
fi
done
if [ ${#DIRTY_FORKS[@]} -gt 0 ]; then
echo "WARNING: uncommitted edits detected in:"
for f in "${DIRTY_FORKS[@]}"; do
echo " - $f"
done
echo ""
if [ "${REDBEAR_ALLOW_DIRTY:-0}" = "1" ]; then
echo ">>> REDBEAR_ALLOW_DIRTY=1 set; proceeding with WIP edits."
else
echo "ERROR: refuse to build with uncommitted edits."
echo " Either commit your changes or set REDBEAR_ALLOW_DIRTY=1 to override."
exit 1
fi
fi
```
For `verify-durable-source-edits.py`, flip the default to strict and make `--no-strict` the override:
```python
# Line 48: change default from non-strict to strict
if not args.no_strict and dirty:
print("ERROR: uncommitted edits detected in upstream-owned source trees.")
print(" Use --no-strict to allow, or commit your changes.")
sys.exit(1)
```
---
### Improvement 5: Stash-all-forks consistency
**Where:** `local/scripts/build-redbear.sh:160-172`
**Impact:** MEDIUM — eliminates the "only relibc is special" asymmetry
**Complexity:** LOW
Combined with Improvement 1, replace the single-relibc stash with a loop over all fork sources. The key insight: `local/sources/<fork>/` is the source of truth, but the cookbook fetches into `recipes/core/<fork>/source` (which is a symlink to `local/sources/<fork>/`). Stashing must happen at the `local/sources/<fork>/` level, not at the recipe-level symlink.
---
### Improvement 6: Failure-cleanup trap
**Where:** `local/scripts/build-redbear.sh` (top-level, after `set -euo pipefail`)
**Impact:** MEDIUM — makes failed builds debuggable
**Complexity:** LOW
```bash
REDBEAR_BUILD_STATE_DIR=$(mktemp -d)
cleanup_on_failure() {
local rc=$?
if [ $rc -ne 0 ]; then
echo ""
echo "========================================"
echo " BUILD FAILED (exit code: $rc)"
echo "========================================"
echo "Diagnostic info preserved at: $REDBEAR_BUILD_STATE_DIR"
echo ""
echo "Last 30 lines of each recipe log:"
for log in /tmp/build-*.log; do
[ -f "$log" ] || continue
echo "--- $log ---"
tail -30 "$log"
echo ""
done 2>/dev/null
echo "Recipe build state:"
for d in "$PROJECT_ROOT"/recipes/*/target/*/; do
[ -d "$d" ] || continue
echo " $d: $(du -sh "$d" 2>/dev/null | cut -f1)"
done
fi
rm -rf "$REDBEAR_BUILD_STATE_DIR"
# Then restore stashes (from Improvement 1)
restore_all_stashes
}
trap cleanup_on_failure EXIT
```
---
### Improvement 7: Pre-cook offline/upstream consistency
**Where:** `local/scripts/build-redbear.sh:339-357`
**Impact:** LOW-MEDIUM — prevents pre-cook from fetching packages the main build can't use
**Complexity:** LOW
```bash
# Replace line 349
if [ "${REDBEAR_ALLOW_UPSTREAM:-0}" = "1" ]; then
COOKBOOK_OFFLINE=false "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1
elif [ -n "${REDBEAR_RELEASE:-}" ]; then
COOKBOOK_OFFLINE=true "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1
else
COOKBOOK_OFFLINE=true "$PROJECT_ROOT/target/release/repo" cook "$pkg" >"$log" 2>&1
fi
```
---
### Improvement 8: Cookbook protection against out-of-band invocations
**Where:** `src/bin/repo.rs` (CLI entry point)
**Impact:** LOW — prevents accidental misuse, gentle guardrail
**Complexity:** LOW
Add a non-blocking warning when `repo cook` is invoked outside `build-redbear.sh`:
```rust
// In src/bin/repo.rs main_inner()
let canonical_marker = std::env::var_os("REDBEAR_CANONICAL_BUILD");
if canonical_marker.is_none() {
eprintln!("WARNING: repo cook invoked outside build-redbear.sh.");
eprintln!(" Cache invalidation, prefix rebuild, and source-fingerprint");
eprintln!(" tracking will not run. Set REDBEAR_CANONICAL_BUILD=1 to suppress");
eprintln!(" this warning if you know what you're doing.");
}
```
In `build-redbear.sh`:
```bash
export REDBEAR_CANONICAL_BUILD=1
```
---
## Implementation Order
Suggested order (each step is independently useful and can be merged separately):
1. **Improvement 1 + 5** (trap-based stash restore, all forks) — eliminates the biggest UX pain point
2. **Improvement 6** (failure-cleanup trap with diagnostics) — makes failures debuggable
3. **Improvement 3** (cookbook binary freshness) — 5-line change, high signal
4. **Improvement 4** (strict-by-default dirty gate) — prevents repeat-accumulation of uncommitted edits
5. **Improvement 7** (pre-cook offline consistency) — small correctness fix
6. **Improvement 8** (cookbook out-of-band warning) — gentle nudge
7. **Improvement 2** (source content hashing) — biggest engineering effort, defer until after the above land
---
## Testing Strategy
Each improvement needs:
1. Unit test in `src/` if it touches cookbook code (cookbook has a test directory)
2. Manual smoke test: `./local/scripts/build-redbear.sh redbear-mini` end-to-end
3. Regression test: dirty `local/sources/relibc/` + `./local/scripts/build-redbear.sh redbear-mini` + verify stash is restored on exit
4. Documentation update: this file's "Problems Encountered" should be empty after all improvements land
---
## Related Documents
- `local/AGENTS.md` — project-wide build and commit policies
- `local/docs/LOCAL-FORK-SUPREMACY-POLICY.md` — why local forks must be complete and committed
- `local/docs/BUILD-CACHE-PLAN.md` — content-hash cache design (Phase 1-3 status, sysroot deferral)
- `local/docs/BUILD-SYSTEM-INVARIANTS.md` — what the build system guarantees
- `local/docs/BUILD-SYSTEM-HARDENING-PLAN.md` — broader hardening roadmap
- `mk/repo.mk` — make-integration of cookbook
- `local/scripts/build-redbear.sh` — canonical entry point
local/docs/BUILD-SYSTEM-IMPROVEMENTS.md
+5
View File
@@ -1,5 +1,10 @@
# Build System Improvements — v6.0 Post-Mortem (2026-06-12)
> **Status (2026-07-18): HISTORICAL POST-MORTEM.** This is a dated record of
> the v6.0 desktop bring-up cycle. Some items below were implemented, others
> abandoned; it is not a list of pending work. For current build-system work
> see `BUILD-SYSTEM-HARDENING-PLAN.md` and `local/scripts/TOOLS.md`.
This document analyzes the build system gaps that surfaced during the v6.0
KDE/Qt/Plasma desktop path bring-up (2026-04 through 2026-06) and
proposes targeted, low-risk improvements. Each improvement is sized as
local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md
-313
View File
@@ -1,313 +0,0 @@
# Red Bear OS Build-System v6.0 Hardening — Post-Mortem
> **Scope.** This document is the durable record of the
> 14-session v6.0 build-system hardening work arc (2026-06-08 to
> 2026-06-12). It captures the 10 build-system improvements
> (9 DONE, 1 OPEN), 32 findings addressed, the Gitea Actions CI
> pipeline, the 149-test suite covering all 17 classifier rules +
> their false-positive inverses plus the 6 new status and 7 new
> scheduler Rust unit tests, and the deferred follow-up work.
> The 7,000+ uncommitted file modifications in the user's working
> tree are not part of this post-mortem — they are ongoing WIP.
>
> **Durability caveat (added 2026-06-12 after final review).**
> The deliverables in this arc are durable **on disk** in the
> working tree, and most are now durable in `git` history. The 13
> most recent commits on `0.2.3` (`b8c1c780d`, `975cda686`,
> `e1c2e7958`, `0f8ad8a50`, `9e5794ea7`, `827895d32`, `693e4d774`,
> `fbc32a6d8`, `5325360b4`, `ae749ffb2`, `97fa3a17a`, `bd18eefc6`,
> `03c8a38a1`) cover the first durable C-7 migration patch
> (`local/patches/kf6-karchive/01-initial-migration.patch`); the
> `make lint-build-system-all` aggregate + 11-job CI; the
> postmortem rebalance to 13-session / 9.5-DONE; the
> `make scratch-rebuild` target wiring; the improvement #10
> scratch-rebuild skeleton + 21 tests + Makefile + Gitea CI
> integration; the migration-dry-run CI integration; the C-7 KF6
> sed migration script v2 + 13 tests + Makefile + Gitea CI
> integration; the postmortem update to 13-session / 9.5-DONE
> / 120-Python-test state; the parallel cook pool; the cook
> status reporter; the build-system hardening arc (5 of 10
> improvements); the BUILD-SYSTEM-IMPROVEMENTS.md doc;
> `classify-cook-failure.py`; `audit-patch-idempotency.py`; and
> the auto-link Qt sysroot dirs patch in `src/cook/script.rs`.
> The remaining v6.0 deliverable still in `git status` is this
> BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md (updated to 14-session
> / 9.5-DONE / 122-Python-test state, with the Session 14
> entry + the b8c1c780d commit-history row). The C-1..C-6 doc
> and code fixes, `boot-logs/README.md`, and
> `migrate-kf6-seds-to-patches.sh` (v1) were committed in
> `ae749ffb2`; v2 rewrite of the script + 13 tests + Makefile +
> Gitea CI integration in `827895d32`; the #10 scratch-rebuild
> skeleton + 21 tests + Makefile + Gitea CI integration in
> `0f8ad8a50`; the migration-dry-run CI integration in
> `9e5794ea7`; the make-wrapper + postmortem rebalance in
> `e1c2e7958`; the lint-build-system-all aggregate + 11-job CI
> in `975cda686`; the first durable C-7 patch + migration
> script v3 fix in `b8c1c780d`. Going forward, any new v6.0
> work should be committed with
> `git add <specific-files>` to avoid sweeping the user's
> 7,000+ unrelated WIP modifications.
## Timeline
| Session | Date (2026) | Focus |
|---------|------------|-------|
| 1 | 06-08 | v6.0 policy compliance pass: 10 build-system improvements, 4 audit scripts, 7 make lint targets, 31 tests |
| 2 | 06-09 | Comprehensive doc cleanup: 12/12 docs pass review, 4 high-priority fixes (AGENTS.md, local/AGENTS.md, README.md, SCRIPT-BEHAVIOR-MATRIX.md), 3 file deletions |
| 3 | 06-10 | P0 audit-script hardening: 5 reviewer findings fixed, doc reconciliation |
| 4 | 06-11 | Audit script review + comprehensive review: 32 findings categorized CRITICAL/HIGH/MEDIUM/LOW, comprehensive fix pass |
| 5 | 06-12 | Final refinements: `local/AGENTS.md:367` reframed, KF6 migration tool created, deferred work documented |
| 6 | 06-12 (cont.) | Hidden risk fix: cub-assessment deleted (874 MB), migrate-kf6-seds-to-patches.sh +x, postmortem accuracy corrections |
| 7 | 06-12 (cont.) | Test coverage gap: added 12 missing positive rule tests + 12 false-positive tests (55/55 pass). Discovered + fixed 6 over-broad multi-pattern rules. Created `.gitea/workflows/build-system.yml` (7-job Gitea Actions pipeline, host-execution, Manjaro/Arch) and `.gitea/RUNNER-SETUP.md` (one-time host setup). Wired into `make test-lint-scripts[-quiet]`. |
| 8 | 06-12 (cont.) | **Build-system improvement #2 shipped**: `local/scripts/repair-cook.sh` (incremental-build optimizer, 134 lines) + 7 unit tests (`local/scripts/tests/test_repair_cook.py`). `make repair.<pkg>` and `make clean-repair.<pkg>` targets wired. Verified P0 audit-script fixes work on real upstream KF6 source (form 1 nested namespace, comment/string strip, .bak timestamp). **Test count: 62/62 pass.** |
| 9 | 06-12 (cont.) | **Build-system improvement #5 shipped**: `local/scripts/lint-recipe.py` (380 lines, 7 rules) + 24 unit tests (`local/scripts/tests/test_lint_recipe.py`). Recipe-index precomputation drops `--all` runtime from 60s+ to 1.1s. `make lint-recipe`, `make lint-recipe.<pkg>`, `make lint-recipe.strict`, `make lint-recipe.<pkg>.strict` wired. New `lint-recipe` Gitea Actions job (job 4 of 8) added to `.gitea/workflows/build-system.yml`. First run on the live tree found: 1 broken-patch reference (`redbear-sessiond/P4-signal-implementations.patch`), 1 dangling `cookbook_apply_patches` call (`tc`), 19 sed -i calls in sddm (warning only — `cookbook_apply_patches` present), 4 sed -i calls in `qt6-wayland-smoke` (uncovers the kind of bug the libwayland fix was preventing). **Test count: 86/86 pass.** |
| 10 | 06-12 (cont.) | **Build-system hardening arc commit + improvement #4 shipped.** First durable commit: `ae749ffb2 build: ship build-system hardening arc (5 of 10 improvements)` — 22 build-system files, including the 5 prior arc deliverables (audit-kf6-deps.py + 13 tests, repair-cook.sh + 7 tests, migrate-kf6-seds-to-patches.sh, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, build-system.yml, Gitea RUNNER-SETUP.md, libdrm/02 sidecar, cache/README cleanup, Makefile lint/repair targets). Then `5325360b4 build: add cook status reporter (improvement #4)` — `src/cook/status.rs` (197 lines, 6 unit tests) + `src/bin/repo.rs` wiring. Auto-enables in `CI=1` mode when stderr is a TTY: one-line `[NN/MM] recipe: phase (Xs)` output for each cook. Verified end-to-end with 3-recipe and 5-recipe real cooks. **Test count: 86/86 Python + 20/20 Rust.** |
| 11 | 06-12 (cont.) | **Build-system improvement #1 shipped.** `src/cook/scheduler.rs` (145 lines, 7 unit tests) + `src/bin/repo.rs` `repo cook --jobs=N` flag. Dep-aware level partition via `dep_levels()` (each recipe's level = `1 + max(level of any direct dep in this vec)`, or 0 if no deps in the vec). For each level, runs all recipes in that level via `std::thread::scope` with up to `N` workers. Drain-after-spawn pattern keeps live-worker count <= jobs. Ratatui TUI path unchanged. 7 unit tests cover empty / single / linear / independent / diamond / dev_dependencies / unknown-dep. Verified end-to-end: 5-recipe batch (redbear-statusnotifierwatcher, redbear-traceroute, redbear-udisks + deps expat, dbus) cooks in level 0 (3 parallel) → level 1 (dbus) → level 2 (redbear-udisks). On clean 3-recipe rebuild: 48s serial vs 45s parallel. Speedup bounded by longest single build (17s) on this small batch — the 2-3x gain from the proposal is on 15-recipe KF6 batches with 5-10 min longest builds. Caveat: `build/qt-host-build` host toolchain not yet locked; v2 mitigation is `flock` in `src/cook/script.rs` (deferred, no current redbear-full test recipe triggers qt-host-build). **Test count: 86/86 Python + 27/27 Rust.** |
| 12 | 06-12 (cont.) | **C-7 KF6 sed migration script v2 + CI integration.** The v1 shipped in `ae749ffb2` was a stub with three structural bugs that made it unrunnable: called `repo cook <recipe_dir>` with a path (cookbook takes bare names); created an empty pristine_dir via mktemp -d but never populated it; Step 4 was `SKIP — manual rewrite pending` so the script wrote no patch even when the inline sed chains actually edited the source. Replaced with a working v2: bare-name cookbook CLI, real pristine-source snapshot (`cp -r source/ source-pristine/`) BEFORE the cook, real diff capture, real patch save to `local/patches/<name>/01-initial-migration.patch`. Added `--dry-run` for safe CI smoke testing, `--recipe=<name>` and `--limit=N` for targeted runs, `--help` for the script's contract. Test escape hatch via `REDBEAR_MIGRATE_RECIPES_DIR` / `REDBEAR_MIGRATE_PATCHES_DIR` env vars so the candidate discovery can be exercised on synthetic trees without touching the live project. 13 unit tests in `local/scripts/tests/test_migrate_kf6_seds.py` — 7 candidate-discovery tests (synthetic tree with `make_recipe()` helper, asserts stdout/stderr + exit code) + 6 script-structure tests (regression guards against the v1 bugs: "uses bare names not paths", "uses release/repo binary", "creates patches dir", "diff includes .git/target excludes", "unfetches after capture", "idempotent SKIP when patch exists"). Wired into `make test-migration-dry-run` and new Gitea Actions job `migration-dry-run` (job 5 of 9, every PR). **Test count: 99/99 Python + 27/27 Rust.** Verified `--dry-run --limit=5` correctly identifies `breeze`, `kde-cli-tools`, `kdecoration`, `kf6-attica`, `kf6-karchive` as the first 5 of 56 candidate recipes. The actual migration run still requires the full KF6 dep chain to be built (qtbase, qtdeclarative, kf6-extra-cmake-modules, plus per-recipe deps); the per-recipe verification + recipe-rewrite remains a manual step (the script's `Next steps:` output documents this). |
| 13 | 06-12 (cont.) | **Improvement #10 (scratch-rebuild) M-sized foundation + CI integration.** The L-sized #10 proposal is split: the M-sized foundation (autotools detection + dep closure + targeted clean + parallel rebuild) is now a runnable 190-line bash script (`local/scripts/scratch-rebuild.sh`); the L-sized remaining work (full integration with `rebuild-cascade.sh`, byte-identical rebuild check via `stage.pkgar` hash diffing, cross-host-toolchain case) is documented but deferred. The script: (1) discovers autotools-using recipes by content regex + AUTOTOOLS_CORE list, (2) computes transitive closure via BFS over the recipe TOML dep graph (both `[build].dependencies` and `[build].dev_dependencies`), (3) deletes `target/<arch>/{build,sysroot,stage.tmp}/` per recipe in the closure (preserves `source/` to avoid re-fetch), (4) re-cooks in dep order via the cookbook's `--jobs=N` flag. Cook errors during step 4 do NOT abort the script — a failed cook may indicate a missing upstream dep on a fresh checkout rather than a real bug. `--dry-run`, `--jobs=N`, `--help` supported. 21 unit tests in `local/scripts/tests/test_scratch_rebuild.py`: 3 autotools-core list tests, 8 regex content-match tests (each canonical autotools command + negative cases), 4 dep-parser tests, 1 help test, 5 script-structure tests (executable bit, uses release/repo, preserves source/, uses --jobs=N, dry-run safe). The test file also surfaced a real Python regex gotcha: `^[[:space:]]*` (POSIX char class with quantifier) silently fails to match the empty string under Python's regex engine; the test uses `^[\s]*` shorthand instead. Wired into `make test-scratch-dry-run`, `make scratch-rebuild`, and new Gitea Actions job `scratch-dry-run` (job 6 of 10). **Test count: 120/120 Python + 27/27 Rust.** Verified `--dry-run` against live tree: discovers 6 autotools users (bison, diffutils, flex, grub, libtool, m4) and computes a 6-recipe closure. `make scratch-rebuild` end-to-end: deletes the 6 recipe dirs' build/sysroot/stage.tmp, runs `repo cook --jobs=4` on the closure. m4 cooks successfully; bison fails (missing host toolchain dep) — the failure is correctly captured to the log without aborting the script. |
| 14 | 06-12 (cont.) | **First durable C-7 KF6 sed migration patch (`b8c1c780d`).** Executed the v2 migration script against `local/recipes/kde/kf6-karchive` and shipped `local/patches/kf6-karchive/01-initial-migration.patch` (24 lines, 1 real hunk). The patch is the result of capturing the diff between the pristine upstream karchive-6.26.0 tarball and the post-cook state — the `ecm_install_po_files_as_qm` line is now commented out (the recipe's `sed -i 's/^ecm_install_po_files_as_qm/#ecm_install_po_files_as_qm/'`). The other 3 sed chains in the recipe (ki18n_install, arg(mode), arg(d->mode)) were no-ops against this upstream version — the migration correctly captures only the real edits. Discovered a real bug in the v2 migration script during the first run: it was producing silently empty diffs on already-cooked recipes because the cookbook's `fetch` re-uses an existing source/ tree. Fix: add explicit `unfetch` BEFORE the `fetch` (so the source/ dir is removed before re-extraction) + set `REDBEAR_ALLOW_LOCAL_UNFETCH=1` (the cookbook's default policy is to never clobber a local-overlay source, and the env var overrides that for the explicit unfetch). The patch was also filtered to drop ECM-autogenerated noise files (`.clang-format`, `.gitignore`, `target/` artifacts); 122-line raw diff → 24-line filtered patch. Added 2 regression tests: `test_sets_local_unfetch_env_var` (guards against forgetting the env var) and `test_unfetches_before_fetching` (guards against the silent-failure mode). **Test count: 122/122 Python + 27/27 Rust.** Next steps for kf6-karchive (manual, not part of this commit): edit `[build].script` to remove the 4 sed chains + add `REDBEAR_PATCHES_DIR` / `cookbook_apply_patches` invocation, re-cook, verify byte-identical `stage.pkgar`, commit recipe rewrite alongside the patch. |## Final state
### 10 build-system improvements — 9 DONE, 1 OPEN
| # | Title | Status | Commit |
|---|-------|--------|--------|
| 3 | Per-recipe patch idempotency auditor | **DONE** | `03c8a38a1` |
| 6 | `recipes/kf6-*` recipe dep audit | **DONE** | `ae749ffb2` |
| 8 | Auto-link Qt sysroot dirs | **DONE** | `03c8a38a1` |
| 9 | Failure classifier | **DONE** | `bd18eefc6` |
| 1 | Parallel-safe cook pool | **DONE** | `fbc32a6d8` |
| 2 | `cook --repair` mode | **DONE** | `ae749ffb2` — `local/scripts/repair-cook.sh` wrapper + `make repair.<pkg>` target |
| 4 | Cook TUI status | **DONE** | `5325360b4` — `src/cook/status.rs` |
| 5 | Build-time recipe lint | **DONE** | `ae749ffb2` — `local/scripts/lint-recipe.py` + 7-rule lint + Gitea CI job |
| 7 | QML gate (4-6 weeks) | open | — (Qt6 engine fix, not a cookbook improvement) |
| 10 | Cookbook scratch-rebuild | open | — (L-sized, 1 week, M risk; separate session) |
### 32 findings — all addressed
**5 P0 audit-script bugs (all fixed):**
- `audit-kf6-deps.py` Form 1 regex truncation of `KF6::Some::Nested::Name` → supports `::`-chained namespaces
- `audit-kf6-deps.py` comment / string-literal false positives → `_strip_cmake_noise` helper
- `audit-kf6-deps.py` `.bak` file silent overwrite on consecutive `--fix` runs → timestamped + collision-resistant
- `classify-cook-failure.py` rule-matching loop duplicated between text and JSON branches → `_match_rules` helper extracted
- `classify-cook-failure.py` `--json` exit-code inversion → documented and tested
**6 additional over-broad multi-pattern rules fixed (Session 7 bonus, found while writing tests):**
- Rules 4 (Qt6::GuiPrivate), 5 (PlasmaWaylandProtocols), 10 (libc.so.6), 12 (Python3), 14 (Package), 16 (fetch denied) each had 2 patterns stored as a list but the matcher uses `all()` semantics. Real cooks fired only one of the two patterns so the rules NEVER fired. Collapsed to 1 pattern each.
### Test coverage — 17/17 classifier rules + 12 false-positive inverses
| Test | Count | Coverage |
|------|-------|----------|
| `test_audit_patch_idempotency.py` | 7 | 3 collect tests, 2 JSON schema tests, 2 name validation tests |
| `test_audit_kf6_deps.py` | 13 | 4 regex-form tests, 5 normalize tests, 1 WIP-skip test, 1 no-fetch honesty test, 1 KF6/Qt6 test, 1 component discovery test |
| `test_classify_cook_failure.py` | 35 | 17 positive rule tests (1 per rule), 12 false-positive tests, 5 existing exit-code/JSON/explain-rule tests, 1 --no-fetch honesty test |
| `test_repair_cook.py` | 7 | synthetic recipe fixtures, fast/slow path logic, --clean-build, REPAIR_FORCE |
| `test_lint_recipe.py` | 24 | 7 rule coverage, 1 recipe-index cache, 1 clean-recipe regression test, 1 error recipe test |
| `test_migrate_kf6_seds.py` | 13 | 7 candidate-discovery tests (synthetic tree, exit-code + stdout/stderr assertions) + 6 script-structure tests (regression guards against v1 bugs) |
| `test_scratch_rebuild.py` | 21 | 3 autotools-core list tests + 8 regex content-match tests (each canonical autotools command + negatives) + 4 dep-parser tests + 1 help test + 5 script-structure tests (executable, release/repo, preserves source/, --jobs=N, dry-run safe) |
| `cook::status` (Rust) | 6 | format_elapsed boundaries, disabled no-op, phase tracking |
| `cook::scheduler::dep_levels` (Rust) | 7 | empty / single / linear / independent / diamond / dev_dependencies / unknown-dep |
**Total: 122/122 Python + 27/27 Rust pass in <1 second (Python) / ~3 seconds (Rust).**
**8 CRITICAL findings (all addressed):**
- C-1 libwayland `patches = [redox.patch]` line removed (was blocking the Wayland stack)
- C-2 libdrm/02 broken hunk documented with sidecar README; regen procedure in `local/patches/libdrm/02-redox-dispatch.patch.README`
- C-3 orphan `local/sources/{pipewire,wireplumber}/` removed (22 MB)
- C-4 kernel `.gitignore` fixed to recursive `/target`
- C-5 broken driver symlinks re-pointed at `local/recipes/drivers/...`
- C-6 sddm stub headers documented as known maintenance debt in `local/recipes/kde/sddm/stubs/README.md`
- C-7 56 KF6 recipes with `sed -i` chains → **FULLY COMPLETE.** Migration script at `local/scripts/migrate-kf6-seds-direct.sh` (working without `repo cook`); cleanup scripts at `local/scripts/cleanup-kf6-noop-seds.sh` (24 recipes, full removal) and `local/scripts/cleanup-kf6-noop-seds-targeted.sh` (6 recipes, targeted removal); edit script at `local/scripts/edit-kf6-recipes-for-patches.sh`. **Result: 29 migration patches in `local/patches/<name>/` (all verified `git apply --check` clean), 24 recipes' `[build].script` rewritten to call `cookbook_apply_patches`, 30 NO-OP recipes with dead sed chains removed, 164 Python tests passing (8 unit test files + 2 e2e test files).** 7 remaining NO-OP recipes (breeze, kde-cli-tools, kf6-kbookmarks, kf6-kded6, kglobalacceld, plasma-desktop, plasma-workspace) had their non-ecm sed chains preserved (they target lines that ARE in upstream) — those chains are real Red Bear edits, not migration candidates.
- C-8 2.8 GB of unzipped source cleanup → deferred until C-7 patches are durable
**7 HIGH findings (all addressed):**
- H-1 AGENTS.md ↔ local/AGENTS.md documentation map cross-references added
- H-2 duplicate `redbear-netctl-console/` removed
- H-3 redbear-meta header: false positive (declaration order matches)
- H-4 `cub/source/cub-assessment/` and `gparted-git/` removed (874 MB + 24 KB on disk)
- H-5/H-6/H-7 KF6 source state captured in C-7 migration plan
**8 MEDIUM findings (all addressed or documented):**
- M-2 dead `validate-patches` Makefile target removed
- M-3 legacy config rename deferred (cosmetic)
- M-4 zbus build-ordering marker deferred (user knows)
- M-5 symlink consistency deferred (cosmetic)
- M-6 `make all` → `build-redbear.sh` routing deferred (preserves advanced/unsafe escape)
- M-7 `APPLY_PATCHES` var: false positive (real use at line 158)
- M-8 .bak files removed (libwayland + ncurses)
**9 LOW findings (all addressed):**
- L-1..L-9: doc cleanup, dead code, cosmetic — all in the doc cleanup pass
### Build-system test infrastructure — fully deployed
| Artifact | Status |
|----------|--------|
| `local/scripts/audit-patch-idempotency.py` | 391 lines, exit 0/1/2 contract, JSON schema doc |
| `local/scripts/audit-kf6-deps.py` | 557 lines (4 regex forms), comment/string stripping, TOML-parser-based `--fix` |
| `local/scripts/classify-cook-failure.py` | 462 lines, 17 rules, `_match_rules` helper, `--explain-rule`, inverted exit code documented |
| `local/scripts/migrate-kf6-seds-to-patches.sh` | 6300-byte migration skeleton (NEW) |
| `local/scripts/tests/test_audit_patch_idempotency.py` | 7 tests |
| `local/scripts/tests/test_audit_kf6_deps.py` | 13 tests |
| `local/scripts/tests/test_classify_cook_failure.py` | 11 tests |
| `make lint-patches`, `make lint-patches-full` | wired to audit-patch-idempotency.py |
| `make lint-kf6-deps` | wired to audit-kf6-deps.py |
| `make lint-cook-failure`, `make lint-cook-failure-explain` | wired to classify-cook-failure.py |
| `make lint-build-system`, `make lint-build-system-full` | aggregate targets |
**Test status:** 31/31 pass in <1 second. CI-safe exit codes (0=clean, 1=failures, 2=all-skipped).
## Deferred to future sessions
1. **C-7**: ✅ **DONE** (this update, 2026-06-12). 29 migration patches shipped + 24 recipes' build scripts rewritten to use `cookbook_apply_patches`. 164 Python tests pass. See C-7 entry above for full status.
2. **C-8**: 2.8 GB of unzipped source cleanup → ready to ship now that C-7 is complete. Run `find local/recipes -maxdepth 4 -name 'source.tar' -size +10M -delete` (verify with `du -sh local/recipes/*/source.tar | sort -h` first). Estimated 1 hour.
2. **C-2 regen**: Regenerate `local/patches/libdrm/02-redox-dispatch.patch` against current libdrm 2.4.125. The 8-step procedure is documented in `local/patches/libdrm/02-redox-dispatch.patch.README`. Estimated 30-60 minutes if a libdrm build host is available.
3. **6 open build-system improvements** (parallel cook, cook --repair, cook TUI, build-time lint, QML gate, cookbook scratch-rebuild) — each is a multi-session project on its own.
4. **User's WIP**: 7,000+ file modifications in the working tree, primarily the user's ongoing KF6 work, cub improvements, redbear-netctl development, and libpciaccess integration. Not in scope for this post-mortem.
## Commit history (v6.0 hardening arc, durable)
The v6.0 deliverables were committed in 3 durable chunks on
2026-06-12, after the post-mortem was first written:
| Commit | Files | What it landed |
|--------|-------|----------------|
| `ae749ffb2` | 22 | Build-system hardening arc: audit-patch-idempotency, audit-kf6-deps, classify-cook-failure, repair-cook, migrate-kf6-seds-to-patches, the 4 Python test files, BUILD-SYSTEM-IMPROVEMENTS.md, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, libdrm/02 sidecar README, cache/README deletion, .gitea/workflows/build-system.yml (8 jobs), .gitea/RUNNER-SETUP.md, Makefile lint + repair targets. C-1..C-6 + 7 HIGH findings all addressed. |
| `5325360b4` | 4 | Cook TUI status reporter (#4): `src/cook/status.rs` (197 lines, 6 unit tests), wired into `src/bin/repo.rs`. One-line `[NN/MM] recipe: phase (Xs)` output for non-TUI cooks. |
| `fbc32a6d8` | 4 | Parallel cook pool (#1): `src/cook/scheduler.rs` (145 lines, 7 unit tests), `--jobs=N` CLI flag in `src/bin/repo.rs`, `dep_levels()` topological partition via `std::thread::scope`. |
| `827895d32` | 2 | C-7 KF6 sed migration script v2: rewrote the v1 stub to actually capture diffs (`cp -r source/ source-pristine/` BEFORE the cook, real `diff -ruN`, real patch save to `local/patches/<name>/01-initial-migration.patch`). 13 unit tests in `test_migrate_kf6_seds.py`. |
| `9e5794ea7` | 4 | CI integration for the migration script: `make test-migration-dry-run` target + Gitea Actions `migration-dry-run` job (job 5 of 9 → 10). |
| `0f8ad8a50` | 5 | Improvement #10 (scratch-rebuild) M-sized foundation: `local/scripts/scratch-rebuild.sh` (190 lines, +x) + 21 unit tests in `test_scratch_rebuild.py`. `make test-scratch-dry-run` + `make scratch-rebuild` targets. Gitea Actions `scratch-dry-run` job (job 6 of 10). |
| `e1c2e7958` | 2 | Wired `make scratch-rebuild` (no-dry-run variant). Postmortem rebalance to 13-session / 9.5-DONE / 120-Python-test state. |
| `975cda686` | 4 | New `make lint-build-system-all` aggregate (offline-safe lints + tests + smoke). New Gitea Actions job (job 7 of 11). |
| `b8c1c780d` | 3 | First durable C-7 migration patch (`local/patches/kf6-karchive/01-initial-migration.patch` — 24 lines, 1 real hunk). Discovered + fixed the v2 script's silent-failure mode: cookbook `fetch` re-uses existing source/; migration must `unfetch` first + set `REDBEAR_ALLOW_LOCAL_UNFETCH=1`. Added 2 regression tests for the env-var + unfetch-before-fetch invariants. |
**This post-mortem itself** is still in `git status` (modified
`local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md`); the
"Durability caveat" at the top of this document tracks its
shipment status.
## What remains uncommitted
| Path | What it is | Why uncommitted |
|------|-----------|-----------------|
| `local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md` | This doc updated for 14-session / 9.5-DONE / 122-Python-test state (Session 14 entry + b8c1c780d commit-history row) | Next user-chosen commit; touches paths the user may have other WIP for |
| `local/docs/BUILD-SYSTEM-FINGERPRINT-HARDENING-PLAN.md` | User WIP plan (Phase 6+, "pending Oracle fingerprint architecture review") | Draft, not for committing in this arc |
| User's `AGENTS.md`, `local/AGENTS.md`, `README.md`, `config/redbear-*.toml`, `local/sources/{base,bootloader,kernel}` | 7,000+ modifications | User WIP; not in this arc |
---
## v6.1 Addendum (2026-06-21) — Cleanup Safety + qtdeclarative qfeatures.h Fix
### Finding C-7.1: Ad-hoc cleanup scripts deleted tracked sources
**Date:** 2026-06-21
**Severity:** HIGH (caused build failure, required manual source restoration)
**Status:** FIXED
#### Problem
An ad-hoc cleanup script (`/tmp/cleanup-redbear.sh`) was used to free
disk space before a full rebuild. The script used `rm -rf` on
patterns that matched **tracked source files** under
`local/recipes/*/source/` and `local/sources/*/source/`. Specifically:
- `local/recipes/libs/libepoxy/source/` (deleted)
- `local/recipes/libs/libdisplay-info/source/` (deleted)
- `local/recipes/libs/libxcvt/source/` (deleted)
- `local/recipes/libs/lcms2/source/` (deleted)
- `local/recipes/system/diskd/source/` (deleted)
- `local/recipes/system/devfsd/source/` (deleted)
- `local/patches/tokio/vendored/src/runtime/task/core.rs` (deleted)
These sources were committed to git history (commits `f36b6a4582`,
`8782592b9a`, `7c49c45e0e`) and had to be manually restored via
`git checkout`.
#### Root cause
The cleanup script did not check whether paths were tracked by git
before deletion. The mainline Redox `make clean` and `make distclean`
targets DO check this, but the ad-hoc script bypassed them.
#### Fix
Added `local/scripts/cleanup-build.sh` — a git-aware cleanup script
that:
1. Uses `git ls-files --error-unmatch` to check every path before
deletion
2. Skips and warns about tracked paths instead of deleting them
3. Supports three modes: `artifacts` (safest), `--all` (also removes
recipe `target/`), `--nuclear` (also removes local recipe
`target/`)
4. Refuses to run outside a git repository
Updated `AGENTS.md` to document the safe cleanup procedure and
explicitly warn against ad-hoc `rm -rf`.
### Finding C-7.2: qtdeclarative missing qfeatures.h caused `division by zero in #if`
**Date:** 2026-06-21
**Severity:** CRITICAL (blocked SDDM and KWin builds)
**Status:** FIXED
#### Problem
The `local/recipes/qt/qtdeclarative/recipe.toml` manually copied
include files from the build directory to the staged sysroot. This
missed the build-time generated `qfeatures.h` (and per-module
`*config.h` files like `qtquick-config.h`).
Without `qfeatures.h`, the preprocessor expansion
`QT_CONFIG(quick_shadereffect)` and `QT_CONFIG(quick_draganddrop)`
in `qquickitem.h` (lines 119, 460) hit a deliberate `1/0` divide-by-zero
trap, producing these errors when compiling `sddm-greeter-qt6`:
```
qquickitem.h:119:5: error: division by zero in #if
119 | #if QT_CONFIG(quick_shadereffect)
qquickitem.h:460:5: error: division by zero in #if
460 | #if QT_CONFIG(quick_draganddrop)
```
#### Root cause
Two issues in the qtdeclarative recipe:
1. **Missing `cmake --install`** — qtbase, qtsvg, and other Qt module
recipes use `cmake --install` to properly stage all generated files
(including `qfeatures.h`). The qtdeclarative recipe skipped this
step and copied files manually.
2. **Empty `qtquick-config.h`** — Even with `cmake --install`, the
`qtquick-config.h` file was being generated empty because the
recipe did not explicitly enable `QT_FEATURE_quick_shadereffect`
or `QT_FEATURE_quick_draganddrop`.
#### Fix
Three changes to `local/recipes/qt/qtdeclarative/recipe.toml`:
1. Added `cmake --install . --prefix "${COOKBOOK_STAGE}/usr"` after
the build step to properly install all generated files.
2. Added explicit `-DQT_FEATURE_quick_shadereffect=ON`,
`-DQT_FEATURE_quick_draganddrop=ON`, `-DQT_FEATURE_draganddrop=ON`,
`-DQT_FEATURE_quick_controls2=ON` to the CMake configuration.
3. Added a safety-net block that regenerates `qtquick-config.h`
with all required feature definitions if CMake produces an empty
file (defensive programming for future Qt version changes).
### Commits
| Commit | What |
|--------|------|
| `57a3ea6c9` | qtdeclarative qfeatures.h fix (cmake --install + safety net) |
| `6a7abe0b87` | qtdeclarative qtquick-config.h fix (explicit feature flags + regen safety net) |
| (pending) | Safe cleanup script + AGENTS.md + CONSOLE-TO-KDE-DESKTOP-PLAN.md v5.8 |
local/docs/CACHYOS-INTEGRATION.md
-313
View File
@@ -1,313 +0,0 @@
# CachyOS Boot Analysis: Reference for Red Bear OS Integration
**Source ISO:** `cachyos-desktop-linux-260628.iso` (CachyOS Desktop, 28 Jun 2026)
**Captured:** 29 Jun 2026, via QEMU/KVM (`qemu-system-x86_64 -m 8G -smp 4 -cpu host -enable-kvm -nographic -kernel ... -initrd ... -append "console=ttyS0,115200n8 earlyprintk=ttyS0,115200 loglevel=7"`)
**Raw log:** `local/docs/boot-logs/cachyos-kernel-boot.log` (441 lines of `dmesg`-grade output)
**Purpose:** Document reference boot sequences for CPU/x86 systems so that the Redox kernel, base, and pcid daemons can be evaluated against the proven Linux paths on the same QEMU target.
---
## 1. Executive summary
CachyOS is an Arch-derived distribution that ships a custom Zen-optimised kernel and an Arch base. Running its kernel under QEMU/KVM gives us a **reference dmesg** for the i440FX + PIIX machine type that Red Bear OS is also designed to run on. The captured log covers:
* Early firmware handover (SeaBIOS → iPXE → ISOLINUX → kernel)
* ACPI table parsing (RSDP, RSDT, FACP, DSDT, FACS, APIC, HPET, WAET)
* Memory zoning (DMA / DMA32 / Normal, GB pages, KASLR)
* PCI enumeration of all 5 QEMU PIIX devices
* Storage discovery (FDC, ATA PIIX for CD-ROM)
* Network init (e1000 82540EM)
* USB stack registration
* APIC / HPET / ACPI PCI hotplug
* Initramfs → systemd-udevd hand-off
The full log is at `local/docs/boot-logs/cachyos-kernel-boot.log`. The lines reproduced below are the
ones that map directly onto Red Bear OS subsystems.
---
## 2. Hardware inventory in QEMU i440FX (and what Red Bear OS must match)
| PCI BDF | Vendor:Device | Class | Linux driver | Red Bear OS mapping |
|---|---|---|---|---|
| 0000:00:00.0 | 8086:1237 (PIIX ISA bridge host) | 060000 bridge | host bridge | `pcid` enumerates, no driver needed |
| 0000:00:01.0 | 8086:7000 (PIIX PCI bridge) | 060100 PCI bridge | bridge | `pcid` enumerates, no driver needed |
| 0000:00:01.1 | 8086:7010 (PIIX IDE) | 010180 IDE controller | ata_piix | `ided` already in base |
| 0000:00:01.3 | 8086:7113 (PIIX ACPI/SMB) | 068000 bridge | PIIX ACPI | `hwd` (ACPI backend) consumes this |
| 0000:00:02.0 | 1234:1111 (Bochs/QEMU VGA) | 030000 VGA | bochs-drm | `vesad` already in base |
| 0000:00:03.0 | 8086:100e (e1000) | 020000 Ethernet | e1000 | `e1000d` in base |
| 0000:00:04.0 | 1af4:1001 (virtio block) | 010000 block | virtio-blk | `virtio-blkd` in base |
Every one of these has a matching userland daemon in `local/sources/base/drivers/`.
No surface-area gap found.
---
## 3. Boot phases observed
### 3.1 SeaBIOS / iPXE (firmware)
```
SeaBIOS (version Arch Linux 1.17.0-2-2)
iPXE (http://ipxe.org) 00:03.0 C900 PCI2.10 PnP PMM+BEFD3CC0+BEF33CC0 C900
Booting from DVD/CD...
ISOLINUX 6.04 6.04-pre3-3-g05ac953c* ETCD Copyright (C) 1994-2015 H. Peter Anvin
Loading /arch/boot/x86_64/vmlinuz-linux-cachyos-lts... ok
Loading /arch/boot/x86_64/initramfs-linux-cachyos-lts.img...ok
```
**Implication for Red Bear OS:** our `recipes/core/bootloader/` package produces `bootloader-live.efi` for the **UEFI** path (Q35 + OVMF), but the i440FX/ISOLINUX path is the one we tested here and the one the Red Bear `redbear-mini.iso` image uses. The kernel loads via `linux` (BIOS-style) GRUB entry, not EFI stub. That is fine for i440FX but is **not** the path OVMF uses; our UEFI image uses a different boot path that we have not exercised in this capture.
### 3.2 Early ACPI / SMP bring-up
```
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] APIC: Static calls initialized
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux 1.17.0-2-2 04/01/2014
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] tsc: Detected 3686.400 MHz processor
```
**Implication for Red Bear OS:** our kernel uses the same `apic` static calls, NX bit, and `kvm-clock` MSRs. The MP-table boot path (`found SMP MP-table at [mem 0x000f66b0-0x000f66bf]`) is the one we use for AP bring-up in `local/sources/kernel/src/acpi/madt/arch/x86.rs`. **No divergence** between the two kernels' expected ACPI surface on this machine.
### 3.3 ACPI tables
```
[ 0.020345] ACPI: Early table checksum verification disabled
[ 0.021036] ACPI: RSDP 0x00000000000F64C0 000014 (v00 BOCHS )
[ 0.021707] ACPI: RSDT 0x00000000BFFE2445 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.022684] ACPI: FACP 0x00000000BFFE22E1 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.023657] ACPI: DSDT 0x00000000BFFE0040 0022A1 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.024623] ACPI: FACS 0x00000000BFFE0000 000040
[ 0.025148] ACPI: APIC 0x00000000BFFE2355 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.026113] ACPI: HPET 0x00000000BFFE23E5 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.027073] ACPI: WAET 0x00000000BFFE241D 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)
[ 0.168090] ACPI: PM-Timer IO Port: 0x608
[ 0.168571] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.170036] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.170765] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.171523] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.172268] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.173034] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.173806] ACPI: Using ACPI (MADT) for SMP configuration information
[ 0.174537] ACPI: HPET id: 0x8086a201 base: 0xfed00000
```
**Implication for Red Bear OS:** our `local/sources/base/drivers/acpid` reads the same RSDP/RSDT/MADT/HPET/WAET table set. The table-set is identical between Linux and our microkernel — there is no missing AML method we should care about for QEMU targets. The `INT_SRC_OVR` block (IRQ overrides 0,5,9,10,11) is exactly what our `hwd` scheme expects.
### 3.4 PCI enumeration
```
[ 0.504336] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 conventional PCI endpoint
[ 0.505163] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 conventional PCI endpoint
[ 0.507163] pci 0000:00:01.1: [8086:7010] type 01 class 0x010180 conventional PCI endpoint
[ 0.512583] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 conventional PCI endpoint
[ 0.515623] pci 0000:00:02.0: [1234:1111] type 00 class 0x030000 conventional PCI endpoint
[ 0.534365] pci 0000:00:03.0: [8086:100e] type 00 class 0x020000 conventional PCI endpoint
[ 0.539589] pci 0000:00:04.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint
```
**Implication for Red Bear OS:** our `pcid` driver already enumerates these (the `redbear-mini.iso` boot log from earlier in this repo confirms `8086:7010 IDE`, `1234:1111 QEMU VGA`, `8086:100e e1000`, `1b36:000d XHCI`, `1af4:1000 virtio-net` — same machine type, same PIIX).
### 3.5 PIIX IDE quirk
```
[ 0.508508] pci 0000:00:01.1: BAR 4 [io 0xc0c0-0xc0cf]
[ 0.509664] pci 0000:00:01.1: BAR 0 [io 0x01f0-0x01f7]: legacy IDE quirk
[ 0.510462] pci 0000:00:01.1: BAR 1 [io 0x03f6]: legacy IDE quirk
[ 0.510643] pci 0000:00:01.1: BAR 2 [io 0x0170-0x0177]: legacy IDE quirk
[ 0.511623] pci 0000:00:01.1: BAR 3 [io 0x0376]: legacy IDE quirk
```
**Implication for Red Bear OS:** this is the standard QEMU PIIX "fixed BAR" emulation. Our `ided` driver must read `BAR0/1/2/3` at the legacy IDE IO ports (1F0/3F6/170/376) plus the BM-DMA port at 0xC0C0. The current `ided` driver reads `0x1F0` for the command block — that matches.
### 3.6 PIIX ACPI/SMB quirk
```
[ 0.513980] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI
[ 0.514631] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB
```
**Implication for Red Bear OS:** IO range 0x600-0x63F is the ACPI embedded controller. Our `hwd` reads from the ACPI embedded controller via the same IO range. We should NOT register a driver for PIIX4 at this address.
### 3.7 Video (VGA) enumeration
```
[ 0.532015] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 0.570799] pci 0000:00:03.0: vgaarb: setting as boot VGA device
[ 0.571424] pci 0000:00:03.0: vgaarb: bridge control possible
[ 0.571619] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
```
(Note: `00:03.0` is `8086:100e` e1000 — the vgaarb message is a quirk where e1000 claims VGA class bit 3 of its PCI config. This is harmless for our `vesad` driver which only uses `00:02.0` `1234:1111` Bochs.)
**Implication for Red Bear OS:** the `00:02.0` Bochs VGA at `fd000000` and BAR2 `febf0000` is the framebuffer MMIO target for `vesad`. Our `redbear-mini.iso` `vesad` driver reads from these BARs and reports `vesad: 1280x720 stride 1280 at 0xC0000000` (matching QEMU's `-vga std` mapping at `0xC0000000`).
### 3.8 USB / SCSI / NIC driver registration order
```
[ 0.561182] usbcore: registered new interface driver usbfs
[ 0.561376] usbcore: registered new interface driver hub
[ 0.562645] usbcore: registered new device driver usb
[ 2.542810] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 2.548031] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0c0 irq 14 lpm-pol 0
[ 2.550428] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0c8 irq 15 lpm-pol 0
[ 2.719342] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
[ 2.778176] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[ 2.802805] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 3.016039] e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56
[ 3.017095] e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection
```
**Implication for Red Bear OS:** driver registration order is `usbcore` → `ata_piix` → `cdrom` → `e1000`. Our `ided` driver (userspace PIO mode) reads the same PATA ports. Our `e1000d` driver reads `0000:00:03.0` (the Bochs e1000). The QEMU MAC `52:54:00:12:34:56` matches what the `redbear-mini.iso` `virtio-netd` configures for the user-mode network.
### 3.9 Input devices
```
[ 0.718125] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[ 0.769816] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
```
**Implication for Red Bear OS:** we have `evdevd` (input event daemon) and the kernel exposes `/scheme/input/` events. Linux names devices like `LNXPWRBN:00` (ACPI power button) and `i8042/serio0` (PS/2 keyboard). Redox's `evdevd` should handle the same ACPI power button events.
### 3.10 Initrd → systemd-udevd
```
Freeing initrd memory: 81472K
Starting systemd-udevd version 261.1-1-arch
:: running early hook [udev]
:: running hook [udev]
:: Triggering uevents...
[ 2.533807] ACPI: \_SB_.LNKC: Enabled at IRQ 10
[ 2.542810] Floppy drive(s): fd0 is 2.88M AMI BIOS
...
:: running hook [memdisk]
:: running hook [archiso]
:: running hook [archiso_loop_mnt]
:: Mounting '' to '/run/archiso/bootmnt'
ERROR: '' device did not show up after 30 seconds...
Falling back to interactive prompt
sh: can't access tty; job control turned off
[rootfs ~]#
```
**Implication for Red Bear OS:** the CachyOS `archiso` hook chain is `early-udev → udev → memdisk → archiso → archiso_loop_mnt → archiso_pxe_*`. It expects the boot medium at a discoverable loop device. Without that, initrd drops to a shell. Red Bear's `initfs` (initramfs) avoids this by hard-mounting the boot device early (via `initfs.toml`). This is a **key design difference**: CachyOS does dynamic discovery, Red Bear OS does explicit configuration. Red Bear's approach is more deterministic but requires the boot device to be known at build time.
---
## 4. Cross-cutting observations and integration gaps
### 4.1 ACPI coverage match
Red Bear OS coverage of the ACPI tables Linux reads (RSDP/RSDT/FACP/DSDT/FACS/APIC/HPET/WAET/MADT):
* `acpid` reads RSDP/RSDT via `hwd` daemon → covers table enumeration
* `hwd` opens `/scheme/acpi/tables/<signature>` and exposes them as files
* AML interpretation via `acpi` crate covers DSDT namespace walk (needed for `_SB_.LNKC`, `_INI`, etc.)
* `hwd` does NOT currently expose `\_SB_.LNKC: Enabled at IRQ 10` device linking — the IRQ 10 link callback is silently dropped. This is a **gap**.
**Action item:** extend `hwd` to interpret IRQ link descriptors in the AML namespace so the cascade `LNK A → LNK B → LNK C` propagates. Linux does this in `acpi_irq.c`.
### 4.2 PCI quirk coverage
Linux applies PIIX4 legacy IDE "fixed BAR" quirks. Our `ided` driver should be aware that BAR 0..3 are at fixed IO 1F0/3F6/170/376 even if the device's config space disagrees.
**Action item:** add a quirk handler in `ided` that pins BAR0..3 to the legacy IDE IO ports on devices matching `8086:7010/7111` (PIIX4/5 IDE).
### 4.3 VGA arbitration
Linux has `vgaarb`. We don't — but we only have one VGA on QEMU. On real multi-GPU systems we'd need this.
**Action item:** add a minimal `vgaarb` in `pcid` that tracks which device owns the VGA routing registers. Currently, on the QEMU test machine, `vesad` simply claims `00:02.0`; if a second VGA class device were present, the boot would be ambiguous.
### 4.4 CPU feature detection vs current Red Bear OS
The CachyOS kernel detects and enables:
```
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] APIC: Static calls initialized
```
Our local kernel fork does the same via `local/sources/kernel/src/arch/x86_shared/`. The boot speed (3 seconds from kernel to rootfs prompt) is mostly spent on `start_kernel()` → `arch_call_rest_init()` → `do_basic_setup()` → `driver_init()`.
### 4.5 No native KVM-clock calibration visible
`kvm-clock: Using msrs 4b564d01 and 4b564d00` is the standard. We do the same via our `kvm-clock` in `local/sources/kernel/src/clock/`. **No action needed.**
### 4.6 USB stack
The log shows `usbcore: registered new interface driver usb` at 0.56s. Our `xhcid` driver (XHCI) attaches on QEMU's `1b36:000d` USB3 controller (we see it in our own boot). The CachyOS log doesn't show a USB device enumeration because the `-device qemu-xhci` flag wasn't used in the kernel cmdline I used; for full USB testing we need `-device usb-ehci` and a USB device.
### 4.7 Framebuffer / graphics
`pci 0000:00:02.0: vgaarb: setting as boot VGA device` is the Bochs/QEMU VGA. Our `vesad` opens `fd000000` (the QEMU BDF 00:02.0 BAR0). Linux uses the same address and produces a working X11/Wayland display. Our `vesad` produces an identical `1280x720` mode and feeds it to the `vesad: 1280x720 stride 1280 at 0xC0000000` line that the `redbear-mini.iso` boot log shows.
### 4.8 Storage stack
```
ata_piix for PIIX4 (00:01.1)
cdrom/scsi for the QEMU DVD-ROM at sr0
```
Our `ided` driver handles the PIIX4 IDE at 00:01.1. The live ISO is mounted via `initfs` from `/usr/lib/boot/initfs`, bypassing the CD-ROM entirely. **No gap.**
### 4.9 Network
`e1000 0000:00:03.0 eth0: 52:54:00:12:34:56` — standard QEMU SLIRP MAC. Our `e1000d` registers the same PCI device. `virtio-netd` on QEMU `1af4:1000` also works. The user-mode SLIRP network is identical to Red Bear.
### 4.10 Init / systemd vs initfs
CachyOS uses `systemd` (PID 1). Red Bear OS uses `initfs` (a stripped-down init that mounts `/usr`, switches to initrd, and runs `/sbin/init` from the userland tree). The initfs approach is faster on QEMU but less flexible. We do not need to switch to systemd — initfs is correct for our use case.
### 4.11 Kernel init ordering difference
| Phase | Linux/CachyOS | Red Bear OS |
|---|---|---|
| Early ACPI | ACPI core reads RSDP, RSDT, FACP, DSDT, FACS, APIC, HPET, WAET | `hwd` reads same |
| SMP | APIC static calls, MP-table | MADT parse + APIC IPI sequence |
| PCI | `pci: Using configuration type 1` + `acpiphp: Slot [N] registered` | `pcid` with `acpiphp`-equivalent |
| USB | `usbcore: registered new interface driver usb` | `xhcid` on `1b36:000d` |
| ATA | `ata_piix` claims PIIX4 IDE | `ided` claims PIIX4 IDE |
| Network | `e1000` for `00:03.0` | `e1000d` for `00:03.0` |
| VGA | `pci 0000:00:02.0: vgaarb: setting as boot VGA device` | `vesad` claims `00:02.0` |
| Input | `i8042`, `AT Translated Set 2 keyboard`, `Power Button` | `evdevd` consumes `i8042` events |
| Initrd | `Freeing initrd memory: 81472K` then `systemd-udevd` | `initfs` mounts `/usr`, hands off to `init` |
The ordering is broadly equivalent. Differences:
* Red Bear uses `initfs` (custom initramfs) instead of `systemd` for PID 1.
* Red Bear's `pcid` is a single daemon vs Linux's multi-call ACPI + PCI + platform drivers.
* Red Bear's ACPI HPET detection is identical (HPET id 0x8086a201 base 0xfed00000).
### 4.12 Why CachyOS can't replace Red Bear OS analysis
CachyOS is a Linux distribution with a full software stack. Red Bear OS is a microkernel with a small userspace. The boot patterns are equivalent at the *firmware and kernel* layer but diverge sharply at *userspace*. The CachyOS log is useful as a *reference* — it tells us what a correctly working x86 kernel does on QEMU, so we can verify Red Bear OS produces an equivalent boot path.
Key equivalence points to verify in Red Bear OS:
1. ACPI tables parsed identically.
2. PCI enumeration covers the same devices.
3. USB stack registered.
4. Network initialized.
5. Framebuffer (VGA) available.
6. Input devices recognized.
Red Bear's own `redbear-mini.iso` boot log (from previous work in this repo) shows all six pass. CachyOS log confirms that what we observe is consistent with a reference Linux boot of the same QEMU machine type.
---
## 5. Files referenced
* `local/docs/boot-logs/cachyos-kernel-boot.log` — the captured log (441 lines)
* `local/docs/boot-logs/README.md` — pointer to the log and the capture command
* `local/docs/CACHYOS-INTEGRATION.md` — this file
## 6. Action items derived from this analysis
1. **`hwd` PIIX4 IRQ linking** — extend `hwd` to interpret the `\_SB_.LNKC` (and related) AML objects so IRQ routing propagates instead of dropping the callback.
2. **`ided` legacy IDE quirk** — add a config-space patcher that pins BAR0..3 to legacy IDE IO ports on `8086:7010/7111`.
3. **`pcid` vgaarb** — add minimal VGA arbitration when multiple VGA devices are present.
4. **`initfs` archiso-compat path** — optionally expose an `archiso` hook so CachyOS-style live ISOs can mount via the same `loop_mnt` pattern (low priority).
5. **MSR exposure** — `cpufreqd` and other tools want `IA32_PERF_CTL` writes. The kernel currently has no MSR scheme. If we want full cpufreqd on real hardware, we need a small userspace `msrd` daemon (or kernel MSR scheme). The QEMU MSR emulation caveat is unrelated.
These are documented as design considerations; no immediate code changes are required because Red Bear OS already boots successfully on QEMU and the integration points listed above are present in some form.
local/docs/CARGO-PATCH-PROPAGATION.md
-161
View File
@@ -1,161 +0,0 @@
# Cargo Patch Propagation — Fork Dependency Policy
**Created:** 2026-07-02
**Status:** Active policy
**Scope:** All Rust recipes that depend on forked packages
## The Rule
Every recipe that uses a forked package **MUST** depend on the local fork — never on
the upstream/crates.io version.
```
Upstream package (gitlab.redox-os.org / crates.io)
│
▼
Our local fork (local/sources/<package>/)
│ (patches applied, version bumps tracked)
▼
ALL downstream recipes depend on THIS fork
```
### Forked packages (as of 2026-07-02)
| Package | Local fork | Version | Symlink |
|---------|-----------|---------|---------|
| `redox_syscall` | `local/sources/syscall/` | 0.8.1 | `recipes/core/base/syscall` |
| `libredox` | `local/sources/libredox/` | 0.1.18 | `recipes/core/base/libredox` |
### Lifecycle: when upstream bumps version
1. **Update fork first** — `git fetch upstream && git rebase upstream/master` in `local/sources/<package>/`
2. **Rebase all patches** — ensure Red Bear patches still apply against the new upstream
3. **Rebuild prefix** — `touch <component> && make prefix` if the fork affects the cross-toolchain sysroot
4. **All dependents automatically use the updated fork** — no per-recipe changes needed
5. **Adapt downstream code** — if the upstream bump changed an API, fix every recipe that breaks (Golden Rule: Red Bear adapts to upstream, never the reverse)
## Why This Matters
### The Problem: Cargo `[patch]` doesn't propagate
Cargo's `[patch.crates-io]` only applies from the **root package** being compiled. When
a recipe depends on a base workspace member (like `daemon`) via path dependency, the base
workspace's own `[patch]` entries are **silently ignored**.
This means:
1. Recipe `redox-drm` depends on `daemon` via `path = ".../base/source/daemon"`
2. `daemon` is part of the `base` workspace, which patches `redox_syscall` and `libredox`
3. But when cargo compiles `redox-drm` as root, only `redox-drm`'s own `[patch]` applies
4. `libredox` from crates.io **vendors its own copy** of `redox_syscall` internally
5. Two different `syscall::Error` types exist at compile time → E0277 type mismatch
### The Fix: Use path dependencies, not version dependencies
Every recipe that needs `redox_syscall` or `libredox` should use the local fork via path
dependency or `[patch.crates-io]` entry. The path is always the same relative depth from
`local/recipes/<category>/<name>/source/Cargo.toml`:
```
../../../../../recipes/core/base/syscall → local fork of redox_syscall
../../../../../recipes/core/base/libredox → local fork of libredox
```
## How to Make a Recipe Depend on the Fork
### Option A: Direct path dependency (preferred for direct deps)
```toml
[dependencies]
redox_syscall = { path = "../../../../../recipes/core/base/syscall", features = ["std"] }
libredox = { path = "../../../../../recipes/core/base/libredox" }
```
### Option B: Version dep + patch redirect (when transitive deps also need redirection)
```toml
[dependencies]
redox_syscall = { version = "0.8", features = ["std"] }
libredox = "0.1"
daemon = { path = "../../../../../recipes/core/base/source/daemon" }
[patch.crates-io]
redox_syscall = { path = "../../../../../recipes/core/base/syscall" }
libredox = { path = "../../../../../recipes/core/base/libredox" }
```
Use Option B when the recipe depends on ANY base workspace member via path (`daemon`,
`scheme-utils`, etc.). The `[patch]` entries redirect ALL transitive `redox_syscall` and
`libredox` dependencies (from `redox-scheme`, etc.) to the local fork.
### Symlinks required
The paths above resolve through symlinks that must exist:
```bash
recipes/core/base/syscall → ../../../local/sources/syscall
recipes/core/base/libredox → ../../../local/sources/libredox
```
These symlinks bridge the base workspace's relative path resolution. Without them, the
base workspace's `path = "../syscall"` resolves to a non-existent directory.
## Validation
Run the validation gate to check all recipes:
```bash
./target/release/repo validate-cargo-deps
```
This scans every `Cargo.toml` in `recipes/` and `local/recipes/`, detects:
- Path dependencies on base workspace members without matching `[patch]` entries
- Version dependencies on forked packages that should use the local fork
- Missing or broken symlinks
Exit code 0 = all OK, exit code 1 = warnings found.
## Recipes Currently Violating This Policy
As of 2026-07-02, the following recipes pull `redox_syscall` or `libredox` from crates.io
instead of the local fork. These are policy violations that should be fixed incrementally:
### `redox_syscall` from crates.io (25 recipes)
Recipes using version 0.8 (straightforward conversion to fork):
- `drivers/ehcid`, `drivers/linux-kpi`, `drivers/redbear-btusb`
- `drivers/redox-driver-sys`
- `system/driver-manager`, `system/evdevd`, `system/firmware-loader`
- `system/hwrngd`, `system/iommu`
- `system/redbear-btctl`, `system/redbear-hwutils`
- `system/redbear-sessiond`, `system/redbear-traceroute`
- `system/redbear-wifictl`, `system/thermald`, `system/udev-shim`
- `tui/tlc`
Recipes using version 0.7 (need API adaptation to 0.8):
- `drivers/virtio-inputd`, `system/devfsd`, `system/diskd`
Recipes using version 0.4 (need significant API adaptation to 0.8):
- `system/redbear-accessibility`, `system/redbear-ime`, `system/redbear-keymapd`
- `gpu/redox-drm` (uses `syscall04` for legacy PCI config — separate concern)
### `libredox` from crates.io (21 recipes)
All recipes using `libredox = "0.1"` or `libredox = "0.1.x"` from crates.io should
switch to the local fork at `local/sources/libredox/` (version 0.1.18).
## Migration Plan
1. **Phase 1 (done):** Fix `redox-drm` — the only recipe with a path dep on `daemon`
2. **Phase 2:** Convert all 0.8.x recipes to use the local fork (straightforward)
3. **Phase 3:** Adapt 0.7.x recipes to 0.8.x API and switch to fork
4. **Phase 4:** Adapt 0.4.x recipes to 0.8.x API and switch to fork
Each conversion is: change `version = "0.x"` to `path = "..."`, cook, fix any API breaks.
## Related
- `local/AGENTS.md` § "GOLDEN RULE — Red Bear adapts to upstream, never the reverse"
- `local/AGENTS.md` § "LOCAL FORK MODEL (CORE COMPONENTS)"
- `local/sources/base/Cargo.toml` lines 133-157 — the base workspace `[patch]` section
that ONLY applies within the base workspace
local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md
+1 -1
View File
@@ -2049,7 +2049,7 @@ This document is the authority. Subsystem plans remain for deep-dive detail:
| `KERNEL-IPC-CREDENTIAL-PLAN.md` | Kernel credential syscalls, IPC, RLIMIT — Phases K1-K2,K4 complete |
| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | PCI/IRQ/MSI-X/IOMMU quality |
| `ACPI-IMPROVEMENT-PLAN.md` | ACPI shutdown, power, sleep states |
| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface |
| `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface |
| `DRM-MODERNIZATION-EXECUTION-PLAN.md` | DRM/KMS modernization |
| `WAYLAND-IMPLEMENTATION-PLAN.md` | Wayland compositor stability |
| `DBUS-INTEGRATION-PLAN.md` | D-Bus architecture |
local/docs/CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md
-158
View File
@@ -1,158 +0,0 @@
# Red Bear OS — CPU/DMA/IRQ/MSI/Scheduler Fix Plan
**Date**: 2026-05-04
**Updated**: 2026-05-04 (MSI T1.1–T2.2 implemented, committed, pushed)
**Status**: Active — MSI Phase 1 complete, DMA/Scheduler pending
**Source of truth**: Linux kernel 7.1 (local/reference/linux-7.1/)
## 1. Problem Statement
Five critical integration gaps in the microkernel architecture:
| Gap | Severity | Impact | Status |
|-----|----------|--------|--------|
| MSI absent from kernel | CRITICAL | All NVMe/GPU/NIC on legacy INTx | ✅ RESOLVED (P8-msi.patch) |
| DMA/IOMMU not integrated | CRITICAL | DMA buffers unprotected | ⏳ Pending |
| PIT tick (148Hz) vs LAPIC (1000Hz) | HIGH | Scheduler 6x slower than Linux | ✅ RESOLVED (P7-scheduler patch) |
| Global scheduler lock | HIGH | Serializes all context switches | ✅ RESOLVED (work-stealing) |
| Thread creation (3 IPC hops) | HIGH | 3x slower than Linux clone() | ⏳ Pending |
## 2. Phase 1: MSI/MSI-X in Kernel (Week 1-3) ✅ COMPLETE
### T1.1: MSI Capability Parsing ✅ DONE
- File: `kernel/src/arch/x86_shared/device/msi.rs` (61 lines)
- Commit: `678980521` in `P8-msi.patch`
- Linux ref: `arch/x86/kernel/apic/msi.c` (391 lines)
- Implements: `MsiMessage` (compose/validate), `MsiCapability` (parse 32/64-bit), `MsixCapability` (parse table/PBA), `is_valid_msi_address`, `is_valid_msi_vector`
- Bounds-safe: all `parse()` methods return `Option<Self>`, using `.get()` instead of raw indexing
### T1.2: Vector Allocation Matrix ✅ DONE
- File: `kernel/src/arch/x86_shared/device/vector.rs` (53 lines)
- Commit: `678980521` in `P8-msi.patch`
- Linux ref: `arch/x86/kernel/apic/vector.c` (1387 lines)
- Implements: per-CPU bitmatrix (7×32-bit banks = 224 vectors 32-255), `allocate_vector`, `free_vector`
- Lock-free CAS-based allocation with `trailing_ones()` find-first-zero
- NOTE: VECTORS table is global (not yet per-CPU sharded) — sufficient for 224 vectors
### T1.3: MSI IRQ Domain (Scheme Integration) ✅ DONE
- File: `kernel/src/scheme/irq.rs`
- Commit: `678980521` in `P8-msi.patch`
- Implements: `msi_vector_is_valid()` (32-0xEF range check), `iommu_validate_msi_irq()` hook (stub: always true), IOMMU gate at `irq_trigger()` for vectors ≥16
### T1.4: Userspace MSI Consumer (driver-sys) ✅ DONE
- File: `local/recipes/drivers/redox-driver-sys/source/src/irq.rs`
- Commit: `678980521`
- Implements: `MsiAllocation` with round-robin CPU allocation, `irq_set_affinity` (scheme write), `program_x86_message` with kernel-mediated address/vector validation (mask `0xFFF0_0000`)
- Quirk-aware fallback retained: FORCE_LEGACY, NO_MSI, NO_MSIX
### T1.5: Kernel-side MSI Affinity Handler ✅ DONE
- File: `kernel/src/scheme/irq.rs`
- Commit: `678980521` in `P8-msi.patch`
- Implements: `Handle::IrqAffinity { irq, mask }` variant, path routing for `<irq>/affinity` and `cpu-XX/<irq>/affinity`, kwrite validates CPU id and stores mask atomically, kfstat/kfpath/kreadoff/close all handle new variant
## 3. Phase 2: DMA/IOMMU Integration (Week 3-5) — AUDITED 2026-05-04
**Status**: IOMMU daemon (1003 lines) and DmaBuffer (261 lines) already exist and are solid. Tasks re-scoped from "create" to "wire."
### T2.1: IommuDmaAllocator (driver-sys) ⏳ P0
- File: `local/recipes/drivers/redox-driver-sys/source/src/dma.rs`
- Add `IommuDmaAllocator` struct: holds IOMMU domain fd, wraps `DmaBuffer::allocate()` with IOMMU MAP opcode
- Uses `scheme:iommu/domain/N` write with MAP request → get IOVA
- Linux ref: `include/linux/dma-mapping.h` — `dma_alloc_coherent()` → `iommu_dma_alloc()`
### T2.2: GPU DMA pass-through ⏳ P0
- Wire `redox-drm` GPU drivers to open IOMMU device endpoint and use IommuDmaAllocator
- amdgpu: VRAM/GTT allocations through IOMMU domain
- Intel i915: GTT pages through IOMMU domain
- Files: `local/recipes/gpu/redox-drm/source/`, `local/recipes/gpu/amdgpu/source/`
### T2.3: Streaming DMA (linux-kpi) ⏳ P1
- `dma_map_single()`: allocate bounce buffer, copy data, map through IOMMU
- `dma_unmap_single()`: copy back, unmap, free bounce buffer
- Linux ref: `kernel/dma/mapping.c` — streaming API
- File: `local/recipes/drivers/linux-kpi/source/`
### T2.4: NVMe DMA pass-through ⏳ P1
- Wire `ahcid`/`nvmed` PRP list physical addresses through IOMMU domain
- Linux ref: `drivers/nvme/host/pci.c` — `nvme_map_data()`
### T2.5: SWIOTLB Fallback (low priority) ⏳ P2
- Linux ref: `kernel/dma/swiotlb.c`
- Bounce buffer for devices with <4GB DMA addressing
- Only needed for ancient hardware; x86_64 modern hardware doesn't need it
## 4. Phase 3: Scheduler Improvements (Week 4-6) — MOSTLY DONE
### T3.1: LAPIC Timer as Primary Tick ✅ DONE
- P7-scheduler-improvements.patch: LAPIC timer calibrated + enabled at vector 48
- TSC-deadline mode, 1000Hz tick drives DWRR scheduler directly
- PIT fallback retained
### T3.2: Per-CPU Scheduler Locks ✅ DONE
- Work-stealing load balancer in switch.rs
- Per-CPU nr_running counter
- Idle CPUs steal work via IPI
### T3.3: Load Balancing ✅ DONE
- RT scheduling class (priority 0-9, skip DWRR, immediate dispatch)
- Threshold reduced: 3→1 ticks for LAPIC-driven mode
- Geometric weights in DWRR
### T3.4: RT Scheduling Class ✅ DONE
### T3.5: NUMA-Aware Scheduling ❌
- Not implemented — low priority for desktop/non-NUMA systems
- Linux ref: kernel/sched/rt.c
- FIFO and Round-Robin classes
- Priority inheritance
- RT throttling: 95% CPU cap/sec
### T3.5: TSC-Deadline Timer
- Use IA32_TSC_DEADLINE MSR for precise tick
- True tickless operation
- TSC calibration via HPET or PIT
## 5. Phase 4: Thread Creation (Week 6-7)
### T4.1: Batched Thread Creation
- Batch new-thread requests (reduce IPC)
- Pre-allocate stack pages during fork
### T4.2: Kernel Thread Pool
- Pre-create idle kernel threads
- Reuse via object pool
### T4.3: Shared Memory IPC
- Use shm for proc scheme bulk ops
- Avoid data copy through IPC channel
## 6. Dependencies
Phase 1 (MSI): T1.1 -> T1.2 -> T1.3 -> T1.4 -> T1.5
Phase 2 (DMA): T2.1 -> T2.2 -> T2.3 -> T2.4 -> T2.5
Phase 3 (Sched): T3.1 -> T3.5 -> T3.2 -> T3.3 -> T3.4
Phase 4 (Thread): T4.1 -> T4.2 -> T4.3
Phase 1+2 independent (parallel). Phase 2.4 needs Phase 1.3.
Phase 3.1 partially done (start immediately).
## 7. Timeline
| Phase | Duration | Cumulative |
|-------|----------|------------|
| Phase 1 (MSI) | 3 weeks | Week 3 |
| Phase 2 (DMA/IOMMU) | 3 weeks | Week 5 |
| Phase 3 (Scheduler) | 3 weeks | Week 7 |
| Phase 4 (Threads) | 2 weeks | Week 7 |
Total: 7 weeks (2 devs parallel Phase 1+2)
## 8. Success Metrics
| Metric | Before | After |
|--------|--------|-------|
| Scheduler tick | 148Hz (PIT) | 1000Hz (LAPIC) |
| NVMe throughput | INTx shared | MSI-X 4+ queues |
| Context switch | ~6.75ms | ~1ms |
| Thread create | 3 IPC hops | 2 IPC hops |
| DMA safety | Unprotected | IOMMU-mapped |
local/docs/CUB-WORKFLOW-ASSESSMENT.md
-144
View File
@@ -1,144 +0,0 @@
# Cub Workflow Integration Assessment
**Status:** Assessment + Implementation complete (2026-05-07)
**Scope:** AUR search → PKGBUILD parse → recipe.toml generation → cook with build tools
## End-to-End Flow Assessment
```
User: "cub -S ripgrep-all"
│
├─ 1. AUR Search ✅ Works. AurClient::search() via AUR RPC v5.
│
├─ 2. Fetch PKGBUILD ✅ Works. Git clone from aur.archlinux.org.
│
├─ 3. Parse PKGBUILD ⚠️ Partial. See Gap #1-3 below.
│
├─ 4. Convert to recipe.toml ⚠️ Partial. See Compatibility Gaps below.
│
└─ 5. Cook recipe ⚠️ Partial. Depends on build tool availability.
```
## Critical Gaps: PKGBUILD → Recipe Conversion
### Gap 1: Install Function Silently Lost (CRITICAL)
PKGBUILD `package()` functions with `install -Dm755` commands are not converted.
The generated recipe.toml has no install instructions. Files are never staged.
**Impact**: Any AUR package using `package()` produces a broken recipe that
builds but installs nothing.
### Gap 2: Multiple Source Entries → Hard Error (CRITICAL)
`cookbook.rs` line 63-67: if a PKGBUILD has >1 source, `generate_recipe()`
returns a hard error. Many AUR packages use multiple source tarballs.
**Impact**: `cub build` fails immediately with "Cookbook recipe generation
currently supports a single primary source."
### Gap 3: SHA-256 Passed as BLAKE3 (HIGH)
PKGBUILD uses SHA-256 checksums. Cookbook expects BLAKE3. The SHA-256 hex
string is copied verbatim into the `blake3` field.
**Impact**: Cookbook hash verification will fail on packages with checksums.
### Gap 4: Dependency Coverage ~15-20% (MEDIUM)
`deps.rs` maps 44 Arch→Redox dependencies. The AUR ecosystem has thousands.
Unmapped deps pass through unchanged (`libxml2` → `libxml2`), which may or
may not resolve at cook time.
### Gap 5: Split Packages Not Generated (HIGH)
AUR packages with `pkgname=('foo' 'foo-docs' 'foo-libs')` and multiple
`package_*()` functions are detected but only the primary package is converted.
`[[optional-packages]]` is never generated.
### Gap 6: Linuxism Detection Incomplete (MEDIUM)
Only `systemctl`, `/usr/lib/systemd`, `systemd`, `/proc` are detected.
Missing: `dbus-daemon`, `udev`, `/sys/`, Python `systemd` imports.
## Recipe.toml Compatibility Gaps
| # | Gap | Severity | Impact |
|---|---|---|---|
| C1 | `dev-dependencies` missing `host:` prefix | CRITICAL | Cross-compilation filtering broken |
| C2 | `[[optional-packages]]` not generated | HIGH | Split packages impossible |
| C3 | `shallow_clone` field missing | MEDIUM | Large git repos clone slowly |
| C4 | `upstream` field missing | LOW | Fork tracking lost |
| C5 | `installs` field not populated | MEDIUM | Install manifest empty |
| C6 | `cargopath`/`cargopackages`/`cargoexamples` missing | MEDIUM | Cargo workspace builds broken |
| C7 | `script` field missing from `[source]` | LOW | Source prep scripts lost |
| C8 | `SameAs`/`Path` source variants not supported | LOW | Recipe reuse impossible |
## Build Tool Availability for Cooking
### ✅ Available (all templates covered)
| Template | Tools Needed | Status |
|----------|-------------|--------|
| `cargo` | rustc + cargo | ✅ rust-native |
| `cmake` | cmake + ninja + gcc | ✅ all present |
| `meson` | meson + ninja + gcc | ✅ all present |
| `configure` | autoconf, automake, libtool, m4, gcc, make | ✅ all present |
| `custom` | whatever script declares | ✅ depends on recipe |
### ❌ Missing / Broken
| Tool | Status | Blocks |
|------|--------|--------|
| **texinfo** | BROKEN (compilation error) | Autotools packages with `makeinfo` |
| **intltool** | WIP (compiled, not tested) | GNOME i18n packages |
| **gobject-introspection** | WIP (not tested) | GTK/GNOME introspection packages |
| **gtk-doc** | WIP (not tested) | Development docs only (low priority) |
### Dependency Mapping Coverage
| Category | Count | Examples |
|----------|-------|----------|
| Explicitly mapped | 44 | glibc→relibc, openssl→openssl3, etc. |
| Dropped (unavailable) | 5 | systemd, xorg-server, linux-api-headers |
| Pass-through (unknown) | Thousands | libxml2, libpcre, etc. |
## Build Flow Integration
### What Works End-to-End
1. Simple Rust packages (cargo template, single source)
2. Simple C packages (configure template, single source)
3. CMake packages (single source)
4. Meson packages (single source)
### What Breaks
1. **Any package with install function** → recipe missing install logic
2. **Multi-source packages** → hard error at generation
3. **Split packages** → only primary package built
4. **Packages with checksums** → BLAKE3 verification mismatch
5. **Packages needing texinfo** → build tool unavailable
6. **Cross-compilation deps** → host: prefix not added
## Recommendations
### Immediate (unblock basic AUR packages)
1. Fix install function conversion — route `package()` content to `BuildKind::Custom.script`
2. Remove multi-source hard error — support multiple source entries or warn gracefully
3. Add `host:` prefix to dev-dependencies when building for target
### Short-term (unblock common packages)
4. Fix texinfo compilation error — unblocks many autotools packages
5. Implement `[[optional-packages]]` generation for split packages
6. Fix SHA-256 → BLAKE3 mapping — use correct hash or document the gap
7. Add `cargopath`/`cargopackages` fields to cargo template generation
### Medium-term (broader coverage)
8. Expand dependency mapping table to cover common AUR libraries
9. Improve linuxism detection (D-Bus, udev, sysfs patterns)
10. Add `shallow_clone`, `upstream`, `installs` fields
11. Validate intltool and gobject-introspection recipes
local/docs/ECOSYSTEM-ADAPTION-POLICY.md
-144
View File
@@ -1,144 +0,0 @@
# Red Bear OS: Ecosystem Adaption Policy
**Rule (per user directive):** Red Bear OS internal projects MUST always update,
adapt, and go inline with Redox ecosystem changes. Red Bear adapts to Redox,
not the other way around.
## How Ecosystem Pinning Works
The Redox crate ecosystem has version coupling: `libredox`, `redox-scheme`,
`redox_syscall` MUST be on a consistent version set across all crates in a
build. Mixing old and new versions causes `call_ro`/`call_wo` and
`syscall::Error` vs `libredox::error::Error` type errors.
### Canonical Versions (as of build session 2026-06-26)
| Crate | Version | Why this version |
|-------|---------|------------------|
| `libredox` | `=0.1.18` | Current upstream version in fork-upstream-map.toml |
| `redox-scheme` | `=0.11.2` | Current upstream version in fork-upstream-map.toml |
| `redox_syscall` | `=0.9.0` | Current upstream version in fork-upstream-map.toml |
**Note on version selection rationale (per "latest versions only" rule):**
Redox ecosystem is mid-migration (2026-05-27 → 2026-06-01 transition). The
transition broke compatibility for upstream consumers. Per research:
- `redox-scheme 0.11.1` (2026-05-27) bumped `redox_syscall ^0.7.3 → ^0.8.0`
AND `libredox ^0.1.12 → ^0.1.17`
- `redoxfs master` (2026-06-24) still uses `redox-scheme 0.11.0` (NOT 0.11.1)
with `redox_syscall 0.7.5` and `libredox 0.1.13`
- `redox_installer master` (2026-05-27) still uses `redox_syscall 0.7`
and `redoxfs 0.9`
- Red Bear syscall fork is currently aligned with upstream `0.9.0` (the first jump past `0.7.x`)
So the current **mutually-aligned** ecosystem is `redox-scheme 0.11.2` on `redox_syscall 0.9.0`. When upstream `redoxfs` and the rest of the fork map move together, keep this table aligned to the values in `local/fork-upstream-map.toml`:
| Crate | New Version |
|-------|-------------|
| `libredox` | `=0.1.18` |
| `redox-scheme` | `=0.11.2` |
| `redox_syscall` | `=0.9.0` |
Use the current map values to keep the ecosystem aligned.
## How to Pin Ecosystem Versions in a Red Bear Recipe
### For Red Bear local recipes (in `local/recipes/`):
**Single-package files** (no `[workspace]`):
```toml
[dependencies]
# These are pinned to latest ecosystem versions.
libredox = "=0.1.18"
redox-scheme = "=0.11.2"
redox_syscall = "=0.9.0"
```
**Workspace roots** (`[workspace]` section):
```toml
[workspace.dependencies]
# Red Bear OS must adapt to upstream Redox ecosystem changes.
# These pins force unified versions across the workspace.
libredox = "=0.1.18"
redox-scheme = "=0.11.2"
redox_syscall = "=0.9.0"
```
Then in member crates:
```toml
[dependencies]
libredox = { workspace = true }
redox-scheme = { workspace = true }
redox_syscall = { workspace = true }
```
### For upstream Redox recipes (kernel, base, relibc, etc.):
Upstream recipes cannot be modified directly. Use the patch system in
`local/patches/<component>/` to bump the ecosystem pins in their
`Cargo.lock` (and Cargo.toml where needed).
Reference patch: `local/patches/base/P9-redox-scheme-latest-deps.patch`.
## Anti-Patterns (Do Not Do)
### ❌ `[patch.crates-io]` with `version = "..."` syntax
```toml
[patch.crates-io]
libredox = { version = "0.1.17" } # WRONG!
```
This is INVALID. Cargo treats `version = "X.Y.Z"` as a patch pointing to the
same source (crates.io), which conflicts with itself. You get the error:
```
error: patch for `libredox` points to the same source, but patches must
point to different sources
```
Use `[workspace.dependencies]` with `=X.Y.Z` for version-only enforcement
instead.
### ❌ `[patch.crates-io]` with `git = ...` for ecosystem crates
```toml
[patch.crates-io]
redox-scheme = { git = "https://..." } # WRONG!
```
Use crates.io directly. Only `ring` (a Redox fork that doesn't exist on
crates.io) should use `git = ...`.
### ❌ Loose version specifiers
```toml
[dependencies]
libredox = "0.1" # WRONG! Allows any 0.1.x
redox-scheme = "^0.11" # WRONG! Allows any 0.11.x
```
Use `=X.Y.Z` to force EXACT versions. This is critical for ecosystem consistency.
## Maintenance Scripts
Three helper scripts in `/tmp/` (during build session, can be moved to
`local/scripts/`):
- `apply_ecosystem_pins.py` — Adds `[patch.crates-io]` (initial step, now superseded)
- `fix_workspace_pins.py` — Detects and fixes `[patch.crates-io]` version-only bugs
- `fix_all_pins_v2.py` — Comprehensive fix: moves version-only patches to
`[workspace.dependencies]`, updates existing entries to `=X.Y.Z`, removes
empty `[patch.crates-io]` blocks
- `dedupe_deps.py` — Removes duplicate ecosystem crate entries in
`[dependencies]`/`[workspace.dependencies]`
## When to Re-pin
Re-check the canonical versions whenever:
- Redox releases a new `libredox`/`redox-scheme`/`redox_syscall` on crates.io
- A Redox upstream recipe bumps its ecosystem pins
- A new Red Bear local recipe is added that needs ecosystem crates
- Build fails with `call_ro`/`call_wo` or `syscall::Error` mismatch errors
local/docs/HOOKS.md
+91 -8
View File
@@ -5,26 +5,85 @@ Red Bear OS. Per AGENTS.md "absolutely NEVER DELETE, NEVER IGNORE"
rule, hooks are always **opt-in** — operators explicitly choose to
install them. No hook is auto-installed.
## Available hooks (2026-07-12, Phase 4.5)
## Available hooks (2026-07-18, Phase 4.5 + release-bump pipeline)
### post-checkout-version-sync.sh (release-bump automation, opt-in)
**File:** `local/scripts/post-checkout-version-sync.sh`
**Purpose:** When an operator switches to a release branch whose name is an
anchored semver (e.g. `0.3.1` → `0.3.2`), automatically synchronise every
Cat 1 (in-house) and Cat 2 (upstream fork) crate version label to match the
new branch. This is the **label-only** half of a release bump — it rewrites
the `version = "..."` fields via `sync-versions.sh --no-regen` and does
nothing else. Source upgrades and lockfile regen remain explicit operator
steps (see `local/docs/RELEASE-BUMP-WORKFLOW.md`).
**Exact guards** (any fail → silent exit 0, no mutations):
1. `$3 == 1` — branch checkouts only (file checkouts are skipped).
2. New branch name matches `^[0-9]+\.[0-9]+\.[0-9]+$` — silently skips
`master`, `submodule/*`, `recovered/*`, and detached HEAD.
3. No in-progress rebase / cherry-pick / merge (`.git/rebase-merge`,
`.git/rebase-apply`, `.git/CHERRY_PICK_HEAD`, `.git/MERGE_HEAD` all
absent).
4. `REDBEAR_NO_AUTO_SYNC` env var unset.
5. Working tree clean (`git status --porcelain` empty) — else warns to
stderr and skips.
**What this hook NEVER does:**
- No network access (no `git fetch`, no upstream lookups).
- No git mutations (no commit, no branch, no push, no stash).
- No lockfile regen (never calls `sync-versions.sh --regen`).
- No source upgrades (never rebases forks or touches `local/sources/*/`).
- Does not run on non-semver branches.
- Does not run on a dirty working tree.
- Never blocks a checkout — always exits 0, even on internal failure.
**Action when all guards pass:** reads the repo-root `Cargo.toml`
`[package] version`. If it differs from the branch version, runs
`sync-versions.sh --no-regen` (labels only) and prints the follow-up hint
pointing the operator at the explicit source-upgrade and lockfile-regen
commands. If already synced, exits silently.
**Install (operator opt-in — NEVER auto-installed):**
```bash
# Via the installer (preferred):
./local/scripts/install-git-hooks.sh # post-checkout only
./local/scripts/install-git-hooks.sh --all # + pre-push + commit-msg
# Or manually:
cp local/scripts/post-checkout-version-sync.sh .git/hooks/post-checkout
chmod +x .git/hooks/post-checkout
```
**Bypass for a single checkout:**
```bash
REDBEAR_NO_AUTO_SYNC=1 git checkout 0.3.2
```
### pre-push-checks.sh (recommended for serious operators)
**File:** `local/scripts/pre-push-checks.sh`
**Purpose:** Run 4 critical pre-flight checks before every `git push`.
Closes the "silent drift" gap identified in AGENTS.md "Daily-upstream-
safe workflow".
**Purpose:** Run 7 pre-flight checks (4 core + 3 audit checks) before every
`git push`. Closes the "silent drift" gap identified in AGENTS.md
"Daily-upstream-safe workflow".
**Checks:**
1. `sync-versions.sh --check` — Cat 0 + Cat 1 + Cat 2 versions
2. `verify-fork-versions.sh` — Cat 2 fork supremacy + content check
3. `verify-patch-content.py` — no orphan patches in `local/patches/`
4. `verify-collision-detection.py` — no config-vs-package conflicts
4. `verify-patch-content.py --report action` — orphan-patch operator decisions
5. `verify-patch-content.py --selftest` — orphan-patch detector self-test
6. `verify-collision-detection.py` — no config-vs-package conflicts
7. `verify-collision-detection.py --selftest` — collision detector self-test
**Install:**
```bash
cp local/scripts/pre-push-checks.sh .git/hooks/pre-push
chmod +x .git/hooks/pre-push
# Or via the installer:
./local/scripts/install-git-hooks.sh --all
```
**Bypass:**
@@ -73,16 +132,35 @@ in `local/scripts/` so operators can opt in by copying it to
## Audit
This directory was created in Phase 4.5 (Round 5). The current
hook inventory is:
This directory was created in Phase 4.5 (Round 5) and extended in the
release-bump pipeline (2026-07-18). The current hook inventory is:
| Hook | File | Function |
|------|------|----------|
| post-checkout | `local/scripts/post-checkout-version-sync.sh` | Auto-sync Cat 1+2 version labels on semver branch switch (label-only, no regen) |
| pre-push | `local/scripts/pre-push-checks.sh` | Runs 7 pre-flight checks before push |
| commit-msg | `local/scripts/commit-msg` | Auto-prepends `phase X.Y:` to commit messages |
| (server-side) | (gitea admin only) | Pre-receive — defense in depth; not yet implemented |
## Full Tool Inventory (Round 10)
## Installer (`install-git-hooks.sh`)
All three hooks are opt-in and can be installed individually with `cp`
(see each section above) or in one shot via the installer:
```bash
./local/scripts/install-git-hooks.sh # post-checkout only (default)
./local/scripts/install-git-hooks.sh --all # post-checkout + pre-push + commit-msg
./local/scripts/install-git-hooks.sh --uninstall # remove default set (restores .bak if present)
./local/scripts/install-git-hooks.sh --uninstall --all
```
The installer is idempotent: if a hook is already up to date it reports so
and does nothing; if an existing hook differs it backs the old one up to
`<name>.bak` (rotating older backups to `<name>.bak.1`, `.2`, …) before
overwriting. `--uninstall` reverses an install and restores the most recent
backup if one exists.
## Full Tool Inventory (Round 14)
The build system has grown organically across 9+ rounds. Here is
the complete tool inventory for reference.
@@ -99,6 +177,11 @@ the complete tool inventory for reference.
| `unblock-base-push.sh` | shell | Base fork deadlock resolver | `path-a`, `path-b`, `path-c` for 3 operator paths |
| `commit-msg` | shell | Auto-phase-prefix hook | opt-in; install to `.git/hooks/commit-msg` |
| `build-preflight.sh` | shell | Build-time validation | Called by `build-redbear.sh` |
| `bump-release.sh` | shell | Canonical release-bump orchestrator | `--with-sources`, `--with-external`, `--check`, `--dry-run` |
| `check-external-versions.sh` | shell↦py | External desktop-stack version reporter | `--strict`, `--no-fetch`, `--json` |
| `bump-graphics-recipes.sh` | shell↦py | Map-driven external source bumper | `--dry-run`, `--no-fetch` |
| `post-checkout-version-sync.sh` | shell | Auto label-sync on semver branch switch | opt-in hook; bypass via `REDBEAR_NO_AUTO_SYNC=1` |
| `install-git-hooks.sh` | shell | Idempotent opt-in hook installer | `--all`, `--uninstall` |
All scripts are under `local/scripts/` and are designed to be
operator-friendly. Hooks are strictly opt-in per AGENTS.md
local/docs/KERNEL-IPC-CREDENTIAL-PLAN.md
+4 -4
View File
@@ -17,7 +17,7 @@ kernel guidance in other docs.
|----------|-------------|
| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Parent: CONSOLE-TO-KDE v4.0 (Kernel & Core Infrastructure) |
| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | Sibling: IRQ/PCI/MSI-X — not duplicated here |
| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Companion: relibc IPC surface — this plan covers kernel side |
| `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Companion: relibc IPC surface — this plan covers kernel side |
| `ACPI-IMPROVEMENT-PLAN.md` | Sibling: ACPI power/shutdown — relevant for §4 (shutdown robustness) |
| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Consumer: desktop stack depends on kernel work here |
@@ -546,7 +546,7 @@ Current state: all three are recipe-applied patches. Goal: upstream into relibc
- Monitor upstream relibc for equivalent implementations
- When upstream absorbs: shrink/drop Red Bear patch chain
- When upstream does NOT absorb after 3+ months: promote to durable Red Bear-maintained
- See `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` Phase I5
- See `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` Phase I5
### Phase K4: Resource Limits and Process Management (Week 4-6)
@@ -662,7 +662,7 @@ Phase K4 (limits/ptrace) ──────────────────
|------|---------------|--------|
| IRQ / MSI-X / IOMMU | `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | Waves 1-6 complete, hardware validation open |
| ACPI shutdown / power | `ACPI-IMPROVEMENT-PLAN.md` | Waves 1-2 complete, sleep states deferred |
| relibc IPC surface | `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Phases I1-I5, message queues deferred |
| relibc IPC surface | `archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | Phases I1-I5, message queues deferred |
| D-Bus / sessiond | `DBUS-INTEGRATION-PLAN.md` | Phase 1 complete, Phase 2 in progress |
| Greeter / login | `GREETER-LOGIN-IMPLEMENTATION-PLAN.md` | Active, bounded proof passing |
| Desktop path | `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Phase 1-5 model, KWin building |
@@ -738,7 +738,7 @@ local/scripts/test-credential-syscalls-guest.sh # In-guest checker
- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — Canonical comprehensive plan
- `docs/01-REDOX-ARCHITECTURE.md` — Architecture reference
- `local/docs/IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` — IRQ/PCI plan (sibling)
- `local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — IPC surface plan (companion)
- `local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` — IPC surface plan (companion)
- `local/docs/ACPI-IMPROVEMENT-PLAN.md` — ACPI/shutdown plan (sibling)
- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — Desktop path plan (consumer)
- `recipes/core/kernel/source/src/syscall/mod.rs` — Syscall dispatch (primary implementation target)
local/docs/LOCAL-FORK-SUPREMACY-POLICY.md
+1 -1
View File
@@ -238,7 +238,7 @@ For relibc/redox-rt specifically:
## Related Documents
- `local/AGENTS.md` — full Red Bear agent guidance (branches, submodules, durability)
- `local/docs/SOURCE-ARCHIVAL-POLICY.md` — how sources are frozen and archived
- `local/docs/archived/archived/SOURCE-ARCHIVAL-POLICY.md` — how sources are frozen and archived
- `local/docs/PATCH-GOVERNANCE.md` (referenced in `local/AGENTS.md`) — how patches
are rebased and applied
- `README.md` (project root) — high-level description including the fork model
local/docs/MC-EDIT-PARITY-ASSESSMENT.md
-488
View File
@@ -1,488 +0,0 @@
# tlcedit vs mcedit — Comprehensive Parity Assessment
**Created:** 2026-06-20
**Source:** MC default keymap (`misc/mc.default.keymap` `[editor]` section) vs tlcedit `handlers.rs`
**Goal:** Every keybinding in mcedit must match in tlcedit and provide the same action
---
## Executive Summary
**Overall parity: ~93%** (up from ~45% pre-P1, ~75% after P1+P2, ~85% after P1+P2+P3, ~90% after P3.5, after P1+P2+P3+P3.5+P4a+P4b implementation, 2026-06-20)
| Category | Count | Severity | Status |
|----------|-------|----------|--------|
| **Key conflicts** (same key, different action) | 0 | ~~🔴 CRITICAL~~ | ✅ All fixed (P1) |
| **Wrong keys** (action exists, different key) | 0 | ~~🟡 HIGH~~ | ✅ All fixed (P1) |
| **Missing actions** (no key, no implementation) | ~4 | 🟡 MEDIUM | P1/P2/P3/P4 reduced from 22 |
| **Correct matches** | ~70 | ✅ | Up from ~28 |
---
## CRITICAL: Key Conflicts
These bindings use the SAME key in both editors but perform DIFFERENT actions.
These MUST be fixed first — they cause user confusion and data loss.
### C1: Ctrl-S — Save vs SyntaxOnOff
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-S` | Toggle syntax highlighting on/off |
| tlcedit | `Ctrl-S` | **Save file** (also F2) |
**Problem:** A user pressing Ctrl-S expecting syntax toggle will unexpectedly save.
**Fix:** tlcedit should use Ctrl-S for SyntaxOnOff. Save is already on F2 (correct).
**Impact:** Remove Ctrl-S as a save shortcut in the editor dispatcher.
### C2: Ctrl-Y — DeleteLine vs Redo
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-Y` | Delete current line |
| tlcedit | `Ctrl-Y` | **Redo** |
**Problem:** Redo on Ctrl-Y is a Windows/VSCode convention, not MC.
**Fix:** Move redo to `Alt-R` (MC's key). Implement DeleteLine on Ctrl-Y.
**Impact:** Add `delete_line()` to buffer; rewire handlers.
### C3: Ctrl-P — SpellCheck vs Macro Playback
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-P` | Spell check current word |
| tlcedit | `Ctrl-P` | **Play macro** |
**Problem:** Macro playback on Ctrl-P conflicts with MC spell check.
**Fix:** MC doesn't have a dedicated macro playback key — macros in MC use
`Ctrl-R` to start/stop recording and then numeric keys (Ctrl-0..9) to replay.
Move macro playback to a different binding.
**Impact:** Change macro system.
### C4: Ctrl-L — Refresh vs Relative Line Numbers
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-L` | Refresh screen (redraw) |
| tlcedit | `Ctrl-L` | **Toggle relative line numbers** |
**Problem:** Relative line numbers is a TLC addition not in MC.
**Fix:** Move relative line toggle to `Alt-N` (MC's ShowNumbers key).
Ctrl-L should trigger a screen refresh.
**Impact:** Minor — rewire two bindings.
### C5: Ctrl-Z — Undo vs WordLeft Alias
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-Z` | Move cursor one word left (alias for Ctrl-Left) |
| tlcedit | `Ctrl-Z` | **Undo** |
**Problem:** Undo on Ctrl-Z is standard everywhere EXCEPT MC.
**Decision needed:** Do we match MC exactly, or keep the universal Ctrl-Z=Undo?
**Recommendation:** Keep Ctrl-Z as Undo (it's a universal convention that even MC
users expect in modern contexts). Document the deviation.
---
## HIGH: Wrong Keys (Action Exists, Different Key)
These actions exist in tlcedit but are triggered by a different key than MC expects.
### W1: Search — F7 vs Alt-F
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `F7` | Open search dialog |
| tlcedit | `Alt-F` | Open search prompt |
**Fix:** Add F7 as primary search key. Keep Alt-F as alias.
**Also:** `F17` for search-continue (repeat last search).
### W2: Replace — F4 vs Alt-%
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `F4` | Open replace dialog |
| tlcedit | `Alt-%` (Alt-Shift-5) | Open replace prompt |
**Problem:** F4 in tlcedit currently has no binding in the editor.
**Fix:** Map F4 to Replace. Keep Alt-% as alias.
**Also:** `F14` for replace-continue.
### W3: SaveAs — F12 vs Shift-F2
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `F12` or `Ctrl-F2` | Save As |
| tlcedit | `Shift-F2` | Save As |
**Fix:** Add F12 as primary SaveAs key. Keep Shift-F2 as alias.
### W4: Undo — Ctrl-U vs Ctrl-Z
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Ctrl-U` | Undo |
| tlcedit | `Ctrl-Z` | Undo |
**Fix:** Add Ctrl-U as MC-compatible undo key. Keep Ctrl-Z as modern alias.
### W5: Redo — Alt-R vs Ctrl-Y
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Alt-R` | Redo |
| tlcedit | `Ctrl-Y` | Redo |
**Fix:** Move redo to Alt-R (matching MC). Free up Ctrl-Y for DeleteLine.
Keep Ctrl-Y as secondary alias for transition.
### W6: Bookmark — Alt-K vs Alt-M
| Editor | Key | Action |
|--------|-----|--------|
| mcedit | `Alt-K` | Set bookmark at cursor |
| tlcedit | `Alt-M` | Set bookmark (BookmarkSet prompt) |
**Fix:** Change BookmarkSet trigger from Alt-M to Alt-K. Free up Alt-M.
**Also:** Alt-O for BookmarkFlush (clear all), Alt-I for BookmarkPrev.
---
## MEDIUM: Missing Actions (22 items)
These MC actions have no tlcedit equivalent at all.
### Navigation (4)
| MC Action | MC Key | Description | Priority |
|-----------|--------|-------------|----------|
| ScrollUp | `Ctrl-Up` | Scroll viewport up 1 line (cursor stays) | P1 |
| ScrollDown | `Ctrl-Down` | Scroll viewport down 1 line (cursor stays) | P1 |
| TopOnScreen | `Ctrl-PgUp` | Move cursor to top visible line | P2 |
| BottomOnScreen | `Ctrl-PgDn` | Move cursor to bottom visible line | P2 |
### Editing (6)
| MC Action | MC Key | Description | Priority |
|-----------|--------|-------------|----------|
| DeleteLine | `Ctrl-Y` | Delete entire current line | P1 |
| DeleteToEnd | `Ctrl-K` | Delete from cursor to end of line | P1 |
| DeleteToWordBegin | `Alt-Backspace` | Delete word before cursor | P1 |
| DeleteToWordEnd | `Alt-D` | Delete word after cursor | P1 |
| Return | `Shift-Enter` / `Ctrl-Enter` | Insert newline above current | P2 |
| InsertOverwrite | `Insert` | Toggle insert/overwrite mode | P2 |
### Selection (8)
| MC Action | MC Key | Description | Priority |
|-----------|--------|-------------|----------|
| MarkPageUp | `Shift-PgUp` | Select one page up | P1 |
| MarkPageDown | `Shift-PgDn` | Select one page down | P1 |
| MarkToWordBegin | `Ctrl-Shift-Left` | Select to word beginning | P1 |
| MarkToWordEnd | `Ctrl-Shift-Right` | Select to word end | P1 |
| MarkToFileBegin | `Ctrl-Shift-Home` | Select to start of file | P2 |
| MarkToFileEnd | `Ctrl-Shift-End` | Select to end of file | P2 |
| MarkColumn | `F13` | Toggle column selection mode | P3 |
| Store / Cut | `Ctrl-Insert` / `Shift-Delete` | MC-style clipboard ops | P1 |
### Editor Features (4)
| MC Action | MC Key | Description | Priority |
|-----------|--------|-------------|----------|
| Help | `F1` | Built-in help screen | P2 |
| Shell | `Ctrl-O` | Switch to subshell (suspend editor) | P1 |
| InsertFile | `F15` | Insert file contents at cursor | P2 |
| InsertLiteral | `Ctrl-Q` | Insert next key literally | P2 |
### File Navigation (2)
| MC Action | MC Key | Description | Priority |
|-----------|--------|-------------|----------|
| FilePrev | `Alt-Minus` | Previous file in edit history | P3 |
| FileNext | `Alt-Plus` | Next file in edit history | P3 |
---
## Viewer (mcview) Parity
### mcview Keybindings (from `[viewer]` section)
| MC Action | MC Key | tlcview Status |
|-----------|--------|----------------|
| Help | `F1` | ❌ Missing |
| WrapMode | `F2` | ✅ Has wrap toggle |
| Quit | `F3` / `F10` / `q` / `Esc` | ✅ Esc/F10 |
| HexMode | `F4` | ✅ Has hex toggle |
| Goto | `F5` | ⚠️ Has `g` key, missing F5 |
| Search | `F7` | ⚠️ Has `/`, missing F7 |
| SearchForward | `/` | ✅ |
| SearchBackward | `?` | ❌ Missing |
| SearchContinue | `F17` / `n` | ✅ Has `n` |
| SearchForwardContinue | `Ctrl-S` | ❌ Missing |
| SearchBackwardContinue | `Ctrl-R` | ❌ Missing |
| MagicMode | `F8` | ❌ Missing |
| NroffMode | `F9` | ✅ Has nroff |
| Home | `Ctrl-A` | ⚠️ Has Home key, missing Ctrl-A |
| End | `Ctrl-E` | ⚠️ Has End key, missing Ctrl-E |
| Left | `h` / `Left` | ✅ Left |
| Right | `l` / `Right` | ✅ Right |
| Up | `k` / `y` / `Insert` / `Up` / `Ctrl-P` | ⚠️ Has Up, missing vim keys |
| Down | `j` / `e` / `Delete` / `Down` / `Enter` / `Ctrl-N` | ⚠️ Has Down, missing vim keys |
| PageDown | `f` / `Space` / `PgDn` / `Ctrl-V` | ⚠️ Has PgDn, missing aliases |
| PageUp | `b` / `PgUp` / `Alt-V` / `Backspace` | ⚠️ Has PgUp, missing aliases |
| Top | `Home` / `Ctrl-Home` / `Ctrl-PgUp` / `g` | ⚠️ Has Home, missing aliases |
| Bottom | `End` / `Ctrl-End` / `Ctrl-PgDn` / `Shift-G` | ⚠️ Has End, missing aliases |
| BookmarkGoto | `m` | ❌ Missing |
| Bookmark | `r` | ❌ Missing |
| FileNext | `Ctrl-F` | ❌ Missing |
| FilePrev | `Ctrl-B` | ❌ Missing |
| Ruler | `Alt-R` | ❌ Missing |
| Shell | `Ctrl-O` | ❌ Missing |
| History | `Alt-Shift-E` | ❌ Missing |
---
## Improvement Plan
### Phase P1: Fix Critical Conflicts + High-Priority Missing ✅ COMPLETE (2026-06-20)
**Objective:** Eliminate key conflicts and add the most common missing actions.
#### Step 1: Fix Key Conflicts
1. **Ctrl-S → SyntaxOnOff** (remove Save shortcut; F2 remains)
2. **Ctrl-Y → DeleteLine** (move Redo to Alt-R)
3. **Ctrl-P → SpellCheckCurrentWord placeholder** (move macro playback)
4. **Ctrl-L → Screen Refresh** (move relative-lines to Alt-N)
5. **Ctrl-U → Undo** (add as alias for Ctrl-Z)
6. **Macro playback → Ctrl-0..9** (MC-style: Ctrl-R records, Ctrl-0 replays)
**Files to modify:**
- `src/editor/handlers.rs` — rewire all conflict bindings
- `src/editor/macro.rs` — new numeric-key macro system
#### Step 2: Add Missing High-Priority Actions
1. **DeleteLine** (`Ctrl-Y`) — `buffer.delete_line()`
2. **DeleteToEnd** (`Ctrl-K`) — delete from cursor to EOL
3. **DeleteToWordBegin** (`Alt-Backspace`) — delete word backward
4. **DeleteToWordEnd** (`Alt-D`) — delete word forward
5. **ScrollUp/ScrollDown** (`Ctrl-Up`/`Ctrl-Down`) — viewport scroll without cursor move
6. **TopOnScreen/BottomOnScreen** (`Ctrl-PgUp`/`Ctrl-PgDn`)
7. **Shell** (`Ctrl-O`) — subshell from editor (already exists in filemanager)
8. **Store/Cut** (`Ctrl-Insert`/`Shift-Delete`) — MC-style clipboard
**Files to modify:**
- `src/editor/buffer.rs` — add `delete_line()`, `delete_to_end_of_line()`
- `src/editor/cursor.rs` — add `delete_word_backward()`, `delete_word_forward()`
- `src/editor/handlers.rs` — add new key bindings
- `src/editor/view.rs` — add `scroll_up()`/`scroll_down()` (viewport-only)
#### Step 3: Fix Wrong Keys (add MC-primary aliases)
1. **F7 → Search** (keep Alt-F as alias)
2. **F4 → Replace** (keep Alt-% as alias)
3. **F12 → SaveAs** (keep Shift-F2 as alias)
4. **F17 → SearchContinue** (repeat last search)
5. **F14 → ReplaceContinue** (repeat last replace)
6. **Alt-R → Redo** (keep Ctrl-Y as secondary during transition)
7. **Alt-K → Bookmark** set (keep Alt-M during transition)
8. **Alt-O → BookmarkFlush** (clear all bookmarks)
9. **Alt-I → BookmarkPrev** (previous bookmark)
10. **Alt-N → ShowNumbers** (toggle line numbers)
**Files to modify:**
- `src/editor/handlers.rs` — add all F-key and Alt-key aliases
#### Step 4: Add Missing Selection Keys
1. **Shift-PgUp / Shift-PgDn** — select page up/down
2. **Ctrl-Shift-Left / Ctrl-Shift-Right** — select word
3. **Ctrl-Shift-Home / Ctrl-Shift-End** — select to file start/end
**Files to modify:**
- `src/editor/cursor.rs` — add `select_page_up()`, `select_page_down()`
- `src/editor/handlers.rs` — add shift+page and ctrl+shift+arrow bindings
### Phase P2: Viewer Parity ✅ COMPLETE (2026-06-20)
1. Add F5/F7 for goto/search in viewer
2. Add `?` backward search
3. Add Ctrl-S/Ctrl-R for search direction continue
4. Add vim movement keys (h/j/k/l) in viewer
5. Add Space/`f` for page down, `b`/Backspace for page up
6. Add `g`/`Shift-G` for top/bottom
7. Add Ctrl-A/Ctrl-E for Home/End
8. Add `m`/`r` for viewer bookmarks
**Files to modify:**
- `src/viewer/mod.rs` — expand key handler
- `src/viewer/search.rs` — add backward search
### Phase P3: Editor Feature Gaps ✅ COMPLETE (2026-06-20)
**Done:**
- ✅ F15 InsertFile — insert file contents at cursor
- ✅ Insert/overwrite toggle — Insert key toggles mode, [OVR] shown in status bar
- ✅ Ctrl-Q InsertLiteral — insert next key literally
- ✅ Shift-Enter — insert newline above current line
- ✅ F1 Help — built-in keybinding reference dialog (6 categories, 30+ bindings)
- ✅ F8 MagicMode in viewer — auto-detect binary vs text via magic::detect_mode
**Deferred P3 items (require significant infrastructure):**
1. **F9 Menu** — editor command menu (needs standalone menu system, separate from filemanager)
2. **F11 UserMenu** — user-defined command menu (needs INI parser integration in editor context)
3. **Alt-Enter Find** — quick file find (needs file manager integration)
4. **Alt-Minus/Alt-Plus** — file history navigation (needs multi-file tab support)
### Phase P4: Advanced Features (future)
1. **Column selection mode** (F13) — block/column selection
2. **Spell check** (Ctrl-P) — integrate spell checker
3. **Block shift** — indent/outdent selected block
4. **Paragraph up/down** — move by paragraph
5. **Window management** — multi-file tabs, split, fullscreen
---
## Summary Table: Complete Keybinding Comparison
### Editor Bindings (sorted by MC key)
| MC Key | MC Action | tlcedit Key | tlcedit Action | Status |
|--------|-----------|-------------|----------------|--------|
| `F1` | Help | `F1` | Keybinding reference | ✅ P3 |
| `F2` | Save | `F2` | Save | ✅ |
| `F3` | Mark (toggle selection) | `F3` | Toggle selection | ✅ |
| `F4` | Replace | `F4` | Replace prompt | ✅ P1 |
| `F5` | Copy block | `F5` | Copy block | ✅ |
| `F6` | Move block | `F6` | Move (cut) block | ✅ |
| `F7` | Search | `F7` | Search prompt | ✅ P1 |
| `F8` | Delete block | `F8` | Delete block | ✅ |
| `F9` | Menu | — | — | ❌ Deferred (needs menu system) |
| `F10` | Quit | `F10` / `Esc` | Close | ✅ |
| `F11` | UserMenu | — | — | ❌ Deferred (needs INI integration) |
| `F12` | SaveAs | `F12` | SaveAs prompt | ✅ P1 |
| `F13` | MarkColumn | — | — | ❌ P4 |
| `F14` | ReplaceContinue | — | — | ❌ P4 |
| `F15` | InsertFile | `F15` | Insert file at cursor | ✅ P3 |
| `F17` | SearchContinue | `F17` | Repeat last search | ✅ P1 |
| `Insert` | InsertOverwrite | `Insert` | Toggle insert/overwrite | ✅ P3 |
| `Enter` | Enter (newline) | `Enter` | Newline + auto-indent | ✅ |
| `Shift-Enter` | Return (newline above) | `Shift-Enter` | Newline above | ✅ P3 |
| `Backspace` | BackSpace | `Backspace` / `Ctrl-H` | delete_back | ✅ P1 |
| `Delete` | Delete | `Delete` / `Ctrl-D` | delete_forward | ✅ P1 |
| `Tab` | Tab | `Tab` | insert tab | ✅ |
| `Up` | Up | `Up` | move_up | ✅ |
| `Down` | Down | `Down` | move_down | ✅ |
| `Left` | Left | `Left` | move_left | ✅ |
| `Right` | Right | `Right` | move_right | ✅ |
| `Home` | Home | `Home` | move_home | ✅ |
| `End` | End | `End` | move_end | ✅ |
| `PgUp` | PageUp | `PgUp` | move_page_up | ✅ |
| `PgDn` | PageDown | `PgDn` | move_page_down | ✅ |
| `Ctrl-Home` | Top of file | `Ctrl-Home` | set_cursor(0) | ✅ |
| `Ctrl-End` | Bottom of file | `Ctrl-End` | set_cursor(end) | ✅ |
| `Ctrl-Up` | ScrollUp | `Ctrl-Up` | scroll viewport | ✅ P1 |
| `Ctrl-Down` | ScrollDown | `Ctrl-Down` | scroll viewport | ✅ P1 |
| `Ctrl-PgUp` | TopOnScreen | `Ctrl-PgUp` | Top of screen | ✅ P1 |
| `Ctrl-PgDn` | BottomOnScreen | `Ctrl-PgDn` | Bottom of screen | ✅ P1 |
| `Ctrl-Left` | WordLeft | `Ctrl-Left` | move_word_backward | ✅ |
| `Ctrl-Right` | WordRight | `Ctrl-Right` | move_word_forward | ✅ |
| `Ctrl-Z` | WordLeft (alias) | `Ctrl-Z` | Undo (universal) | ✅ (kept) |
| `Ctrl-X` | WordRight (alias) | — | — | ❌ P4 |
| `Ctrl-U` | Undo | `Ctrl-U` / `Ctrl-Z` | Undo | ✅ P1 |
| `Ctrl-Y` | DeleteLine | `Ctrl-Y` | DeleteLine | ✅ P1 |
| `Ctrl-K` | DeleteToEnd | `Ctrl-K` | Delete to end of line | ✅ P1 |
| `Ctrl-D` | Delete (alias) | `Ctrl-D` | Delete (alias) | ✅ P1 |
| `Ctrl-H` | BackSpace (alias) | `Ctrl-H` | BackSpace (alias) | ✅ P1 |
| `Ctrl-S` | SyntaxOnOff | `Ctrl-S` | Toggle syntax highlight | ✅ P1 |
| `Ctrl-L` | Refresh | `Ctrl-L` | Refresh screen | ✅ P1 |
| `Ctrl-R` | MacroStartStopRecord | `Ctrl-R` | Macro toggle | ✅ |
| `Ctrl-P` | SpellCheckCurrentWord | `Ctrl-P` | Macro play | ⚠️ P4 (spell check) |
| `Ctrl-O` | Shell | — | — | ❌ P4 |
| `Ctrl-Q` | InsertLiteral | `Ctrl-Q` | Insert next key literally | ✅ P3 |
| `Ctrl-N` | EditNew | — | — | ❌ P4 |
| `Ctrl-F` | BlockSave | — | — | ❌ P4c |
| `Ctrl-Insert` | Store (copy) | `Ctrl-Insert` | Copy selection | ✅ P4a |
| `Ctrl-N` | EditNew | `Ctrl-N` | New file (clear buffer) | ✅ P4a |
| `Shift-Insert` | Paste | `Ctrl-V` / `Shift-Insert` | Paste | ✅ P1 |
| `Shift-Delete` | Cut | `Shift-Delete` | Cut selection | ✅ P4a |
| `Shift-Tab` | Tab (alias / unindent) | `Shift-Tab` | Unindent selection | ✅ P4a |
| `Alt-R` | Redo | `Alt-R` | Redo | ✅ P1 |
| `Alt-L` | Goto line | `Alt-L` | GotoLine prompt | ✅ |
| `Alt-B` | MatchBracket | `Alt-B` | match_bracket | ✅ |
| `Alt-P` | ParagraphFormat | `Alt-P` | format_paragraph | ✅ |
| `Alt-K` | Bookmark | `Alt-K` | BookmarkSet | ✅ P1 |
| `Alt-J` | BookmarkNext | `Alt-J` | BookmarkJump | ✅ (different action name) |
| `Alt-I` | BookmarkPrev | `Alt-I` | BookmarkPrev | ✅ P1 |
| `Alt-O` | BookmarkFlush | `Alt-O` | BookmarkClear | ✅ P1 |
| `Alt-N` | ShowNumbers | `Alt-N` | Toggle relative lines | ✅ P1 |
| `Alt-D` | DeleteToWordEnd | `Alt-D` | Delete word forward | ✅ P1 |
| `Alt-Backspace` | DeleteToWordBegin | `Alt-Backspace` | Delete word backward | ✅ P1 |
| `Alt-Tab` | Complete | `Alt-Tab` | word_complete | ✅ |
| `Alt-T` | Sort | — | — | ❌ P4 |
| `Alt-M` | Mail | — | — | ❌ P4 |
| `Alt-U` | ExternalCommand | — | — | ❌ P4 |
| `Alt-Enter` | Find | — | — | ❌ Deferred (needs FM) |
| `Alt-Minus` | FilePrev | — | — | ❌ Deferred (needs multi-file) |
| `Alt-Plus` | FileNext | — | — | ❌ Deferred (needs multi-file) |
| `Alt-E` | SelectCodepage | — | — | ❌ P4 |
| `Alt-Shift-E` | History | — | — | ❌ P4 |
| `Shift-Left` | MarkLeft | `Shift-Left` | select_left | ✅ |
| `Shift-Right` | MarkRight | `Shift-Right` | select_right | ✅ |
| `Shift-Up` | MarkUp | `Shift-Up` | start_sel + move_up | ✅ |
| `Shift-Down` | MarkDown | `Shift-Down` | start_sel + move_down | ✅ |
| `Shift-Home` | MarkToHome | `Shift-Home` | select_to_home | ✅ |
| `Shift-End` | MarkToEnd | `Shift-End` | select_to_end | ✅ |
| `Shift-PgUp` | MarkPageUp | `Shift-PgUp` | Select page up | ✅ P1 |
| `Shift-PgDn` | MarkPageDown | `Shift-PgDn` | Select page down | ✅ P1 |
| `Ctrl-Shift-Left` | MarkToWordBegin | `Ctrl-Shift-Left` | Select word backward | ✅ P1 |
| `Ctrl-Shift-Right` | MarkToWordEnd | `Ctrl-Shift-Right` | Select word forward | ✅ P1 |
| `Ctrl-Shift-Home` | MarkToFileBegin | `Ctrl-Shift-Home` | Select to file begin | ✅ P1 |
| `Ctrl-Shift-End` | MarkToFileEnd | `Ctrl-Shift-End` | Select to file end | ✅ P1 |
| `Alt-Underline` | ShowTabTws | — | — | ❌ P4 |
### TLC-only features (not in MC)
| Key | Action | Keep? |
|-----|--------|-------|
| `Ctrl-]` | Jump to tag | ✅ Keep (TLC advantage) |
| `Ctrl-T` | Pop tag stack | ✅ Keep |
| `Ctrl-F1` | Toggle code fold | ✅ Keep |
| `Alt-W` | Toggle word wrap | ✅ Keep |
| `Shift-F2` | SaveAs (alias) | ✅ Keep as alias |
| `Ctrl-V` | Paste (alias) | ✅ Keep as alias |
| `Ctrl-Z` | Undo (universal) | ✅ Keep as alias |
---
## Conflict Resolution Decision Matrix
| Conflict | MC Key/Action | TLC Key/Action | Resolution | Status |
|----------|---------------|----------------|------------|--------|
| Ctrl-S | SyntaxOnOff | Save | Ctrl-S → SyntaxOnOff. Save = F2 only. | ✅ P1 |
| Ctrl-Y | DeleteLine | Redo | Ctrl-Y → DeleteLine. Redo → Alt-R. | ✅ P1 |
| Ctrl-P | SpellCheck | Macro play | Kept as Macro play. Spell check → P4. | ⚠️ P4 |
| Ctrl-L | Refresh | Relative lines | Ctrl-L → Refresh. Relative → Alt-N. | ✅ P1 |
| Ctrl-Z | WordLeft | Undo | Kept Ctrl-Z = Undo (universal). Ctrl-U alias. | ✅ P1 |
| Alt-K | Bookmark | BookmarkSet (was Alt-M) | Alt-K → BookmarkSet. | ✅ P1 |
| Alt-O | BookmarkFlush | BookmarkClear (was Alt-K) | Alt-O → BookmarkClear. | ✅ P1 |
| Ctrl-Q | InsertLiteral | Close (was close shortcut) | Ctrl-Q → InsertLiteral. Close = F10/Esc. | ✅ P3 |
---
## Effort Estimate
| Phase | Duration | Deliverables | Status |
|-------|----------|--------------|--------|
| P1: Conflicts + Core Missing | 1-2 weeks | 5 conflict fixes, 15+ new actions, 6 selection keys | ✅ COMPLETE |
| P2: Viewer Parity | 1 week | F-key aliases, vim keys, backward search, Ctrl-A/E | ✅ COMPLETE |
| P3: Feature Gaps | 2-3 weeks | Help, InsertFile, InsertLiteral, overwrite, Shift-Enter, MagicMode | ✅ COMPLETE |
| P3.5: Handoff + Buttonbar | 1 week | start_line plumbing, +N CLI, editor/viewer buttonbar | ✅ COMPLETE |
| P4a: More Bindings | 1 week | Ctrl-Insert (Copy), Shift-Delete (Cut), Ctrl-N (EditNew), Shift-Tab (Unindent) | ✅ COMPLETE |
| P4b: Cursor Position | 1 week | filepos save/restore on close/open (MC `~/.mc/filepos` parity) | ✅ COMPLETE |
| P4c: Advanced | Future | Column select, sort, file-nav, spell check, Mail/ExternalCommand | 📋 Planned |
| **Total P1-P4b** | **7-9 weeks** | **~93% MC parity** | **✅ DONE** |
local/docs/MULTITHREADING-COMPREHENSIVE-ASSESSMENT-AND-PLAN.md
-763
View File
@@ -1,763 +0,0 @@
# Red Bear OS — Multi-Threading Comprehensive Assessment and Implementation Plan
**Date:** 2026-07-02 (initial assessment); 2026-07-02 (Phase 0c patch recovery complete)
**Scope:** Full-stack multi-threading audit: hardware/SMP, kernel scheduler, kernel futex, kernel syscall ABI, relibc pthreads, userspace threading correctness and performance
**Status:** Authoritative — supersedes `local/docs/archived/KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` and the restored `local/docs/KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` lineage for all threading matters
**Validation levels:** `builds` → `enumerates` → `usable` → `validated` → `hardware-validated`
---
## UPDATE — Phase 0c Patch Recovery (2026-07-02)
The assessment in §1 below was accurate as of the time of writing. The Phase 0c
patch recovery was then executed in the same session, landing the following
commits on the local kernel fork (`local/sources/kernel/`):
| Commit | Effect |
|--------|--------|
| `ed3f0e1` | **P6-futex-sharding**: replaces single global `Mutex<L1, FutexList>` with 64-shard hash table |
| `5fb42fc` | **RUN_QUEUE_COUNT pre-flight**: defines `pub const RUN_QUEUE_COUNT: usize = 40;` (was missing from patch chain) |
| `cbf051e` | **P7-cache-affine-context (manual)**: surgically inserts `SchedPolicy` enum, `SCHED_PRIORITY_LEVELS`, helper functions, and 9 new Context fields (last_cpu, sched_policy, sched_rt_priority, sched_rr_ticks_consumed, sched_static_prio, sched_rr_quantum, vruntime, futex_pi_*, PhysicalAddress import) |
| `f7652fc` | **P5-context-mod-sched + P8-percpu-sched + P8-percpu-wiring**: PerCpuSched struct with SyncUnsafeCell-wrapped per-CPU run queues, get_percpu_block helper, full per-CPU scheduler wiring in switch.rs (pick_next_from_queues, pick_next_from_global_queues, select_next_context) |
| `7fc8bbf` | **P8-initial-placement + P9-numa-topology + P9-proc-lock-ordering**: least-loaded-CPU spawn, NUMA topology hints, proc scheme lock order fix |
| `327c150` | **`set_sched_policy` + `set_sched_other_prio`**: missing Context methods called by proc scheme handles |
| `e8ec916` | **fadt usize/u32 type mismatch fix**: changes FADT_MIN_SIZE constants to `u32` to match `Sdt::length()` |
| `4789d54` | **SchedPolicy/Name/Priority proc scheme handles**: adds `/proc/<tid>/{name, sched-policy, priority}` paths and read/write handlers |
**Upstream check (bg_27f3578a, 2026-07-02):** verified that `gitlab.redox-os.org/redox-os/kernel`
master (commit `aa7e7d2f44ba7cd9d1b007d37db139b345d46b8a`) has **NONE** of these features. The
local fork is the sole implementation. No upstream cherry-picks are available.
**Plan-vs-actual state:**
| Plan claim (Section 1) | Actual state after Phase 0c |
|---|---|
| "Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine)" | ✅ All 5 features now present. Per-CPU `PerCpuSched` with `run_queues`, `steal_work()`, `migrate_one_context()`, `maybe_balance_queues()`, `vruntime` CFS-style weighting, `last_cpu` cache-affine vruntime bonus, `SchedPolicy::Fifo`/`RoundRobin` RT scanning in `pick_next_from_queues` |
| "Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET)" | 🟡 **Sharding done** (64-shard hash). REQUEUE/PI/robust/WAKE_OP/BITSET still missing. |
| "relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent" | 🟡 relibc fork only has the `pthread_cond_signal` POSIX fix so far. `sched_*`, robust, PI still pending in relibc. |
| "❌ Missing from relibc: CPU affinity API, Thread naming" | 🟡 **Kernel side done** — `/proc/<tid>/{sched-policy, name, priority}` handles. relibc pthread_setname_np / pthread_setaffinity_np / sched_setscheduler still pending. |
| "cargo check has 1 pre-existing error" | ✅ **Fixed** — `cargo check` now exits 0 with 0 errors. |
**Phase 0c status: kernel side complete (all 8 of 8 applicable kernel P5–P9 patches
re-applied or made obsolete by the existing refactored scheduler). Remaining work
is in the relibc fork (Phase 0e) and the futex-REQUEUE/PI/robust work (Phase 1).**
The detailed analysis in §1–§9 below is preserved as historical record. The status
column "🚧 Missing" in §1 should be re-read as "now present in the local kernel
fork, pending relibc userspace wiring."
---
---
## 1. Executive Summary
### The Critical Finding — Lost Threading Work
The P5–P9 scheduler and futex enhancement work (documented as "complete" in the archived
plans) was **lost during the local fork migration** (2026-06). The local forks at
`local/sources/kernel/` and `local/sources/relibc/` were created from **upstream Redox
baselines** that did NOT include the Red Bear enhancement patches. The patches exist in
`local/patches/kernel/` and `local/patches/relibc/` but are **not wired into the recipes**
(both `recipe.toml` files use `path = "..."` with no `patches = [...]` list).
**Impact:** The running kernel has:
- Baseline DWRR scheduler only (no per-CPU queues, no work stealing, no load balancing, no vruntime, no RT scheduling, no cache-affine)
- Baseline futex only (WAIT/WAIT64/WAKE — no sharding, no PI, no REQUEUE, no robust, no WAKE_OP, no BITSET)
- relibc `sched_*` are all `todo!()`, `pthread_setschedparam` is a no-op, robust mutexes are `todo_skip!`, PI is absent
**Recovery:** 13 of 18 kernel P5–P9 patches apply cleanly to the current fork. 5 fail due to
patch-chain dependencies (they expect earlier patches applied first). The bulk of the work is
recoverable by re-applying patches to the forks and committing them.
### What Actually Works Today
| Layer | Status | Detail |
|-------|--------|--------|
| **SMP boot** | ✅ Solid | INIT→SIPI sequence correct, per-CPU PCR via GS_BASE, x2APIC support |
| **Context switching** | ✅ Solid | FPU/SIMD/AVX state save via XSAVE, FSBASE/GSBASE swap (FSGSBASE or MSR), correct callee-saved register save |
| **TLB shootdown protocol** | ✅ Correct | AtomicBool flag + IPI + ack counter with `fence(SeqCst)` race prevention |
| **Basic thread lifecycle** | ✅ Functional | pthread_create/join/detach/exit through proc scheme + redox_rt clone |
| **Basic synchronization** | ✅ Functional | Futex-backed mutex, condvar, rwlock, barrier, spinlock, once |
| **TLS** | ✅ Functional | ELF PT_TLS + pthread_key_create/getspecific/setspecific |
| **Per-CPU data** | ✅ Functional | PercpuBlock via GS_BASE, all per-CPU state accessible |
| **Signal delivery** | ✅ Functional | Shared-memory Sigcontrol pages, per-thread masks, trampoline |
| **Scheduler algorithm** | 🚧 Basic DWRR | 40 priority levels, geometric weights, cooperative preemption (3-tick quantum) |
| **Futex operations** | 🚧 Basic only | WAIT/WAIT64/WAKE with single global mutex |
| **SMP load balancing** | ❌ Missing | No work stealing, no migration, contexts stuck on birth CPU |
| **RT scheduling** | ❌ Missing | No SCHED_FIFO/SCHED_RR, no kernel policy dispatch |
| **Futex REQUEUE** | ❌ Missing | Condvar broadcast causes thundering herd |
| **Robust mutexes** | ❌ Missing | Thread death while holding mutex → permanent deadlock |
| **PI futexes** | ❌ Missing | No priority inheritance → priority inversion risk |
| **CPU affinity API** | ❌ Missing from relibc | Kernel supports sched_affinity field but no userspace API |
| **Thread naming** | ❌ Missing from relibc | Kernel supports name field but no userspace API |
| **Per-page TLB flush** | ❌ Missing | `invalidate_all()` = full CR3 reload on every shootdown |
| **NUMA awareness** | ❌ Missing | No SRAT/SLIT, no proximity domains, flat memory model |
| **IRQ balancing** | ❌ Missing | All legacy IRQs hardwired to BSP |
---
## 2. Layer-by-Layer Assessment
### 2.1 Hardware / SMP Layer
**Files:** `src/acpi/madt/arch/x86.rs`, `src/arch/x86_shared/start.rs`,
`src/arch/x86_shared/device/local_apic.rs`, `src/arch/x86_shared/device/ioapic.rs`,
`src/arch/x86_shared/ipi.rs`, `src/arch/x86_shared/interrupt/ipi.rs`, `src/percpu.rs`,
`src/arch/x86_shared/gdt.rs`
**Verdict: Functional foundation, performance gaps.**
| Component | Status | Detail |
|-----------|--------|--------|
| AP boot (INIT/SIPI) | ✅ validated | Correct trampoline at 0x8000, per-AP PCR/IDT/stack allocation |
| x2APIC mode | ✅ builds | Detected via CPUID, MSR-based access, APIC ID detection |
| Per-CPU PCR via GS_BASE | ✅ validated | `PercpuBlock::current()` reads from PCR, SWAPGS protocol correct |
| IPI send/receive | ✅ functional | 5 IPI kinds (Wakeup/Tlb/Switch/Pit/Profile), broadcast + unicast |
| TLB shootdown | ✅ correct | AtomicBool + IPI + ack with `fence(SeqCst)` race prevention |
| TLB granularity | ❌ coarse | Full CR3 reload (`mov cr3, cr3`) on every shootdown — no INVLPG |
| TLB broadcast | 🚧 sequential | Iterates CPUs individually, doesn't use ICR "all excluding self" shorthand |
| IRQ routing | ❌ BSP-only | Legacy I/O APIC entries hardcode `dest: bsp_apic_id` |
| NUMA | ❌ absent | No SRAT/SLIT, no proximity domains |
| SMT/HT topology | ❌ absent | No cache hierarchy, no hyperthread awareness |
| Idle loop | ✅ functional | MWAIT with deepest C-state or HLT fallback |
| W^X for trampoline | 🚧 minor | Trampoline page briefly W+X, unmapped after AP boot |
### 2.2 Kernel Scheduler Layer
**Files:** `src/context/switch.rs`, `src/context/mod.rs`, `src/context/context.rs`,
`src/context/timeout.rs`
**Verdict: Correct but primitive — DWRR only, no SMP balancing, no RT classes.**
**Algorithm:** Deficit Weighted Round Robin (DWRR)
- 40 priority levels, each a `VecDeque<WeakContextRef>`
- Geometric weights: `SCHED_PRIO_TO_WEIGHT[i] ≈ 1.25^i` (88761 → 15)
- Per-CPU `balance` accumulator drives dequeue decisions
- Quantum: 3 PIT ticks (~12.2ms) per scheduling round
- Cooperative preemption: `preempt_locks > 0` disables preemption
**Global locks:**
- `RUN_CONTEXTS: Mutex<L1, RunContextData>` — all 40 priority queues under one L1 lock
- `IDLE_CONTEXTS: Mutex<L2, VecDeque<WeakContextRef>>` — sleeping contexts
- `CONTEXT_SWITCH_LOCK: AtomicBool` — global CAS spinlock serializing all context switches
**What's missing (all was in lost P5–P9 work):**
| Gap | Lost Patch | Recoverable? |
|-----|-----------|-------------|
| Per-CPU run queues (eliminate global L1) | P6-percpu-runqueues, P8-percpu-sched, P8-percpu-wiring | ✅ applies cleanly |
| Work stealing | P8-work-stealing | ❌ needs rebase (depends on per-CPU wiring) |
| Initial placement (least-loaded CPU) | P8-initial-placement | ✅ applies cleanly |
| Load balancing | P8-load-balance (absorbed) | needs verification |
| Vruntime tracking + min-vruntime selection | P6-vruntime-switch | ✅ applies cleanly |
| SchedPolicy enum (FIFO/RR/Other) | P5-sched-rt-policy | ✅ applies cleanly |
| RT scheduling dispatch | P5-sched-rt-policy | ✅ applies cleanly |
| Cache-affine scheduling | P7-cache-affine-switch | ✅ applies cleanly |
| NUMA topology hints | P9-numa-topology | ✅ applies cleanly |
### 2.3 Kernel Futex Layer
**File:** `src/syscall/futex.rs`
**Verdict: Baseline only — critical operations missing for desktop workloads.**
| Operation | Status | Impact of Absence |
|-----------|--------|-------------------|
| `FUTEX_WAIT` (32-bit) | ✅ | — |
| `FUTEX_WAIT64` (64-bit) | ✅ | — |
| `FUTEX_WAKE` | ✅ | — |
| `FUTEX_REQUEUE` | ❌ returns EINVAL | `pthread_cond_broadcast` wakes ALL waiters (thundering herd) |
| `FUTEX_CMP_REQUEUE` | ❌ not defined | Same + atomicity gap |
| `FUTEX_WAKE_OP` | ❌ not defined | glibc mutex fast path unavailable |
| `FUTEX_WAIT_BITSET` | ❌ not defined | `pselect`/`ppoll` optimization unavailable |
| `FUTEX_WAKE_BITSET` | ❌ not defined | Targeted wake unavailable |
| `FUTEX_LOCK_PI` / `UNLOCK_PI` | ❌ not defined | Priority inversion unprotected |
| Robust futex list | ❌ not defined | Thread death → permanent deadlock |
| Futex sharding (per-futex lock) | ❌ single global L1 mutex | All futex ops on all CPUs contend on one lock |
| Process-private futexes | ❌ global table | Unnecessary cross-process visibility |
**Architecture:**
```
static FUTEXES: Mutex<L1, FutexList> // single global lock
type FutexList = HashMap<PhysicalAddress, Vec<FutexEntry>>
```
Physical address is the key (enables cross-address-space futex via MAP_SHARED).
Virtual address + Weak<AddrSpaceWrapper> used for CoW disambiguation.
**Recoverable work (lost patches):**
| Feature | Lost Patch | Applies? |
|---------|-----------|----------|
| 64-shard hash table | P6-futex-sharding | ✅ cleanly |
| FUTEX_REQUEUE + CMP_REQUEUE | P8-futex-requeue | ❌ needs rebase |
| PI futex (LOCK_PI/UNLOCK_PI/TRYLOCK_PI) | P8-futex-pi | ❌ needs rebase |
| PI CAS fix | P9-futex-pi-cas-fix | ❌ needs rebase |
| Robust futex list | P8-futex-robust | ❌ needs rebase |
The 4 failing patches likely fail because they depend on sharding (P6-futex-sharding) being
applied first. Apply in order: P6-sharding → P8-requeue → P8-pi → P8-robust → P9-pi-cas-fix.
### 2.4 Kernel Syscall ABI Layer
**Files:** `src/syscall/mod.rs`, `src/syscall/futex.rs`, `src/syscall/time.rs`,
`src/syscall/process.rs`, `local/sources/syscall/src/number.rs`, `src/scheme/proc.rs`
**Verdict: Minimal surface — most threading done via proc scheme, not syscalls.**
The kernel defines only ~35 syscall numbers. Threading-relevant ones:
| Syscall | Status | Notes |
|---------|--------|-------|
| `SYS_FUTEX` (240) | ✅ partial | WAIT/WAIT64/WAKE only |
| `SYS_YIELD` (158) | ✅ | `context::switch()` + signal handler |
| `SYS_FMAP` (900) | ✅ | Anonymous + file-backed mmap |
| `SYS_FUNMAP` (92) | ✅ | munmap |
| `SYS_MPROTECT` (125) | ✅ | |
| `SYS_MREMAP` (155) | ✅ | |
| `SYS_NANOSLEEP` (162) | ✅ | EINTR-aware |
| `SYS_CLOCK_GETTIME` (265) | ✅ partial | REALTIME + MONOTONIC only |
**Threading done via proc scheme (not syscalls):**
| Operation | Mechanism |
|-----------|-----------|
| Thread/process creation | `proc:` scheme: open "new-context", share addr_space + files via kdup |
| waitpid | `proc:` scheme: `EVENT_READ` on context fd |
| getpid/gettid | `proc:` scheme: read "attrs" handle |
| kill/tkill | `proc:` scheme: `ForceKill` / `Interrupt` ContextVerb |
| CPU affinity | `proc:` scheme: write "sched-affinity" handle |
| Priority | `proc:` scheme: write "attrs" prio field |
| Signal setup | `proc:` scheme: write "sighandler" + shared Sigcontrol pages |
| TLS base (FSBASE) | `proc:` scheme: write "regs/env" EnvRegisters |
**Completely missing syscalls (no number, no handler):**
`clone`, `fork`, `vfork`, `waitpid`, `wait4`, `kill`, `tkill`, `tgkill`, `arch_prctl`,
`set_thread_area`, `set_tid_address`, `set_robust_list`, `get_robust_list`,
`sched_setaffinity`, `sched_getaffinity`, `sched_setscheduler`, `sched_getparam`,
`sigaction`, `sigprocmask`, `sigpending`, `sigsuspend`, `sigtimedwait`,
`timer_create`, `timer_settime`, `timer_delete`, `timerfd_create`,
`getrusage`, `setrlimit`, `getrlimit`, `times`
### 2.5 relibc Pthread Layer
**Files:** `src/pthread/mod.rs`, `src/sync/*.rs`, `src/header/pthread/*.rs`,
`src/header/sched/mod.rs`, `src/ld_so/tcb.rs`, `src/platform/redox/mod.rs`
**Verdict: Core pthreads solid, scheduling/robust/PI absent, several POSIX gaps.**
#### Fully Working (futex-backed)
| API Group | Backend | Notes |
|-----------|---------|-------|
| `pthread_create/join/detach/exit` | redox_rt clone + Waitval | Stack via mmap, TLS via Tcb::new() |
| `pthread_cancel/setcancelstate/testcancel` | SIGRT_RLCT_CANCEL (33) | Deferred cancellation only |
| `pthread_mutex_*` (normal/recursive/errorcheck) | AtomicU32 CAS + futex_wait/wake | 3-state: unlocked/locked/waiters |
| `pthread_cond_*` | Two-counter futex design | CLOCK_REALTIME only (monotonic = stub) |
| `pthread_rwlock_*` | AtomicU32 + futex | Reader count + WAITING_WR bit |
| `pthread_barrier_*` | Mutex + Cond | gen_id wrapping counter |
| `pthread_spin_*` | AtomicI32 CAS | No futex, pure spinning |
| `pthread_once` | 3-state futex (UNINIT→INITING→INIT) | |
| `pthread_key_create/getspecific/setspecific/delete` | BTreeMap global + thread_local values | Destructor iteration per POSIX |
| `pthread_sigmask` | Delegates to sigprocmask | |
| `pthread_kill` | redox_rt::rlct_kill | |
| `pthread_atfork` | Thread-local LinkedList hooks | |
| ELF TLS (`__thread` / `#[thread_local]`) | PT_TLS + Tcb | Static + dynamic DTV for dlopen |
| `pthread_attr_*` (getters/setters) | RlctAttr struct | |
#### Stubs / No-ops / Missing
| API | Status | Root Cause |
|-----|--------|------------|
| `sched_get_priority_max/min` | `todo!()` | Kernel has no scheduling policy API |
| `sched_getparam/setparam` | `todo!()` | Same |
| `sched_setscheduler` | `todo!()` | Same |
| `sched_rr_get_interval` | `todo!()` | Same |
| `pthread_setschedparam` | No-op (returns Ok) | Kernel ignores policy |
| `pthread_setschedprio` | No-op (returns Ok) | Kernel ignores priority change |
| `pthread_getschedparam` | `todo!()` | |
| `pthread_getcpuclockid` | ENOENT | No per-thread CPU clock |
| `pthread_mutex_consistent` | `todo_skip!` | Robust mutex not implemented |
| `pthread_mutex_getprioceiling` | `todo_skip!` | Priority ceiling not implemented |
| `pthread_mutex_setprioceiling` | `todo_skip!` | Same |
| `pthread_mutexattr_setprotocol` (PRIO_INHERIT) | Accepted, no-op | PI futex missing |
| `pthread_mutexattr_setrobust` (ROBUST) | Accepted, no-op | Robust futex missing |
| `pthread_cond_init` CLOCK_MONOTONIC | `todo_skip!` | |
| `pthread_cond_signal` | Calls broadcast (wakes ALL) | Missing FUTEX_REQUEUE optimization |
| `pthread_setaffinity_np` | Not defined | |
| `pthread_getaffinity_np` | Not defined | |
| `pthread_setname_np` | Not defined | |
| `pthread_getname_np` | Not defined | |
| `pthread_setcanceltype` | Always returns DEFERRED | ASYNC not tracked |
| Guard pages | Attribute stored, not mapped | No PROT_NONE page before stack |
| PTHREAD_KEYS_MAX limit | Not checked | |
---
## 3. Gap Classification
### 3.1 Correctness Gaps (Must Fix — Silent Data Corruption or Deadlock)
| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| C1 | **No robust mutexes** | Thread death while holding mutex → permanent deadlock for all waiters | Kernel: robust futex list + relibc: pthread_mutex_consistent |
| C2 | **No PI futexes** | Priority inversion: low-prio thread blocks high-prio thread indefinitely | Kernel: FUTEX_LOCK_PI/UNLOCK_PI + relibc: mutexattr_setprotocol |
| C3 | **`pthread_cond_signal` wakes ALL** | Correctness: wastes CPU. Performance: thundering herd on every signal | relibc: use true wake(1) — may need FUTEX_REQUEUE |
| C4 | **`fork()` not thread-safe** | `pthread_atfork` hooks exist but child inherits locked mutexes | relibc: implement atfork child handlers properly |
### 3.2 Performance Gaps (Must Fix for Desktop Responsiveness)
| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| P1 | **No SMP load balancing** | Cores sit idle while others are overloaded | Kernel: work stealing + initial placement |
| P2 | **No futex sharding** | Single global L1 mutex for ALL futex ops on ALL CPUs | Kernel: 64-shard hash table |
| P3 | **No FUTEX_REQUEUE** | `pthread_cond_broadcast` wakes all → thundering herd | Kernel: REQUEUE + CMP_REQUEUE |
| P4 | **Full TLB flush on every shootdown** | Per-page mprotect/munmap flushes entire TLB on all cores | Kernel: INVLPG-based selective flush |
| P5 | **Global context switch lock** | Serialization bottleneck beyond ~8 cores | Kernel: per-CPU context switch (needs per-CPU run queues) |
| P6 | **All IRQs to BSP** | CPU 0 handles all interrupts, cache thrash, latency | Kernel: IRQ steering in I/O APIC + MSI/MSI-X dest field |
| P7 | **No RT scheduling** | Audio/compositor threads can't get priority | Kernel: SchedPolicy + RT dispatch + relibc: sched_setscheduler |
### 3.3 POSIX Completeness Gaps (Must Fix for Application Compatibility)
| # | Gap | Impact | Fix Location |
|---|-----|--------|-------------|
| X1 | `sched_*` all `todo!()` | Applications calling sched_setscheduler panic | relibc: implement via proc scheme |
| X2 | `pthread_setschedparam` no-op | Apps can't change thread priority | relibc: wire to proc scheme prio write |
| X3 | `pthread_setaffinity_np` missing | Apps can't pin threads to CPUs | relibc: implement via proc scheme affinity write |
| X4 | `pthread_setname_np` missing | Debugging harder (no thread names in /proc) | relibc: implement via proc scheme name write |
| X5 | `pthread_getcpuclockid` ENOENT | Per-thread profiling impossible | relibc + kernel: expose cpu_time via clock |
| X6 | Guard pages not mapped | Stack overflow → silent corruption, no SIGSEGV | relibc: mmap PROT_NONE guard page in pthread_create |
| X7 | `pthread_cond_init` monotonic stub | CLOCK_MONOTONIC condvars use REALTIME (affected by wall clock jumps) | relibc: implement monotonic condvar |
---
## 4. Implementation Plan
### Phase 0: Patch Recovery — Re-Apply Lost Threading Work (Week 1–2)
**Goal:** Recover the P5–P9 work that was lost during the local fork migration.
**This is the highest-priority phase — it restores ~6 months of work with minimal new code.**
#### 0.1 — Re-apply kernel scheduler patches to local fork
Apply in dependency order to `local/sources/kernel/`:
| Order | Patch | Status | Action |
|-------|-------|--------|--------|
| 1 | P6-futex-sharding | ✅ applies | Commit directly |
| 2 | P6-percpu-runqueues | ✅ applies | Commit directly |
| 3 | P8-percpu-sched | ✅ applies | Commit directly |
| 4 | P8-percpu-wiring | ✅ applies | Commit directly |
| 5 | P8-initial-placement | ✅ applies | Commit directly |
| 6 | P5-sched-rt-policy | ✅ applies | Commit directly |
| 7 | P5-context-mod-sched | ✅ applies | Commit directly |
| 8 | P6-vruntime-switch | ✅ applies | Commit directly |
| 9 | P7-cache-affine-switch | ✅ applies | Commit directly |
| 10 | P9-numa-topology | ✅ applies | Commit directly |
| 11 | P9-proc-lock-ordering | ✅ applies | Commit directly |
| 12 | P8-work-stealing | ❌ needs rebase | Rebase against 1–11, then apply |
| 13 | P8-futex-requeue | ❌ needs rebase | Rebase against P6-sharding (#1), then apply |
| 14 | P8-futex-pi | ❌ needs rebase | Rebase against #13, then apply |
| 15 | P8-futex-robust | ❌ needs rebase | Rebase against #14, then apply |
| 16 | P9-futex-pi-cas-fix | ❌ needs rebase | Rebase against #14, then apply |
| 17 | P7-scheduler-improvements | ❌ needs rebase | Rebase against 1–11, then apply |
**Verification after each patch:**
```bash
cd local/sources/kernel
cargo check # must pass
```
#### 0.2 — Re-apply relibc threading patches to local fork
Apply to `local/sources/relibc/`:
| Patch | Action |
|-------|--------|
| P3-threads.patch | ✅ applies — commit |
| P3-barrier-smp-futex (from absorbed/) | Verify already in fork; if not, apply |
| P3-pthread-signal-races (from absorbed/) | Verify already in fork |
| P3-pthread-yield (from absorbed/) | Verify already in fork |
| P5-robust-mutexes (from absorbed/) | Verify; re-apply if missing |
| P5-robust-mutex-enotrec-fix (from absorbed/) | Same |
| P5-sched-api (from absorbed/) | Same |
| P7-pthread-affinity (from absorbed/) | Same |
| P7-pthread-setname (from absorbed/) | Same |
| P7-setpriority (from absorbed/) | Same |
| P9-spin-and-barrier (from absorbed/) | Same |
| P9-spin-fix (from absorbed/) | Same |
| P3-semaphore-comprehensive | ✅ applies |
**Verification:**
```bash
cd local/sources/relibc
make all # must pass
touch relibc && make prefix # rebuild prefix with new libc
```
#### 0.3 — Build and smoke test
```bash
export REDBEAR_ALLOW_PROTECTED_FETCH=1
./local/scripts/build-redbear.sh --upstream redbear-mini
make qemu # verify boot + basic operation
```
**Success criteria:** redbear-mini boots, multi-threaded daemons (pcid, xhcid) start, no kernel panic.
---
### Phase 1: Futex Completeness (Week 2–4)
**Goal:** Close the futex operation gaps that affect correctness and performance.
**Depends on:** Phase 0 complete (sharding applied first).
#### 1.1 — FUTEX_REQUEUE + FUTEX_CMP_REQUEUE
**Kernel:** `src/syscall/futex.rs`
- Add `FUTEX_REQUEUE` and `FUTEX_CMP_REQUEUE` to the futex dispatcher
- Implement: move up to `val` waiters from addr1 → addr2, optionally compare `*addr1 == val2`
- Requires locking TWO shards (acquire both in deterministic order to avoid deadlock)
**relibc:** `src/sync/cond.rs`
- Change `pthread_cond_broadcast` to use `FUTEX_REQUEUE` (move waiters from condvar futex to mutex futex)
- Change `pthread_cond_signal` to wake exactly 1 (not all)
**Impact:** Eliminates thundering herd on every `pthread_cond_broadcast`. Major win for Qt event loop, KWin compositor, Mesa worker threads.
#### 1.2 — PI Futexes (FUTEX_LOCK_PI / FUTEX_UNLOCK_PI / FUTEX_TRYLOCK_PI)
**Kernel:** `src/syscall/futex.rs`
- Add `PiState` tracking per futex: owner context + waiter list with priorities
- On `LOCK_PI` block: boost owner's priority to waiter's priority
- On `UNLOCK_PI`: restore original priority, wake highest-priority waiter
- Requires kernel RT scheduling (Phase 0.1 #6–7: P5-sched-rt-policy)
**relibc:** `src/sync/pthread_mutex.rs`
- Implement `PTHREAD_PRIO_INHERIT` protocol path using PI futex
- Replace `todo_skip!` in `pthread_mutex_consistent` with real implementation
#### 1.3 — Robust Futex List
**Kernel:** `src/syscall/futex.rs` + `src/context/context.rs`
- Add `robust_list_head: Option<usize>` to `Context` struct
- Implement `set_robust_list` / `get_robust_list` via proc scheme or syscall
- On thread exit (`exit_this_context`): walk robust list, set `FUTEX_OWNER_DIED` bit, wake one waiter with `EOWNERDEAD`
**relibc:** `src/sync/pthread_mutex.rs`
- Implement robust list registration in `pthread_mutex_lock`
- Implement `pthread_mutex_consistent`: clear `EOWNERDEAD` state
- Replace `todo_skip!` with real implementation
#### 1.4 — FUTEX_WAKE_OP
**Kernel:** `src/syscall/futex.rs`
- Implement atomic op + wake: perform op on addr2, then wake up to `val` waiters on addr1
- Operations: set, add, or, andn, xor, with comparison condition
**Impact:** glibc mutex fast path optimization. Not critical for relibc but helps ported glibc-linked binaries.
---
### Phase 2: SMP Scheduling Quality (Week 3–6)
**Goal:** Make multi-core actually distribute work.
**Depends on:** Phase 0 complete (per-CPU queues applied).
#### 2.1 — Work stealing (recover + fix)
**Kernel:** `src/context/switch.rs`
- On `select_next_context()` empty local queue: steal from victim CPU
- Pick victim by round-robin, steal highest-priority runnable context
- Limit steal batch size (1–2 contexts per steal attempt)
- Send `IpiKind::Wakeup` to target CPU if stealing woke it from idle
**Recovery:** P8-work-stealing needs rebase against per-CPU wiring.
#### 2.2 — Load balancing (recover + verify)
**Kernel:** `src/context/switch.rs`
- Periodic balance trigger (every N ticks or when queue depth difference > threshold)
- Migrate contexts from overloaded CPU to most-idle CPU
- Respect `sched_affinity` mask during migration
**Recovery:** P8-load-balance is in absorbed/ — verify it's in the fork after Phase 0.
#### 2.3 — Reschedule IPI
**Kernel:** `src/arch/x86_shared/ipi.rs` + `src/context/switch.rs`
- When waking a context on a different CPU, send `IpiKind::Switch` to that CPU
- Currently the Switch IPI exists but is not used by the scheduler
#### 2.4 — Per-page TLB flush (INVLPG)
**Kernel:** `rmm/src/arch/x86_64.rs` + `src/context/memory.rs`
- Add `invalidate_page(addr)` using `invlpg` instruction
- Modify `Flusher` to track individual pages and use INVLPG when ≤ N pages affected
- Fall back to CR3 reload only for large-scale invalidations
**Impact:** Every `mprotect`/`mmap`/`munmap` on a multi-threaded process currently flushes the ENTIRE TLB on every core. This is one of the most impactful single fixes.
#### 2.5 — TLB broadcast optimization
**Kernel:** `src/percpu.rs`
- Replace per-CPU sequential `shootdown_tlb_ipi(Some(id))` loop with ICR "all excluding self" (destination shorthand 0b11)
- Single IPI + global ack counter instead of N individual IPIs + N ack counters
---
### Phase 3: RT Scheduling (Week 4–6)
**Goal:** Allow applications to request real-time scheduling for latency-sensitive threads.
**Depends on:** Phase 0 (SchedPolicy applied) + Phase 2 (per-CPU queues).
#### 3.1 — Kernel RT scheduling dispatch
**Kernel:** `src/context/switch.rs` (from P5-sched-rt-policy — recovered in Phase 0)
- `select_next_context()` passes:
1. SCHED_FIFO contexts (highest RT priority first, no preemption within same prio)
2. SCHED_RR contexts (highest RT priority first, round-robin within same prio)
3. SCHED_OTHER contexts (existing DWRR/vruntime)
- SCHED_RR quantum: configurable per-context (default 100ms)
#### 3.2 — relibc sched_* API completion
**relibc:** `src/header/sched/mod.rs`
Replace ALL `todo!()` stubs:
| Function | Implementation |
|----------|---------------|
| `sched_getscheduler(pid)` | Read policy from proc scheme attrs |
| `sched_setscheduler(pid, policy, param)` | Write policy + RT priority via proc scheme |
| `sched_getparam(pid, param)` | Read RT priority from proc scheme |
| `sched_setparam(pid, param)` | Write RT priority via proc scheme |
| `sched_get_priority_max(policy)` | Return 99 for FIFO/RR, 0 for OTHER |
| `sched_get_priority_min(policy)` | Return 1 for FIFO/RR, 0 for OTHER |
| `sched_rr_get_interval(pid, tp)` | Return SCHED_RR quantum (100ms default) |
#### 3.3 — pthread_setschedparam wiring
**relibc:** `src/pthread/mod.rs`
- Replace `set_sched_param` no-op with real proc scheme call
- Replace `set_sched_priority` no-op with real proc scheme call
---
### Phase 4: POSIX Pthread Completeness (Week 5–8)
**Goal:** Close remaining POSIX gaps that block application compatibility.
**Depends on:** Phase 0 + Phase 3 (for sched API).
#### 4.1 — pthread_setaffinity_np / pthread_getaffinity_np
**relibc:** `src/header/pthread/mod.rs` + `src/header/sched/mod.rs`
- Implement using proc scheme "sched-affinity" write/read
- Define `cpu_set_t` type and `CPU_SET/CPU_CLR/CPU_ZERO/CPU_ISSET` macros
#### 4.2 — pthread_setname_np / pthread_getname_np
**relibc:** `src/header/pthread/mod.rs`
- Implement using proc scheme name write/read (kernel already supports 32-char name field)
#### 4.3 — pthread_cond_init CLOCK_MONOTONIC
**relibc:** `src/sync/cond.rs`
- Replace `todo_skip!` with real monotonic clock support
- Store clock choice in cond struct, use `CLOCK_MONOTONIC` for deadline calculations
#### 4.4 — Guard pages
**relibc:** `src/pthread/mod.rs`
- In `pthread_create`, when allocating stack via mmap:
- Map `[stack_base, stack_base + guard_size)` with `PROT_NONE`
- Map `[stack_base + guard_size, stack_base + guard_size + stack_size)` with `PROT_READ | PROT_WRITE`
- On thread exit, munmap both regions
#### 4.5 — pthread_getcpuclockid
**relibc:** `src/header/pthread/mod.rs`
- Return `CLOCK_THREAD_CPUTIME_ID` (requires kernel support — add clock to `clock_gettime`)
**Kernel:** `src/syscall/time.rs`
- Add `CLOCK_THREAD_CPUTIME_ID` → read `context.cpu_time`
#### 4.6 — PTHREAD_KEYS_MAX enforcement
**relibc:** `src/header/pthread/tls.rs`
- Check `NEXTKEY` against `PTHREAD_KEYS_MAX` (1024) before allocating
---
### Phase 5: IRQ Steering and NUMA (Week 8–12)
**Goal:** Distribute interrupt load and respect memory locality.
**Depends on:** Phase 2 (per-CPU infrastructure).
#### 5.1 — IRQ steering
**Kernel:** `src/arch/x86_shared/device/ioapic.rs` + `src/arch/x86_shared/idt.rs`
- Change I/O APIC redirection `dest` from `bsp_apic_id` to round-robin or RSS hash
- Add per-CPU legacy IRQ handlers in IDT (not just BSP)
- For MSI/MSI-X: set destination CPU in Message Address register
#### 5.2 — NUMA topology discovery
**Kernel:** `src/acpi/` (from P9-numa-topology — recovered in Phase 0)
- Parse SRAT (Static Resource Affinity Table) for proximity domains
- Parse SLIT (System Locality Distance Information Table) for inter-node distances
- Store `NumaTopology` in kernel for O(1) scheduling lookups
#### 5.3 — NUMA-aware memory allocation
**Kernel:** `src/memory/` + frame allocator
- Track frame NUMA node in `Frame` or `PageInfo`
- On allocation, prefer frames from requesting CPU's NUMA node
- Fallback to remote node when local node is exhausted
---
## 5. Dependency Chain
```
Phase 0 (Patch Recovery) ← BLOCKING FOR ALL OTHERS
│
├──► Phase 1 (Futex Completeness)
│ │
│ ├──► 1.1 REQUEUE ──► condvar performance
│ ├──► 1.2 PI ──► priority inversion fix (needs Phase 3.1)
│ ├──► 1.3 Robust ──► deadlock prevention
│ └──► 1.4 WAKE_OP ──► glibc compat
│
├──► Phase 2 (SMP Scheduling)
│ │
│ ├──► 2.1 Work stealing ──► core utilization
│ ├──► 2.2 Load balancing ──► fair distribution
│ ├──► 2.3 Reschedule IPI ──→ cross-CPU wakeup
│ ├──► 2.4 Per-page TLB ──► mmap/mprotect performance
│ └──► 2.5 TLB broadcast ──► IPI efficiency
│
├──► Phase 3 (RT Scheduling)
│ │
│ ├──► 3.1 Kernel RT dispatch (from Phase 0)
│ ├──► 3.2 relibc sched_* API ──► POSIX compat
│ └──► 3.3 pthread_setschedparam ──► app priority control
│
├──► Phase 4 (POSIX Pthread Completeness)
│ │
│ ├──► 4.1 Affinity API ──► CPU pinning
│ ├──► 4.2 Thread naming ──► debuggability
│ ├──► 4.3 Monotonic condvar ──► clock correctness
│ ├──► 4.4 Guard pages ──► stack overflow detection
│ ├──► 4.5 CPU clock ──► per-thread profiling
│ └──► 4.6 Keys max ──► resource limit
│
└──► Phase 5 (IRQ + NUMA)
│
├──► 5.1 IRQ steering ──► interrupt distribution
├──► 5.2 NUMA topology ──► (from Phase 0)
└──► 5.3 NUMA allocator ──► memory locality
```
**Parallel work possible:**
- Phase 1 + Phase 2 + Phase 3 can run in parallel after Phase 0
- Phase 4 items are independent of each other
- Phase 5 depends on Phase 2 but not on Phase 1/3/4
---
## 6. Validation Plan
### 6.1 Build Evidence
| Check | Command |
|-------|---------|
| Kernel compiles | `make r.kernel` |
| relibc compiles | `make r.relibc` |
| Prefix rebuilt | `touch relibc kernel && make prefix` |
| Full OS builds | `./local/scripts/build-redbear.sh redbear-mini` |
### 6.2 Runtime Evidence (QEMU)
| Test | Verification |
|------|-------------|
| Multi-threaded boot | `make qemu QEMUFLAGS="-smp 4"` — all 4 CPUs active |
| pthread smoke test | Guest: compile + run simple pthread_create/join/mutex test |
| Work stealing | Guest: spawn 8 threads on 4-CPU QEMU, verify all CPUs utilized |
| Futex REQUEUE | Guest: condvar broadcast benchmark — waiters wake in ≤2 batches, not N |
| PI futex | Guest: priority inversion test — high-prio thread unblocked within 1 tick |
| Robust mutex | Guest: kill thread holding mutex, verify EOWNERDEAD recovery |
| RT scheduling | Guest: SCHED_FIFO thread preempts SCHED_OTHER within 100μs |
| CPU affinity | Guest: pin thread to CPU 1, verify it never runs on CPU 0 |
| Thread naming | Guest: `cat /scheme/proc/*/name` shows set names |
| Guard pages | Guest: overflow stack, verify SIGSEGV (not silent corruption) |
| TLB efficiency | Guest: mprotect benchmark — compare TLB miss rate before/after |
### 6.3 Validation Scripts (to create)
```bash
local/scripts/test-threading-qemu.sh # Comprehensive threading smoke test
local/scripts/test-futex-requeue-qemu.sh # REQUEUE-specific test
local/scripts/test-futex-pi-qemu.sh # PI futex test
local/scripts/test-futex-robust-qemu.sh # Robust mutex test
local/scripts/test-sched-rt-qemu.sh # RT scheduling latency test
local/scripts/test-sched-balance-qemu.sh # Load balancing on multi-vCPU
local/scripts/test-threading-baremetal.sh # Bare metal multi-threaded stress
```
---
## 7. Estimated Effort
| Phase | Duration | New Code | Recovery | Dependencies |
|-------|----------|----------|----------|-------------|
| Phase 0: Patch Recovery | 1–2 weeks | Minimal (rebase 5 patches) | 13 patches apply directly | None |
| Phase 1: Futex Completeness | 2–3 weeks | REQUEUE impl + WAKE_OP | PI/robust from P8 patches | Phase 0 |
| Phase 2: SMP Scheduling | 3–4 weeks | TLB INVLPG + broadcast opt | Work stealing from P8 | Phase 0 |
| Phase 3: RT Scheduling | 1–2 weeks | relibc sched_* API | RT dispatch from P5 | Phase 0 |
| Phase 4: POSIX Pthread | 2–3 weeks | Affinity/naming/guard/clock | Partial from P7 patches | Phase 0, 3 |
| Phase 5: IRQ + NUMA | 3–4 weeks | IRQ steering + NUMA allocator | NUMA topology from P9 | Phase 0, 2 |
**Total:** 12–18 weeks with 1–2 developers. Phase 0 alone recovers the majority of the value in 1–2 weeks.
---
## 8. Integration with Existing Plans
| Plan | Relationship |
|------|-------------|
| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | **Consumer** — Phase 3 (KWin) needs PI futex + RT scheduling; Phase 2 (compositor) needs work stealing |
| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | **Sibling** — IRQ steering (Phase 5.1) belongs to both plans |
| `DRM-MODERNIZATION-EXECUTION-PLAN.md` | **Consumer** — GPU worker threads benefit from load balancing + affinity |
| `IMPLEMENTATION-MASTER-PLAN.md` | **Parent** — this plan covers the kernel threading substrate |
| `CPU-DMA-IRQ-MSI-SCHEDULER-FIX-PLAN.md` | **Sibling** — overlaps on scheduler/IRQ delivery |
---
## 9. Bottom Line
The Red Bear OS threading stack is **functional for basic single-threaded and lightly-threaded
workloads**. The SMP boot, context switching, TLB shootdown, and basic futex operations are
correct.
The **critical problem** is that 6 months of threading enhancement work (P5–P9 patches) was
lost during the local fork migration. This work exists as patch files that apply cleanly to
the current fork — **Phase 0 (Patch Recovery) is the single highest-ROI action**.
After Phase 0, the remaining gaps are:
1. **Futex REQUEUE/PI/robust** — for condvar performance and deadlock prevention
2. **SMP work stealing + load balancing** — for multi-core utilization
3. **RT scheduling** — for audio/compositor thread priority
4. **POSIX pthread completeness** — for application compatibility
5. **IRQ steering + NUMA** — for multi-socket performance
The **desktop-critical path** (KWin responsiveness) requires Phases 0–3. The
**server-critical path** (multi-socket, NUMA) adds Phase 5. Phase 4 (POSIX completeness)
benefits all paths but is not desktop-blocking.
local/docs/REDBEAR-BOOT-EXPERIENCE-PLAN.md
-483
View File
@@ -1,483 +0,0 @@
# Red Bear OS — CachyOS-Class Boot Experience Implementation Plan
**Version:** 1.0 · 2026-06-11 · Branch: `0.2.3`
**Status:** Canonical plan for boot visual quality, display handoff, and boot speed
**Depends on:** existing `redox-drm`, `inputd`, `vesad`, `fbbootlogd`, `fbcond`, `bootloader`
**Supersedes:** boot-comfort fragments in `CONSOLE-TO-KDE-DESKTOP-PLAN.md` (boot pipeline layer only)
---
## 0. Architecture Decision
**The Linux model is correct: once DRM driver becomes available, it realizes handoff automatically.**
No daemon-side config awareness. No polling. No inter-daemon handshakes. When `redox-drm` registers
`scheme:drm/card0`, the display path switches through the existing `inputd` ESTALE mechanism. Init
orchestrates the lifecycle — staging the splash, detecting DRM, withdrawing the earlyfb, forwarding
traffic to the new path.
### Target Pipeline (Post-Plan)
```
UEFI GOP framebuffer (bootloader paints Red Bear logo)
→ kernel boots, passes FB env vars to init
→ init starts vesad (20_vesad.service) ← registers display.vesa (earlyfb)
→ init starts redbear-bootanim (20_bootanim.service) ← paints splash on earlyfb
→ init starts fbbootlogd (quiet mode, hidden behind splash)
→ init starts fbcond (VT 2, behind splash)
→ redox-drm loads (04_drivers.target), registers scheme:drm/card0
→ inputd signals ESTALE on all display.* handles
→ 50_drm-handoff.service runs ← atomic swap: vesad → DRM
• bootanim re-parents onto DRM FB (memcpy, no redraw)
• fbbootlogd/fbcond reconnect to DRM
• vesad releases bootloader FB, exits
→ SDDM/KWin start (08_userland.target)
→ bootanim fades out as greeter paints
Visible result: black → red bear logo + spinner → silent handoff → SDDM fade-in
No log text unless user presses Esc. No flicker. No blank screen.
```
### Linux Mechanism Mapping
| CachyOS / Linux | Red Bear equivalent |
|---|---|
| `simpledrm` (kernel) | `vesad` earlyfb + bootanim mmap |
| `Plymouth` (userspace splash) | `redbear-bootanim` (Rust, per AGENTS.md "system-critical must be Rust") |
| Plymouth two-step (pre-DRM → post-DRM) | bootanim `Surface::Vesad` → `Surface::Drm` state machine |
| `drm_aperture_remove_conflicting_framebuffers()` | init-managed via `50_drm-handoff.service` + `98_release_vesad.service` |
| `CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER` | bootanim holds firmware FB visible until DRM handoff completes |
| Plymouth Esc-to-reveal | bootanim SIGUSR2 → fbbootlogd reconnects, paints log overlay |
| Plymouth fade-out on greeter ready | bootanim SIGTERM → 200ms fade → exit |
---
## 1. Current State Assessment
### What Exists
| Component | Location | Scheme | Status |
|---|---|---|---|
| Bootloader | `local/sources/bootloader/` | UEFI GOP text menu | Text-only, no logo/splash |
| Kernel debug display | `local/sources/kernel/src/devices/graphical_debug/` | `scheme:debug` | Immediately overwrites bootloader FB |
| vesad | `local/sources/base/drivers/graphics/vesad/` | `display.vesa` | ✅ Registers earlyfb. No handoff code. Stays alive. |
| fbbootlogd | `local/sources/base/drivers/graphics/fbbootlogd/` | `fbbootlog` | ✅ Overwrites FB with log text immediately. Has handoff path. VT 1. |
| fbcond | `local/sources/base/drivers/graphics/fbcond/` | `fbcon` | ✅ Text console VTs. Handoff with 4-retry limit. VT 2+. |
| inputd | `local/sources/base/drivers/inputd/` | `scheme:input` | ✅ Display/input multiplexer. Signals ESTALE on handoff. |
| redox-drm | `local/recipes/gpu/redox-drm/source/` | `scheme:drm` | 🚧 Registers DRM. Calls inputd/handle/ to announce itself. |
| virtio-gpud | `local/sources/base/drivers/graphics/virtio-gpud/` | `display.virtio-gpu` | ⚠️ Legacy, uses old GraphicsScheme API |
| ihdgd | `local/sources/base/drivers/graphics/ihdgd/` | `display.ihdg.*` | ⚠️ Legacy Intel driver |
| Branding assets | `local/Assets/images/` | n/a | PNGs exist, NOT integrated anywhere |
### What's Missing (Gap Analysis)
| # | Gap | Impact |
|---|-----|--------|
| 1 | No boot splash/logo | User sees raw kernel/init log text from the first millisecond |
| 2 | fbbootlogd overwrites bootloader FB immediately | Any bootloader-painted pixels are destroyed within milliseconds |
| 3 | No smooth display handoff | vesad stays alive, doesn't release FB memory, no coordinated transition |
| 4 | No "quiet boot" mode | Kernel/init log is always shown, no way to hide it behind splash |
| 5 | Boot is slow (4 barrier syncs before SDDM) | 00→02→04→06→08 target chain; each waits for all services |
| 6 | No progress indicator | No animated spinner or progress bar during boot |
| 7 | No bootloader branding | UEFI bootloader shows text mode selection menu only |
| 8 | vesad doesn't release FB on DRM handoff | Bootloader FB stays mapped, wasting ~8MB memory |
| 9 | `29_activate_console` is a mess | Overridden to no-op in legacy-base, then overridden again in mini. 200ms sleep hack. |
| 10 | fbcond gives up after 4 handoff retries | If DRM is slow (firmware load), console silently stops |
| 11 | Legacy virtio-gpud/ihdgd may conflict | Could race with redox-drm for display scheme |
### Init Service Order (Current)
```
INITFS STAGE:
00_runtime.target → 10_inputd → 20_vesad → 20_fbbootlogd → 20_fbcond
→ 40_drivers.target → 50_rootfs → 90_initfs.target → switch_root
ROOTFS STAGE:
00_base.target → 02_early_hw.target → 04_drivers.target → 06_services.target
→ 08_userland.target → 29_activate_console → 30_console (getty 2) → login
For redbear-full:
Same + 12_sddm → kwin_wayland → KDE Plasma
```
---
## 2. Phased Implementation Plan
### PHASE 1 — Branding Infrastructure
**Goal:** Single source of truth for Red Bear visual assets with deterministic conversion.
**Effort:** 1–4 hours
**Files:**
| Path | Type | Purpose |
|---|---|---|
| `local/Assets/scripts/render-assets.sh` | script | PNG → BMP/RAW conversion via `imagemagick` (host-side) |
| `local/Assets/MANIFEST.sha256` | text | Deterministic checksums for all generated assets |
| `local/recipes/system/redbear-assets/recipe.toml` | recipe (Rule 1) | Stages assets to `/usr/share/redbear/assets/` |
| `local/sources/redbear-assets/` | source (Rule 1) | Trivial install crate |
| `local/docs/BOOT-BRANDING-SPEC.md` | doc | Resolution policy, color profile, animation budget |
**Generated assets (from existing PNGs):**
| Asset | Format | Resolution | Consumer |
|---|---|---|---|
| `bootlogo-1080p.bmp` | 32-bit BGRA BMP | 1920×1080 | Bootloader UEFI `Blt()` |
| `bootlogo-720p.bmp` | 32-bit BGRA BMP | 1280×720 | Bootloader fallback |
| `bootlogo-tiny.bmp` | 32-bit BGRA BMP | 640×480 | VESA-only firmware |
| `splash-1080p.raw` | Raw BGRA scanout | 1920×1080 | bootanim direct mmap |
| `splash-1080p.anim.json` | JSON | n/a | Animation timeline |
**Verification:**
- `render-assets.sh` produces all assets, byte-identical across rebuilds
- `redbear-assets` recipe stages them into sysroot
---
### PHASE 2 — `redbear-bootanim`: Plymouth Equivalent
**Goal:** Rust userspace daemon that owns the framebuffer from vesad registration until greeter focus,
rendering the Red Bear brand consistently across both earlyfb and DRM.
**Effort:** 1–2 days
**Files:**
| Path | Type | Purpose |
|---|---|---|
| `local/sources/redbear-bootanim/` | source (Rule 1) | Bootanim daemon source |
| `local/sources/redbear-bootanim/src/main.rs` | Rust | Daemon entry, signal handlers |
| `local/sources/redbear-bootanim/src/surface.rs` | Rust | Surface abstraction over vesad earlyfb + DRM |
| `local/sources/redbear-bootanim/src/anim.rs` | Rust | Animation loop (logo + spinner + progress) |
| `local/sources/redbear-bootanim/src/progress.rs` | Rust | Unix datagram socket for progress updates from init |
| `local/recipes/system/redbear-bootanim/recipe.toml` | recipe (Rule 1) | Depends on redbear-assets, inputd |
| `config/redbear-bootanim.toml` | config fragment | 20_bootanim.service + 50_drm-handoff + 98_release_vesad |
**Service wiring:**
```toml
# 20_bootanim.service — runs on earlyfb, transitions to DRM
[[files]]
path = "/etc/init.d/20_bootanim.service"
data = """
[unit]
description = "Red Bear boot animation (splash)"
requires_weak = ["10_inputd.service", "20_vesad.service"]
[service]
cmd = "/usr/bin/redbear-bootanim"
args = ["--surface=vesad", "--vt=1"]
type = "simple"
respawn = false
"""
```
**Behavior:**
| State | Surface | Renders | Input |
|---|---|---|---|
| `Surface::Vesad` | mmap'd bootloader FB | Logo + spinner + progress | Pass-through to fbcond |
| `Surface::Drm` | `/scheme/drm/card0` | Same pixels (memcpy, no redraw) | Pass-through |
| `Reveal` (SIGUSR2/Esc) | Both | Translucent log overlay on splash | Log scrollback |
| `Exit` (SIGTERM) | n/a | 200ms fade to black, exit | n/a |
**Key design property:** Handoff is a memcpy, not a redraw. bootanim holds a cached `Box<[u32]>` of the last frame (~8MB). On handoff, it copies this to the DRM FB. Both surfaces end up pixel-identical — zero flicker.
**Verification:**
- `redbear-mini`: logo appears in UEFI FB, continues through init, transitions to fbbootlogd
- `redbear-full`: logo → smooth DRM handoff → SDDM fade-in (no blank gap >1 frame)
- Esc reveals log; Esc again hides it
---
### PHASE 3 — Atomic DRM Handoff (Linux `drm_aperture` Equivalent)
**Goal:** One-shot helper that orchestrates vesad → DRM transition in a single transaction.
**Effort:** 4–8 hours
**Files:**
| Path | Type | Purpose |
|---|---|---|
| `local/sources/redbear-bootanim/src/bin/handoff.rs` | Rust | Handoff orchestrator binary |
| `local/sources/redbear-bootanim/src/bin/release_fb.rs` | Rust | Sends RELEASE_EARLYFB to vesad |
**Handoff sequence (in `handoff.rs`):**
```
1. Send PREPARE_HANDOFF to bootanim → bootanim flushes scanout, snapshots frame, pauses animation
2. bootanim opens /scheme/drm/card0, performs ModeSetCrtc + first present
3. bootanim returns HANDOFF_READY
4. Send RELEASE_EARLYFB to vesad → vesad munmaps bootloader FB, signals ESTALE, exits
5. Send POST_HANDOFF to bootanim → bootanim resumes animation on DRM surface exclusively
6. Send REBIND_DISPLAY drm to inputd → promotes DRM to primary, ESTALE to remaining consumers
7. Exit 0
```
**Why a separate binary:** Init can enforce ordering and timeout. If handoff hangs, init moves on — user still gets a working system (stuck splash, compositor paints over it).
**Timeout/fallback:** If `redox-drm` doesn't register within 30s, handoff helper falls back to keeping splash on vesad, shows "GPU driver did not load" overlay.
**Linux mapping:**
| Linux | Red Bear |
|---|---|
| `drm_aperture_remove_conflicting_framebuffers()` | Init via `handoff.rs` (driver doesn't do implicit aperture management) |
| `CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER` | bootanim holds firmware FB visible until handoff step 4 |
| Plymouth `show-splash` / `hide-splash` | bootanim exit + sessiond Seat transition signal |
> **See also:** `local/docs/boot-logs/REDBEAR-MINI-BOOT-PS2D-INPUTD-LOG-FIX.md`
> for the 2026-06-30 fix that added `inputd: scheme:input registered` and
> `ps2d: registered producer handle, listening on serio/0 (keyboard) and
> serio/1 (mouse)` startup log lines. The handoff binary's step 6
> (`REBIND_DISPLAY drm to inputd`) now has companion observability —
> operators can confirm `inputd` is alive before the handoff completes.
**Verification:**
- `redbear-full` QEMU: screen never black for >1 frame during handoff
- Disable redox-drm: fallback message appears, user can still log in via getty
- Kill bootanim mid-handoff: handoff helper detects and recovers
---
### PHASE 4 — Quiet Boot (Log Suppression Behind Splash)
**Goal:** Normal boot shows only splash. Kernel/init log hidden unless user presses Esc or boot fails.
**Effort:** 1 day
**Files to modify:**
| Path | Change |
|---|---|
| `local/sources/base/drivers/graphics/fbbootlogd/src/main.rs` | Add `--quiet` flag (don't open display, write to logd only) |
| `local/sources/base/drivers/graphics/fbbootlogd/src/scheme.rs` | Quiet mode: no display painting until SIGUSR2 |
| `local/sources/base/drivers/inputd/src/main.rs` | Separate "log sink" consumer role from "display" consumer |
| `config/redbear-full.toml` | fbbootlogd args `["--quiet"]` |
| `config/redbear-mini.toml` | fbbootlogd args `[]` (no quiet — text target shows log) |
| `local/docs/QUIET-BOOT-SPEC.md` | Kernel cmdline `redbear_quiet=0|1`, key bindings, failure modes |
**Reveal key:** Esc (configurable in `/etc/redbear/bootanim.toml`) → bootanim sends SIGUSR2 to fbbootlogd → fbbootlogd connects to display, paints log. Esc again → disconnects, clears overlay.
**Force-reveal conditions (always show log, no quiet):**
- Kernel panic
- `redox-drm` register timeout
- Init restart loop > 2 times
- `redbear_quiet=0` kernel cmdline
**Verification:**
- `redbear-full`: no log text during normal boot. Esc reveals, Esc hides.
- `redbear-mini`: log always visible (no quiet).
- Daemon crash during boot: log auto-reveals for 5s.
---
### PHASE 5 — Boot Speed: Flatten the Stage Graph
**Goal:** Parallelize display path with hardware enumeration. Remove the 200ms sleep hack.
**Effort:** 1–2 days
**Current chain (4 barrier syncs):**
```
00_base → 02_early_hw → 04_drivers → 06_services → 08_userland → SDDM
```
**Proposed chain (parallel branches):**
```
00_base.target (10_inputd is the ONLY hard dep)
├─ [branch A — display] [branch B — hardware]
│ 10_bootanim 50_rootfs
│ 20_vesad 02_early_hw.target
│ 20_fbbootlogd 04_drivers.target
│ 20_fbcond redox-drm, xhcid, e1000d, ...
│ 06_services.target
│ dbus, sessiond, dhcpd
│
└──────────────┬───────────────────┘
│
08_userland.target
12_sddm (requires 50_drm-handoff, not 04_drivers.target)
29_activate_console (no sleep — waits on handoff FD)
30_console (getty 2)
```
**Key changes:**
- Display services and driver services run in parallel
- `29_activate_console` uses FD-barrier instead of `sleep 0.2` (the FD-handoff pattern from existing pcid patches)
- SDDM requires `50_drm-handoff.service`, not `04_drivers.target`
- fbcond retry limit removed — handoff helper retries DRM internally with exponential backoff (30s budget)
**Benchmark targets:**
| Metric | QEMU target | Bare-metal target |
|---|---|---|
| kernel_entry → bootanim started | < 300ms | < 200ms |
| bootanim → SDDM visible | < 2.0s | < 4.0s |
| kernel_entry → SDDM painted | < 5.0s | < 7.0s |
| Regression threshold | >10% fails CI | >10% fails CI |
**Verification:**
- `measure-boot-stages.sh` produces CSV of stage timestamps
- QEMU video recording: splash from start to SDDM, no black gap
- `redbear-mini` unchanged (speedup is redbear-full specific)
---
### PHASE 6 — Bootloader Branding & Live Progress
**Goal:** Red Bear logo visible from UEFI handoff. Branded boot menu with auto-boot countdown.
**Effort:** 1–2 days
**Files to add/modify:**
| Path | Change |
|---|---|
| `local/sources/bootloader/src/os/uefi/boot_logo.rs` | New module: `Blt()` bootlogo BMP at native resolution |
| `local/sources/bootloader/src/os/uefi/display.rs` | Extend Output to support `Blt()` with 32-bit BGRA |
| `local/sources/bootloader/src/os/uefi/video_mode.rs` | Prefer largest available mode, paint bootlogo |
| `local/sources/bootloader/src/main.rs` | Add `--quiet` (default on), `--menu-timeout=3` config |
| `local/sources/bootloader/mk/uefi.mk` | Embed BMPs at compile time via `include_bytes!` |
| `recipes/core/bootloader/recipe.toml` | Add redbear-assets as dependency |
| `local/docs/BOOTLOADER-BRANDING-SPEC.md` | Menu layout, timeout, key bindings, text fallback |
**Bootloader progress bar:**
- Logo + thin progress bar at bottom (0% at start)
- Bar fills to 10% when kernel is read from disk
- Bar fills to 100% when kernel entry is reached
- Same logo persists through kernel → init transition (no visible gap)
**Fallback:** If UEFI GOP doesn't support `Blt()`, bootloader falls back to text mode. Splash from Phase 2 still works.
**Verification:**
- `redbear-full` ISO in QEMU: red bear logo in UEFI FB, 3s menu, smooth transition to kernel FB
- Bare metal AMD + Intel: same behavior
- Firmware without Blt(): text fallback works
---
### PHASE 7 — Early Graphical Greeter
**Goal:** Something graphical appears before full SDDM/KWin is ready (~2s splash → ~3s minimal greeter).
**Effort:** 1–2 days
**Files:**
| Path | Type | Purpose |
|---|---|---|
| `local/recipes/wayland/redbear-compositor/source/src/bin/mini.rs` | Rust | Minimal Wayland greeter (user selector on black bg) |
| `config/redbear-greeter-services.toml` | config | `11_mini-greeter.service` between handoff and SDDM |
**The mini greeter:**
- Tiny Wayland compositor (few hundred lines Rust)
- Shows single user selector per configured user
- Owns the `wl_display` before KWin
- On user selection: calls `org.freedesktop.login1.Manager.SwitchToUser(uid)`, exits
- Init then starts `12_sddm` which inherits the Wayland display
**Verification:**
- `redbear-full`: splash → mini greeter (~500ms) → user selection → KWin/Plasma
- Total time < 7s on QEMU
- `redbear-mini`: unchanged
---
### PHASE 8 — Clean FB Resource Management
**Goal:** vesad releases bootloader FB on handoff. Memory accounting is auditable.
**Effort:** 4–8 hours
**Files to modify:**
| Path | Change |
|---|---|
| `local/sources/base/drivers/graphics/vesad/src/main.rs` | On RELEASE_EARLYFB: munmap FB, close FD, log freed bytes, exit 0 |
| `local/sources/base/drivers/graphics/vesad/src/scheme.rs` | Track FB lifetime in `Resource` struct |
| `local/sources/base/drivers/inputd/src/main.rs` | On handoff: query vesad resource, log freed bytes, 30s kill watchdog |
| `config/redbear-bootanim.toml` | Add vesad-release-timeout watchdog service |
| `local/docs/FB-RESOURCE-LIFECYCLE.md` | Full lifecycle diagram with byte counts |
**FB lifecycle:**
```
Bootloader → vesad mmap (8MB) → redox-drm allocates DRM FB (8MB)
→ handoff: both mapped briefly (16MB) → release vesad → only DRM (8MB)
```
**Verification:**
- `/var/log/logd` shows FB byte counts through lifecycle
- Watchdog kills vesad if release hangs >30s
- `redbear-mini`: vesad stays alive (no DRM, no release)
---
## 3. Dependency Graph
```
Phase 1 (branding assets) ← everything downstream
│
Phase 2 (bootanim daemon) ← needs Phase 1 assets
│
Phase 3 (atomic handoff) ← needs Phase 2 state machine
│
Phase 4 (quiet boot) ← independent, parallelizable
│
Phase 5 (boot speed graph) ← needs Phase 3 (handoff is the barrier)
│
Phase 6 (bootloader branding) ← independent, parallelizable
│
Phase 7 (mini greeter) ← needs Phase 3 + Phase 5
│
Phase 8 (FB resource mgmt) ← needs Phase 3 (release step)
Critical path: 1 → 2 → 3 → 5 → 7
Parallelizable: 4, 6, 8
```
---
## 4. Effort Summary
| Phase | Effort | Risk | Rollback |
|---|---|---|---|
| 1. Branding assets | 1–4 h | Trivial (host-side imagemagick) | Delete recipe + config |
| 2. bootanim daemon | 1–2 d | Handoff correctness is subtle | Disable service; log/console still works |
| 3. Atomic handoff | 4–8 h | Low (thin orchestrator) | Fallback to vesad if handoff fails |
| 4. Quiet boot | 1 d | Reveal key must work pre-fbcond | Per-config opt-in; mini unchanged |
| 5. Boot speed | 1–2 d | Invasive stage graph restructure | Revert config; one git checkout |
| 6. Bootloader branding | 1–2 d | UEFI Blt() varies by firmware | Text mode fallback preserved |
| 7. Mini greeter | 1–2 d | New UI; keyboard handling | Opt-in per config; SDDM still works |
| 8. FB resource mgmt | 4–8 h | Force-killing vesad could break consumers | Disable watchdog service |
**Total: ~7–10 working days** for a single engineer to land all 8 phases.
**First visible improvement:** Phase 1 + Phase 2 (~2 days) → bootloader logo + splash on earlyfb.
**Full CachyOS-class experience:** All 8 phases.
---
## 5. Watch-Outs
1. **Bootloader `Blt()` is firmware-dependent.** Test on ≥2 bare-metal firmwares + QEMU OVMF. If GOP doesn't support `Blt()`, text fallback kicks in.
2. **Resolution mismatch on handoff.** If DRM mode differs from vesad earlyfb, bootanim resamples the cached frame (Lanczos). Worst case: Intel i915 at 1366×768 panel + 1920×1080 DRM mode.
3. **Init FD-handoff semantics** assumed by Phase 5 (`pass_fds = [3]`) must be verified in init source before restructuring the boot graph.
4. **No patches in `local/patches/`.** All changes are direct edits in `local/sources/<component>/` (Rule 1) or tracked config fragments.
5. **Actual source paths:** `local/sources/base/drivers/graphics/<daemon>/`, not `local/sources/base/src/daemon/`. Verify before editing.
6. **KWin QML gate:** If full Plasma can't boot, Phase 7's mini greeter is the graceful degradation. Working graphical session without Plasma is better than stuck boot.
7. **Legacy virtio-gpud/ihdgd conflict:** Verify `config/redbear-full.toml` excludes these. If they ship alongside redox-drm, they'll race for the display scheme.
---
## 6. Immediate Next Steps (Blocking Issues)
Before starting Phase 1, fix these existing issues that block a clean boot:
1. **Init stops at thermald** — why console services (29-31) never start. Need runtime debug output from init.
2. **`29_activate_console.service` no-op** — redbear-legacy-base.toml overrides to `cmd = "true"`. VT 2 never activated.
3. **Remove temporary debug code** from init main.rs (INIT_LOG_LEVEL=DEBUG, debug_log function).
4. **Fix `00_acpid.service` reference** — `00_driver-manager.service` references non-existent `00_acpid.service` (should be `30_acpid.service`).
local/docs/RELEASE-BUMP-WORKFLOW.md
+368
View File
@@ -0,0 +1,368 @@
# Red Bear OS — Release-Bump Workflow
**Status:** canonical operator runbook (2026-07-18)
**Audience:** operators performing a Red Bear OS release bump
**Scope:** moving version labels, fork sources, and the external desktop
stack forward across a release-branch switch.
This document is the **runbook** for the release-bump pipeline. It is the
companion to two contracts:
- `local/AGENTS.md` § "Version conventions — two categories" and
§ "No-fake-version-label rule" — the *policies* this pipeline enforces.
- `local/scripts/TOOLS.md` — the *tool inventory* (modes, flags, where each
tool runs).
If the docs disagree, AGENTS.md is the policy authority and TOOLS.md is the
behavioural contract of each script. This document is the workflow that
sits on top of both.
---
## TL;DR — the 30-second version
A Red Bear OS release bump is four steps. The first is human; the rest are
mechanical or operator-judgement.
| Step | What | Who/What | Touches source? |
|------|------|----------|-----------------|
| **1. Cut the branch** | Create the release branch on gitea (`0.3.2`) | Operator only | No |
| **2. Switch + label-sync** | `git checkout 0.3.2` → hook rewrites all version *labels* to match the branch (or run `./local/scripts/bump-release.sh` by hand) | post-checkout hook (opt-in) or operator | **No** — labels only |
| **3. Source upgrades** | `./local/scripts/bump-release.sh --with-sources --with-external` rebases eligible forks to newer upstream tags and bumps the Qt6/KF6/Plasma recipes | Operator | **Yes** — real rebases + recipe bumps |
| **4. Rebuild + stabilize** | Rebuild, fix patch fallout, regenerate lockfiles | Operator | Yes |
Nothing in steps 2–4 ever auto-commits. Every step that changes the tree
prints the exact `git add` / `git commit` sequence the operator should run.
---
## The four-step release model in detail
### Step 1 — Cut the release branch (operator only)
Per `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY", release branches are
created **by the operator only** — one per Red Bear OS release cycle. Agents
must never create branches. The branch name MUST be an anchored semver:
`0.3.1`, `0.3.2`, `0.4.0`. Non-semver names (`master`, `submodule/*`,
`recovered/*`) are never release branches and are ignored by every tool in
this pipeline.
The branch is created on the canonical gitea repo
(`https://gitea.redbearos.org/vasilito/RedBear-OS.git`) from the prior
release tip, then fetched locally:
```bash
git fetch origin
git checkout 0.3.2 # tracks origin/0.3.2
```
Nothing else happens at branch creation time. The bump machinery fires on
the *checkout*, not on the branch creation.
### Step 2 — Switch and label-sync
When the operator checks out the new release branch, two things can happen:
**A. If the post-checkout hook is installed** (opt-in; see
`local/docs/HOOKS.md`):
```bash
git checkout 0.3.2
# post-checkout-version-sync.sh fires automatically:
# - guards pass (semver branch, clean tree, no rebase in progress, …)
# - detects root Cargo.toml is at 0.3.1, not 0.3.2
# - runs: sync-versions.sh --no-regen (labels only)
# - prints the follow-up hint pointing at step 3
```
The hook does the **label-only** half of the bump:
- Cat 0 (cookbook root `Cargo.toml`): `version = "0.3.1"` → `"0.3.2"`.
- Cat 1 (in-house crates under `local/recipes/*/source/`): same.
- Cat 2 (upstream forks under `local/sources/*/`): the *suffix* moves —
`0.9.0+rb0.3.1` → `0.9.0+rb0.3.2`. The upstream base (`0.9.0`) is
preserved because no source rebase has happened yet.
The hook explicitly does NOT:
- regenerate lockfiles (`--regen` is never passed — it is a separate
explicit step in step 4),
- touch fork source content,
- commit anything,
- run on non-semver branches or dirty trees,
- block the checkout under any circumstance (always exits 0).
**B. If the hook is not installed**, run the orchestrator by hand:
```bash
./local/scripts/bump-release.sh # label sync + reports, no mutations beyond labels
# or equivalently for labels only:
./local/scripts/sync-versions.sh --no-regen
```
`bump-release.sh` (no args) delegates the label rewrite to
`sync-versions.sh --no-regen` and additionally prints a fork-upstream
report and an external-version report so the operator can see what step 3
will need to touch.
After step 2 the tree has correct labels but possibly stale sources and
lockfiles. The operator should commit the label bump:
```bash
git add -A
git commit -m "release: sync version labels to 0.3.2"
```
### Step 3 — Source upgrades
This is the expensive half. It is always operator-initiated via explicit
flags and never triggered by the hook.
#### 3a. Cat 2 fork source bumps — `--with-sources`
```bash
./local/scripts/bump-release.sh --with-sources
```
For each fork in `local/fork-upstream-map.toml`, `bump-release.sh`:
1. **Classifies** the fork by its map `mode` column (decision tree below).
2. For eligible forks (mode `snapshot` or `tracked`), `git ls-remote --tags`
the upstream URL, find the latest anchored-semver tag, and compare to
the fork's current base (the `<X.Y.Z>` before `+rb`).
3. If upstream moved: invoke `upgrade-forks.sh --to=<tag> <fork>`, which
fetches + pins `upstream_ref` to `refs/tags/<tag>`, creates a backup
branch (`rb-backup/<fork>`), resets, re-applies the Red Bear net-diff,
falls back to cherry-pick if needed, regenerates the fork lockfile, and
runs `verify-fork-functions.sh`.
4. On success: set the fork `Cargo.toml` `version = "<tag>+rb<branch>"`,
update map column 3 in place, and print the exact commit commands for
the parent repo and the `submodule/<fork>` branch.
5. On failure: report, leave map/label untouched. No partial state.
Per the **no-fake-label rule** (`local/AGENTS.md` § "No-fake-version-label
rule (STRICT)"), labels never move without matching source content. When
upstream is unchanged, only the `+rb<branch>` suffix moves (already done in
step 2). When upstream moved, the fork is really rebased first, then the
label moves to match.
#### 3b. External desktop stack — `--with-external`
```bash
./local/scripts/bump-release.sh --with-external
# or just the version report first:
./local/scripts/check-external-versions.sh
```
This drives the rewritten, map-driven `bump-graphics-recipes.sh` over the
Qt6/KF6/Plasma desktop stack and singletons (mesa, libdrm, libwayland,
sddm, …) declared in `local/external-upstream-map.toml`. Groups resolve
their version **once** (e.g. all KF6 frameworks move to the same
`frameworks/X.Y/`), symlinked recipes are deduped by realpath so qtbase is
bumped exactly once, and each recipe is validated with
`repo validate-patches <recipe>` after the bump. Results land in
`.redbear-recipe-bump/last-report.txt` with the line format:
```
recipe= old= new= patches_total= patches_pass= patches_fail= fail_details=
```
Failing patch validation is **reported, not auto-fixed** and never causes a
recipe to be removed (per the AGENTS.md ABSOLUTE RULE). Failures become
work items for step 4 (stabilization).
The two flags compose:
```bash
./local/scripts/bump-release.sh --with-sources --with-external
```
### Step 4 — Rebuild and stabilize
After sources move, the build invalidation machinery fires automatically:
- **Source fingerprints** — each recipe's content-hash cache (BLAKE3 of
build deps) detects that the fork tarball changed and forces a rebuild.
- **relibc ABI wipe** — when relibc itself moved, every downstream recipe
is rebuilt against the new `libc.a`.
- **Prefix auto-rebuild** — `build-redbear.sh` detects that relibc/kernel/
base have commits newer than `prefix/.../libc.a` and rebuilds the prefix
before any recipe build begins.
- **Preflight gates** — `build-preflight.sh` re-runs `sync-versions.sh
--check`, `verify-fork-versions.sh`, `verify-patch-content.py`, and
`verify-collision-detection.py` before the build proceeds.
Stabilization work then proceeds in the normal build-fix loop: patch
failures surfaced by step 3's report are fixed at the root cause (real
implementation, no stubs — see `local/AGENTS.md` § "STUB AND WORKAROUND
POLICY"). Lockfile regeneration is a separate explicit operator step:
```bash
./local/scripts/sync-versions.sh --regen # after labels + sources settle
```
---
## Fork decision tree (for `--with-sources`)
The fork's `mode` column in `local/fork-upstream-map.toml` decides what
`bump-release.sh --with-sources` is allowed to do. Modes currently in use:
| Mode | Meaning | Source-bump eligible? |
|------|---------|-----------------------|
| `snapshot` | Imported from an archived upstream snapshot; git history tracks a single upstream tag. (syscall, libredox, redoxfs, redox-scheme, relibc, userutils) | **Yes** — auto source-bump when upstream semver moves. |
| `tracked` | Fork follows an upstream ref (e.g. `main`); content checked but not byte-pinned. (base) | **Yes** — but `base`'s map tag is `main`, so it always resolves to suffix-only (see below). |
| `diverged` | Fork has diverged materially from upstream; no clean rebase path without operator work. (kernel, bootloader, installer) | **No** — report-only. Escape hatch: `--force-diverged`. |
Decision logic, per fork:
```
mode == snapshot or tracked:
latest_upstream = git ls-remote --tags <url> | latest anchored semver
if latest_upstream > current_base:
action = source-bump (upgrade-forks.sh --to=<latest_upstream>)
else:
action = suffix-only (already done in step 2)
mode == diverged:
action = report-only
# operator may pass --force-diverged to attempt a rebase anyway
tag == main (base special case):
action = suffix-only, always
# base member crates carry no +rb suffix and the verifier accepts that
# (see AGENTS.md § "Version conventions — two categories")
```
### Bootloader note (special case)
`bootloader` is `diverged` and additionally has **genuinely unrelated git
history** — there is no merge-base between the fork HEAD and upstream
`master`. `git merge-base HEAD upstream/master` fails. This means
`upgrade-forks.sh` cannot compute a net-diff to reapply and any attempt to
source-bump it would destroy committed work.
`bump-release.sh` NEVER attempts a bootloader source bump, even with
`--force-diverged`. It reports bootloader as `action=report-only` with the
guidance:
> Bootloader fork has no merge-base with upstream. A one-time
> `git replace --graft <fork-head> <upstream-tag>` intervention is required
> to synthesize a common ancestor before any rebase can be attempted. This
> is future work; the orchestrator never performs it.
The graft path (documented for the future, not run by any script today):
```bash
cd local/sources/bootloader
git replace --graft $(git rev-parse HEAD) <upstream-tag-sha>
# now upgrade-forks.sh can compute a merge-base and reapply the Red Bear delta
```
Until that graft lands, bootloader stays at its current upstream base and
only its `+rb<branch>` suffix moves with each release bump.
---
## Notes and invariants
### `Cargo.toml.orig` files are never touched
Every Cat 2 fork may carry a `Cargo.toml.orig` that mirrors the upstream
`Cargo.toml` byte-for-byte (Cargo leaves it behind when the fork edits the
real `Cargo.toml`). `sync-versions.sh`, `bump-release.sh`, and the
post-checkout hook all edit ONLY the live `Cargo.toml`. `Cargo.toml.orig`
is a read-only witness of upstream state and MUST stay byte-identical to
upstream so that `verify-fork-versions.sh` can compute the Red Bear delta.
### Lockfile regeneration is a separate explicit step
`sync-versions.sh --no-regen` (the default, and what the hook calls) rewrites
version labels but does not regenerate `Cargo.lock`. Lockfile regen is
opt-in via `sync-versions.sh --regen` and should be run after *both* labels
and sources have settled, because a source rebase can pull new transitive
deps that an earlier label-only regen would miss.
### Nothing auto-commits
Every step that changes the tree prints the exact commit sequence:
- **Parent repo** (`RedBear-OS` on the release branch): label bumps, map
column-3 updates, recipe tarball/blake3/rev updates.
- **`submodule/<fork>` branches**: fork source rebases.
The operator runs `git add` / `git commit` / `git push` themselves. This is
non-negotiable per `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY" and
the project's general "agents never commit unless asked" rule.
### Role of `provision-release.sh` and `REDBEAR_RELEASE`
`provision-release.sh` is the **freeze-time** tool: it provisions a new
immutable release archive from a Redox ref into `sources/redbear-<release>/`.
It is invoked explicitly and human-initiated, never by the bump pipeline.
`REDBEAR_RELEASE=<x.y.z>` is the env var that switches the build into
**sealed/release mode**: sources are extracted from the immutable archive,
online fetching is completely disabled, and local forks are ignored. In
development mode `REDBEAR_RELEASE` MUST be unset (see `.config` rules in
`AGENTS.md` § "BUILD COMMANDS"). The bump pipeline runs in development mode;
it has no behaviour in sealed mode.
---
## Cross-references
| Topic | Where |
|-------|-------|
| Version categories (Cat 1 / Cat 2), `+rb` suffix rules | `local/AGENTS.md` § "Version conventions — two categories" |
| No-fake-version-label rule (labels match source content) | `local/AGENTS.md` § "No-fake-version-label rule" |
| Latest-upstream-before-freeze rule | `local/AGENTS.md` § "Local fork dependency rule" |
| Local fork dependency rule (path deps, no version strings) | `local/AGENTS.md` § "Local fork dependency rule" |
| Branch and submodule policy (operator-only branches) | `local/AGENTS.md` § "BRANCH AND SUBMODULE POLICY" |
| Tool inventory + modes | `local/scripts/TOOLS.md` |
| Hook install / bypass / inventory | `local/docs/HOOKS.md` |
| Fork → upstream tag/mode map | `local/fork-upstream-map.toml` |
| External desktop-stack version map | `local/external-upstream-map.toml` |
| Post-checkout hook source | `local/scripts/post-checkout-version-sync.sh` |
| Installer source | `local/scripts/install-git-hooks.sh` |
| Orchestrator source | `local/scripts/bump-release.sh` |
---
## Appendix — full worked example
Bumping from `0.3.1` to `0.3.2` with the hook installed:
```bash
# Step 1 — operator creates 0.3.2 on gitea (out of band), then:
git fetch origin
git checkout 0.3.2
# ⟶ post-checkout hook fires:
# post-checkout-version-sync: branch '0.3.2' root Cargo.toml at '0.3.1' -> '0.3.2'
# post-checkout-version-sync: running sync-versions.sh --no-regen (labels only, no lockfile regen)
# ... sync output ...
# post-checkout-version-sync: label sync complete for branch '0.3.2'.
# follow-up steps (NOT run automatically): ...
git add -A && git commit -m "release: sync version labels to 0.3.2"
# Step 3 — source upgrades (operator decides)
./local/scripts/bump-release.sh --with-sources --with-external
# ⟶ per-fork decision lines, per-recipe validation report at
# .redbear-recipe-bump/last-report.txt. Prints commit commands.
# Step 4 — rebuild + stabilize
./local/scripts/sync-versions.sh --regen # lockfiles
./local/scripts/build-redbear.sh redbear-mini # build + preflight gates
# fix any patch fallout at the root cause; commit on 0.3.2 / submodule/<fork>
```
Without the hook, step 2 becomes:
```bash
git checkout 0.3.2
./local/scripts/bump-release.sh # labels + reports
git add -A && git commit -m "release: sync version labels to 0.3.2"
```
The rest is identical.
local/docs/RELIBC-AGAINST-GLIBC-ASSESSMENT.md
-207
View File
@@ -1,207 +0,0 @@
# Relibc vs GNU libc — Cross-Reference Assessment
**Date:** 2026-05-05
**Reference:** glibc 2.41 (2026-05-05 clone from sourceware.org)
**Relibc pinned:** commit 861bbb0 with Red Bear patch chain (26 patches)
---
## 1. eventfd
### glibc reference
```c
// sysdeps/unix/sysv/linux/eventfd.c (not cloned yet — syscall wrapper)
// bits/eventfd.h:
EFD_SEMAPHORE = 00000001 // octal 1
EFD_CLOEXEC = 02000000 // octal 0x80000
EFD_NONBLOCK = 00004000 // octal 0x800
```
glibc calls `INLINE_SYSCALL(eventfd2, 2, initval, flags)` — a kernel syscall. The kernel creates an anonymous file descriptor for event notification. Supports `EFD_SEMAPHORE` (semaphore-like counting), `EFD_CLOEXEC`, `EFD_NONBLOCK`.
### relibc current state (updated 2026-05-05 — S1-S4 implemented)
```rust
// src/header/sys_eventfd/mod.rs — 30 lines
// Full eventfd() implementation with EFD_SEMAPHORE/CLOEXEC/NONBLOCK.
// Opens scheme:event/eventfd/{initval}/{sem} via Sys::open.
```
**Implementation shipped** ✅
**Kernel support**: `P0-eventfd-kernel.patch` extends event scheme with eventfd path parsing ✅
### Gaps
| Gap | Severity | Detail |
|-----|----------|--------|
| No `eventfd()` in relibc | Medium | libwayland has its own inline, but relibc should be canonical |
| No `eventfd_read()`/`eventfd_write()` | Low | POSIX-adjacent convenience wrappers (glibc provides them) |
---
## 2. signalfd
### glibc reference
```c
// sysdeps/unix/sysv/linux/signalfd.c
int signalfd(int fd, const sigset_t *mask, int flags) {
return INLINE_SYSCALL(signalfd4, 4, fd, mask, __NSIG_BYTES, flags);
}
// bits/signalfd.h:
SFD_CLOEXEC = 02000000 // octal 0x80000
SFD_NONBLOCK = 00004000 // octal 0x800
```
glibc is a thin syscall wrapper. Kernel handles signal mask, fd management, and non-blocking reads returning `struct signalfd_siginfo`.
### relibc current state
```rust
// src/header/signal/signalfd.rs — 103 lines
// Full implementation: opens /scheme/event, applies CLOEXEC/NONBLOCK via fcntl,
// calls sigprocmask(SIG_BLOCK, mask), returns fd.
// signalfd4 supports modifying existing fd's flags.
```
**Flags match glibc** ✅
**signalfd_siginfo struct matches** ✅
**signalfd4 with existing fd** ✅ (fcntl-based flag modification)
### Prowess vs glibc
| Aspect | glibc | relibc |
|--------|-------|--------|
| Implementation | Syscall wrapper (5 lines) | Userspace via `/scheme/event` (100 lines) |
| Existing fd support | Kernel handles | fcntl O_CLOEXEC/O_NONBLOCK modification ✅ |
| Errno mapping | Kernel errno | Wraps in Errno, proper EINVAL/EFAULT |
| Signal blocking | Kernel auto-blocks on read | `sigprocmask(SIG_BLOCK, mask)` called explicitly ✅ |
### Gaps
| Gap | Severity | Detail |
|-----|----------|--------|
| No read path | High | Nothing reads `signalfd_siginfo` from the fd — the `/scheme/event` fd is opened but signals aren't delivered through it |
| Signal delivery unverified | High | The `sigprocmask(SIG_BLOCK)` blocks signals but there's no evidence the kernel delivers them via the event fd |
| `signalfd_siginfo` read not implemented | Critical | `struct signalfd_siginfo` is defined but never populated via read(2) |
---
## 3. Semaphores
### glibc reference
```c
// sysdeps/pthread/sem_open.c — 216 lines
// Uses:
// __shm_get_name(name) → canonical path in /dev/shm
// O_CREAT+O_EXCL path: creates temp file, writes semaphore header, ftruncate to sizeof(sem_t), mmap
// Non-create path: open existing, __sem_check_add_mapping(name, fd) → reuse or mmap
// pthread_setcancelstate(PTHREAD_CANCEL_DISABLE) — cancellation-safe
// va_arg for mode_t when O_CREAT
```
glibc uses a sophisticated named semaphore implementation:
1. **Name canonicalization**: `__shm_get_name` transforms `/name` → `/dev/shm/sem.name`
2. **Existing mapping reuse**: `__sem_check_add_mapping` checks global list of already-mapped semaphores
3. **Atomic creation**: O_CREAT+O_EXCL with temp file, then writes header, ftruncate, mmap
4. **Cancellation safety**: `pthread_setcancelstate(PTHREAD_CANCEL_DISABLE)` around file operations
5. **Proper mode_t**: va_arg for mode when O_CREAT
6. **Reference counting**: `__sem_check_add_mapping` increments refcount, `sem_close` decrements
### relibc current state
```rust
// src/header/semaphore/mod.rs — 176 lines
// Uses: shm_open(name, O_CREAT|O_EXCL|O_RDWR, mode) → ftruncate → mmap → init
// NamedSemaphore struct with RlctSempahore (futex-based)
```
**Core mechanism works** ✅ (shm_open + mmap)
**sem_init/destroy/post/wait/trywait/timedwait/clockwait** ✅
**sem_open/close/unlink** ✅ (implemented in P3-semaphore-comprehensive.patch)
### Gaps vs glibc
| Gap | Severity | Detail |
|-----|----------|--------|
| **No name canonicalization** | ~~Medium~~ ✅ FIXED | Names now prefixed with `sem.` before `shm_open`. glibc uses `/dev/shm/sem.NAME` equivalent. |
| **No existing mapping reuse** | ~~High~~ ✅ FIXED | Global `BTreeMap<String, NamedSemEntry>` with `AtomicUsize` refcount. `sem_open` reuses existing mappings, increments refcount. |
| **No refcounting** | ~~High~~ ✅ FIXED | `sem_close` decrements `AtomicUsize`, munmaps only when zero. |
| **No cancellation safety** | Low | No `pthread_setcancelstate` around file ops |
| **va_list not parsed** | Medium | `sem_open` hardcodes `value=0` when O_CREAT, ignoring mode and initial value from varargs |
| **No `__sem_check_add_mapping` equivalent** | High | Opens named sem every time instead of reusing existing mapping |
| **No O_NOFOLLOW** | Low | glibc uses `O_NOFOLLOW` for security |
---
## 4. Cross-Cutting Gaps
### Error Handling
| Area | glibc | relibc |
|------|-------|--------|
| errno thread-safety | TLS errno via kernel | `Cell<c_int>` per platform ✅ |
| errno after close | Preserved (close may overwrite) | `let _ = Sys::close(fd)` — ignores errors ✅ |
| EINTR | Handled in syscall wrappers | `Semaphore::wait` returns `Result<(), c_int>`. sem_wait/timedwait loop on EINTR ✅ |
| `sem_wait` | AS-safe (futex wait, EINTR) | EINTR retry loop ✅ |
| sem_open refcount | Mutex-protected global list | `BTreeMap<String, NamedSemEntry>` with `AtomicUsize` ✅ |
| sem_close/sem_unlink | Mutex-protected | `Mutex<Option<BTreeMap<...>>>` protects registry ✅ |
| signalfd mask | Per-process (kernel) | Per-call sigprocmask ✅ |
---
## 5. Priority Improvement Plan
### Phase S1: Critical Correctness (1-2 weeks)
1. **sem_open refcounting** — Add global `HashMap<String, (Arc<NamedSemaphore>, AtomicUsize)>` to reuse existing mappings. `sem_close` decrements refcount, munmaps only when zero.
2. **Eventfd implementation** — Implement `eventfd()` via `/scheme/event/eventfd/` using the existing scheme mechanism. Remove libwayland's inline copy.
### Phase S2: Completeness (2-3 weeks)
3. **signalfd read path** — Implement read(2) → `signalfd_siginfo` struct population. The `/scheme/event` fd must deliver signal info formatted as `signalfd_siginfo`.
4. **sem_open va_list** — Parse `mode_t` and `value` from varargs when O_CREAT. Requires `crate::header::stdarg` or manual stack walking.
5. **sem_open name canonicalization** — Prefix names with `/scheme/shm/sem.` for namespace isolation.
### Phase S3: Robustness (3-4 weeks)
6. **EINTR handling** — Wrap futex waits to retry on `EINTR`.
7. **sem_open cancellation safety** — Add `pthread_setcancelstate` around file ops (if pthread cancellation is supported).
8. **eventfd semaphore mode** — Implement `EFD_SEMAPHORE` counting semantics (decrements on read, blocks at 0).
### Phase S4: POSIX Conformance (2-3 weeks)
9. **eventfd_read/eventfd_write** — Convenience wrappers.
10. **sem_open existing-semaphore reopening** — Handle `(oflag & O_CREAT) == 0` path (open existing without O_EXCL).
11. **Signalfd signal delivery verification** — Runtime tests proving signals arrive via signalfd.
---
## 6. Eventfd Kernel Requirement
Unlike signalfd and sem_open which can be implemented in userspace via existing schemes (`/scheme/event`, `shm_open`), eventfd currently relies on libwayland's inline implementation that opens `/scheme/event/eventfd/`. A canonical relibc implementation should use the same path.
The `/scheme/event` kernel scheme needs:
- Support for `eventfd` sub-path
- EFD_SEMAPHORE counting semantics in kernel
- Non-blocking reads returning `u64` count
This is a **kernel change** and should be tracked separately from relibc.
---
## 7. Summary
| Component | relibc Status | Matches glibc | Critical Gaps |
|-----------|--------------|---------------|---------------|
| eventfd | Full implementation | Constants ✅ | eventfd() implemented via /scheme/event/eventfd/ ✅ |
| signalfd | 103-line impl | Flags ✅, struct ✅, signalfd4 ✅ | Read path needs kernel signal delivery ⚠️ |
| sem_open | 226-line impl with refcount | Core works ✅, shm+mmap ✅ | va_list ✅, refcounting ✅, mapping reuse ✅ |
| sem_close | Refcounted munmap | Semantics correct ✅ | Atomic refcount decrement ✅ |
| sem_wait/post | Futex-based, EINTR retry | Works ✅ | EINTR loop ✅, errno returned on other errors |
**Total estimated effort: 8-12 weeks for all gaps.**
**Critical path: eventfd kernal + signalfd read + sem_open refcounting (Phase S1).**
local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md
-456
View File
@@ -1,456 +0,0 @@
# Red Bear OS /scheme/ Namespace Population Plan
**Version**: 1.0 (2026-06-12)
**Status**: Draft — pending review
**Canonical**: `local/docs/SCHEME-NAMESPACE-POPULATION-PLAN.md`
**Blocks**: Writable rootfs on live ISO, `redoxfs` disk discovery, `ls /scheme/` in shell
**Cross-references**: Linux kobject/uevent, Fuchsia Zircon/Component Manager, seL4 CSpace, Plan 9 per-process namespace, Genode capability routing, MINIX 3 driver model
## 1. Problem Statement
`ls /scheme/` hangs or returns empty in Red Bear OS. Three root causes:
1. **initnsmgr `getdents` depends on daemons registering** — but boot ordering means some schemes
haven't registered yet when `redoxfs` calls `fs::read_dir("/scheme")` to find disk devices.
2. **No aggregator for block devices** — `redoxfs` must enumerate all `disk.*` schemes individually,
but `/scheme/disk.live` may not exist yet when the rootfs mount runs at priority 50.
3. **driver-block `getdents` returns `EOPNOTSUPP`** — individual disk schemes use legacy text-based
listing, not proper `getdents`.
The result: `redoxfs` can't discover disks, rootfs fails to mount read-write, and `/scheme/`
listing is incomplete.
## 2. Design Principles (Informed by Cross-Reference)
### 2.1 Microkernel Principle (seL4, Red Bear OS)
The kernel tracks scheme IDs (integers), not names. All name→ID mapping happens in userspace
(`initnsmgr`). This is correct per the user's explicit correction:
> "Kernel does not have to track id-name mapping! Kernel only knows about IDs. It's a microkernel
> and stuff like this must be done in userspace"
**Implication**: We never modify the kernel to "export" scheme names. The namespace is purely
a userspace construct managed by `initnsmgr`.
### 2.2 Aggregator Pattern (Linux devtmpfs + Fuchsia devcoordinator)
Linux populates `/dev` via two mechanisms:
- **devtmpfs** — kernel auto-creates basic `/dev/null`, `/dev/sda1` etc. at boot
- **udev** — userspace daemon receives uevents via netlink, applies rules, creates additional nodes
Fuchsia uses **devcoordinator** (now driver-index + device-finder):
- Drivers register devices with the driver manager
- devcoordinator exposes them via `devfs` (listable, browsable)
- Component Manager routes specific devices to components via capability declarations
Red Bear OS should follow the **aggregator** pattern: userspace daemons that discover,
enumerate, and expose device categories through listable scheme namespaces.
### 2.3 Bootstrap Ordering (Plan 9, Fuchsia)
Plan 9 bootstraps namespace incrementally:
1. Kernel boots with `#` device drivers (kernel-resident, like Red Bear's `GlobalSchemes`)
2. `boot(8)` script binds drivers into the namespace
3. `init(8)` builds the per-process namespace from `/lib/namespace`
Fuchsia bootstraps similarly:
1. Zircon boots, creates root job + resource handles
2. component_manager starts, receives boot info (device handles from ZBI)
3. driver_index enumerates drivers, binds them to devices
4. devfs provides the listable namespace
Red Bear OS boot sequence (current):
```
bootstrap → initnsmgr (initial schemes: 10 kernel globals + "proc" + "initfs")
→ init starts service targets
→ 10_lived.service (priority 10): registers "disk.live"
→ 40_drivers.target: pcid, graphics, etc.
→ 45_diskd.service (NEW): scans disk.* schemes, registers "diskd"
→ 50_rootfs.service: redoxfs uses diskd to find root device
```
### 2.4 Separation of Discovery and Access (Genode, seL4)
Genode separates:
- **Platform session** — device discovery (what hardware exists)
- **I/O session** — device access (read/write/mmio)
seL4 separates:
- **Device Untyped caps** — raw hardware access
- **Platform description** — structured description of what devices exist
In Red Bear OS terms: `diskd` provides discovery (listing), but actual block I/O goes through
the original `disk.live`/`disk.sata0` schemes directly. `diskd` returns `OpenResult::OtherScheme`
so the kernel hands the caller a raw fd to the underlying scheme — zero overhead.
## 3. Current Architecture
### 3.1 Kernel Global Schemes (10)
Registered by bootstrap in `exec.rs` → `initnsmgr::run()`:
| Scheme | GlobalSchemes Variant | Kernel Source |
|--------|-----------------------|---------------|
| debug | Debug | `scheme/debug.rs` |
| event | Event | `scheme/event.rs` |
| memory | Memory | `scheme/memory.rs` |
| pipe | Pipe | `scheme/pipe.rs` |
| serio | Serio | `scheme/serio.rs` |
| irq | Irq | `scheme/irq.rs` |
| time | Time | `scheme/time.rs` |
| sys | Sys | `scheme/sys/mod.rs` |
| proc | Proc | `scheme/proc/mod.rs` |
| acpi | Acpi | `scheme/acpi.rs` |
| dtb | Dtb | `scheme/dtb.rs` |
These are registered in the `KernelSchemes` enum (kernel/src/scheme/mod.rs:438) and
exposed to initnsmgr during bootstrap.
### 3.2 initnsmgr Namespace Manager
Located at `local/sources/base/bootstrap/src/initnsmgr.rs`.
Key structures:
```rust
struct Namespace {
schemes: HashMap<String, Arc<FdGuard>>, // name → fd
}
```
- `open("")` → `Handle::List` (directory listing handle)
- `getdents(Handle::List)` → iterates `schemes` HashMap, returns `DirEntry` for each name
- Daemons register via `NsDup::IssueRegister` + sendfd mechanism
- Bootstrap passes initial set: kernel globals + "proc" + "initfs"
### 3.3 Userspace Scheme Registration
Daemons register via:
1. `Socket::create()` → creates scheme socket
2. `NsDup::IssueRegister` → tells initnsmgr the scheme name
3. `sendfd` → sends the scheme socket fd to initnsmgr
4. initnsmgr stores in `schemes: HashMap<String, Arc<FdGuard>>`
### 3.4 Current Userspace Schemes (at boot)
| Scheme | Daemon | Priority | Source |
|--------|--------|----------|--------|
| initfs | bootstrap | 0 | bootstrap exec.rs |
| proc | kernel | 0 | GlobalSchemes |
| disk.live | lived | 10 | init.initfs.d/10_lived.service |
| disk.sata0 | ahcid | 40 | pcid-spawner |
| disk.virtio0 | virtio-blkd | 40 | pcid-spawner |
| display | vesad | 20 | init.initfs.d/20_vesad.service |
| drm | redox-drm | 30 | init.initfs.d/30_graphics.service |
| net | e1000d / virtio-netd | 40 | pcid-spawner |
| orbital | orbital | rootfs | (legacy, not used in redbear-full) |
### 3.5 The Root Cause Chain
```
redoxfs mount (priority 50)
→ fs::read_dir("/scheme") → initnsmgr getdents
→ iterates schemes HashMap → finds "disk.live" (registered at priority 10)
→ is_scheme_category("disk") → true
→ Fd::open("/scheme/disk.live") → reads text listing
→ finds block device → opens /scheme/disk.live/0 → reads UUID
→ UUID matches → mounts as rootfs
```
**The bug**: `redoxfs` retries 20×200ms = 4 seconds. If disk discovery takes longer than
4 seconds (e.g., AHCI probe on real hardware), rootfs mount fails → read-only fallback.
**The fix**: `diskd` aggregator + longer timeout + event-driven notification.
## 4. Solution Architecture
### 4.1 Component Overview
```
┌─────────────────────────────────────────────────────────┐
│ /scheme/ namespace │
│ (initnsmgr) │
│ │
│ Kernel globals: │
│ debug, event, memory, pipe, serio, irq, │
│ time, sys, proc, acpi, dtb │
│ │
│ Boot schemes (initfs): │
│ initfs, disk.live, display │
│ │
│ Aggregators: │
│ diskd ← /scheme/diskd lists ALL block devices │
│ │
│ Hardware daemons (post-drivers.target): │
│ disk.sata0..7 (ahcid) │
│ disk.virtio0..7 (virtio-blkd) │
│ disk.nvme0..7 (nvmed) │
│ disk.usb0..7 (usbscsid) │
│ disk.ide0..3 (ideid) │
│ net (e1000d, virtio-netd, ixgbed, rt8169d) │
│ drm (redox-drm) │
│ │
│ System daemons (post-rootfs): │
│ audio (audiod) │
│ firmware (firmware-loader) │
│ input (evdevd) │
│ udev (udev-shim) │
│ ... │
└─────────────────────────────────────────────────────────┘
```
### 4.2 diskd — Disk Aggregator Daemon (IMPLEMENTED)
**Location**: `local/recipes/system/diskd/`
**Scheme name**: `diskd`
**Binary**: `/usr/bin/diskd`
**Status**: Code complete, cargo check/clippy/fmt clean
**How it works**:
1. At boot (priority 45), diskd starts
2. Probes `/scheme/disk.live`, `/scheme/disk.sata0`..7, `/scheme/disk.virtio0`..7, etc.
3. For each found scheme, reads its text listing to discover devices and partitions
4. Registers scheme `diskd` with initnsmgr
5. `getdents` on `diskd:` returns real `DirEntry` with `DirentKind::BlockDev`
6. `open("0")` or `open("0p1")` opens the underlying scheme and returns `OtherScheme`
(zero-copy — caller talks directly to the block device)
**Why this solves the root cause**:
- `redoxfs` currently must enumerate ALL `/scheme/disk.*` individually — 50+ `Fd::open` calls
- With `diskd`, `redoxfs` does ONE `read_dir("/scheme/diskd")` to get all block devices
- diskd already did the probing and enumeration
- Even if AHCI hasn't registered yet, diskd's retry logic handles late registration
- `redoxfs` timeout only needs to wait for `diskd` to be ready, not all individual schemes
### 4.3 Changes Required to Existing Components
#### 4.3.1 redoxfs — Use diskd for disk discovery
**File**: `local/sources/redoxfs/src/bin/mount.rs` (function `filesystem_by_uuid`)
**Current behavior**:
```rust
// Line 224: fs::read_dir("/scheme") → filter is_scheme_category("disk")
// For each disk.* scheme: open, read listing, find block devices, check UUID
// Retry 20×200ms = 4 seconds total
```
**New behavior** (two-path approach):
```rust
fn filesystem_by_uuid(uuid: &[u8; 16]) -> Option<File> {
// Path A: Try diskd aggregator first (fast, single enumeration)
if let Some(f) = try_diskd_uuid(uuid) {
return Some(f);
}
// Path B: Fall back to legacy per-scheme enumeration
// (for backwards compat and environments without diskd)
try_legacy_uuid_search(uuid)
}
fn try_diskd_uuid(uuid: &[u8; 16]) -> Option<File> {
// Wait for diskd scheme to appear
for _ in 0..50 { // 50 × 200ms = 10 seconds
if let Ok(dir) = fs::read_dir("/scheme/diskd") {
for entry in dir {
let entry = entry.ok()?;
let name = entry.file_name().to_string_lossy().into_owned();
// Open the block device via diskd (which proxies to underlying scheme)
let path = format!("/scheme/diskd/{name}");
if let Ok(mut f) = File::open(&path) {
if check_uuid(&mut f, uuid) {
return Some(f);
}
}
}
}
thread::sleep(Duration::from_millis(200));
}
None
}
```
#### 4.3.2 init.initfs.d — Add diskd service
**New file**: `local/sources/base/init.initfs.d/45_diskd.service`
```ini
[[service]]
name = "diskd"
command = "/usr/bin/diskd"
priority = 45
requires = ["lived"]
```
This ensures diskd starts after lived (which provides disk.live at priority 10) and before
rootfs mount (priority 50).
#### 4.3.3 config/redbear-mini.toml — Add diskd package
Add `diskd` to the `[packages]` section so it's included in the image.
### 4.4 /scheme/ Namespace Completeness Matrix
After all changes, `/scheme/` will expose:
| Category | Scheme Name | Provider | getdents | Notes |
|----------|-------------|----------|----------|-------|
| **Kernel globals** | | | | |
| Debug | `debug` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/debug.rs |
| Event | `event` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/event.rs |
| Memory | `memory` | kernel GlobalSchemes | EOPNOTSUPP | No sub-entries expected |
| Pipe | `pipe` | kernel GlobalSchemes | EOPNOTSUPP | Anonymous, no listing |
| Serio | `serio` | kernel GlobalSchemes | ✅ real DirEntry | kernel/src/scheme/serio.rs |
| IRQ | `irq` | kernel GlobalSchemes | ✅ real DirEntry | cpu-XX entries |
| Time | `time` | kernel GlobalSchemes | ✅ real DirEntry | CLOCK_* entries |
| Sys | `sys` | kernel GlobalSchemes | ✅ real DirEntry | scheme:/scp/ sub-entries |
| Proc | `proc` | kernel GlobalSchemes | ✅ real DirEntry | pid entries |
| ACPI | `acpi` | kernel GlobalSchemes | ✅ real DirEntry | rxsdt, kstop |
| DTB | `dtb` | kernel GlobalSchemes | EOPNOTSUPP | Single blob |
| **Bootstrap** | | | | |
| InitFS | `initfs` | bootstrap | ✅ real DirEntry | initramfs contents |
| **Storage** | | | | |
| Live disk | `disk.live` | lived | ✅ text listing | virtio/ahci backend |
| SATA disk | `disk.sata0..7` | ahcid | ✅ text listing | per-disk scheme |
| VirtIO disk | `disk.virtio0..7` | virtio-blkd | ✅ text listing | per-disk scheme |
| NVMe disk | `disk.nvme0..7` | nvmed | ✅ text listing | per-disk scheme |
| USB disk | `disk.usb0..7` | usbscsid | ✅ text listing | per-disk scheme |
| IDE disk | `disk.ide0..3` | ideid | ✅ text listing | per-disk scheme |
| **Aggregators** | | | | |
| Disk aggregator | `diskd` | diskd | ✅ real DirEntry BlockDev | THIS PLAN |
| **Display** | | | | |
| Framebuffer | `display` | vesad | EOPNOTSUPP | Legacy text listing |
| DRM/KMS | `drm` | redox-drm | ✅ real DirEntry | card0, card0-*, connectors |
| **Network** | | | | |
| Ethernet | `net` | e1000d/virtio-netd | ✅ real DirEntry | interface entries |
| **Input** | | | | |
| Input events | `input` | evdevd | ✅ real DirEntry | event0, event1, ... |
| **Audio** | | | | |
| Audio | `audio` | audiod | ✅ text listing | Audio streams |
| **System** | | | | |
| Firmware | `firmware` | firmware-loader | ✅ real DirEntry | GPU/device blobs |
| Udev | `udev` | udev-shim | ✅ real DirEntry | Linux-compatible device nodes |
### 4.5 initnsmgr getdents — Already Correct
The `initnsmgr` `getdents` implementation at line 402-439 of `initnsmgr.rs` iterates
`schemes: HashMap<String, Arc<FdGuard>>` and emits a `DirEntry` for each registered scheme.
This is already correct — it will list any scheme that has been registered, including `diskd`.
**The `/scheme/` listing issue was NOT a getdents bug** — it was a timing issue:
- Daemons hadn't registered yet when `fs::read_dir("/scheme")` was called
- The fix is proper boot ordering (diskd at priority 45) and the diskd aggregator
## 5. Future Enhancements (Beyond Current Scope)
### 5.1 Event-Driven Discovery (uevent Equivalent)
Currently `diskd` probes statically at startup. For hotplug (USB drives, PCIe hot-add):
- **pcid** sends a `uevent`-like notification when a new PCI device appears
- **diskd** listens for these notifications and re-scans
- Alternative: inotify-like watch on `/scheme/` (would need kernel support)
This mirrors Linux's `uevent` netlink broadcast → `udev` listener pattern.
### 5.2 devfs-Style Aggregation
A future `devfsd` could provide Linux-compatible `/dev` paths:
```
/scheme/devfs/sda → /scheme/diskd/0
/scheme/devfs/sda1 → /scheme/diskd/0p1
/scheme/devfs/null → /scheme/debug (write sink)
/scheme/devfs/zero → /scheme/memory (zero-filled read)
/scheme/devfs/random → /scheme/entropy
/scheme/devfs/tty0 → /scheme/display.0
/scheme/devfs/input/event0 → /scheme/input/event0
```
This would be the Fuchsia devcoordinator equivalent — a unified, Linux-compatible
device namespace. The `udev-shim` already provides parts of this.
### 5.3 Per-Process Namespace (Plan 9 Style)
Plan 9's `bind` and `mount` allow per-process namespace customization. Red Bear OS's
`setrens` syscall provides a basic version (switch namespace fd). Future enhancement:
- Per-container namespaces (for `contain` and future container runtime)
- Namespace inheritance rules (like Fuchsia's `.cml` capability routing)
- `chroot`-like namespace restriction for sandboxed applications
### 5.4 Capability-Based Access (seL4 Style)
seL4 uses CSpace (capability spaces) for device access. Each process has a CSpace that
contains only the capabilities it should have access to. Red Bear OS could evolve toward
this model:
- `initnsmgr` tracks which schemes each process can access
- `open("/scheme/net")` checks the caller's capability set
- `setrens` evolves from "switch namespace" to "restrict to capability subset"
This would require kernel changes (per-process scheme allowlists), which is beyond current
scope but worth keeping in mind for security hardening.
## 6. Implementation Plan
### Phase 1 — Immediate Fix (This Session)
| Step | Action | Files | Status |
|------|--------|-------|--------|
| 1 | diskd daemon implementation | `local/recipes/system/diskd/` | ✅ Done |
| 2 | Add diskd init service | `local/sources/base/init.initfs.d/45_diskd.service` | Pending |
| 3 | Add diskd to config | `config/redbear-mini.toml` | Pending |
| 4 | Modify redoxfs to use diskd | `local/sources/redoxfs/src/bin/mount.rs` | Pending |
| 5 | Commit uncommitted changes | driver-manager, config | Pending |
| 6 | Remove pcid debug logging | `local/sources/base/drivers/pcid/src/cfg_access/fallback.rs` | Pending |
| 7 | Make C++ header fix durable | `mk/prefix.mk` | Pending |
| 8 | Build and test ISO | `./local/scripts/build-redbear.sh redbear-mini` | Pending |
| 9 | Boot test in QEMU | `scripts/run_mini1.sh` | Pending |
### Phase 2 — Hotplug Support (Future)
| Step | Action | Dependencies |
|------|--------|--------------|
| 1 | pcid uevent notification | pcid-spawner enhancement |
| 2 | diskd dynamic re-scan | uevent listener |
| 3 | devfsd Linux-compatible /dev | udev-shim + diskd integration |
### Phase 3 — Namespace Security (Future)
| Step | Action | Dependencies |
|------|--------|--------------|
| 1 | Per-process scheme allowlist | kernel scheme access control |
| 2 | Container namespace isolation | contain enhancement |
| 3 | Capability routing | initnsmgr capability model |
## 7. Cross-Reference Summary
| System | Mechanism | Red Bear Equivalent | Status |
|--------|-----------|---------------------|--------|
| **Linux** | kobject/uevent → udev → /dev | pcid → diskd → /scheme/diskd | Phase 1 |
| **Fuchsia** | devcoordinator → devfs | initnsmgr → diskd | Phase 1 |
| **seL4** | CSpace capabilities | setrens (basic) | Phase 3 |
| **Plan 9** | bind/mount per-process | setrens (basic) | Phase 3 |
| **Genode** | Platform session | redox-driver-sys | Existing |
| **MINIX 3** | driver announce → devfs | daemon register → initnsmgr | Existing |
## 8. Risk Assessment
| Risk | Mitigation |
|------|------------|
| diskd probe takes too long on real hardware | Increase retry count (50×200ms = 10s), add event-driven re-scan |
| diskd crashes and disk namespace disappears | init service auto-restart (`restart = true` in service file) |
| redoxfs legacy path broken by diskd changes | Two-path approach: try diskd first, fall back to legacy |
| Boot ordering regression (diskd starts before lived) | Explicit `requires = ["lived"]` in service file |
| diskd returns stale device list after hotplug | Phase 2: event-driven re-scan; Phase 1: manual re-trigger via signal |
## 9. Acceptance Criteria
1. `ls /scheme/` in shell shows all registered schemes (no hang, no empty)
2. `ls /scheme/diskd/` shows all block devices discovered by diskd
3. `redoxfs` mounts rootfs read-write via diskd path
4. `/tmp` is writable by non-root users
5. Boot completes to login prompt with zero warnings
6. QEMU boot test passes: `scripts/run_mini1.sh` reaches login prompt
7. `./local/scripts/build-redbear.sh redbear-mini` produces working ISO
local/docs/SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md
+18 -18
View File
@@ -12,11 +12,11 @@ versioning, upstream sync). It delegates subsystem-specific detail to specialize
| Plan Document | Covers | When to Consult |
|---|---|---|
| `IMPROVEMENT-PLAN.md` | USB/Wi-Fi/Bluetooth code quality audit findings (P0–P3) | USB, Wi-Fi, BT quality remediation |
| `IMPLEMENTATION-MASTER-PLAN.md` | Driver/subsystem feature gaps (storage, audio, input, CPU, virtio) | Driver feature implementation |
| `archived/IMPROVEMENT-PLAN.md` | USB/Wi-Fi/Bluetooth code quality audit findings (P0–P3) | USB, Wi-Fi, BT quality remediation |
| `archived/IMPLEMENTATION-MASTER-PLAN.md` | Driver/subsystem feature gaps (storage, audio, input, CPU, virtio) | Driver feature implementation |
| `CONSOLE-TO-KDE-DESKTOP-PLAN.md` | Desktop/KDE path from console to hardware-accelerated Plasma | Wayland, Mesa, KWin, SDDM |
| `UPSTREAM-SYNC-PROCEDURE.md` | Per-component sync procedure for local forks | Executing individual fork syncs |
| `STUBS-FIX-PROGRESS.md` | Stub→real-code rewrite tracking | Replacing stubs with real implementations |
| `archived/STUBS-FIX-PROGRESS.md` | Stub→real-code rewrite tracking | Replacing stubs with real implementations |
| `BUILD-SYSTEM-IMPROVEMENTS.md` | Build system hardening, collision detection, manifests | Build system changes |
| `ACPI-IMPROVEMENT-PLAN.md` | ACPI sleep, thermal, EC, power | ACPI improvements |
| `IRQ-AND-LOWLEVEL-CONTROLLERS-ENHANCEMENT-PLAN.md` | PCI IRQ, MSI-X, IOMMU, controllers | IRQ/PCI quality |
@@ -446,11 +446,11 @@ git commit -m "submodule: sync <component> to upstream HEAD"
### 3.4 Integrate USB Quirk Enforcement
**Context:** The existing `IMPROVEMENT-PLAN.md` Section 3.3 identifies 49/50 xHCI quirks declared but not enforced at runtime. This is a critical gap for supporting real hardware.
**Context:** The existing `archived/IMPROVEMENT-PLAN.md` Section 3.3 identifies 49/50 xHCI quirks declared but not enforced at runtime. This is a critical gap for supporting real hardware.
**File:** `local/sources/base/drivers/usb/xhcid/src/xhci/quirks.rs`
**Priority enforcement gaps (from IMPROVEMENT-PLAN.md):**
**Priority enforcement gaps (from archived/IMPROVEMENT-PLAN.md):**
| Quirk | Affected HW | Action |
|-------|-------------|--------|
@@ -466,7 +466,7 @@ if (pdev->vendor == PCI_VENDOR_ID_ASMEDIA && pdev->device == 0x1042)
xhci->quirks |= XHCI_ASMEDIA_MODIFY_FLOWCONTROL;
```
**See also:** `IMPROVEMENT-PLAN.md` § 3.3 for the complete quirk enforcement gap analysis.
**See also:** `archived/IMPROVEMENT-PLAN.md` § 3.3 for the complete quirk enforcement gap analysis.
**Estimated time:** 4–8 hours
**Dependencies:** Phase 1.5, Phase 3.1.
@@ -544,7 +544,7 @@ syscalls needed by modern software. Input device handling is comprehensive.
### 4.3 Stub Replacement: Identify and Replace Remaining Stubs
**Context:** The project has a zero-tolerance stub policy (`local/AGENTS.md` § STUB AND WORKAROUND POLICY). The `STUBS-FIX-PROGRESS.md` tracks the stub→real-code rewrite campaign. Audit the remaining stubs and prioritize fixes.
**Context:** The project has a zero-tolerance stub policy (`local/AGENTS.md` § STUB AND WORKAROUND POLICY). The `archived/STUBS-FIX-PROGRESS.md` tracks the stub→real-code rewrite campaign. Audit the remaining stubs and prioritize fixes.
**Action:**
1. Run a workspace-wide grep for stub patterns:
@@ -560,15 +560,15 @@ syscalls needed by modern software. Input device handling is comprehensive.
| Stub Pattern | Correct Fix |
|---|---|
| `unimplemented!("event ring growth")` | Implement `grow_event_ring()` per `IMPROVEMENT-PLAN.md` § 3.1 |
| `todo!("BOS descriptor")` | Un-comment `fetch_bos_desc()` per `IMPROVEMENT-PLAN.md` § 3.2 |
| `unimplemented!("event ring growth")` | Implement `grow_event_ring()` per `archived/IMPROVEMENT-PLAN.md` § 3.1 |
| `todo!("BOS descriptor")` | Un-comment `fetch_bos_desc()` per `archived/IMPROVEMENT-PLAN.md` § 3.2 |
| `return Err(ENOSYS)` for a known syscall | Implement the syscall in kernel or relibc |
| `#![allow(warnings)]` in xhcid | Fix the underlying warnings, then remove |
| `rate_idx = 0` hardcoded | Implement Minstrel rate scaling per `IMPROVEMENT-PLAN.md` § 6.3 |
| `rate_idx = 0` hardcoded | Implement Minstrel rate scaling per `archived/IMPROVEMENT-PLAN.md` § 6.3 |
**See also:**
- `IMPROVEMENT-PLAN.md` — 35 items covering USB/WiFi/BT stubs
- `STUBS-FIX-PROGRESS.md` — ~517 TODO/FIXME items tracked
- `archived/IMPROVEMENT-PLAN.md` — 35 items covering USB/WiFi/BT stubs
- `archived/STUBS-FIX-PROGRESS.md` — ~517 TODO/FIXME items tracked
**Estimated time:** 16–32 hours (ongoing across Phase 4)
**Dependencies:** Phase 3.1 (synced forks provide the correct APIs to implement against).
@@ -671,10 +671,10 @@ baseline established.
**Linux reference:** `drivers/usb/storage/usb.c:1050-1100` — `usb_stor_control_thread()` with proper error recovery on device disconnect. Red Bear's usbscsid must handle the same scenario: device removed mid-transfer → error, not panic.
**See also:** `IMPROVEMENT-PLAN.md` § 2.1 (usbscsid `.unwrap()` removal) — this must be done before hardware testing.
**See also:** `archived/IMPROVEMENT-PLAN.md` § 2.1 (usbscsid `.unwrap()` removal) — this must be done before hardware testing.
**Estimated time:** 4–8 hours
**Dependencies:** Phase 5.1, `IMPROVEMENT-PLAN.md` P0 items complete.
**Dependencies:** Phase 5.1, `archived/IMPROVEMENT-PLAN.md` P0 items complete.
### 5.4 Wi-Fi and Bluetooth Maturity Assessment
@@ -695,11 +695,11 @@ baseline established.
- `drivers/net/wireless/intel/iwlwifi/mvm/fw.c:300-500` — firmware init sequence
**See also:**
- `IMPROVEMENT-PLAN.md` § 6 — Wi-Fi subsystem improvements
- `archived/IMPROVEMENT-PLAN.md` § 6 — Wi-Fi subsystem improvements
- `WIFI-IMPLEMENTATION-PLAN.md` — Wi-Fi architecture plan
**Estimated time:** 4–8 hours
**Dependencies:** Phase 5.1, `IMPROVEMENT-PLAN.md` P1-P2 Wi-Fi items.
**Dependencies:** Phase 5.1, `archived/IMPROVEMENT-PLAN.md` P1-P2 Wi-Fi items.
---
@@ -795,12 +795,12 @@ preferable to inventing heuristics.
1. **Run `sync-versions.sh --check`** — identify version drift immediately
2. **`cd local/sources/base && git status --short`** — catalog WIP changes
3. **`cargo check` on xhcid** — confirm orphan `}` issue exists
4. **Read `IMPROVEMENT-PLAN.md` P0 items** — these are the USB safety fixes that block hardware testing
4. **Read `archived/IMPROVEMENT-PLAN.md` P0 items** — these are the USB safety fixes that block hardware testing
5. **Read `UPSTREAM-SYNC-PROCEDURE.md`** — prepare for Phase 3 fork syncs
### Ongoing Discipline
- Every commit to a local fork MUST also update the parent repo's submodule pointer
- Every upstream cherry-pick MUST be tested with `cargo check` + `repo cook` before push
- Every stub fixed MUST update `STUBS-FIX-PROGRESS.md`
- Every stub fixed MUST update `archived/STUBS-FIX-PROGRESS.md`
- Every new Linux algorithm ported MUST include a comment referencing the specific Linux 7.1 file and line range
local/docs/IMPLEMENTATION-MASTER-PLAN.md → local/docs/archived/IMPLEMENTATION-MASTER-PLAN.md
View File
local/docs/IMPROVEMENT-PLAN.md → local/docs/archived/IMPROVEMENT-PLAN.md
View File
local/docs/archived/README.md
+12 -2
View File
@@ -1,6 +1,6 @@
# Archived Documentation
Last synced: 2026-07-13 — this archive inventory is dynamically maintained and
Last synced: 2026-07-18 — this archive inventory is dynamically maintained and
tracks only files that currently exist in this directory.
These documents were written during earlier phases of Red Bear OS development.
@@ -12,10 +12,15 @@ current plans. They are kept for reference only.
| Archived | Status |
|----------|--------|
| `ACPI-I2C-HID-IMPLEMENTATION-PLAN.md` | Deferred; USB HID remains the primary input path. |
| `IMPLEMENTATION-MASTER-PLAN.md` | Completion report of the 2026-07 audit sprint; superseded by `../CONSOLE-TO-KDE-DESKTOP-PLAN.md`. |
| `IMPROVEMENT-PLAN.md` | RESOLVED quality-audit plan (38/38); still referenced by `../SYSTEM-STABILITY-AND-UPSTREAM-SYNC-PLAN.md` § xHCI quirks (§3.3). |
| `INTEL-HDA-IMPLEMENTATION-PLAN.md` | Deferred; audio remains a later priority. |
| `KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` | Historical scheduler analysis kept for reference. |
| `README.md` | Archive metadata only. |
| `RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md` | relibc IPC surface plan; credential work resolved in `../KERNEL-IPC-CREDENTIAL-PLAN.md`. |
| `repo-governance.md` | Historical governance note kept for reference. |
| `SOURCE-ARCHIVAL-POLICY.md` | Absorbed into `../../AGENTS.md` § RELEASE MODEL / SOURCE-OF-TRUTH RULE. |
| `STUBS-FIX-PROGRESS.md` | Final-state log of the completed stub→real-code campaign. |
| `USB-BOOT-INPUT-PLAN.md` | Superseded by the current USB implementation planning. |
| `USB-VALIDATION-RUNBOOK-2026-07.md` | Historical validation runbook for the earlier USB phase. |
| `XHCID-DEVICE-IMPROVEMENT-PLAN.md` | Superseded by the current USB implementation planning. |
@@ -28,5 +33,10 @@ current plans. They are kept for reference only.
remain as deferred historical references.
- `KERNEL-SCHEDULER-MULTITHREAD-IMPROVEMENT-PLAN.md` and `repo-governance.md`
are retained as archive-only context.
- 2026-07-18 archival (release-bump cleanup): `IMPROVEMENT-PLAN.md`,
`IMPLEMENTATION-MASTER-PLAN.md`, `STUBS-FIX-PROGRESS.md`,
`RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md`, `SOURCE-ARCHIVAL-POLICY.md`
moved here from `local/docs/`. All inbound references were updated to the
`archived/` paths in the same pass.
## Date archived: 2026-05-03
## Date archived: 2026-05-03 (first batch); 2026-07-18 (second batch)
local/docs/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md → local/docs/archived/RELIBC-IPC-ASSESSMENT-AND-IMPROVEMENT-PLAN.md
View File
local/docs/SOURCE-ARCHIVAL-POLICY.md → local/docs/archived/SOURCE-ARCHIVAL-POLICY.md
View File
local/docs/STUBS-FIX-PROGRESS.md → local/docs/archived/STUBS-FIX-PROGRESS.md
View File
local/docs/cub-assessment-and-improvement-plan.md
-508
View File
@@ -1,508 +0,0 @@
# cub Assessment and Improvement Plan (v6.0 2026)
**Date:** 2026-06
**Status:** Assessment complete, critical fixes in progress
**Scope:** Comprehensive empirical review of cub's AUR → RBPKGBUILD → Redox recipe.toml conversion pipeline, with concrete bug fixes and a forward improvement plan.
---
## Executive Summary
cub (Red Bear OS package manager, located at `local/recipes/system/cub/source/`) is a substantial 17-module Rust workspace with a real, working AUR → RBPKGBUILD → recipe.toml conversion pipeline. The architecture is sound: a 937-line `pkgbuild` parser, a 408-line `rbpkgbuild` serializer, a 465-line `cookbook` recipe generator, plus AUR fetching, dependency mapping, sandbox, cook, storage, and resolver modules. All 70 unit tests pass and the source compiles cleanly.
However, the conversion pipeline has **8 known bugs** that would prevent any real-world Arch package from converting to a working Red Bear recipe. This assessment is based on running the pipeline against 12 representative real-world PKGBUILDs and observing the output.
**Verdict:** cub is the right tool for the job. The architecture is correct, the code is mostly right, and the bugs are surgical fixes — not architectural rewrites. The first 7 bugs are being fixed in commit `<hash>` (this PR). The 8th is deferred as a cookbook-integration concern.
---
## What cub does well
The 12-PKGBUILD assessment found that cub's conversion pipeline handles these cases correctly:
| Case | Status |
|---|---|
| Simple meson package (libevdev) | ✅ clean conversion |
| Cargo package (fd-find) | ✅ clean conversion |
| CMake package (fmt) | ✅ clean conversion |
| Configure package (libpciaccess 0.19) | ✅ clean conversion |
| Git source with `git+url#tag=` (gzip, mesa 24.3 single-source) | ✅ clean conversion |
| Split package with `pkgbase` + `package_<name>()` (openssl) | ⚠️ converts primary only, warns about split |
| Custom build template (raw shell script) | ✅ extracts build/package bodies |
| `pkgver()` function (dynamic version) | ⚠️ ignored with warning |
| `sha256sums=('SKIP')` for git sources | ✅ correct handling |
| TOML validity of generated recipe | ✅ always valid |
| Detection of Linux-only deps (libx11, libxcb, libinput, alsa) | ✅ correctly flags as unmapped |
| Linuxism detection (glibc, x11, gtk, etc.) | ✅ warns + actions_required |
The unit tests (70 in total) cover the rbpkgbuild, rbsrcinfo, sandbox, storage, recipe, resolver modules. The test coverage is solid for the serializer/parser side.
---
## The 8 bugs
### Bug 1 — `glibc` mapped to `relibc` (CRITICAL)
**Location:** `cub-lib/src/deps.rs:13`
**Severity:** Critical — affects every Linux C library consumer
**Symptom:** `depends=('glibc')` produces `[package] dependencies = ["relibc"]` in the generated recipe
**Root cause:** The dependency mapping hard-codes:
```rust
"glibc" => ("relibc".to_string(), false),
```
**Why wrong:** glibc is the Linux C library; relibc is the Redox C library, which is part of the OS itself (not a package). Putting relibc as a runtime dependency in a Redox recipe is a tautology — every Redox program already links against relibc implicitly. Worse, it suggests relibc is a separately-installable package, which it isn't.
**Fix:** Drop the mapping. Use the standard "empty string" pattern that cub already uses for other Linux-only dependencies (systemd, xorgproto, libcap, libpulse, etc.):
```rust
"glibc" => (String::new(), false),
```
The `map_dep_list` function already filters out empty mappings.
**Status:** Fixed in this commit.
### Bug 2 — `gcc-libs` mapped to `gcc` (CRITICAL)
**Location:** `cub-lib/src/deps.rs:15`
**Severity:** Critical — affects every Arch package that links libstdc++/libgcc
**Symptom:** `depends=('gcc-libs')` produces `[package] dependencies = ["gcc"]` in the generated recipe
**Root cause:** The mapping hard-codes:
```rust
"gcc-libs" => ("gcc".to_string(), false),
```
**Why wrong:** gcc-libs is the **runtime** libgcc + libstdc++ (analogous to Arch's `gcc-libs`). gcc is the **compiler**. They're entirely different artifacts. Mapping runtime C++ libraries to a compiler creates broken cross-compilation graphs (the cookbook would try to build gcc as a runtime dep).
**Fix:** Drop the mapping, same as Bug 1. relibc provides the runtime; the compiler gcc is a host-only build tool (covered by Bug 3):
```rust
"gcc-libs" => (String::new(), false),
```
**Status:** Fixed in this commit.
### Bug 3 — Build tools not prefixed with `host:` (CRITICAL)
**Location:** `cub-lib/src/deps.rs` (entire `map_dependency` function)
**Severity:** Critical — affects every package that uses standard build tools
**Symptom:** Generated recipes list `meson`, `ninja`, `cmake`, `make`, `pkg-config`, `git`, `perl`, `python`, `rust`, `cargo`, `autoconf`, `automake`, `libtool`, etc. as bare `[build] dependencies`, without the `host:` prefix
**Root cause:** The hard-coded mapping table returns bare names. The Redox cookbook convention is to mark build tools as `host:<tool>` so the cookbook knows they're host-only and not part of the cross-compiled Redox target.
The current `prefix_host_deps` function in `cookbook.rs` already handles `host:` prefixing for `dev-dependencies` (the `check` deps) but not for `dependencies` (the `makedepends`).
**Why wrong:** Without `host:`, the Redox cookbook would try to *cook* meson, ninja, and cmake as Redox packages, which is wrong — they're host tools that run on the build host, not target tools that get cross-compiled.
**Fix:** Add a `BUILD_TOOLS: &[&str]` constant containing all known build tools. In `map_dependency`, when the name matches a build tool, return `MappedDep { mapped: "host:<name>", is_exact: true }` instead of the bare name.
The set of build tools (~30 entries): `make`, `cmake`, `ninja`, `meson`, `pkg-config`, `pkgconf`, `pkgconfig`, `autoconf`, `automake`, `libtool`, `m4`, `git`, `svn`, `mercurial`, `perl`, `python`, `python2`, `python3`, `rust`, `cargo`, `rustc`, `go`, `golang`, `bison`, `flex`, `yacc`, `gperf`, `gettext`, `intltool`, `help2man`, `gengetopt`, `xmlto`, `asciidoc`, `doxygen`, `graphviz`, `swig`.
**Status:** Fixed in this commit.
### Bug 4 — AUR `git+url#tag=branch` source syntax not parsed (CRITICAL)
**Location:** `cub-lib/src/pkgbuild.rs` (the `source_from_arch` function or equivalent)
**Severity:** Critical — affects every AUR git package
**Symptom:** Generated recipes contain literal `git+https://...git#tag=mesa-24.3.0` as the source URL, which is invalid Redox cookbook format
**Root cause:** AUR PKGBUILDs use Arch-style source notation. `source_from_arch` does not strip the `git+` prefix or extract the `#tag=branch` fragment.
**Real example:** `mesa 24.3` PKGBUILD has:
```bash
source=("git+https://gitlab.freedesktop.org/mesa/mesa.git#tag=mesa-${pkgver}")
```
cub produces: `git = "git+https://gitlab.freedesktop.org/mesa/mesa.git#tag=mesa-${pkgver}"` in the recipe.
**Fix:** Modify `source_from_arch` (or wherever source URLs are normalized) to:
1. Check if the source starts with `git+` — if so, set `source_type = SourceType::Git` and strip the prefix
2. Check for a `#` fragment — if present, split off and parse the fragment. Common fragment keys:
- `tag=...` → `SourceEntry::branch` (since the Redox cookbook uses `branch` for tags)
- `branch=...` → `SourceEntry::branch`
- `commit=...` or `revision=...` → `SourceEntry::rev`
3. For git sources without a fragment, set `branch = "HEAD"` as a reasonable default
The Redox cookbook's git source format is:
```toml
[source]
git = "https://..."
branch = "v1.0" # or rev = "..."
```
**Status:** Fixed in this commit.
### Bug 5 — Multi-source PKGBUILDs not supported (HIGH)
**Location:** `cub-lib/src/cookbook.rs:80-100`
**Severity:** High — blocks mesa, ffmpeg, and many real packages
**Symptom:** Multi-source PKGBUILDs (e.g., mesa with 2 sources: git repo + llvmpipe_generated.h.tar.xz) error with:
```
Error: multiple sources not yet supported (found 2: https://..., https://...).
Use single-source packages or manually create recipe.toml
```
**Root cause:** The `generate_recipe` function explicitly errors on multi-source. The `convert_pkgbuild` function preserves all sources in the `RbPkgBuild` but `generate_recipe` only handles 1.
**Why wrong:** Many real packages have multiple sources (a primary git repo + an auxiliary file). Forcing the user to manually create the recipe for these defeats the purpose of automated conversion.
**Fix:** In `convert_pkgbuild` (in `pkgbuild.rs`), after the full source array is parsed, **truncate to the first source only** and add a warning to the `ConversionReport`:
```rust
if sources.len() > 1 {
warnings.push(format!(
"PKGBUILD has {} sources; using the first as primary ({}). Auxiliary sources dropped: {}",
sources.len(),
sources[0],
sources[1..].join(", ")
));
actions_required.push(
"review multi-source PKGBUILD and re-add auxiliary sources manually".to_string()
);
sources.truncate(1);
}
```
This way `generate_recipe` doesn't need to know about multi-source, and the user gets a clear warning + action item.
**Status:** Fixed in this commit.
### Bug 6 — `options=()` flags silently dropped (MEDIUM)
**Location:** `cub-lib/src/pkgbuild.rs` (no extraction at all)
**Severity:** Medium — silent data loss
**Symptom:** PKGBUILDs with `options=('!lto' '!emptydirs' '!strip')` lose all option flags. The generated recipe has no trace of them.
**Root cause:** The current `convert_pkgbuild` doesn't extract `options=()`.
**Why wrong:** Options like `!lto` (disable link-time optimization) and `!strip` (don't strip binaries) can be required for a package to build correctly. Silently dropping them creates recipes that fail in ways the user can't predict.
**Fix:** Extend the `CompatSection` struct in `rbpkgbuild.rs` to include an `options: Vec<String>` field. Extract `options=()` in `convert_pkgbuild` and store it. The recipe generator doesn't need to consume it — the user reads the `compat.options` field in the RBPKGBUILD and re-adds the relevant flags to their Redox recipe.
**Status:** Fixed in this commit.
### Bug 7 — `${pkgver}` literal in source URL (HIGH)
**Location:** `cub-lib/src/pkgbuild.rs` (source array not passed through `resolve_shell_vars`)
**Severity:** High — affects every AUR package that uses `${pkgver}` in its source URL
**Symptom:** Generated recipes have `tar = ".../libevdev-${pkgver}.tar.xz"` with the literal `${pkgver}` string. The Redox cookbook doesn't do shell variable substitution at source-URL time, so the resulting tar URL is invalid.
**Root cause:** The `convert_pkgbuild` function calls `resolve_shell_vars` on `pkgname`, `pkgver`, `pkgdesc`, `url` but NOT on the `source` array.
**Why wrong:** The recipe is broken as soon as it's generated. The user would have to manually fix every source URL.
**Fix:** After extracting the `source` array, apply `resolve_shell_vars` to each entry. Since `pkgver` is already resolved (it's a top-level scalar in the PKGBUILD), the substitution should be straightforward. The existing `resolve_shell_vars` function is the right tool.
**Status:** Fixed in this commit.
### Bug 8 — Recipes lack cookbook boilerplate (DEFERRED)
**Location:** `cub-lib/src/cookbook.rs` (the `custom_script` function)
**Severity:** Medium — affects every Custom-template package
**Symptom:** Custom-template recipes generate raw shell scripts without the Redox cookbook's `DYNAMIC_INIT` prelude or the `cookbook_apply_patches` helper that the Rule 2 migration recipes use.
**Root cause:** `custom_script` (in `cookbook.rs:244-282`) joins `prepare`, `build_script`, `check`, `install_script`, and install lines into a single script. It does not prepend `DYNAMIC_INIT` (which is required for any Redox cookbook shell script to find the cookbook helpers like `cookbook_meson`, `cookbook_apply_patches`, etc.).
**Why wrong:** A real Redox recipe that doesn't start with `DYNAMIC_INIT` will fail at build time because the shell helpers aren't loaded.
**Fix:** Prepend `DYNAMIC_INIT\n` to the generated `custom_script`. Add an optional `redox_compat/` shim dir or `cookbook_apply_patches "${REDBEAR_PATCHES_DIR}"` call if the package has patches (this requires the user to declare patches in the RBPKGBUILD).
**Status:** Deferred. This is a cookbook-integration concern that intersects with the project's Rule 2 patch policy. A separate task should:
1. Decide whether the cub-generated recipe's `patches` field is treated as the source-of-truth (per Rule 2) or as auxiliary patches
2. Add a `local/patches/<component>/` directory generation step to cub
3. Update the `custom_script` to include `DYNAMIC_INIT` and `cookbook_apply_patches`
---
## Test methodology
The `cub-assessment` test binary at `local/recipes/system/cub/source/cub-assessment/src/main.rs` is the integration test that exercises 12 representative real-world PKGBUILDs:
1. `libevdev` — simple meson (reference case)
2. `fd-find` — cargo
3. `libpciaccess 0.18.1` — meson
4. `fmt` — cmake
5. `wlroots-git` — git source, complex deps
6. `libpciaccess 0.19` (extra/-style) — meson + ninja
7. `ffmpeg` — configure + options
8. `mesa 24.3` — git+url + multi-source + pkgver() (used to fail pre-fix)
9. `gzip` — configure + git source + check
10. `zlib` — simple C, configure
11. `openssl` — pkgbase split package
12. `glib2` — complex deps, real-world
**Pre-fix results (before this commit):**
- 11 of 12 converted successfully
- 1 of 12 errored (mesa 24.3, multi-source)
- All recipes valid TOML
- Several recipes had wrong deps (glibc→relibc, gcc-libs→gcc)
- All git+url sources kept literal prefix
- All multi-source attempts failed
**Post-fix results (after this commit):**
- All 12 convert successfully
- mesa 24.3 produces a valid recipe with a multi-source warning
- glibc, gcc-libs, x11, alsa, libinput are correctly dropped/flagged
- Build tools are correctly `host:`-prefixed
- git+url sources parse correctly into branch/rev fields
- 70 existing unit tests still pass
---
## Test cases by category
### A. Conversion path success (verifies the core pipeline)
- ✅ `convert_pkgbuild` returns Ok for all 12 cases
- ✅ `generate_recipe` returns Ok for all 12 cases post-fix
- ✅ All generated recipes parse as valid TOML
### B. Dependency mapping correctness
- ✅ `glibc` → dropped (empty string)
- ✅ `gcc-libs` → dropped (empty string)
- ✅ `glib2` → `glib` (exact)
- ✅ `openssl` → `openssl3` (inexact, warning)
- ✅ `zlib` → `zlib` (exact)
- ✅ `meson`, `ninja`, `cmake` → `host:meson`, etc. (post-fix)
- ✅ `libinput` → dropped with action_required
- ✅ `libx11`, `libxcb`, `libxrandr` → dropped (no Redox mapping)
### C. Source URL handling
- ✅ Tar source with `${pkgver}` → resolved to actual version (post-fix)
- ✅ Git source with `git+url#tag=v1.0` → git+ stripped, branch=v1.0 (post-fix)
- ✅ Git source without fragment → branch=HEAD (post-fix)
- ✅ Multi-source → first kept, others warned (post-fix)
- ✅ `sha256sums=('SKIP')` → ignored, source_type=Git
### D. Build template detection
- ✅ meson template (cookbook_meson path)
- ✅ cmake template (cookbook_cmake path)
- ✅ cargo template (cookbook_cargo path)
- ✅ configure template (cookbook_configure path)
- ✅ Custom template (raw script path)
### E. Edge cases
- ✅ Split package (pkgbase + pkgname array + package_* functions)
- ✅ pkgver() function (ignored with warning)
- ✅ options=() (preserved in compat section, post-fix)
- ✅ check() function (extracted when allow_tests=true)
### F. Validation
- ✅ RbPkgBuild.validate() passes for all
- ✅ Generated RBSRCINFO is valid
- ✅ All arch values are "x86_64-unknown-redox"
- ✅ All licenses are preserved
---
## Forward improvement plan (post-fix)
After the 7 critical bug fixes, the conversion pipeline produces recipes that are *valid TOML with correct deps* but still need work to be *buildable in the Redox cookbook*. Here's the prioritized follow-up list:
### Tier 1: Cookbook integration (Bugs 8 + others)
1. **Custom template boilerplate** — prepend `DYNAMIC_INIT` to all `custom_script` outputs. Without this, the recipe will fail at runtime because cookbook helpers like `cookbook_meson` aren't loaded.
2. **Patch application hook** — for recipes with `patches.files` non-empty, automatically emit `cookbook_apply_patches "${REDBEAR_PATCHES_DIR}"` in the custom script (matching the Rule 2 migration pattern established in commits `1a291fbb9` and `b0f440c47`).
3. **Systematic dep audit** — the 80-entry hard-coded dep map in `deps.rs` is a maintenance burden. Migrate to a data-driven table that lives in `local/docs/dependency-mapping.md` (or a similar canonical source) and is loaded at runtime. This makes the mapping auditable and easy to extend.
4. **Per-package overrides** — sometimes a single package needs a non-standard dep. For example, `mesa` needs `llvm21` not `llvm`. Add a `compat.dependency_overrides: HashMap<String, String>` field in `RBPKGBUILD` that overrides the hard-coded mapping.
### Tier 2: Conversion quality
5. **Better Linuxism detection** — currently flags `glibc`, `x11`, `alsa`, `libpulse` as Linux-only. Many AUR packages have implicit Linux-only paths (e.g., `/proc/cpuinfo`, `/sys/class/...`, `/dev/...`) that the heuristic doesn't catch.
6. **Multi-source preservation** — when truncating to 1 source, the user loses access to auxiliary files (e.g., a gitignore or systemd unit). Generate a `redox/aux-sources.txt` next to the recipe listing the dropped sources, so the user can manually re-add them.
7. **`pkgver()` execution** — for git sources, the Arch convention is to have a `pkgver()` function that calls `git describe` to compute the version. This is impossible to execute in cub (we don't have a git clone at conversion time). Generate a static `version = "GIT"` and add a `compat.notes = "git source — version detected dynamically"` annotation.
8. **License mapping** — Arch licenses like `('GPL3' 'Apache-2.0')` are SPDX-like but not exact. Map to SPDX identifiers and warn when an Arch license doesn't have a Red Bear SPDX equivalent.
### Tier 3: UX
9. **`cub -S <pkg>` should run end-to-end** — currently the AUR → recipe → cook flow is fragmented (separate `get-aur`, `build` subcommands). The user should be able to say `cub -S mesa` and get a built Redox package.
10. **Conversion report** — the `ConversionReport` already exists but is only printed by `get-aur` and `import-aur`. The `cook` and `install` subcommands should print the report after the recipe is generated so the user can see warnings and action items.
11. **TUI integration** — the cub TUI at `cub-tui/` should show a "Convert from AUR" action that drives the full conversion and shows the report.
12. **BUR (Redox User Repo) sync** — `cub -Sy` syncs from BUR at `https://gitlab.redox-os.org/redox-os/bur.git`. This works but the BUR is sparse. A "publish" subcommand would let users contribute their converted recipes back to BUR.
### Tier 4: Infrastructure
13. **Network access policy** — AUR fetching requires network access. The project's `REPO_OFFLINE=1` default blocks this. Add a `cub config set aur-fetch off` knob to disable AUR fetches in offline mode while keeping BUR sync (which uses git's local cache) working.
14. **Cache management** — cub caches AUR metadata in `/tmp/cub_cache/`. On Redox this maps to ramfs and is volatile. A persistent cache in `~/.cub/cache/` (or in the `var/lib/packages` directory) would survive reboots.
15. **Test coverage** — 70 unit tests is good, but the integration tests (the 12-PKGBUILD assessment) should be moved into `cargo test` so they run in CI. Currently they're in a separate `cub-assessment` sub-crate that nobody runs automatically.
16. **Sparse AUR metadata** — instead of cloning the full AUR git repo for every package, use the AUR RPC API (`https://aur.archlinux.org/rpc/?v=5&type=info&arg=<pkg>`) to get just the metadata, then clone only the source. This is already partially implemented in `aur.rs` but the full flow isn't exercised in tests.
---
## Files involved
- `local/recipes/system/cub/source/cub-lib/src/deps.rs` — bug 1, 2, 3
- `local/recipes/system/cub/source/cub-lib/src/pkgbuild.rs` — bugs 4, 5, 6, 7
- `local/recipes/system/cub/source/cub-lib/src/cookbook.rs` — bug 5, 8
- `local/recipes/system/cub/source/cub-lib/src/rbpkgbuild.rs` — bug 6 (CompatSection extension)
- `local/recipes/system/cub/source/cub-lib/src/converter.rs` — bug 4 (also has its own `convert_pkgbuild`)
- `local/recipes/system/cub/source/cub-assessment/src/main.rs` — integration test (no changes)
---
## Conclusion
cub is the right tool for the job. The architecture is correct (PKGBUILD → RbPkgBuild → recipe.toml is a clean three-stage pipeline), the existing code is mostly right (70 unit tests pass, all 12 real-world PKGBUILDs convert to valid TOML), and the bugs are surgical fixes that don't require architectural changes.
After the 7 critical fixes in this commit, cub will be a useful Red Bear package tool. The remaining 16 follow-up items in the forward plan address cookbook integration, conversion quality, UX, and infrastructure — they're substantial but well-scoped.
The recommendation is to ship the 7 fixes, then use cub to bootstrap the next 5-10 missing recipes (zlib, libffi, pcre2, freetype, libpng, etc.) as a real-world validation. If the generated recipes cook successfully, we proceed to scale up cub-driven recipe generation. If they fail, the next round of fixes will be more targeted because we'll have concrete error messages.
---
## Appendix A: cub architecture map
```
cub-cli (binary)
├── main.rs
│ ├── Cli (clap derive)
│ ├── Commands: Install, Search, Info, Sync, SystemUpgrade, Build,
│ │ Get, GetAur, GetPkgbuild, Inspect, ImportAur, UpdateAll,
│ │ Remove, QueryLocal, QueryInfo, QueryList, CleanCache, CleanAll
│ ├── Aliases: -S, -Ss, -Si, -Sy, -Syu, -B, -G, -R, -Q, -Qi, -Ql,
│ │ -Pi, --import-aur, -Sua, -Sc, -Scc, -Gp
│ └── launch_tui_or_help() / run_command()
│
├── CookbookAdapter → calls cub::cookbook::generate_recipe()
├── PkgbuildConverter → calls pkgbuild::convert_pkgbuild()
└── PackageCreator → calls cub::package (gated on `full` feature)
cub-lib (library)
├── lib.rs → re-exports all modules
├── aur.rs → AUR RPC client + AurPackage
├── converter.rs → simpler convert_pkgbuild (used by tests)
├── cook.rs → invokes `repo cook` on the generated recipe
├── cookbook.rs → RbPkgBuild → cookbook recipe.toml
│ (THE PIPELINE ENDPOINT)
├── deps.rs → Arch → Red Bear dep mapping
│ (Bugs 1, 2, 3 live here)
├── depresolve.rs → dependency resolution across packages
├── error.rs → CubError enum
├── package.rs → pkgar creation (gated on `full` feature)
├── pkgbuild.rs → PKGBUILD → RbPkgBuild (937 lines)
│ (Bugs 4, 5, 6, 7 live here)
├── rbpkgbuild.rs → RbPkgBuild TOML serialization (408 lines)
├── rbpkgbuild tests
├── rbsrcinfo.rs → .SRCINFO format reader/writer
├── recipe.rs → save_recipe_to_store, etc.
├── resolver.rs → check_missing_header, library, pkgconfig
├── sandbox.rs → build sandboxing for repo cook
└── storage.rs → CubStore: ~/.cub/ directory management
cub-tui (Ratatui TUI)
├── app.rs → TUI state machine
├── lib.rs → public API: cub_tui::run()
└── theme.rs → colors and styles
```
## Appendix B: RBPKGBUILD format
cub's intermediate format (between PKGBUILD and recipe.toml):
```toml
format = 1
[package]
name = "mesa"
version = "24.3.0"
release = 1
description = "Open-source implementation of the OpenGL specification"
homepage = "https://mesa.freedesktop.org"
license = ["MIT"]
architectures = ["x86_64-unknown-redox"]
maintainers = []
[[source]]
type = "git"
url = "https://gitlab.freedesktop.org/mesa/mesa.git"
sha256 = ""
rev = ""
branch = "mesa-24.3.0"
[dependencies]
build = ["host:meson", "host:ninja", "llvm"]
runtime = ["libdrm", "wayland"]
check = []
optional = []
provides = []
conflicts = []
[build]
template = "meson"
release = true
features = []
args = []
build_dir = ""
prepare = []
build_script = []
check = []
install_script = []
[install]
bins = []
libs = []
headers = []
docs = []
man = []
[compat]
imported_from = "aur"
original_pkgbuild = "..." # full PKGBUILD text
conversion_status = "Full" # or "Partial"
target = "x86_64-unknown-redox"
split_packages = []
options = [] # added in this commit
[policy]
allow_network = false
allow_tests = false
review_required = false
```
## Appendix C: Generated recipe.toml format
What cub produces (post-fix):
```toml
[source]
git = "https://gitlab.freedesktop.org/mesa/mesa.git"
shallow_clone = true
branch = "mesa-24.3.0"
[build]
template = "meson"
dependencies = [
"host:meson",
"host:ninja",
"llvm",
]
[package]
dependencies = [
"libdrm",
"wayland",
]
version = "24.3.0-1"
description = "Open-source implementation of the OpenGL specification"
```
What a real Red Bear recipe looks like (e.g., `local/recipes/libs/mesa/recipe.toml`):
```toml
# Comprehensive header comment block explaining Rule 2 + migration rationale
[source]
git = "https://gitlab.redox-os.org/redox-os/mesa"
branch = "redox-24.0"
[build]
template = "custom"
dependencies = [
"expat",
"libdrm",
"libwayland",
"llvm21",
"wayland-protocols",
"zlib",
]
dev-dependencies = [
"llvm21.dev",
]
script = """
DYNAMIC_INIT
# Apply Red Bear OS external patches (per local/AGENTS.md Rule 2)
cookbook_apply_patches "${REDBEAR_PATCHES_DIR}"
# TODO: Should be CPPFLAGS but cookbook_meson isn't reading it
export CFLAGS+=" -DHAVE_PTHREAD=1 -I${COOKBOOK_SYSROOT}/include/libdrm"
...
cookbook_meson -Dplatforms=wayland -Dvirgl=enabled ...
"""
[package]
version = "0.2.3"
description = "mesa 0.2.3 (v6.0 2026) — Mesa 3-D graphics library, Red Bear OS fork..."
```
The cub-generated recipe is **structurally correct** but **stylistically and contextually different** from a real Red Bear recipe. The forward plan addresses this gap.
local/external-upstream-map.toml
+355
View File
@@ -0,0 +1,355 @@
# ============================================================================
# external-upstream-map.toml — Red Bear OS graphical/desktop external sources
# ============================================================================
#
# Purpose
# Declarative description of every external (non-fork, non-path) upstream
# source consumed by the graphical / desktop build surface. Drives two
# tools:
# - local/scripts/check-external-versions.sh (read-only status report)
# - local/scripts/bump-graphics-recipes.sh (rewrites tar=/rev=/blake3=)
#
# Schema
# [groups.<id>] a version-coherent family (one version resolved, applied
# to every member). Qt6 / KF6 / Plasma each resolve their
# upstream version ONCE per group.
# version_index URL listing available versions (HTML or JSON).
# version_regex Python regex; capture group 1 holds the version token.
# resolve "version" (default; index lists full X.Y.Z) or
# "majmin" (index lists X.Y series dirs; full version
# is constructed as "<majmin>.0").
# recipe_root directory holding every member recipe (e.g.
# "local/recipes/kde"). Members are "<recipe_root>/<member>".
# tar_template default tar URL; supports {ver}, {majmin}, {name}.
# url_name_strip_prefix strip this prefix from a member's directory name
# to get its URL name (e.g. "kf6-" → "attica"). Members
# not starting with the prefix are used verbatim.
# members explicit recipe directory-name list.
# member_url_names overrides: { "<member>" = "<url_name>" } for irregulars
# (kf6-kded6 → kded, qt6-sensors → qtsensors, ...).
# member_templates overrides: { "<member>" = { tar_template = "..." } }
# for members whose URL family differs from the group
# default (kirigami on invent.kde.org, plasma members
# on invent.kde.org, ...).
#
# [packages.<id>] a singleton (one recipe, independently versioned).
# kind "tar" — version_index + tar_template bump model.
# "git-rev" — git ls-remote --tags + tag_regex bump model.
# recipe canonical recipe directory ("local/recipes/...").
# For kind = "tar": version_index, version_regex, resolve, tar_template.
# For kind = "git-rev": git_url, tag_regex.
#
# Update policy
# - Operator-edited. Agents MUST NOT edit this file without operator approval.
# - Every entry MUST be verifiable against the real recipe.toml before commit
# (run check-external-versions.sh and confirm CURRENT/OUTDATED is accurate).
# - When a series moves (e.g. Qt 6.11 → 6.12, KF6 6.28 → 6.29, GNOME series
# bump), update the affected version_index URL. The bumper then resolves
# the new series automatically.
# - series-locked indexes (glib/pango point at a specific X.Y subdir of
# download.gnome.org) are intentional: GNOME ships many patch releases
# within a series; locking the index avoids cross-series noise. Bump the
# series in this map when GNOME moves.
#
# Exclusions (deliberately NOT in this map)
# - Git branch-followers (curl, openssl, sdl2, seatd, polkit-qt6, icon themes):
# they track a branch tip, not a versioned release.
# - path = recipes (libepoxy, libdisplay-info, libpciaccess, libudev,
# libxcvt, zbus, pam-redbear, qt6-wayland-smoke): in-tree sources, no
# upstream tarball.
# - recipes/core/*: fork territory, managed by the fork pipeline.
# - icu: non-standard tag/tarball separator transform (release-75-1 tag vs
# icu4c-75_1-src.tgz); cannot be expressed by {ver}/{majmin} cleanly.
# Manually managed until the templating supports separator transforms.
# - fontconfig, gdk-pixbuf: no local/recipes/ entry exists (only mainline
# recipes/dev/*). Added to the map when a local fork/recipe is created.
#
# Verified 2026-07-18 against actual recipe.toml tar=/rev= lines.
# ============================================================================
# ---------------------------------------------------------------------------
# Groups
# ---------------------------------------------------------------------------
# ---- KF6 Frameworks -------------------------------------------------------
# 45 members: 44 tar from download.kde.org/stable/frameworks/<majmin>/ +
# kirigami (same 6.28.0 version, different host/template).
# Irregulars verified against recipe.toml:
# kf6-extra-cmake-modules → extra-cmake-modules
# kf6-kded6 → kded
# kf6-pty → kpty
# kf6-notifyconfig → knotifyconfig
# kf6-parts → kparts
# kf6-syntaxhighlighting → syntax-highlighting
# NOTE: kf6-kwayland and kf6-plasma-activities are Plasma-group members
# (they pull from stable/plasma/) and are listed under [groups.plasma].
[groups.kf6]
version_index = "https://download.kde.org/stable/frameworks/"
version_regex = 'href="([0-9]+\.[0-9]+)/"'
resolve = "majmin"
recipe_root = "local/recipes/kde"
url_name_strip_prefix = "kf6-"
tar_template = "https://download.kde.org/stable/frameworks/{majmin}/{name}-{ver}.tar.xz"
members = [
"kf6-attica", "kf6-extra-cmake-modules", "kf6-karchive", "kf6-kauth",
"kf6-kbookmarks", "kf6-kcmutils", "kf6-kcodecs", "kf6-kcolorscheme",
"kf6-kcompletion", "kf6-kconfig", "kf6-kconfigwidgets", "kf6-kcoreaddons",
"kf6-kcrash", "kf6-kdbusaddons", "kf6-kdeclarative", "kf6-kded6",
"kf6-kglobalaccel", "kf6-kguiaddons", "kf6-ki18n", "kf6-kiconthemes",
"kf6-kidletime", "kf6-kimageformats", "kf6-kio", "kf6-kitemmodels",
"kf6-kitemviews", "kf6-kjobwidgets", "kf6-knewstuff", "kf6-knotifications",
"kf6-kpackage", "kf6-kservice", "kf6-ksvg", "kf6-ktexteditor",
"kf6-ktextwidgets", "kf6-kwallet", "kf6-kwidgetsaddons",
"kf6-kwindowsystem", "kf6-kxmlgui", "kf6-notifyconfig", "kf6-parts",
"kf6-prison", "kf6-pty", "kf6-solid", "kf6-sonnet",
"kf6-syntaxhighlighting", "kirigami",
]
[groups.kf6.member_url_names]
"kf6-extra-cmake-modules" = "extra-cmake-modules"
"kf6-kded6" = "kded"
"kf6-notifyconfig" = "knotifyconfig"
"kf6-parts" = "kparts"
"kf6-pty" = "kpty"
"kf6-syntaxhighlighting" = "syntax-highlighting"
# kirigami ships the same framework version but via invent.kde.org archive.
[groups.kf6.member_templates.kirigami]
tar_template = "https://invent.kde.org/frameworks/kirigami/-/archive/v{ver}/kirigami-v{ver}.tar.gz"
# ---- Qt6 ------------------------------------------------------------------
# 6 members. qt6-sensors recipe → "qtsensors" URL submodule name.
[groups.qt6]
version_index = "https://download.qt.io/official_releases/qt/6.11/"
version_regex = 'href="([0-9]+\.[0-9]+\.[0-9]+)/"'
resolve = "version"
recipe_root = "local/recipes/qt"
url_name_strip_prefix = ""
tar_template = "https://download.qt.io/official_releases/qt/{majmin}/{ver}/submodules/{name}-everywhere-src-{ver}.tar.xz"
members = ["qtbase", "qtdeclarative", "qtsvg", "qtwayland", "qtshadertools", "qt6-sensors"]
[groups.qt6.member_url_names]
"qt6-sensors" = "qtsensors"
# ---- Plasma ---------------------------------------------------------------
# Version resolved ONCE from download.kde.org/stable/plasma/, applied to:
# - download.kde.org/stable/plasma/ family (default template):
# breeze, kf6-kwayland(→kwayland), kf6-plasma-activities(→plasma-activities),
# plasma-workspace
# - invent.kde.org/plasma/<name>/ family (per-member overrides):
# kwin, kdecoration, kglobalacceld, kde-cli-tools, plasma-desktop
# - invent.kde.org/frameworks/plasma-framework/ (different group path):
# plasma-framework
# plasma-wayland-protocols is intentionally OUT of this group (independent
# 1.21.0 versioning); it is a singleton below.
[groups.plasma]
version_index = "https://download.kde.org/stable/plasma/"
version_regex = 'href="([0-9]+\.[0-9]+\.[0-9]+)/"'
resolve = "version"
recipe_root = "local/recipes/kde"
url_name_strip_prefix = "kf6-"
tar_template = "https://download.kde.org/stable/plasma/{ver}/{name}-{ver}.tar.xz"
members = [
"breeze", "kf6-kwayland", "kf6-plasma-activities", "plasma-workspace",
"kwin", "kdecoration", "kglobalacceld", "kde-cli-tools",
"plasma-desktop", "plasma-framework",
]
[groups.plasma.member_url_names]
"kf6-kwayland" = "kwayland"
"kf6-plasma-activities" = "plasma-activities"
[groups.plasma.member_templates.kwin]
tar_template = "https://invent.kde.org/plasma/kwin/-/archive/v{ver}/kwin-v{ver}.tar.gz"
[groups.plasma.member_templates.kdecoration]
tar_template = "https://invent.kde.org/plasma/kdecoration/-/archive/v{ver}/kdecoration-v{ver}.tar.gz"
[groups.plasma.member_templates.kglobalacceld]
tar_template = "https://invent.kde.org/plasma/kglobalacceld/-/archive/v{ver}/kglobalacceld-v{ver}.tar.gz"
[groups.plasma.member_templates.kde-cli-tools]
tar_template = "https://invent.kde.org/plasma/kde-cli-tools/-/archive/v{ver}/kde-cli-tools-v{ver}.tar.gz"
[groups.plasma.member_templates.plasma-desktop]
tar_template = "https://invent.kde.org/plasma/plasma-desktop/-/archive/v{ver}/plasma-desktop-v{ver}.tar.gz"
[groups.plasma.member_templates.plasma-framework]
tar_template = "https://invent.kde.org/frameworks/plasma-framework/-/archive/v{ver}/plasma-framework-v{ver}.tar.gz"
# ---------------------------------------------------------------------------
# Singletons — tar
# ---------------------------------------------------------------------------
# mesa — software 3D / LLVMpipe / virgl. GitLab archive URL.
[packages.mesa]
kind = "tar"
recipe = "local/recipes/libs/mesa"
version_index = "https://archive.mesa3d.org/"
version_regex = 'href="mesa-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://gitlab.freedesktop.org/mesa/mesa/-/archive/mesa-{ver}/mesa-mesa-{ver}.tar.gz"
# libdrm — DRM user-space library. Static dri.freedesktop.org listing for
# version discovery; tarball from GitLab archive.
[packages.libdrm]
kind = "tar"
recipe = "local/recipes/libs/libdrm"
version_index = "https://dri.freedesktop.org/libdrm/"
version_regex = 'href="libdrm-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://gitlab.freedesktop.org/mesa/libdrm/-/archive/libdrm-{ver}/libdrm-libdrm-{ver}.tar.gz"
# libwayland — Wayland protocol C library.
[packages.libwayland]
kind = "tar"
recipe = "local/recipes/wayland/libwayland"
version_index = "https://gitlab.freedesktop.org/wayland/wayland/-/tags"
version_regex = '/wayland/wayland/-/tags/([0-9]+\.[0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://gitlab.freedesktop.org/wayland/wayland/-/releases/{ver}/downloads/wayland-{ver}.tar.xz"
# wayland-protocols — Wayland protocol XML.
[packages.wayland-protocols]
kind = "tar"
recipe = "local/recipes/wayland/wayland-protocols"
version_index = "https://gitlab.freedesktop.org/wayland/wayland-protocols/-/tags"
version_regex = '/wayland/wayland-protocols/-/tags/([0-9]+\.[0-9]+(?:\.[0-9]+)?)'
resolve = "version"
tar_template = "https://gitlab.freedesktop.org/wayland/wayland-protocols/-/releases/{ver}/downloads/wayland-protocols-{ver}.tar.xz"
# libevdev — evdev wrapper. Static freedesktop.org listing.
[packages.libevdev]
kind = "tar"
recipe = "local/recipes/libs/libevdev"
version_index = "https://www.freedesktop.org/software/libevdev/"
version_regex = 'libevdev-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://www.freedesktop.org/software/libevdev/libevdev-{ver}.tar.xz"
# libinput — input device management. GitLab tags page.
[packages.libinput]
kind = "tar"
recipe = "local/recipes/libs/libinput"
version_index = "https://gitlab.freedesktop.org/libinput/libinput/-/tags"
version_regex = '/libinput/libinput/-/tags/([0-9]+\.[0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://gitlab.freedesktop.org/libinput/libinput/-/archive/{ver}/libinput-{ver}.tar.bz2"
# libxkbcommon — keyboard description library. GitHub archive tag.
[packages.libxkbcommon]
kind = "tar"
recipe = "local/recipes/libs/libxkbcommon"
version_index = "https://github.com/xkbcommon/libxkbcommon/tags"
version_regex = 'xkbcommon-([0-9]+\.[0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://github.com/xkbcommon/libxkbcommon/archive/refs/tags/xkbcommon-{ver}.tar.gz"
# dbus — system + session message bus. Static freedesktop.org releases listing.
[packages.dbus]
kind = "tar"
recipe = "local/recipes/system/dbus"
version_index = "https://dbus.freedesktop.org/releases/dbus/"
version_regex = 'dbus-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://dbus.freedesktop.org/releases/dbus/dbus-{ver}.tar.xz"
# glib — GNOME core library. Series-locked index (operator bumps series here).
[packages.glib]
kind = "tar"
recipe = "local/recipes/libs/glib"
version_index = "https://download.gnome.org/sources/glib/2.89/"
version_regex = 'glib-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://download.gnome.org/sources/glib/{majmin}/glib-{ver}.tar.xz"
# cairo — 2D graphics. Static cairographics.org releases listing.
[packages.cairo]
kind = "tar"
recipe = "local/recipes/libs/cairo"
version_index = "https://www.cairographics.org/releases/"
version_regex = 'cairo-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://www.cairographics.org/releases/cairo-{ver}.tar.xz"
# pango — text layout. Series-locked GNOME index.
[packages.pango]
kind = "tar"
recipe = "local/recipes/libs/pango"
version_index = "https://download.gnome.org/sources/pango/1.56/"
version_regex = 'pango-([0-9]+\.[0-9]+\.[0-9]+)\.tar'
resolve = "version"
tar_template = "https://download.gnome.org/sources/pango/{majmin}/pango-{ver}.tar.xz"
# harfbuzz — text shaping. GitHub tags HTML is JS-only; use the API (JSON).
[packages.harfbuzz]
kind = "tar"
recipe = "local/recipes/libs/harfbuzz"
version_index = "https://api.github.com/repos/harfbuzz/harfbuzz/tags"
version_regex = '"name":\s*"([0-9]+\.[0-9]+\.[0-9]+)"'
resolve = "version"
tar_template = "https://github.com/harfbuzz/harfbuzz/releases/download/{ver}/harfbuzz-{ver}.tar.xz"
# freetype2 — font rendering. GitLab tags use VER-X-Y-Z format; ver_transform
# converts the captured dashes to dots. Tarball via sourceforge (canonical URL
# used by the recipe).
[packages.freetype2]
kind = "tar"
recipe = "local/recipes/libs/freetype2"
version_index = "https://gitlab.freedesktop.org/freetype/freetype/-/tags"
version_regex = '/freetype/freetype/-/tags/VER-([0-9]+-[0-9]+-[0-9]+)'
ver_transform = "s/-/./g"
resolve = "version"
tar_template = "https://sourceforge.net/projects/freetype/files/freetype2/{ver}/freetype-{ver}.tar.xz/download"
# lcms2 — Little CMS. GitHub archive tag (tag "lcms2.19" → version "2.19").
[packages.lcms2]
kind = "tar"
recipe = "local/recipes/libs/lcms2"
version_index = "https://github.com/mm2/Little-CMS/tags"
version_regex = 'lcms([0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://github.com/mm2/Little-CMS/archive/refs/tags/lcms{ver}.tar.gz"
# konsole — KDE terminal. Independent release cycle (KDE Gear, not Frameworks).
# invent.kde.org tags page.
[packages.konsole]
kind = "tar"
recipe = "local/recipes/kde/konsole"
version_index = "https://invent.kde.org/utilities/konsole/-/tags"
version_regex = '/utilities/konsole/-/tags/v([0-9]+\.[0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://invent.kde.org/utilities/konsole/-/archive/v{ver}/konsole-v{ver}.tar.gz"
# plasma-wayland-protocols — INDEPENDENT versioning (1.21.0, NOT plasma 6.7.x).
[packages.plasma-wayland-protocols]
kind = "tar"
recipe = "local/recipes/kde/plasma-wayland-protocols"
version_index = "https://invent.kde.org/libraries/plasma-wayland-protocols/-/tags"
version_regex = '/libraries/plasma-wayland-protocols/-/tags/v([0-9]+\.[0-9]+\.[0-9]+)'
resolve = "version"
tar_template = "https://invent.kde.org/libraries/plasma-wayland-protocols/-/archive/v{ver}/plasma-wayland-protocols-v{ver}.tar.gz"
# ---------------------------------------------------------------------------
# Singletons — git-rev
# ---------------------------------------------------------------------------
# sddm — display manager. rev is a commit sha; resolve latest tag's sha.
[packages.sddm]
kind = "git-rev"
recipe = "local/recipes/kde/sddm"
git_url = "https://github.com/sddm/sddm.git"
tag_regex = '^v?[0-9]+\.[0-9]+\.[0-9]+$'
# pipewire — audio/video server. rev is a tag name (e.g. "0.3.85").
[packages.pipewire]
kind = "git-rev"
recipe = "local/recipes/libs/pipewire"
git_url = "https://gitlab.freedesktop.org/pipewire/pipewire.git"
tag_regex = '^[0-9]+\.[0-9]+\.[0-9]+$'
# wireplumber — PipeWire session manager. rev is a tag name (e.g. "0.4.14").
[packages.wireplumber]
kind = "git-rev"
recipe = "local/recipes/libs/wireplumber"
git_url = "https://gitlab.freedesktop.org/pipewire/wireplumber.git"
tag_regex = '^[0-9]+\.[0-9]+\.[0-9]+$'
local/scripts/bump-fork.sh
+60
View File
@@ -1,4 +1,35 @@
#!/usr/bin/env bash
#
# ============================================================================
# DEPRECATED — DO NOT USE (2026-07-18, release-bump-pipeline.md Deliverable 4)
# ============================================================================
# This script is superseded by:
# - local/scripts/bump-release.sh (canonical orchestrator)
# - local/scripts/upgrade-forks.sh --to=... (per-fork source bump)
#
# WHY IT IS DEPRECATED:
# bump-fork.sh uses a WHOLESALE-REPLACEMENT model: it clones a fresh
# upstream tag, applies patch files from local/patches/<comp>/, and then
# SWAPS the entire local/sources/<comp>/ directory with the new tree.
# This DESTROYS any committed fork work that is not also present as a
# patch file under local/patches/<comp>/ — including commits made
# directly to the submodule/<comp> branch (the path-fork model that
# replaced overlay patches in 2026-06).
#
# Concretely: kernel, relibc, base, bootloader, installer, redoxfs, and
# userutils now carry Red Bear changes as direct commits on their
# `submodule/<component>` branches, NOT as patch files. Running
# bump-fork.sh on any of them would silently discard that work.
#
# RETENTION POLICY:
# This file is KEPT for historical reference only, per the project's
# never-delete rule. The runtime guard below prevents accidental use.
# Its logic is otherwise unchanged.
#
# ESCAPE HATCH (operators only, with eyes open):
# REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1 ./local/scripts/bump-fork.sh ...
# ============================================================================
#
# bump-fork.sh — Auto-bump a local fork to match upstream version
#
# Usage: bump-fork.sh <component> [--version=X.Y.Z]
@@ -11,6 +42,35 @@
set -euo pipefail
# ---- Runtime deprecation guard -------------------------------------------
# Refuse to run unless the operator explicitly acknowledges the
# deprecation by exporting REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1.
# Without the env var, print the deprecation explanation to stderr and
# exit 1 BEFORE any potentially destructive operation runs.
if [[ "${REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED:-0}" != "1" ]]; then
cat >&2 <<'EOF'
============================================================================
bump-fork.sh is DEPRECATED (release-bump-pipeline.md Deliverable 4).
Its wholesale-replacement model destroys committed fork work that is not
also present as a patch file under local/patches/<comp>/. Under the current
path-fork model, kernel/relibc/base/bootloader/installer/redoxfs/userutils
carry Red Bear changes as direct commits on submodule/<component> branches
— running this script on any of them would silently discard that work.
Use instead:
./local/scripts/bump-release.sh # canonical orchestrator
./local/scripts/upgrade-forks.sh --to=<tag> # per-fork source bump
This file is retained for historical reference per the never-delete rule.
To override (operators only, with full understanding of the risk):
REDBEAR_I_KNOW_BUMP_FORK_IS_DEPRECATED=1 ./local/scripts/bump-fork.sh ...
============================================================================
EOF
exit 1
fi
# Derive Red Bear OS version from the current git branch name.
# Branch "0.3.1" → BRANCH_VERSION="0.3.1". Same pattern as
# local/scripts/build-redbear.sh:158-165. Falls back to "0.0.0" if
local/scripts/bump-graphics-recipes.sh
+260 -83
View File
@@ -1,121 +1,298 @@
#!/usr/bin/env bash
# bump-graphics-recipes.sh — Update tarball URLs/BLAKE3 for graphics recipes.
# Scope: recipes/libs, recipes/gpu, recipes/kde, recipes/qt, recipes/wip/{wayland,gpu,kde,qt}.
# Excludes: recipes/core/* (mini-ISO path), recipes/system/* (services).
# bump-graphics-recipes.sh — Map-driven external source-bump engine.
#
# Usage: bump-graphics-recipes.sh [--dry-run] [--no-fetch]
# Rewritten 2026-07-18 from the legacy hardcoded UPSTREAM_INDEX approach.
# The single source of truth is local/external-upstream-map.toml; this script
# never scans directories (symlinked duplicates like recipes/qt/qtbase ↔
# recipes/wip/qt/qtbase are handled by iterating the MAP, which lists each
# canonical local/recipes/ path exactly once).
#
# What it does:
# - Resolves each group's latest version ONCE, then applies to all members.
# - Resolves each singleton's latest version individually.
# - tar bumps: download new tarball → b3sum → sed tar=/blake3= in recipe.toml
# (+ sync [package] version = "..." if present, semver, and differs).
# - git-rev bumps: git ls-remote --tags → latest matching tag → resolve its
# commit sha (annotated tags dereferenced via ^{}) → sed rev = "...".
# - After each non-dry-run bump: runs ./target/release/repo validate-patches
# <recipe> and captures [PASS]/[FAIL] counts.
# - Writes a machine-parseable stabilization report to stdout AND
# .redbear-recipe-bump/last-report.txt.
#
# Usage:
# bump-graphics-recipes.sh [--dry-run] [--no-fetch]
#
# --dry-run resolve and print intended changes; perform no downloads,
# edits, or validate-patches runs.
# --no-fetch reuse cached version indexes under .redbear-recipe-bump/;
# error on entries whose cache is missing.
#
# Never commits. Never deletes recipes. Never creates branches.
# See local/external-upstream-map.toml for the package inventory and schema.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
MAP="$ROOT/local/external-upstream-map.toml"
CACHE_DIR="$ROOT/.redbear-recipe-bump"
REPORT="$CACHE_DIR/last-report.txt"
DRY_RUN=0
NO_FETCH=0
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=1 ;;
--no-fetch) NO_FETCH=1 ;;
--help|-h) echo "Usage: $0 [--dry-run] [--no-fetch]"; exit 0 ;;
--help|-h) grep -E '^#' "$0" | head -30; exit 0 ;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
declare -A UPSTREAM_INDEX=(
[cairo]=https://cairographics.org/releases/
[freetype2]=https://download.savannah.gnu.org/freetype/
[glib]=https://download.gnome.org/sources/glib/
[harfbuzz]=https://github.com/harfbuzz/harfbuzz/releases/
[libepoxy]=https://github.com/anholt/libepoxy/releases/
[libxkbcommon]=https://github.com/xkbcommon/libxkbcommon/releases/
[pango]=https://download.gnome.org/sources/pango/
[libjpeg]=https://github.com/libjpeg-turbo/libjpeg-turbo/releases/
[shared-mime-info]=https://gitlab.gnome.org/GNOME/shared-mime-info/-/tags/
[gdk-pixbuf]=https://download.gnome.org/sources/gdk-pixbuf/
[libxml2]=https://download.gnome.org/sources/libxml2/
[kf6-karchive]=https://download.kde.org/stable/frameworks/
[kf6-attica]=https://download.kde.org/stable/frameworks/
[kf6-kcodecs]=https://download.kde.org/stable/frameworks/
if [ ! -f "$MAP" ]; then
echo "ERROR: map not found: $MAP" >&2
exit 2
fi
REPO_BIN="$ROOT/target/release/repo"
NO_FETCH_FLAG=""
[ "$NO_FETCH" = "1" ] && NO_FETCH_FLAG="--no-fetch"
mkdir -p "$CACHE_DIR"
# ---------------------------------------------------------------------------
# Phase 1: Python resolves every entry and emits a TSV action plan.
# Fields (tab-separated):
# ACTION RECIPE_NAME RECIPE_DIR KIND OLD NEW NEW_VERSION NEW_PKG_VERSION NOTE
# ACTION ∈ {bump, current, failed}
# ---------------------------------------------------------------------------
ACTIONS_FILE="$(mktemp)"
trap 'rm -f "$ACTIONS_FILE"' EXIT
python3 - "$ROOT" "$NO_FETCH_FLAG" <<'PYEOF' > "$ACTIONS_FILE"
import re
import sys
import tomllib
from pathlib import Path
ROOT = Path(sys.argv[1])
NO_FETCH = "--no-fetch" in sys.argv[2:]
MAP_PATH = ROOT / "local/external-upstream-map.toml"
CACHE_DIR = ROOT / ".redbear-recipe-bump"
with open(MAP_PATH, "rb") as f:
MAP = tomllib.load(f)
sys.path.insert(0, str(ROOT / "local/scripts/lib"))
import external_version_lib
external_version_lib.configure(CACHE_DIR, NO_FETCH)
from external_version_lib import (
fetch, version_sort_key, apply_ver_transform, resolve_index,
render_template, member_url_name, member_template, read_recipe_text,
current_tar_url, current_rev, resolve_git_rev,
)
GRAPHICS_DIRS=(
"$ROOT/recipes/libs" "$ROOT/recipes/gpu" "$ROOT/recipes/kde" "$ROOT/recipes/qt"
"$ROOT/recipes/wip/wayland" "$ROOT/recipes/wip/gpu" "$ROOT/recipes/wip/kde" "$ROOT/recipes/wip/qt"
)
VER_RE='-v?[0-9]+(\.[0-9]+){1,3}(-rc[0-9]+)?(-alpha[0-9]+)?(-beta[0-9]+)?'
resolve_latest() {
local cached="$1" prefix="$2"
grep -oE "${prefix}${VER_RE}\.tar\.(gz|xz|bz2|zst)" "$cached" | sort -uV | tail -1
}
def current_pkg_version(text):
m = re.search(r'^\[package\]\s*\n(?:[^\[]*?\n)*?\s*version\s*=\s*"([^"]+)"',
text, re.MULTILINE)
return m.group(1) if m else None
def is_semver(v):
return bool(re.match(r'^\d+\.\d+(\.\d+)?$', v or ""))
def emit(action, name, rdir, kind, old, new, new_ver, pkg_ver, note):
print("\t".join([action, name, str(rdir), kind,
old or "", new or "", new_ver or "",
pkg_ver or "", note or ""]))
# Groups
for gid, g in MAP.get("groups", {}).items():
v, _src, err = resolve_index(g["version_index"], g["version_regex"],
g.get("resolve", "version"))
for member in g["members"]:
rdir = ROOT / g["recipe_root"] / member
text = read_recipe_text(rdir)
cur = current_tar_url(text)
tmpl = member_template(g, member)
uname = member_url_name(g, member)
if cur is None:
emit("failed", f"{gid}:{member}", rdir, "tar", "", "", "", "",
"no-tar-in-recipe")
continue
if v is None:
emit("failed", f"{gid}:{member}", rdir, "tar", cur, "", "", "",
f"index:{err}")
continue
new_url = render_template(tmpl, {**v, "name": uname})
if new_url == cur:
emit("current", f"{gid}:{member}", rdir, "tar", cur, new_url,
v["ver"], "", "")
else:
pkg = current_pkg_version(text)
new_pkg = v["ver"] if (pkg and is_semver(pkg) and pkg != v["ver"]) else ""
emit("bump", f"{gid}:{member}", rdir, "tar", cur, new_url,
v["ver"], new_pkg, "")
# Singletons
for pid, p in MAP.get("packages", {}).items():
rdir = ROOT / p["recipe"]
text = read_recipe_text(rdir)
if p["kind"] == "tar":
cur = current_tar_url(text)
if cur is None:
emit("failed", pid, rdir, "tar", "", "", "", "", "no-tar-in-recipe")
continue
v, _src, err = resolve_index(p["version_index"], p["version_regex"],
p.get("resolve", "version"),
p.get("ver_transform"))
if v is None:
emit("failed", pid, rdir, "tar", cur, "", "", "", f"index:{err}")
continue
new_url = render_template(p["tar_template"], {**v, "name": pid})
if new_url == cur:
emit("current", pid, rdir, "tar", cur, new_url, v["ver"], "", "")
else:
pkg = current_pkg_version(text)
new_pkg = v["ver"] if (pkg and is_semver(pkg) and pkg != v["ver"]) else ""
emit("bump", pid, rdir, "tar", cur, new_url, v["ver"], new_pkg, "")
elif p["kind"] == "git-rev":
cur = current_rev(text)
if cur is None:
emit("failed", pid, rdir, "git-rev", "", "", "", "", "no-rev-in-recipe")
continue
res, err = resolve_git_rev(p["git_url"], p["tag_regex"])
if res is None:
emit("failed", pid, rdir, "git-rev", cur, "", "", "", f"git:{err}")
continue
new_sha = res["sha"]
if cur == new_sha or cur.startswith(new_sha[:12]) or cur == res["tag"]:
emit("current", pid, rdir, "git-rev", cur, new_sha, res["tag"], "", "")
else:
emit("bump", pid, rdir, "git-rev", cur, new_sha, res["tag"], "", "")
PYEOF
# ---------------------------------------------------------------------------
# Phase 2: bash applies the action plan.
# ---------------------------------------------------------------------------
UPDATED=0
SKIPPED=0
CURRENT=0
FAILED=0
REPORT_LINES=()
for dir in "${GRAPHICS_DIRS[@]}"; do
[ -d "$dir" ] || continue
for rdir in "$dir"/*/; do
[ -f "$rdir/recipe.toml" ] || continue
name=$(basename "$rdir")
run_validate_patches() {
# $1 = recipe name (basename). Outputs patches_total/pass/fail on stdout
# as "total=N pass=N fail=N". Returns non-zero only if the binary is
# missing (caller treats that as a warning, not a failure).
local recipe="$1"
if [ ! -x "$REPO_BIN" ]; then
echo "total=0 pass=0 fail=0"
echo "WARN: $REPO_BIN not found; skipped validate-patches for $recipe" >&2
return 1
fi
local out
out="$(REDBEAR_CANONICAL_BUILD=1 "$REPO_BIN" validate-patches "$recipe" 2>&1)" || true
local pass fail
pass="$(printf '%s\n' "$out" | grep -c '\[PASS\]' || true)"
fail="$(printf '%s\n' "$out" | grep -c '\[FAIL\]' || true)"
local total=$((pass + fail))
echo "total=${total} pass=${pass} fail=${fail}"
}
if [ -z "${UPSTREAM_INDEX[$name]:-}" ]; then
SKIPPED=$((SKIPPED+1))
continue
fi
# Read the TSV action plan line by line.
while IFS=$'\t' read -r ACTION RNAME RDIR KIND OLD NEW NVER NPKG NOTE; do
[ -n "$ACTION" ] || continue
RTOML="$RDIR/recipe.toml"
current=$(grep '^tar = ' "$rdir/recipe.toml" | head -1 | sed -E 's/^tar = "//;s/"$//')
[ -z "$current" ] && { SKIPPED=$((SKIPPED+1)); continue; }
case "$ACTION" in
current)
CURRENT=$((CURRENT + 1))
REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=")
if [ "$DRY_RUN" = "1" ]; then
printf '[current] %s (already at %s)\n' "$RNAME" "$NVER"
fi
;;
case "$name" in
glib|pango|shared-mime-info|gdk-pixbuf|libxml2) prefix="$name" ;;
freetype2) prefix="freetype" ;;
libepoxy) prefix="libepoxy" ;;
libxkbcommon) prefix="libxkbcommon" ;;
libjpeg) prefix="libjpeg-turbo" ;;
cairo) prefix="cairo" ;;
harfbuzz) prefix="harfbuzz" ;;
kf6-*) prefix="$(echo $name | sed 's/^kf6-//')" ;;
*) prefix="$name" ;;
esac
failed)
FAILED=$((FAILED + 1))
REPORT_LINES+=("recipe=$RNAME old=$OLD new= patches_total= patches_pass= patches_fail= fail_details=$NOTE")
printf '[FAILED] %s: %s\n' "$RNAME" "$NOTE" >&2
;;
cached="$ROOT/.redbear-recipe-bump/$name.html"
mkdir -p "$(dirname "$cached")"
if [ "$NO_FETCH" = "0" ]; then
if ! curl -sLf "${UPSTREAM_INDEX[$name]}" -o "$cached"; then
echo "WARN: $name: failed to fetch ${UPSTREAM_INDEX[$name]}" >&2
FAILED=$((FAILED+1))
bump)
if [ "$DRY_RUN" = "1" ]; then
if [ "$KIND" = "tar" ]; then
printf '[would-bump] %s\n %s\n -> %s\n (would download + b3sum + sed tar=/blake3=)\n' \
"$RNAME" "$OLD" "$NEW"
else
printf '[would-bump] %s\n rev=%s\n -> rev=%s (%s)\n' \
"$RNAME" "$OLD" "$NEW" "$NVER"
fi
UPDATED=$((UPDATED + 1))
REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=(dry-run)")
continue
fi
fi
[ -f "$cached" ] || continue
latest=$(resolve_latest "$cached" "$prefix")
[ -z "$latest" ] && { SKIPPED=$((SKIPPED+1)); continue; }
# --- tar bump ---
if [ "$KIND" = "tar" ]; then
tmp="$(mktemp)"
if ! curl -sLf --retry 3 --retry-delay 2 --connect-timeout 20 --max-time 300 "$NEW" -o "$tmp" 2>/dev/null; then
FAILED=$((FAILED + 1))
REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total= patches_pass= patches_fail= fail_details=download-failed")
printf '[FAILED] %s: download failed for %s\n' "$RNAME" "$NEW" >&2
rm -f "$tmp"
continue
fi
new_hash="$(b3sum "$tmp" | awk '{print $1}')"
rm -f "$tmp"
latest_url=$(grep -m1 -oE "${UPSTREAM_INDEX[$name]}${latest}" "$cached" | head -1)
[ -z "$latest_url" ] && latest_url="${UPSTREAM_INDEX[$name]}${latest}"
# Escape sed-special replacement chars (& \ |) so URLs containing
# query strings or other metacharacters do not corrupt the rewrite.
esc_new=$(printf '%s' "$NEW" | sed 's/[&\\|]/\\&/g')
sed -i -E "s|^\\s*tar\\s*=\\s*\"[^\"]+\"|tar = \"$esc_new\"|" "$RTOML"
sed -i -E "s|^\\s*blake3\\s*=\\s*\"[a-f0-9]+\"|blake3 = \"$new_hash\"|" "$RTOML"
if [ "$current" = "$latest_url" ]; then
SKIPPED=$((SKIPPED+1))
continue
fi
if [ -n "$NPKG" ]; then
sed -i -E "s|^(\\s*version\\s*=\\s*)\"[^\"]+\"|\\1\"$NPKG\"|" "$RTOML"
fi
echo "[$name] $current -> $latest_url"
if [ "$DRY_RUN" = "0" ]; then
tmp=$(mktemp)
curl -sLf "$latest_url" -o "$tmp"
new_hash=$(b3sum "$tmp" | awk '{print $1}')
sed -i -E "s|^tar = \"[^\"]+\"|tar = \"$latest_url\"|" "$rdir/recipe.toml"
sed -i -E "s|^blake3 = \"[a-f0-9]+\"|blake3 = \"$new_hash\"|" "$rdir/recipe.toml"
rm -f "$tmp"
fi
UPDATED=$((UPDATED+1))
UPDATED=$((UPDATED + 1))
# --- git-rev bump ---
elif [ "$KIND" = "git-rev" ]; then
sed -i -E "s|^\\s*rev\\s*=\\s*\"[^\"]+\"|rev = \"$NEW\"|" "$RTOML"
UPDATED=$((UPDATED + 1))
fi
# --- validate-patches ---
validate_out="$(run_validate_patches "$(basename "$RDIR")" 2>/dev/null || true)"
vt="$(printf '%s' "$validate_out" | grep -oE 'total=[0-9]*' | head -1 | cut -d= -f2)"
vp="$(printf '%s' "$validate_out" | grep -oE 'pass=[0-9]*' | head -1 | cut -d= -f2)"
vf="$(printf '%s' "$validate_out" | grep -oE 'fail=[0-9]*' | head -1 | cut -d= -f2)"
vt="${vt:-0}"; vp="${vp:-0}"; vf="${vf:-0}"
REPORT_LINES+=("recipe=$RNAME old=$OLD new=$NEW patches_total=$vt patches_pass=$vp patches_fail=$vf fail_details=")
;;
esac
done < "$ACTIONS_FILE"
# ---------------------------------------------------------------------------
# Phase 3: report + summary.
# ---------------------------------------------------------------------------
{
for line in "${REPORT_LINES[@]}"; do
printf '%s\n' "$line"
done
done
} | tee "$REPORT"
echo ""
echo "=== Bump summary ==="
echo "Updated (or would-update): $UPDATED"
echo "Skipped (no index / already-current / non-tar): $SKIPPED"
echo "Failed (network/parse): $FAILED"
[ "$DRY_RUN" = "1" ] && echo "Dry-run only. No files changed."
echo "Already current: $CURRENT"
echo "Failed (network/parse): $FAILED"
[ "$DRY_RUN" = "1" ] && echo "(dry-run; no files changed)"
echo "Report written to: $REPORT"
local/scripts/bump-release.sh
Executable
+579
View File
@@ -0,0 +1,579 @@
#!/usr/bin/env bash
# bump-release.sh — Canonical release-bump orchestrator for Red Bear OS.
#
# Policy (release-bump-pipeline.md):
# - bump-release.sh DECIDES per fork.
# - sync-versions.sh EXECUTES suffix-only label rewrites (single code path).
# - refresh-fork-upstream-map.sh maintains the map's column-3 tag.
# - upgrade-forks.sh --to=<tag> performs actual fork source bumps.
#
# Fork routing by map mode (release-bump-pipeline.md design constraint #2):
# - snapshot/tracked → eligible for source bump via upgrade-forks.sh --to
# - diverged → report-only; operator-manual rebase (Phase 2.4+).
# Override with --force-diverged.
# - base (tag=main) → always suffix-only; member crates untouched.
# - bootloader → NEVER source-bumped (no merge-base with upstream;
# one-time 'git replace --graft' is the documented path).
# Upgrade candidates are resolved by latest upstream semver tag EXCLUDING any
# tag already reachable from the fork's HEAD (ancestor tags are history, not
# upgrades — see latest_tag_for_url).
#
# Modes:
# (no args) Fork upstream discovery REPORT + label sync (delegates to
# sync-versions.sh --no-regen) + hint to rerun with
# --with-sources / --with-external when upgrades exist.
# --with-sources For each fork whose map mode is snapshot/tracked AND whose
# latest upstream semver tag is newer than the current base
# (current base = fork Cargo.toml version stripped of +rb*):
# upgrade-forks.sh --to=<tag> <fork>
# On success: set fork Cargo.toml version to
# "<tag>+rb<branch>", update the map row's column 3 in
# place, regen that fork's Cargo.lock (best-effort), report
# the rb-backup/* branch. On failure: report, leave label
# and map untouched, continue with remaining forks. After
# all source bumps: re-run sync-versions.sh --no-regen.
# --with-external Invoke local/scripts/bump-graphics-recipes.sh (pass
# through --dry-run / --no-fetch). Tolerate its absence
# with a warning (a sibling unit is rewriting it
# concurrently).
# --check Fully read-only; print the same report; exit 1 if any
# label drift (via sync-versions.sh --check) OR any fork
# has an available source bump; exit 0 when fully synced.
# --dry-run Print decisions, mutate nothing (implies no delegation
# that mutates).
# --no-fetch Skip all network (ls-remote); use the map's current tags
# as "latest" (actions become report-only).
# --regen Pass --regen instead of --no-regen to sync-versions.sh.
# --force-diverged Pass --force-diverged to upgrade-forks.sh for diverged
# forks (still reports them first).
# --fork=<name> Scope all fork operations to one fork.
#
# GUARDS:
# - Current git branch must match ^[0-9]+\.[0-9]+\.[0-9]+$ (anchored semver).
# Otherwise refuse — release bumps only happen on release branches.
# - REDBEAR_CI=1: only --check / --dry-run are allowed. --with-sources and
# --with-external are refused.
#
# NEVER AUTO-COMMITS. The script ends with an exact copy-paste commit guide
# listing parent-repo files changed (map, Cargo.toml of cookbook + Cat 1
# crates) and per-fork git -C local/sources/<fork> commit + push to
# submodule/<fork> reminders.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
SCRIPT_DIR="$ROOT/local/scripts"
MAP="$ROOT/local/fork-upstream-map.toml"
cd "$ROOT"
# ---- argument parsing ------------------------------------------------------
WITH_SOURCES=0
WITH_EXTERNAL=0
CHECK_ONLY=0
DRY_RUN=0
NO_FETCH=0
REGEN=0
FORCE_DIVERGED=0
FORK_FILTER=""
for arg in "$@"; do
case "$arg" in
--with-sources) WITH_SOURCES=1 ;;
--with-external) WITH_EXTERNAL=1 ;;
--check) CHECK_ONLY=1 ;;
--dry-run) DRY_RUN=1 ;;
--no-fetch) NO_FETCH=1 ;;
--regen) REGEN=1 ;;
--force-diverged) FORCE_DIVERGED=1 ;;
--fork=*) FORK_FILTER="${arg#*=}" ;;
-h|--help)
sed -n '2,55p' "$0"
exit 0
;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
# ---- branch guard ----------------------------------------------------------
BRANCH="$(git branch --show-current 2>/dev/null || echo "")"
if [[ -z "$BRANCH" ]]; then
echo "ERROR: cannot determine current git branch" >&2
exit 1
fi
if [[ ! "$BRANCH" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "ERROR: branch '$BRANCH' is not anchored semver (X.Y.Z)" >&2
echo " Release bumps only happen on release branches." >&2
exit 1
fi
# ---- CI guard --------------------------------------------------------------
if [[ "${REDBEAR_CI:-0}" == "1" ]]; then
if [[ $WITH_SOURCES -eq 1 || $WITH_EXTERNAL -eq 1 ]]; then
echo "ERROR: REDBEAR_CI=1 forbids --with-sources / --with-external" >&2
echo " Use --check or --dry-run only." >&2
exit 1
fi
fi
# Aggregate read-only flag: anything that must NOT mutate the tree.
READ_ONLY=0
[[ $CHECK_ONLY -eq 1 || $DRY_RUN -eq 1 ]] && READ_ONLY=1
# ---- helpers ---------------------------------------------------------------
SEMVER_RE='^[0-9]+\.[0-9]+\.[0-9]+$'
# Resolve the newest GENUINE upgrade tag for a remote via `git ls-remote`.
# Output: "<tag>|<raw-count>" — tag empty when nothing qualifies.
# Two filters: (a) tags already reachable from the fork's HEAD are history,
# not upgrades (relibc's 0.6.0/2020 outranks its newer 0.2.x API line
# lexically); (b) tags not version-greater than the current base are not
# upgrades either (the base tag itself is usually an ancestor of HEAD).
latest_tag_for_url() {
local url="$1" fork="${2:-}" base="${3:-}"
[[ $NO_FETCH -eq 0 ]] || { echo "|0"; return 0; }
local tag forkdir="" raw=0
[[ -n "$fork" && -d "$ROOT/local/sources/$fork/.git" ]] \
&& forkdir="$ROOT/local/sources/$fork"
while IFS= read -r tag; do
[[ -z "$tag" ]] && continue
raw=$((raw + 1))
if [[ -n "$forkdir" ]] \
&& git -C "$forkdir" rev-parse --verify --quiet "refs/tags/$tag" >/dev/null 2>&1 \
&& git -C "$forkdir" merge-base --is-ancestor "refs/tags/$tag" HEAD 2>/dev/null; then
continue
fi
if [[ -n "$base" ]] && ! version_greater "$tag" "$base"; then
continue
fi
echo "${tag}|${raw}"
return 0
done < <(git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \
| awk '{print $2}' \
| sed 's|refs/tags/||' \
| sed 's|\^{}$||' \
| grep -E "$SEMVER_RE" || true)
echo "|${raw}"
return 0
}
# version_greater <a> <b> → true iff a is version-greater than b (sort -V).
version_greater() {
[[ "$1" != "$2" ]] \
&& [[ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | tail -1)" == "$1" ]]
}
# Compare two semvers. Echo -1 / 0 / 1. Echo "ERR" if either is non-semver.
semver_cmp() {
local a="$1" b="$2"
if [[ ! "$a" =~ $SEMVER_RE || ! "$b" =~ $SEMVER_RE ]]; then
echo "ERR"; return
fi
local a1 a2 a3 b1 b2 b3
IFS=. read -r a1 a2 a3 <<<"$a"
IFS=. read -r b1 b2 b3 <<<"$b"
if (( a1 < b1 )); then echo "-1"; return; fi
if (( a1 > b1 )); then echo "1"; return; fi
if (( a2 < b2 )); then echo "-1"; return; fi
if (( a2 > b2 )); then echo "1"; return; fi
if (( a3 < b3 )); then echo "-1"; return; fi
if (( a3 > b3 )); then echo "1"; return; fi
echo "0"
}
# Strip any +rb<x.y.z> build-metadata suffix from a version string.
strip_rb_suffix() {
echo "$1" | sed -E 's/\+rb[0-9]+(\.[0-9]+\.[0-9]+)?$//'
}
# Read a fork's current base version from its Cargo.toml (stripped of +rb*).
fork_current_base_version() {
local fork="$1"
local ct="$ROOT/local/sources/$fork/Cargo.toml"
[[ -f "$ct" ]] || { echo ""; return; }
local ver
ver=$(grep -E '^version[[:space:]]*=' "$ct" | head -1 | sed 's/.*"\(.*\)".*/\1/' || true)
strip_rb_suffix "$ver"
}
# ---- parse the map into 4 parallel arrays ----------------------------------
declare -a MAP_FORKS=()
declare -a MAP_URLS=()
declare -a MAP_TAGS=()
declare -a MAP_MODES=()
parse_map() {
MAP_FORKS=()
MAP_URLS=()
MAP_TAGS=()
MAP_MODES=()
[[ -f "$MAP" ]] || return 0
while IFS=$'\t' read -r fork url tag mode; do
[[ -z "$fork" ]] && continue
MAP_FORKS+=("$fork")
MAP_URLS+=("$url")
MAP_TAGS+=("$tag")
MAP_MODES+=("${mode:-}")
done < <(awk '
/^[^#[:space:]]/ {
fork = $1; url = $2; tag = $3; mode = $4
sub(/[ \t]*#.*/, "", mode)
print fork "\t" url "\t" tag "\t" mode
}
' "$MAP")
}
# In-place column-3 update for a single fork (byte-exact awk rewrite).
# A sidecar tmp file is consumed by `mv`; no trap needed (if mv fails the
# tmp file lingers harmlessly under $MAP.tmp.<pid>).
update_map_tag_inplace() {
local fork="$1" new_tag="$2"
[[ -f "$MAP" ]] || return 0
local tmp="${MAP}.tmp.$$"
awk -v fork="$fork" -v new_tag="$new_tag" '
$1 == fork && /^[^#[:space:]]/ {
prefix_re = fork "[ \t]+[^ \t]+[ \t]+"
if (match($0, prefix_re)) {
prefix = substr($0, 1, RLENGTH)
rest = substr($0, RLENGTH + 1)
if (match(rest, /^[^ \t]+/)) {
suffix = substr(rest, RLENGTH + 1)
$0 = prefix new_tag suffix
}
}
}
{ print }
' "$MAP" > "$tmp" && mv "$tmp" "$MAP"
# Clean up a straggler if mv failed for some reason.
[[ -e "$tmp" ]] && rm -f "$tmp" || true
}
# ---- counters --------------------------------------------------------------
SOURCE_BUMPS_AVAILABLE=0
SOURCE_BUMPS_APPLIED=0
SOURCE_BUMPS_FAILED=0
LABEL_DRIFT=0
# Latest-tag cache filled by phase_report; phase_source_bumps reuses it so
# each fork costs exactly one ls-remote per run.
declare -A LATEST_FOR=()
# ---- phase: report ---------------------------------------------------------
phase_report() {
echo "=== Fork upstream discovery report ==="
echo "branch=$BRANCH read_only=$READ_ONLY no_fetch=$NO_FETCH"
echo ""
local idx fork url current mode latest cargo_base basis cmp action
for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do
fork="${MAP_FORKS[$idx]}"
url="${MAP_URLS[$idx]}"
current="${MAP_TAGS[$idx]}"
mode="${MAP_MODES[$idx]}"
if [[ -n "$FORK_FILTER" && "$fork" != "$FORK_FILTER" ]]; then
continue
fi
if [[ "$current" == "main" ]]; then
echo "fork=$fork mode=$mode current=$current latest=(skip) action=skip-main"
continue
fi
if [[ "$fork" == "bootloader" ]]; then
echo "fork=$fork mode=$mode current=$current latest=(skip) action=report-only note=no-merge-base-needs-git-replace-graft"
continue
fi
if [[ "$mode" == "diverged" ]]; then
echo "fork=$fork mode=diverged current=$current latest=(skip) action=report-only"
continue
fi
# Decision basis: the fork's Cargo.toml base is ground truth; the map
# tag is the fallback when the fork has no Cargo.toml version.
cargo_base=$(fork_current_base_version "$fork")
basis="$current"
[[ -n "$cargo_base" ]] && basis="$cargo_base"
local result raw
latest=""
if [[ $NO_FETCH -eq 0 ]]; then
result=$(latest_tag_for_url "$url" "$fork" "$basis")
latest="${result%%|*}"
raw="${result##*|}"
# Tags existed upstream but none qualified as newer → already current.
[[ -z "$latest" && "$raw" -gt 0 ]] && latest="$basis"
else
latest="$basis"
fi
LATEST_FOR[$fork]="$latest"
if [[ -z "$latest" ]]; then
echo "fork=$fork mode=$mode current=$current latest=(unknown) action=skip-no-tag"
continue
fi
cmp=$(semver_cmp "$basis" "$latest")
case "$cmp" in
-1)
action="upgrade"
SOURCE_BUMPS_AVAILABLE=$((SOURCE_BUMPS_AVAILABLE + 1))
;;
0) action="ok" ;;
1) action="map-ahead-of-upstream" ;;
*) action="skip-non-semver" ;;
esac
echo "fork=$fork mode=$mode current=$current latest=$latest cargo=$cargo_base basis=$basis action=$action"
done
echo ""
echo "source_bumps_available=$SOURCE_BUMPS_AVAILABLE"
}
# ---- phase: label sync -----------------------------------------------------
phase_label_sync() {
local sync_flag="--no-regen"
[[ $REGEN -eq 1 ]] && sync_flag="--regen"
if [[ $READ_ONLY -eq 1 ]]; then
echo ""
echo "=== Label sync (read-only: sync-versions.sh --check) ==="
if "$SCRIPT_DIR/sync-versions.sh" --check >/tmp/br-label-sync.$$ 2>&1; then
LABEL_DRIFT=0
echo " OK: no label drift"
else
LABEL_DRIFT=1
echo " DRIFT detected (sync-versions.sh --check exit non-zero)"
grep -E '^(DRIFT|FAIL)' /tmp/br-label-sync.$$ | head -20 | sed 's/^/ /' || true
fi
rm -f /tmp/br-label-sync.$$
return
fi
echo ""
echo "=== Label sync: sync-versions.sh $sync_flag ==="
"$SCRIPT_DIR/sync-versions.sh" "$sync_flag" || true
}
# ---- phase: source bumps ---------------------------------------------------
phase_source_bumps() {
if [[ $WITH_SOURCES -eq 0 ]]; then
return
fi
if [[ $READ_ONLY -eq 1 ]]; then
echo ""
echo "(--dry-run / --check: source bumps NOT executed)"
if [[ $SOURCE_BUMPS_AVAILABLE -gt 0 ]]; then
echo "(rerun with --with-sources to perform $SOURCE_BUMPS_AVAILABLE bump(s))"
fi
return
fi
echo ""
echo "=== Fork source bumps (--with-sources) ==="
local idx fork url current mode latest cargo_base basis cmp ct old_ver new_ver backup_branch
local -a upgrade_args=()
for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do
fork="${MAP_FORKS[$idx]}"
url="${MAP_URLS[$idx]}"
current="${MAP_TAGS[$idx]}"
mode="${MAP_MODES[$idx]}"
if [[ -n "$FORK_FILTER" && "$fork" != "$FORK_FILTER" ]]; then continue; fi
if [[ "$current" == "main" ]]; then continue; fi
# Bootloader is never source-bumped, even with --force-diverged: its
# git history is genuinely unrelated to upstream (no merge-base), so
# the rebase machinery cannot work. The documented path is a one-time
# operator 'git replace --graft' intervention (RELEASE-BUMP-WORKFLOW.md).
if [[ "$fork" == "bootloader" ]]; then
echo " (bootloader: never auto-bumped — no merge-base with upstream;"
echo " one-time operator 'git replace --graft' intervention required;"
echo " see local/docs/RELEASE-BUMP-WORKFLOW.md)"
continue
fi
case "$mode" in
snapshot|tracked) ;;
diverged)
# phase_report already showed action=report-only; with
# --force-diverged we could attempt, but the map's current
# tag is the manual base. Skip unless --force-diverged.
if [[ $FORCE_DIVERGED -eq 1 ]]; then
echo " ($fork: diverged + --force-diverged; attempting)"
else
continue
fi
;;
*) continue ;;
esac
cargo_base=$(fork_current_base_version "$fork")
basis="$current"
[[ -n "$cargo_base" ]] && basis="$cargo_base"
# Reuse the report phase's ls-remote result; only fetch when this fork
# was not covered (e.g. report filtered it out).
if [[ -v "LATEST_FOR[$fork]" ]]; then
latest="${LATEST_FOR[$fork]}"
elif [[ $NO_FETCH -eq 0 ]]; then
local result raw
result=$(latest_tag_for_url "$url" "$fork" "$basis")
latest="${result%%|*}"
raw="${result##*|}"
[[ -z "$latest" && "$raw" -gt 0 ]] && latest="$basis"
else
latest="$basis"
fi
[[ -z "$latest" ]] && continue
cmp=$(semver_cmp "$basis" "$latest")
[[ "$cmp" == "-1" ]] || continue
echo ""
echo "--- Bumping $fork: $basis -> $latest ---"
upgrade_args=(--to="$latest")
[[ $FORCE_DIVERGED -eq 1 ]] && upgrade_args+=(--force-diverged)
upgrade_args+=("$fork")
if ! "$SCRIPT_DIR/upgrade-forks.sh" "${upgrade_args[@]}"; then
echo " FAIL: upgrade-forks.sh ${upgrade_args[*]} failed" >&2
echo " Leaving label + map untouched for $fork" >&2
SOURCE_BUMPS_FAILED=$((SOURCE_BUMPS_FAILED + 1))
continue
fi
# On success: set fork Cargo.toml version to <tag>+rb<branch>.
# Quote-splice via awk with literal index() — sed regexes break on the
# '.'/'+' metacharacters present in versions like 0.9.0+rb0.3.1.
ct="$ROOT/local/sources/$fork/Cargo.toml"
if [[ -f "$ct" ]]; then
old_ver=$(grep -E '^version[[:space:]]*=' "$ct" | head -1 | sed 's/.*"\(.*\)".*/\1/' || true)
new_ver="${latest}+rb${BRANCH}"
if [[ -n "$old_ver" ]]; then
awk -v old="$old_ver" -v new="$new_ver" '
$0 ~ "^version[[:space:]]*=" {
q1 = index($0, "\"")
q2 = index(substr($0, q1 + 1), "\"") + q1
if (q1 > 0 && q2 > q1 && substr($0, q1 + 1, q2 - q1 - 1) == old) {
$0 = substr($0, 1, q1) new substr($0, q2)
}
print; next
}
{ print }
' "$ct" > "$ct.tmp.$$" && mv "$ct.tmp.$$" "$ct"
[[ -e "$ct.tmp.$$" ]] && rm -f "$ct.tmp.$$" || true
echo " Cargo.toml: $old_ver -> $new_ver"
fi
fi
# Regen the fork's Cargo.lock (best-effort: warn on failure, continue)
if [[ -f "$ROOT/local/sources/$fork/Cargo.toml" ]]; then
if (cd "$ROOT/local/sources/$fork" && cargo generate-lockfile >/dev/null 2>&1); then
echo " Cargo.lock: regenerated"
else
echo " WARN: cargo generate-lockfile failed for $fork (continuing)" >&2
fi
fi
# Update the map row's column 3 in place
update_map_tag_inplace "$fork" "$latest"
echo " Map: column 3 ($current -> $latest)"
# Report the rb-backup/* branch created by upgrade-forks.sh
backup_branch=$(git -C "$ROOT/local/sources/$fork" branch --list 'rb-backup/*' 2>/dev/null | tail -1 | sed 's/^[* ]*//' || true)
if [[ -n "$backup_branch" ]]; then
echo " Backup branch: $backup_branch"
fi
SOURCE_BUMPS_APPLIED=$((SOURCE_BUMPS_APPLIED + 1))
done
# Re-sync labels after all source bumps land
echo ""
echo "=== Post-bump label sync (sync-versions.sh --no-regen) ==="
"$SCRIPT_DIR/sync-versions.sh" --no-regen || true
}
# ---- phase: external bumps -------------------------------------------------
phase_external() {
[[ $WITH_EXTERNAL -eq 0 ]] && return
local script="$SCRIPT_DIR/bump-graphics-recipes.sh"
if [[ ! -e "$script" ]]; then
echo ""
echo "WARN: bump-graphics-recipes.sh not present; skipping external bump" >&2
echo " (a sibling unit is rewriting it concurrently)" >&2
return
fi
local -a args=()
[[ $DRY_RUN -eq 1 ]] && args+=(--dry-run)
[[ $NO_FETCH -eq 1 ]] && args+=(--no-fetch)
echo ""
echo "=== External bumps (bump-graphics-recipes.sh ${args[*]:-}) ==="
"$script" "${args[@]}" || true
}
# ---- phase: commit guide ---------------------------------------------------
print_commit_guide() {
echo ""
echo "=== Commit guide (NEVER executed automatically by this script) ==="
echo ""
echo "Parent-repo files potentially changed:"
echo " local/fork-upstream-map.toml (column-3 tag updates)"
echo " Cargo.toml (cookbook root, on label drift)"
echo " local/recipes/*/source/Cargo.toml (Cat 1 crates, on label drift)"
echo " local/sources/<fork>/Cargo.toml (fork base version, on source bump)"
echo " local/sources/<fork>/Cargo.lock (fork lockfile, on source bump)"
echo ""
echo "Commit them on branch '$BRANCH':"
echo " git -C \"$ROOT\" add local/fork-upstream-map.toml Cargo.toml \\"
echo " local/recipes/ local/sources/"
echo " git -C \"$ROOT\" commit -m \"release: bump fork tags + labels to $BRANCH\""
echo ""
echo "Per-fork commits + pushes (for source bumps; replace <tag> per fork):"
if [[ -n "$FORK_FILTER" ]]; then
echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" add -A"
echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" commit -m \"bump: $FORK_FILTER to <tag>+rb$BRANCH\""
echo " git -C \"$ROOT/local/sources/$FORK_FILTER\" push origin submodule/$FORK_FILTER"
else
local idx fork mode current
for ((idx = 0; idx < ${#MAP_FORKS[@]}; idx++)); do
fork="${MAP_FORKS[$idx]}"
mode="${MAP_MODES[$idx]}"
current="${MAP_TAGS[$idx]}"
[[ "$current" == "main" ]] && continue
[[ "$mode" == "diverged" ]] && continue
echo " # $fork (current tag: $current)"
echo " git -C \"$ROOT/local/sources/$fork\" add -A"
echo " git -C \"$ROOT/local/sources/$fork\" commit -m \"bump: $fork to <tag>+rb$BRANCH\""
echo " git -C \"$ROOT/local/sources/$fork\" push origin submodule/$fork"
done
fi
}
# ---- main ------------------------------------------------------------------
parse_map
phase_report
phase_label_sync
phase_source_bumps
phase_external
print_commit_guide
if [[ $CHECK_ONLY -eq 1 ]]; then
echo ""
if [[ $LABEL_DRIFT -gt 0 ]]; then
echo "CHECK FAILED: label drift detected"
exit 1
fi
if [[ $SOURCE_BUMPS_AVAILABLE -gt 0 ]]; then
echo "CHECK FAILED: $SOURCE_BUMPS_AVAILABLE fork(s) have available source bumps"
exit 1
fi
echo "CHECK PASSED: no drift, no available source bumps"
exit 0
fi
exit 0
local/scripts/check-external-versions.sh
Executable
+268
View File
@@ -0,0 +1,268 @@
#!/usr/bin/env bash
# check-external-versions.sh — Report current vs latest upstream versions for
# every entry in local/external-upstream-map.toml.
#
# Read-only. Network is used to resolve latest versions unless --no-fetch is
# passed (in which case the on-disk cache under .redbear-recipe-bump/ is used
# and the script errors on a missing cache entry).
#
# Usage:
# check-external-versions.sh [--no-fetch] [--strict] [--json] [--help]
#
# Output (default): a fixed-width table
# name current latest STATUS
# qtbase 6.11.1 6.11.1 CURRENT
# ...
# Groups expand to one row per member; the group's resolved version is shown
# for every member, so any member whose embedded URL version differs from the
# group's is flagged OUTDATED.
#
# Exit codes:
# 0 always (default); --strict exits 1 if any entry is OUTDATED or failed
# to resolve.
#
# Map-driven: the TOML at local/external-upstream-map.toml is the single source
# of truth. See that file's header for schema and update policy.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
MAP="$ROOT/local/external-upstream-map.toml"
if [ ! -f "$MAP" ]; then
echo "ERROR: map not found: $MAP" >&2
exit 2
fi
# Delegate everything to an embedded Python3 script. Python (stdlib tomllib)
# gives correct TOML parsing; curl (called via subprocess) is used for the
# actual network fetches per the plan's requirement.
exec python3 - "$ROOT" "$@" <<'PYEOF'
import json
import re
import sys
import tomllib
from pathlib import Path
ROOT = Path(sys.argv[1])
MAP_PATH = ROOT / "local/external-upstream-map.toml"
CACHE_DIR = ROOT / ".redbear-recipe-bump"
args = sys.argv[2:]
NO_FETCH = "--no-fetch" in args
STRICT = "--strict" in args
AS_JSON = "--json" in args
if "--help" in args or "-h" in args:
print(__doc__.replace("check-external-versions.sh", Path(sys.argv[0]).name))
sys.exit(0)
with open(MAP_PATH, "rb") as f:
MAP = tomllib.load(f)
sys.path.insert(0, str(ROOT / "local/scripts/lib"))
import external_version_lib
external_version_lib.configure(CACHE_DIR, NO_FETCH)
from external_version_lib import (
cache_key_for, fetch, version_sort_key, apply_ver_transform,
resolve_index, render_template, member_url_name, member_template,
read_recipe_text, current_tar_url, current_rev, resolve_git_rev,
)
def template_to_regex(template: str) -> re.Pattern:
"""Convert a tar_template into a regex that captures {ver}/{majmin}/{name}.
Handles templates where the same variable appears more than once: the first
occurrence becomes a named capture group, subsequent ones become
backreferences. {name} is greedy; {ver}/{majmin} are anchored to a leading
digit so '{name}-{ver}' doesn't split 'extra-cmake-modules' wrong.
"""
escaped = re.escape(template)
# re.escape emits '\{ver\}' for '{ver}' on Python 3.7+
for var in ("ver", "majmin", "name"):
token = "\\{" + var + "\\}" # literal \{var\} as it appears post-escape
if token not in escaped:
continue
if var == "name":
capture = r'(?P<name>[^/]+)'
else:
capture = f'(?P<{var}>\\d[^/]*?)'
escaped = escaped.replace(token, capture, 1)
# remaining occurrences → backreference
escaped = escaped.replace(token, f"(?P={var})")
return re.compile(escaped)
def extract_version_vars(url: str, template: str):
"""Reverse-match url against template to recover {ver}/{majmin}."""
try:
rx = template_to_regex(template)
except re.error:
return {}
m = rx.search(url)
if not m:
return {}
d = m.groupdict()
out = {}
if d.get("ver"):
out["ver"] = d["ver"]
if d.get("majmin"):
out["majmin"] = d["majmin"]
return out
# --- main -----------------------------------------------------------------
rows = [] # list of dicts: name, current, latest, status, note
summary = {"current": 0, "outdated": 0, "failed": 0}
# Groups: resolve the group version ONCE, then expand to members.
for gid, g in MAP.get("groups", {}).items():
vars_dict, src, err = resolve_index(
g["version_index"], g["version_regex"], g.get("resolve", "version"))
group_latest = vars_dict["ver"] if vars_dict else "?"
for member in g["members"]:
recipe_path = ROOT / g["recipe_root"] / member
tmpl = member_template(g, member)
uname = member_url_name(g, member)
text = read_recipe_text(recipe_path)
cur_url = current_tar_url(text)
if cur_url is None:
rows.append({
"name": f"{gid}:{member}", "current": "?",
"latest": group_latest, "status": "FAILED",
"note": "no-tar-line",
})
summary["failed"] += 1
continue
# construct what the latest tar URL would be
if vars_dict is None:
status = "FAILED"
note = f"index:{err}"
else:
render_vars = dict(vars_dict)
render_vars["name"] = uname
latest_url = render_template(tmpl, render_vars)
cur_vars = extract_version_vars(cur_url, tmpl)
cur_ver = cur_vars.get("ver", "?")
if latest_url == cur_url:
status = "CURRENT"
note = ""
else:
status = "OUTDATED"
note = ""
rows.append({
"name": f"{gid}:{member}", "current": cur_ver,
"latest": group_latest, "status": status, "note": note,
})
if status == "CURRENT":
summary["current"] += 1
else:
summary["outdated"] += 1
continue
# failed resolution path
cur_vars = extract_version_vars(cur_url, tmpl)
rows.append({
"name": f"{gid}:{member}", "current": cur_vars.get("ver", "?"),
"latest": "?", "status": status, "note": note,
})
summary["failed"] += 1
# Packages (singletons)
for pid, p in MAP.get("packages", {}).items():
recipe_path = ROOT / p["recipe"]
text = read_recipe_text(recipe_path)
kind = p["kind"]
if kind == "tar":
cur_url = current_tar_url(text)
tmpl = p["tar_template"]
cur_vars = extract_version_vars(cur_url or "", tmpl)
cur_ver = cur_vars.get("ver", "?")
vars_dict, src, err = resolve_index(
p["version_index"], p["version_regex"], p.get("resolve", "version"),
p.get("ver_transform"))
if vars_dict is None:
rows.append({
"name": pid, "current": cur_ver, "latest": "?",
"status": "FAILED", "note": f"index:{err}",
})
summary["failed"] += 1
continue
latest_url = render_template(tmpl, {**vars_dict, "name": pid})
status = "CURRENT" if latest_url == cur_url else "OUTDATED"
rows.append({
"name": pid, "current": cur_ver, "latest": vars_dict["ver"],
"status": status, "note": "",
})
if status == "CURRENT":
summary["current"] += 1
else:
summary["outdated"] += 1
elif kind == "git-rev":
cur = current_rev(text) or "?"
resolved, err = resolve_git_rev(p["git_url"], p["tag_regex"])
if resolved is None:
rows.append({
"name": pid, "current": cur[:12], "latest": "?",
"status": "FAILED", "note": f"git:{err}",
})
summary["failed"] += 1
continue
latest_sha = resolved["sha"]
latest_tag = resolved["tag"]
# current may be either a sha or a tag name
is_current = (
cur == latest_sha or cur.startswith(latest_sha[:12]) or
cur == latest_tag
)
status = "CURRENT" if is_current else "OUTDATED"
rows.append({
"name": pid, "current": cur[:12] if len(cur) >= 40 else cur,
"latest": f"{latest_tag}@{latest_sha[:12]}",
"status": status, "note": "",
})
if status == "CURRENT":
summary["current"] += 1
else:
summary["outdated"] += 1
else:
rows.append({
"name": pid, "current": "?", "latest": "?",
"status": "FAILED", "note": f"unknown-kind:{kind}",
})
summary["failed"] += 1
# --- output ---------------------------------------------------------------
if AS_JSON:
payload = {
"summary": summary,
"rows": rows,
}
print(json.dumps(payload, indent=2))
else:
# fixed-width table
name_w = max(len("name"), max((len(r["name"]) for r in rows), default=0))
cur_w = max(len("current"), max((len(str(r["current"])) for r in rows), default=0))
lat_w = max(len("latest"), max((len(str(r["latest"])) for r in rows), default=0))
hdr = (f"{'name':<{name_w}} {'current':<{cur_w}} "
f"{'latest':<{lat_w}} STATUS")
print(hdr)
print("-" * len(hdr))
for r in rows:
note = f" ({r['note']})" if r.get("note") else ""
print(f"{r['name']:<{name_w}} {str(r['current']):<{cur_w}} "
f"{str(r['latest']):<{lat_w}} {r['status']}{note}")
print()
print(f"summary: current={summary['current']} "
f"outdated={summary['outdated']} failed={summary['failed']}")
if NO_FETCH:
print("(from cache; --no-fetch)")
exit_code = 0
if STRICT and (summary["outdated"] > 0 or summary["failed"] > 0):
exit_code = 1
sys.exit(exit_code)
PYEOF
local/scripts/install-git-hooks.sh
Executable
+200
View File
@@ -0,0 +1,200 @@
#!/usr/bin/env bash
# install-git-hooks.sh — Idempotent installer for Red Bear OS optional git hooks.
#
# Hooks in Red Bear OS are ALWAYS opt-in (per AGENTS.md "absolutely NEVER
# DELETE, NEVER IGNORE" rule — operators explicitly choose to install them).
# This script is a convenience wrapper around the manual `cp` instructions
# documented in local/docs/HOOKS.md. It never silently clobbers an existing
# hook: if a hook file already exists and differs from the canonical copy,
# it is backed up to `<name>.bak` before the new one is written, and a
# warning is printed.
#
# Usage:
# ./local/scripts/install-git-hooks.sh # install post-checkout only (default)
# ./local/scripts/install-git-hooks.sh --all # post-checkout + pre-push + commit-msg
# ./local/scripts/install-git-hooks.sh --uninstall # remove hooks installed by this script
# ./local/scripts/install-git-hooks.sh --uninstall --all
#
# Exit codes:
# 0 success (or already-installed, or nothing to uninstall)
# 1 fatal misconfiguration (not a git repo, scripts missing, etc.)
#
# See: local/docs/HOOKS.md (hook inventory + bypass instructions),
# local/docs/RELEASE-BUMP-WORKFLOW.md (release-bump pipeline).
set -euo pipefail
# ---------------------------------------------------------------------------
# Resolve repo root. Must be a Red Bear OS checkout (has local/scripts/).
# ---------------------------------------------------------------------------
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [[ -z "$ROOT" ]]; then
echo "install-git-hooks: not inside a git work tree" >&2
exit 1
fi
SCRIPTS_DIR="$ROOT/local/scripts"
HOOKS_DIR="$(git rev-parse --git-path hooks 2>/dev/null)"
# `git rev-parse --git-path hooks` resolves correctly for linked worktrees
# and custom $GIT_DIR. Fall back to the conventional path if it fails.
if [[ -z "$HOOKS_DIR" ]]; then
GIT_DIR="$(git rev-parse --git-dir 2>/dev/null || echo .git)"
[[ "$GIT_DIR" = /* ]] || GIT_DIR="$ROOT/$GIT_DIR"
HOOKS_DIR="$GIT_DIR/hooks"
fi
mkdir -p "$HOOKS_DIR"
# ---------------------------------------------------------------------------
# Parse flags.
# ---------------------------------------------------------------------------
INSTALL_ALL=0
UNINSTALL=0
for arg in "$@"; do
case "$arg" in
--all) INSTALL_ALL=1 ;;
--uninstall) UNINSTALL=1 ;;
-h|--help)
sed -n '1,30p' "$0"
exit 0
;;
*)
echo "install-git-hooks: unknown option: $arg" >&2
echo " usage: $0 [--all] [--uninstall]" >&2
exit 1
;;
esac
done
# ---------------------------------------------------------------------------
# Hook manifest. The default install set is intentionally minimal (only
# post-checkout) so that a bare `install-git-hooks.sh` does the least
# surprising thing. `--all` extends to the pre-push and commit-msg hooks.
# ---------------------------------------------------------------------------
# Each entry: <hook-name>;<source-script-relative-to-local/scripts>
HOOKS_DEFAULT=(
"post-checkout;post-checkout-version-sync.sh"
)
HOOKS_EXTRA=(
"pre-push;pre-push-checks.sh"
"commit-msg;commit-msg"
)
# Build the active set from the flags.
ACTIVE_HOOKS=()
if [[ $UNINSTALL -eq 1 ]]; then
# Uninstall needs to know the full set that --all would have installed,
# so a default-install-then---all-uninstall cleans everything up.
for entry in "${HOOKS_DEFAULT[@]}" "${HOOKS_EXTRA[@]}"; do
ACTIVE_HOOKS+=("$entry")
done
if [[ $INSTALL_ALL -eq 0 ]]; then
# Without --all, only uninstall the default set (the ones a default
# install would have placed). Operators who used --all should pass
# --uninstall --all.
ACTIVE_HOOKS=("${HOOKS_DEFAULT[@]}")
fi
else
ACTIVE_HOOKS=("${HOOKS_DEFAULT[@]}")
[[ $INSTALL_ALL -eq 1 ]] && ACTIVE_HOOKS+=("${HOOKS_EXTRA[@]}")
fi
# ---------------------------------------------------------------------------
# Install / uninstall loop.
# ---------------------------------------------------------------------------
install_one() {
local hook_name="$1"
local src_rel="$2"
local src="$SCRIPTS_DIR/$src_rel"
local dst="$HOOKS_DIR/$hook_name"
if [[ ! -f "$src" ]]; then
echo "install-git-hooks: WARN source script not found: $src_rel — skipping $hook_name" >&2
return 0
fi
if [[ -e "$dst" || -L "$dst" ]]; then
# Content-based, not path-based: a copied hook has a different path
# than the source, so only byte comparison detects "already current".
if cmp -s "$dst" "$src" 2>/dev/null; then
echo "install-git-hooks: $hook_name already up to date ($dst)"
chmod +x "$dst" 2>/dev/null || true
return 0
fi
# Existing file differs — back it up before overwriting.
local bak="${dst}.bak"
# Rotate any pre-existing .bak so we don't destroy the first backup.
if [[ -e "$bak" ]]; then
local i=1
while [[ -e "${dst}.bak.${i}" && $i -le 100 ]]; do i=$((i+1)); done
if [[ $i -gt 100 ]]; then
bak="${dst}.bak.$(date +%Y%m%d%H%M%S)-$$"
else
bak="${dst}.bak.${i}"
fi
fi
echo "install-git-hooks: WARN existing $hook_name differs — backing up to $bak" >&2
mv "$dst" "$bak"
fi
cp "$src" "$dst"
chmod +x "$dst"
echo "install-git-hooks: installed $hook_name -> $dst"
}
uninstall_one() {
local hook_name="$1"
local dst="$HOOKS_DIR/$hook_name"
if [[ ! -e "$dst" && ! -L "$dst" ]]; then
echo "install-git-hooks: $hook_name not present ($dst) — nothing to remove"
return 0
fi
rm -f "$dst"
echo "install-git-hooks: removed $hook_name ($dst)"
# Restore the most recent backup if one exists, so uninstall is reversible.
if [[ -e "${dst}.bak" ]]; then
mv "${dst}.bak" "$dst"
chmod +x "$dst" 2>/dev/null || true
echo "install-git-hooks: restored previous $hook_name from ${dst}.bak"
fi
}
echo "=== Red Bear OS git hook installer ==="
echo " repo root: $ROOT"
echo " hooks dir: $HOOKS_DIR"
echo " mode: $([[ $UNINSTALL -eq 1 ]] && echo 'uninstall' || echo 'install') ($([[ $INSTALL_ALL -eq 1 ]] && echo 'all hooks' || echo 'default: post-checkout only'))"
echo ""
if [[ $UNINSTALL -eq 1 ]]; then
for entry in "${ACTIVE_HOOKS[@]}"; do
hook_name="${entry%%;*}"
uninstall_one "$hook_name"
done
echo ""
echo "Uninstall complete. Remaining hooks (if any):"
ls -1 "$HOOKS_DIR" 2>/dev/null | grep -vE '\.bak(\.[0-9]+)?$' || echo " (none)"
exit 0
fi
for entry in "${ACTIVE_HOOKS[@]}"; do
hook_name="${entry%%;*}"
src_rel="${entry##*;}"
install_one "$hook_name" "$src_rel"
done
echo ""
echo "=== Install complete ==="
echo ""
echo "Installed hooks are OPT-IN and never auto-commit, auto-push, or block"
echo "checkouts. Bypass / configuration:"
echo " post-checkout: REDBEAR_NO_AUTO_SYNC=1 git checkout <branch>"
echo " pre-push: REDBEAR_SKIP_PRE_PUSH=1 git push (or git push --no-verify)"
echo ""
echo "Full documentation: local/docs/HOOKS.md"
echo "Release-bump runbook: local/docs/RELEASE-BUMP-WORKFLOW.md"
local/scripts/lib/external_version_lib.py
+213
View File
@@ -0,0 +1,213 @@
"""Shared upstream-version resolution helpers for Red Bear external-package tooling.
Consumers (add local/scripts/lib to sys.path, then import):
- local/scripts/check-external-versions.sh (embedded Python)
- local/scripts/bump-graphics-recipes.sh (embedded Python)
Single source of truth for: index fetching (cache + fallback), version sort
keys, sed-style version transforms, index resolution, tar-template rendering,
group member helpers, recipe.toml source pins, and git ls-remote tag
resolution. Behavior changes belong here, not in the consumers.
"""
import hashlib
import os
import re
import subprocess
from pathlib import Path
from typing import Optional
_CACHE_DIR: Optional[Path] = None
_NO_FETCH = False
def configure(cache_dir: Path, no_fetch: bool) -> None:
global _CACHE_DIR, _NO_FETCH
_CACHE_DIR = Path(cache_dir)
_NO_FETCH = bool(no_fetch)
def cache_key_for(url: str) -> Path:
assert _CACHE_DIR is not None, "configure() must be called before fetch use"
safe = re.sub(r'[^a-zA-Z0-9_.-]', '_', url)
if len(safe) > 180:
# Stable hash (blake2b) — NOT Python's hash(), which is randomized
# per-process via PYTHONHASHSEED and would orphan cache entries.
digest = hashlib.blake2b(url.encode(), digest_size=8).hexdigest()
safe = safe[:80] + "___" + digest + "___" + safe[-80:]
return _CACHE_DIR / f"{safe}.cache"
def fetch(url: str):
"""Return (text, source); source is fetch/cache/cache-fallback*/error."""
assert _CACHE_DIR is not None, "configure() must be called before fetch use"
cache = cache_key_for(url)
if _NO_FETCH:
if cache.exists():
return cache.read_text(errors="replace"), "cache"
return None, "no-cache"
try:
result = subprocess.run(
["curl", "-sLf", "--retry", "3", "--retry-delay", "2",
"--connect-timeout", "20", "--max-time", "90", url],
capture_output=True, text=True, timeout=120,
)
if result.returncode == 0 and result.stdout:
_CACHE_DIR.mkdir(parents=True, exist_ok=True)
tmp = cache.with_suffix(cache.suffix + ".tmp")
tmp.write_text(result.stdout)
os.replace(str(tmp), str(cache))
return result.stdout, "fetch"
if cache.exists():
return cache.read_text(errors="replace"), "cache-fallback"
err = (result.stderr or "").strip()
return None, err or f"curl-exit-{result.returncode}"
except subprocess.TimeoutExpired:
if cache.exists():
return cache.read_text(errors="replace"), "cache-fallback-timeout"
return None, "timeout"
except Exception as e:
if cache.exists():
return cache.read_text(errors="replace"), "cache-fallback-error"
return None, str(e)
def version_sort_key(v: str):
"""Sort key that orders pre-releases BEFORE the corresponding final.
Handles forms like 1.0.0, 1.0.0-rc1, 1.0.0-alpha1, 1.0.0-beta2.
A final release sorts after any pre-release of the same X.Y.Z base.
"""
m = re.match(r'^(\d+(?:[._]\d+)*)(?:[-.]([0-9A-Za-z]+))?$', v)
if not m:
parts = re.split(r'[._+-]', v)
return [(1, 0, p) if not p.isdigit() else (0, int(p), "") for p in parts]
main_str, pre = m.group(1), m.group(2)
main_key = [(0, int(p), "") if p.isdigit() else (1, 0, p)
for p in re.split(r'[._]', main_str)]
if pre is None or pre == "":
pre_key = [(1, 0, "zzzzzz_final")]
else:
pre_key = [(-1, 0, pre)]
return main_key + pre_key
def apply_ver_transform(ver: str, transform: str) -> str:
"""Apply a sed-style transform like 's/-/./g' to a version token."""
m = re.match(r'^s/(.*)/(.*)/([gi]*)$', transform)
if not m:
return ver
pat, repl, flags = m.group(1), m.group(2), m.group(3)
count = 0 if 'g' in flags else 1
try:
return re.sub(pat, repl, ver, count=count)
except re.error:
return ver
def resolve_index(index_url: str, regex_str: str, resolve_kind: str,
ver_transform: Optional[str] = None):
"""Resolve latest version vars from an index page -> (vars, source, error)."""
text, src = fetch(index_url)
if text is None:
return None, src, "fetch-failed"
try:
pattern = re.compile(regex_str)
except re.error as e:
return None, src, f"bad-regex: {e}"
matches = pattern.findall(text)
if not matches:
return None, src, "no-match"
norm = []
for mm in matches:
if isinstance(mm, tuple):
mm = "-".join(str(x) for x in mm)
norm.append(mm)
unique = list(dict.fromkeys(norm))
unique.sort(key=version_sort_key, reverse=True)
best = unique[0]
if ver_transform:
best = apply_ver_transform(best, ver_transform)
if resolve_kind == "majmin":
return {"majmin": best, "ver": f"{best}.0"}, src, None
parts = best.split(".")
majmin = ".".join(parts[:2]) if len(parts) >= 2 else best
return {"majmin": majmin, "ver": best}, src, None
def render_template(template: str, variables: dict) -> str:
out = template
for k in ("ver", "majmin", "name"):
out = out.replace("{" + k + "}", variables.get(k, ""))
return out
def member_url_name(group: dict, member: str) -> str:
overrides = group.get("member_url_names", {}) or {}
if member in overrides:
return overrides[member]
strip = group.get("url_name_strip_prefix", "") or ""
if strip and member.startswith(strip):
return member[len(strip):]
return member
def member_template(group: dict, member: str) -> str:
mts = group.get("member_templates", {}) or {}
if member in mts and "tar_template" in mts[member]:
return mts[member]["tar_template"]
return group["tar_template"]
def read_recipe_text(recipe_path: Path) -> str:
toml = recipe_path / "recipe.toml"
if not toml.exists():
return ""
return toml.read_text(errors="replace")
def current_tar_url(text: str):
m = re.search(r'^\s*tar\s*=\s*"([^"]+)"', text, re.MULTILINE)
return m.group(1) if m else None
def current_rev(text: str):
m = re.search(r'^\s*rev\s*=\s*"([^"]+)"', text, re.MULTILINE)
return m.group(1) if m else None
def resolve_git_rev(git_url: str, tag_regex_str: str):
"""Resolve latest tag + commit sha via git ls-remote --tags."""
try:
result = subprocess.run(
["git", "ls-remote", "--tags", git_url],
capture_output=True, text=True, timeout=90,
)
except subprocess.TimeoutExpired:
return None, "timeout"
except Exception as e:
return None, str(e)
if result.returncode != 0:
return None, (result.stderr or "git-error").strip()
try:
pattern = re.compile(tag_regex_str)
except re.error as e:
return None, f"bad-regex: {e}"
shas = {}
for line in result.stdout.splitlines():
parts = line.split("\t")
if len(parts) != 2:
continue
sha, ref = parts
if not ref.startswith("refs/tags/"):
continue
tag = ref[len("refs/tags/"):]
deref = tag.endswith("^{}")
clean = tag[:-3] if deref else tag
if pattern.fullmatch(clean):
if clean not in shas or deref:
shas[clean] = sha.strip()
if not shas:
return None, "no-tags-matched"
best = max(shas.keys(), key=version_sort_key)
return {"tag": best, "sha": shas[best]}, None
local/scripts/post-checkout-version-sync.sh
Executable
+214
View File
@@ -0,0 +1,214 @@
#!/usr/bin/env bash
# post-checkout-version-sync.sh — Optional post-checkout hook for Red Bear OS.
#
# Purpose:
# When an operator switches to a release branch whose name is an anchored
# semver (e.g. `0.3.1`, `0.3.2`), automatically synchronise every Cat 1
# (in-house) and Cat 2 (upstream fork) crate version label to match that
# branch. This is the label-only half of a release bump — it does NOT
# touch fork source content, regenerate lockfiles, or commit anything.
#
# Why label-only:
# Labels are cheap and mechanical (sed rewrites performed by
# sync-versions.sh). Source upgrades (rebasing a fork onto a newer upstream
# tag) and lockfile regen are expensive and operator-judgement steps, so
# they remain explicit (`bump-release.sh --with-sources --with-external`
# and `sync-versions.sh --regen`). See
# local/docs/RELEASE-BUMP-WORKFLOW.md for the full release model.
#
# Git post-checkout contract (git invokes this hook with three arguments):
# $1 = previous HEAD sha
# $2 = new HEAD sha
# $3 = 1 for a branch checkout, 0 for a file checkout
#
# Install (operator opt-in — NEVER auto-installed):
# cp local/scripts/post-checkout-version-sync.sh .git/hooks/post-checkout
# chmod +x .git/hooks/post-checkout
#
# Or via the installer (preferred):
# ./local/scripts/install-git-hooks.sh # post-checkout only
# ./local/scripts/install-git-hooks.sh --all # + pre-push + commit-msg
#
# Bypass for a single checkout:
# REDBEAR_NO_AUTO_SYNC=1 git checkout 0.3.2
#
# Hard guarantees (a failing hook MUST NEVER block a checkout):
# - Always exits 0. Any internal failure degrades to a stderr note + exit 0.
# - No network access. No git mutations (no commit / branch / push / stash).
# - No lockfile regeneration (NEVER calls `sync-versions.sh --regen`).
# - Does not run on non-semver branches (master, submodule/*, recovered/*,
# detached HEAD).
# - Does not run inside an in-progress rebase / cherry-pick / merge.
# - Does not run when the working tree is dirty (warns + skips).
# - Total runtime budget: < ~5 seconds on a warm tree.
#
# See: local/docs/HOOKS.md, local/docs/RELEASE-BUMP-WORKFLOW.md,
# local/AGENTS.md § "Version conventions — two categories".
# NOTE: deliberately `set -uo pipefail` and NOT `-e`. A post-checkout hook
# must never abort mid-way and leave the operator staring at a non-zero exit
# that blocks the checkout. Every fallible step is either guarded or wrapped
# so that failure degrades to an exit 0 with a stderr note.
set -uo pipefail
# ---------------------------------------------------------------------------
# Locate the repo root. We prefer `git rev-parse --show-toplevel` over a
# dirname-relative derivation because git places the hook under `.git/hooks/`
# which may itself be a gitdir pointer (worktrees, linked checkouts).
# ---------------------------------------------------------------------------
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || true)"
if [[ -z "$ROOT" ]]; then
# Not inside a git work tree — nothing to do.
exit 0
fi
# Git passes exactly three arguments to post-checkout. If invoked manually
# without them, treat $3 as unknown (skip the branch-flag guard).
BRANCH_FLAG="${3:-}"
# ---------------------------------------------------------------------------
# Guard (a): branch checkouts only. $3 == 1 means a branch checkout;
# $3 == 0 means a file checkout (e.g. `git checkout -- path`). Silently
# skip file checkouts — there is no branch context to sync against.
# ---------------------------------------------------------------------------
if [[ "$BRANCH_FLAG" != "1" ]]; then
exit 0
fi
# ---------------------------------------------------------------------------
# Guard (b): the new branch name MUST be an anchored semver
# `^[0-9]+\.[0-9]+\.[0-9]+$`. This silently excludes master, main,
# submodule/* branches, recovered/* branches, and detached HEAD.
# ---------------------------------------------------------------------------
BRANCH="$(git branch --show-current 2>/dev/null || true)"
if [[ -z "$BRANCH" ]]; then
# Detached HEAD — no branch name to anchor a version against.
exit 0
fi
if ! printf '%s' "$BRANCH" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
# Non-semver branch (master, submodule/*, recovered/*, feature/*, …).
# Silently skip — version labels are only meaningful on release branches.
exit 0
fi
# ---------------------------------------------------------------------------
# Guard (c): never run inside an in-progress rebase / cherry-pick / merge.
# Touching version labels mid-sequence would corrupt the operator's
# in-flight history rewrite. Silently skip in all such states.
# ---------------------------------------------------------------------------
GIT_DIR="$(git rev-parse --git-dir 2>/dev/null || true)"
if [[ -n "$GIT_DIR" ]]; then
# Resolve relative to the worktree root for linked worktrees.
[[ "$GIT_DIR" = /* ]] || GIT_DIR="$ROOT/$GIT_DIR"
for marker in rebase-merge rebase-apply CHERRY_PICK_HEAD MERGE_HEAD; do
if [[ -e "$GIT_DIR/$marker" ]]; then
# An interactive operation is in progress — stay out of the way.
exit 0
fi
done
fi
# ---------------------------------------------------------------------------
# Guard (d): honour the explicit opt-out env var.
# ---------------------------------------------------------------------------
if [[ -n "${REDBEAR_NO_AUTO_SYNC:-}" ]]; then
exit 0
fi
# ---------------------------------------------------------------------------
# Guard (e): working tree MUST be clean. A bump rewrites Cargo.toml files in
# place; running it over uncommitted operator edits would silently swallow
# those edits into the bump. Warn + skip instead.
# ---------------------------------------------------------------------------
if ! git -C "$ROOT" diff --quiet --ignore-submodules HEAD 2>/dev/null \
|| [[ -n "$(git -C "$ROOT" status --porcelain 2>/dev/null)" ]]; then
echo "post-checkout-version-sync: skipping version auto-sync: uncommitted changes present" >&2
echo " (commit or stash, then run: ./local/scripts/sync-versions.sh --no-regen)" >&2
exit 0
fi
# ---------------------------------------------------------------------------
# All guards passed. Compare the root cookbook Cargo.toml `[package] version`
# against the branch version. They are the canonical indicator of whether a
# label sync is needed: sync-versions.sh Phase 0 keeps them in lock-step,
# so if the root is already at the branch version, every Cat 1/Cat 2 label
# is already correct too.
# ---------------------------------------------------------------------------
ROOT_CARGO="$ROOT/Cargo.toml"
if [[ ! -f "$ROOT_CARGO" ]]; then
# Not a Red Bear OS checkout (no root Cargo.toml). Stay silent.
exit 0
fi
CURRENT_VERSION=""
if grep -qE '^\[package\]' "$ROOT_CARGO" 2>/dev/null; then
# Must match sync-versions.sh Phase 0's extraction so both scripts agree.
CURRENT_VERSION="$(awk '
/^\[package\]/ { in_pkg=1; next }
/^\[/ { in_pkg=0 }
in_pkg && /^version[[:space:]]*=/ { print; exit }
' "$ROOT_CARGO" 2>/dev/null | sed 's/.*"\(.*\)".*/\1/' || true)"
fi
if [[ "$CURRENT_VERSION" == "$BRANCH" ]]; then
# Labels already synced for this branch — nothing to do.
exit 0
fi
# ---------------------------------------------------------------------------
# Drift detected. Run the label-only sync (NEVER --regen — lockfile regen is
# an explicit operator step). sync-versions.sh is the single executor of
# suffix-only label rewrites; this hook never duplicates that sed logic.
# ---------------------------------------------------------------------------
SYNC="$ROOT/local/scripts/sync-versions.sh"
if [[ ! -x "$SYNC" ]]; then
echo "post-checkout-version-sync: $SYNC not found or not executable — skipping label sync" >&2
echo " run manually: ./local/scripts/sync-versions.sh --no-regen" >&2
exit 0
fi
echo "post-checkout-version-sync: branch '$BRANCH' root Cargo.toml at '$CURRENT_VERSION' -> '$BRANCH'"
echo "post-checkout-version-sync: running sync-versions.sh --no-regen (labels only, no lockfile regen)"
# Run label sync. Capture output so the operator sees a concise summary.
# We do NOT propagate a non-zero exit: even if sync fails, the checkout
# itself must not be blocked.
SYNC_OUTPUT="$("$SYNC" --no-regen 2>&1)" || true
if [[ -n "$SYNC_OUTPUT" ]]; then
printf '%s\n' "$SYNC_OUTPUT"
fi
echo "post-checkout-version-sync: label sync complete for branch '$BRANCH'."
# ---------------------------------------------------------------------------
# Follow-up hint. The hook handles ONLY mechanical label rewrites. The
# expensive half of a release bump — fork source upgrades to newer upstream
# tags and external desktop-stack version refresh — is the operator's
# explicit decision and is never triggered automatically.
# ---------------------------------------------------------------------------
cat >&2 <<EOF
post-checkout-version-sync: follow-up steps (NOT run automatically):
- source upgrades (Cat 2 forks to newer upstream tags):
./local/scripts/bump-release.sh --with-sources --with-external
- lockfile regeneration (after labels settle):
./local/scripts/sync-versions.sh --regen
- check-only verification:
./local/scripts/sync-versions.sh --check
See local/docs/RELEASE-BUMP-WORKFLOW.md for the full runbook.
EOF
# Always exit 0. A hook that blocks the checkout is strictly worse than a
# hook that no-ops; git treats a non-zero exit from post-checkout as a
# warning and clutters the operator's terminal on every checkout.
exit 0
local/scripts/refresh-fork-upstream-map.sh
+292 -43
View File
@@ -1,8 +1,33 @@
#!/usr/bin/env bash
# refresh-fork-upstream-map.sh — Update local/fork-upstream-map.toml
# with the current upstream URLs and latest release tags for each
# Cat 2 fork. This is the canonical source of truth for
# verify-fork-versions.sh.
# refresh-fork-upstream-map.sh — Update column 3 (the upstream release tag)
# of local/fork-upstream-map.toml in place, preserving every other byte.
#
# Policy (release-bump-pipeline.md, Deliverable C):
# - APPEND-ONLY over column 3 (the upstream release tag).
# - Preserve columns 1 (fork), 2 (url), 4 (mode) and ALL comments / header
# text byte-for-byte. The curated header comments and the `base` row are
# never dropped.
# - SKIP ls-remote for rows with mode `diverged` — their rebase is operator
# manual (Phase 2.4+); the map tag documents the CURRENT base, not the
# latest upstream.
# - SKIP ls-remote for rows whose tag is `main` (the `base` row, which has
# no upstream releases).
# - For forks present in local/sources/*/ with an upstream remote but
# MISSING from the map: APPEND a new row
# `<fork> <url> <latest-tag> # TODO: classify mode`
# at the end of the file.
#
# Usage:
# ./local/scripts/refresh-fork-upstream-map.sh # Update col 3 in place
# ./local/scripts/refresh-fork-upstream-map.sh --check # Read-only report, exit 1 on drift
# ./local/scripts/refresh-fork-upstream-map.sh --no-fetch # Use map's current tags (no network)
# ./local/scripts/refresh-fork-upstream-map.sh --help # This header
#
# Acceptance:
# - `git diff local/fork-upstream-map.toml` shows only column-3 token changes
# (if any) plus any newly appended rows.
# - `grep -c '^base\b' local/fork-upstream-map.toml` = 1.
# - `grep -c diverged local/fork-upstream-map.toml` unchanged vs before.
set -euo pipefail
@@ -11,45 +36,269 @@ cd "$ROOT"
MAP="$ROOT/local/fork-upstream-map.toml"
# fork_name upstream_url latest_tag
declare -A FORKS=(
[syscall]="https://gitlab.redox-os.org/redox-os/syscall.git"
[libredox]="https://gitlab.redox-os.org/redox-os/libredox.git"
[redoxfs]="https://gitlab.redox-os.org/redox-os/redoxfs.git"
[redox-scheme]="https://gitlab.redox-os.org/redox-os/redox-scheme.git"
[relibc]="https://gitlab.redox-os.org/redox-os/relibc.git"
[kernel]="https://gitlab.redox-os.org/redox-os/kernel.git"
[bootloader]="https://gitlab.redox-os.org/redox-os/bootloader.git"
[installer]="https://gitlab.redox-os.org/redox-os/installer.git"
[userutils]="https://gitlab.redox-os.org/redox-os/userutils.git"
)
CHECK_ONLY=0
NO_FETCH=0
{
echo "# fork-upstream-map.toml — Auto-generated by"
echo "# local/scripts/refresh-fork-upstream-map.sh. Do not edit by hand."
echo "# Format: <fork-name> <upstream-git-url> <upstream-release-tag>"
for arg in "$@"; do
case "$arg" in
--check) CHECK_ONLY=1 ;;
--no-fetch) NO_FETCH=1 ;;
-h|--help) sed -n '2,35p' "$0"; exit 0 ;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
if [[ ! -f "$MAP" ]]; then
echo "ERROR: map file not found: $MAP" >&2
exit 1
fi
# Anchored semver regex (release-bump-pipeline.md design constraint #2).
SEMVER_RE='^[0-9]+\.[0-9]+\.[0-9]+$'
# ---- latest_tag_for_url <url> [fork] [base] --------------------------------
# Resolve the newest GENUINE upgrade tag for a remote via `git ls-remote`.
# Output: "<tag>|<raw-count>" — tag empty when nothing qualifies.
# Two filters: (a) tags already reachable from the fork's HEAD are history,
# not newer releases (relibc's 0.6.0/2020 outranks its newer 0.2.x API line
# lexically); (b) tags not version-greater than the current base are not
# newer either (the base tag itself is usually an ancestor of HEAD).
latest_tag_for_url() {
local url="$1" fork="${2:-}" base="${3:-}"
local tag forkdir="" raw=0
[[ -n "$fork" && -d "$ROOT/local/sources/$fork/.git" ]] \
&& forkdir="$ROOT/local/sources/$fork"
while IFS= read -r tag; do
[[ -z "$tag" ]] && continue
raw=$((raw + 1))
if [[ -n "$forkdir" ]] \
&& git -C "$forkdir" rev-parse --verify --quiet "refs/tags/$tag" >/dev/null 2>&1 \
&& git -C "$forkdir" merge-base --is-ancestor "refs/tags/$tag" HEAD 2>/dev/null; then
continue
fi
if [[ -n "$base" ]] && ! version_greater "$tag" "$base"; then
continue
fi
echo "${tag}|${raw}"
return 0
done < <(git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \
| awk '{print $2}' \
| sed 's|refs/tags/||' \
| sed 's|\^{}$||' \
| grep -E "$SEMVER_RE" || true)
echo "|${raw}"
return 0
}
# version_greater <a> <b> → true iff a is version-greater than b (sort -V).
version_greater() {
[[ "$1" != "$2" ]] \
&& [[ "$(printf '%s\n%s\n' "$2" "$1" | sort -V | tail -1)" == "$1" ]]
}
# ---- parse existing map into associative arrays ----------------------------
# Sets MAP_FORK_URL[fork]=url, MAP_FORK_TAG[fork]=tag, MAP_FORK_MODE[fork]=mode.
declare -A MAP_FORK_URL=()
declare -A MAP_FORK_TAG=()
declare -A MAP_FORK_MODE=()
# Match a data row: first non-whitespace, non-# token = fork; followed by
# whitespace-separated url, tag, optional mode + trailing comment.
# We use awk to emit TAB-separated columns; bash reads them in.
while IFS=$'\t' read -r fork url tag mode; do
[[ -z "$fork" ]] && continue
MAP_FORK_URL["$fork"]="$url"
MAP_FORK_TAG["$fork"]="$tag"
MAP_FORK_MODE["$fork"]="$mode"
done < <(awk '
/^[^#[:space:]]/ {
fork = $1
url = $2
tag = $3
mode = $4
sub(/[ \t]*#.*/, "", mode)
print fork "\t" url "\t" tag "\t" mode
}
' "$MAP")
# ---- collect on-disk forks with upstream remote ----------------------------
declare -A DISK_FORK_URL=()
for d in "$ROOT"/local/sources/*/; do
[[ -d "$d/.git" ]] || continue
name=$(basename "$d")
url=$(git -C "$d" remote get-url upstream 2>/dev/null || true)
[[ -n "$url" ]] && DISK_FORK_URL["$name"]="$url"
done
# ---- plan column-3 updates + appends ---------------------------------------
declare -A NEW_TAG_FOR=()
declare -a APPEND_ROWS=()
DRIFT_COUNT=0
UPDATE_COUNT=0
echo "=== refresh-fork-upstream-map ==="
if [[ "$NO_FETCH" -eq 1 ]]; then
echo "(--no-fetch: skipping git ls-remote; using current map tags)"
fi
echo ""
# Iterate in map-order (preserve stable output for --check reports).
MAP_ORDER=()
while IFS= read -r line; do
if [[ "$line" =~ ^[^#[:space:]] ]]; then
fork=$(awk '{print $1}' <<<"$line")
MAP_ORDER+=("$fork")
fi
done < <(awk 1 "$MAP")
for fork in "${MAP_ORDER[@]}"; do
url="${MAP_FORK_URL[$fork]:-}"
current="${MAP_FORK_TAG[$fork]:-}"
mode="${MAP_FORK_MODE[$fork]:-}"
# Skip the base row (tag=main).
if [[ "$current" == "main" ]]; then
echo "fork=$fork mode=$mode current=$current latest=(skip) action=skip-main"
continue
fi
# Skip diverged forks: tag documents CURRENT base.
if [[ "$mode" == "diverged" ]]; then
echo "fork=$fork mode=diverged current=$current latest=(skip) action=skip-diverged"
continue
fi
latest=""
if [[ "$NO_FETCH" -eq 0 ]]; then
result=$(latest_tag_for_url "$url" "$fork" "$current")
latest="${result%%|*}"
raw="${result##*|}"
# Tags existed upstream but none qualified as newer → already current.
[[ -z "$latest" && "$raw" -gt 0 ]] && latest="$current"
fi
if [[ -z "$latest" ]]; then
echo "fork=$fork mode=$mode current=$current latest=(unknown) action=skip-no-tag"
continue
fi
if [[ "$latest" != "$current" ]]; then
NEW_TAG_FOR["$fork"]="$latest"
if [[ $CHECK_ONLY -eq 1 ]]; then
echo "fork=$fork mode=$mode current=$current latest=$latest action=DRIFT"
DRIFT_COUNT=$((DRIFT_COUNT + 1))
else
echo "fork=$fork mode=$mode current=$current latest=$latest action=UPDATE"
UPDATE_COUNT=$((UPDATE_COUNT + 1))
fi
else
echo "fork=$fork mode=$mode current=$current latest=$latest action=ok"
fi
done
# Forks on disk but missing from the map → append.
# Iterate in sorted order for stable output.
for fork in $(printf '%s\n' "${!DISK_FORK_URL[@]}" | sort); do
if [[ -z "${MAP_FORK_URL[$fork]:-}" ]]; then
url="${DISK_FORK_URL[$fork]}"
latest=""
if [[ "$NO_FETCH" -eq 0 ]]; then
result=$(latest_tag_for_url "$url" "$fork")
latest="${result%%|*}"
fi
latest="${latest:-unknown}"
row="$fork $url $latest # TODO: classify mode"
if [[ $CHECK_ONLY -eq 1 ]]; then
echo "fork=$fork mode=(missing) current=(none) latest=$latest action=DRIFT-MISSING"
DRIFT_COUNT=$((DRIFT_COUNT + 1))
else
echo "fork=$fork mode=(missing) current=(none) latest=$latest action=APPEND"
APPEND_ROWS+=("$row")
fi
fi
done
# ---- --check exit ----------------------------------------------------------
if [[ $CHECK_ONLY -eq 1 ]]; then
echo ""
for fork in syscall libredox redoxfs redox-scheme relibc kernel bootloader installer userutils; do
url="${FORKS[$fork]:-}"
if [ -z "$url" ]; then
echo "WARN: no upstream URL for $fork, skipping" >&2
continue
fi
# Get latest tag matching the project's version scheme
# (e.g. 0.9.0, 0.1.18, 0.5.12, 1.0.0). Exclude pre-release and
# non-stable tags.
latest=$(cd /tmp && git ls-remote --tags --sort=-v:refname "$url" 2>/dev/null \
| awk '{print $2}' \
| sed 's|refs/tags/||' \
| grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' \
| head -1)
if [ -z "$latest" ]; then
echo "WARN: couldn't determine latest tag for $fork from $url" >&2
continue
fi
printf "%-13s %-55s %s\n" "$fork" "$url" "$latest"
done
} > "$MAP"
echo "Summary: drift=$DRIFT_COUNT"
if [[ $DRIFT_COUNT -gt 0 ]]; then
echo "FAIL: $DRIFT_COUNT drift(s) detected" >&2
exit 1
fi
echo "OK: no drift"
exit 0
fi
echo "Updated $MAP"
cat "$MAP"
# ---- apply column-3 rewrites in place (byte-exact awk) ---------------------
if [[ ${#NEW_TAG_FOR[@]} -eq 0 && ${#APPEND_ROWS[@]} -eq 0 ]]; then
echo ""
echo "No changes needed ($MAP already current)"
exit 0
fi
TMP="${MAP}.tmp.$$"
APPEND_FILE=""
trap 'rm -f "$TMP" "$APPEND_FILE"' EXIT
# Sidecar file with append rows (avoids awk quoting issues).
if [[ ${#APPEND_ROWS[@]} -gt 0 ]]; then
APPEND_FILE="${TMP}.append"
: > "$APPEND_FILE"
for row in "${APPEND_ROWS[@]}"; do
printf '%s\n' "$row" >> "$APPEND_FILE"
done
fi
# Two-pass awk:
# pass 1 (control): UPDATES[fork] = newtag
# pass 2 (map): rewrite column 3 in matching rows; everything else
# is printed verbatim. Append sidecar rows at END.
#
# The column-3 rewrite uses match() to locate the prefix
# `^fork<ws>url<ws>` and the next non-whitespace run (the tag), so all
# leading whitespace, the URL, and everything after the tag (mode,
# comments, EOL) are preserved byte-for-byte.
#
# A sentinel record guarantees the control stream always has at least one
# line. Without it, when CONTROL is empty (only appends, no col-3
# rewrites), awk's `NR == FNR` test would misidentify every map line as a
# control line and corrupt the output.
CONTROL=$'#__rbum_sentinel__\t0.0.0\n'
for fork in "${!NEW_TAG_FOR[@]}"; do
CONTROL+="${fork}"$'\t'"${NEW_TAG_FOR[$fork]}"$'\n'
done
awk -v APPEND_FILE="$APPEND_FILE" '
NR == FNR {
UPDATES[$1] = $2
next
}
{
if ($0 ~ /^[^#[:space:]]/) {
fork = $1
if (fork in UPDATES) {
prefix_re = fork "[ \t]+[^ \t]+[ \t]+"
if (match($0, prefix_re)) {
prefix = substr($0, 1, RLENGTH)
rest = substr($0, RLENGTH + 1)
if (match(rest, /^[^ \t]+/)) {
suffix = substr(rest, RLENGTH + 1)
$0 = prefix UPDATES[fork] suffix
}
}
}
}
print
}
END {
if (APPEND_FILE != "") {
while ((getline line < APPEND_FILE) > 0) print line
close(APPEND_FILE)
}
}
' <(printf '%s' "$CONTROL") "$MAP" > "$TMP"
mv "$TMP" "$MAP"
echo ""
echo "Updated $MAP ($UPDATE_COUNT col-3 rewrite(s), ${#APPEND_ROWS[@]} append(s))"
local/scripts/upgrade-forks.sh
+111 -7
View File
@@ -5,7 +5,7 @@
# 1. Fetch latest upstream (full depth for accurate merge base)
# 2. Identify Red Bear commits (ahead of upstream merge-base)
# 3. Save RB commits to a patch file
# 4. Reset to upstream/master (or upstream/main)
# 4. Reset to upstream/master (or upstream/main, or refs/tags/<tag> with --to)
# 5. Reapply RB commits via cherry-pick (or patch fallback)
# 6. Verify no upstream functions were dropped (verify-fork-functions.sh)
# 7. Report success/failure per fork
@@ -13,14 +13,29 @@
# Usage:
# ./local/scripts/upgrade-forks.sh # Upgrade all forks
# ./local/scripts/upgrade-forks.sh kernel redoxfs # Upgrade specific forks
# ./local/scripts/upgrade-forks.sh --to=0.5.12 kernel # Pin to a specific tag
# ./local/scripts/upgrade-forks.sh --dry-run # Show plan, don't execute
# ./local/scripts/upgrade-forks.sh --force # Skip safety checks
# ./local/scripts/upgrade-forks.sh --no-fetch # Use cached upstream refs (no network)
# ./local/scripts/upgrade-forks.sh --force-reapply # Reapply even when 0 commits behind
# ./local/scripts/upgrade-forks.sh --force-diverged # Allow running on diverged-mode forks
#
# --to=<X.Y.Z> (release-bump-pipeline.md Deliverable B):
# After fetching upstream, pin upstream_ref to refs/tags/<tag>. The script
# tries refs/tags/<tag> first, then refs/tags/v<tag>, resolving via
# `git rev-parse`. When --to is given, the master/main default-branch
# logic is skipped entirely.
#
# Diverged-mode guard (release-bump-pipeline.md Deliverable B):
# At the top of each fork's iteration, the script looks up the fork's mode
# (4th column of local/fork-upstream-map.toml). If mode is `diverged` and
# neither --force nor --force-diverged is given, the script REFUSES with
# exit 1 and a message explaining that diverged rebases are operator-manual
# per policy (Phase 2.4+). A missing map entry is a warning, not a refusal.
#
# Safety:
# - Requires clean working tree in each fork (no uncommitted changes)
# - Creates backup branch before reset
# - Creates backup branch (rb-backup/<fork>-<ts>-<pid>) before reset
# - Stops on first conflict (interactive resolution or --abort)
# - Post-upgrade: verifies all upstream functions are present
#
@@ -28,6 +43,7 @@
# REDBEAR_UPGRADE_FORCE=1 Same as --force
# REDBEAR_UPGRADE_NO_FETCH=1 Same as --no-fetch
# REDBEAR_UPGRADE_FORCE_REAPPLY=1 Same as --force-reapply
# REDBEAR_UPGRADE_FORCE_DIVERGED=1 Same as --force-diverged
set -euo pipefail
@@ -38,6 +54,8 @@ DRY_RUN=0
FORCE=0
NO_FETCH=0
FORCE_REAPPLY=0
FORCE_DIVERGED=0
PIN_TO_TAG=""
declare -a TARGET_FORKS=()
while [[ $# -gt 0 ]]; do
@@ -46,8 +64,17 @@ while [[ $# -gt 0 ]]; do
--force) FORCE=1 ;;
--no-fetch) NO_FETCH=1 ;;
--force-reapply) FORCE_REAPPLY=1 ;;
--force-diverged) FORCE_DIVERGED=1 ;;
--to=*)
PIN_TO_TAG="${1#*=}"
if [[ ! "$PIN_TO_TAG" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "ERROR: --to requires an anchored semver X.Y.Z (got: $PIN_TO_TAG)" >&2
exit 1
fi
;;
-h|--help)
echo "Usage: $0 [--dry-run] [--force] [--no-fetch] [--force-reapply] [fork1 fork2 ...]"
echo "Usage: $0 [--dry-run] [--force] [--no-fetch] [--force-reapply]"
echo " [--to=<X.Y.Z>] [--force-diverged] [fork1 fork2 ...]"
echo ""
echo "Upgrades local forks to latest upstream and reapplies Red Bear patches."
echo ""
@@ -56,6 +83,12 @@ while [[ $# -gt 0 ]]; do
echo " --force Skip safety checks (dirty tree, etc.)"
echo " --no-fetch Use cached upstream refs (no network required)"
echo " --force-reapply Reapply even when 0 commits behind upstream"
echo " --to=<X.Y.Z> Pin upstream_ref to refs/tags/<tag> (or refs/tags/v<tag>)"
echo " instead of upstream/master or upstream/main."
echo " Resolved via git rev-parse. Skips default-branch logic."
echo " --force-diverged Allow running on forks whose map mode is 'diverged'."
echo " Without this (or --force), diverged forks are refused"
echo " because their rebase is operator-manual (Phase 2.4+)."
echo ""
echo "Forks (default: all in local/sources/ with upstream remotes):"
for d in "$PROJECT_ROOT"/local/sources/*/; do
@@ -72,6 +105,7 @@ done
[[ "${REDBEAR_UPGRADE_FORCE:-0}" == "1" ]] && FORCE=1
[[ "${REDBEAR_UPGRADE_NO_FETCH:-0}" == "1" ]] && NO_FETCH=1
[[ "${REDBEAR_UPGRADE_FORCE_REAPPLY:-0}" == "1" ]] && FORCE_REAPPLY=1
[[ "${REDBEAR_UPGRADE_FORCE_DIVERGED:-0}" == "1" ]] && FORCE_DIVERGED=1
cd "$PROJECT_ROOT"
@@ -131,9 +165,48 @@ for fork in "${TARGET_FORKS[@]}"; do
continue
fi
# ---- Diverged-mode guard (release-bump-pipeline.md Deliverable B) ----
# Look up the fork's mode (4th column of local/fork-upstream-map.toml).
# If mode is `diverged` and neither --force nor --force-diverged is given,
# REFUSE: diverged rebases are operator-manual per policy (Phase 2.4+).
# A missing map entry is a warning, not a refusal.
fork_mode=""
if [[ -f "$PROJECT_ROOT/local/fork-upstream-map.toml" ]]; then
fork_mode=$(awk -v want="$fork" '
$1 == want && /^[^#[:space:]]/ {
mode = $4
sub(/[ \t]*#.*/, "", mode)
print mode
exit
}
' "$PROJECT_ROOT/local/fork-upstream-map.toml" 2>/dev/null || true)
fi
if [[ -z "$fork_mode" ]]; then
echo -e " ${YELLOW}WARN: '$fork' not found in local/fork-upstream-map.toml${NC}"
echo -e " ${YELLOW} (proceeding without diverged-mode check)${NC}"
elif [[ "$fork_mode" == "diverged" && "$FORCE" -eq 0 && "$FORCE_DIVERGED" -eq 0 ]]; then
echo -e " ${RED}REFUSE: '$fork' is mode=diverged in the upstream map${NC}"
echo -e " ${YELLOW}Diverged forks track upstream via merge commits and cannot be${NC}"
echo -e " ${YELLOW}auto-rebased by this script. Their rebase is operator-manual${NC}"
echo -e " ${YELLOW}(Phase 2.4+ work). Override with --force-diverged (or --force).${NC}"
FAIL_COUNT=$((FAIL_COUNT + 1))
FAILED_FORKS+=("$fork")
echo ""
continue
elif [[ "$fork_mode" == "diverged" ]]; then
echo -e " ${YELLOW}WARN: '$fork' is mode=diverged; proceeding due to --force-diverged${NC}"
fi
# Safety: check working tree is clean
if [[ "$FORCE" -eq 0 ]]; then
dirty=$(cd "$fork_dir" && git status --porcelain 2>/dev/null | wc -l)
if ! cd "$fork_dir" 2>/dev/null; then
echo -e " ${RED}FAIL: cannot enter fork dir $fork_dir${NC}"
FAIL_COUNT=$((FAIL_COUNT + 1))
FAILED_FORKS+=("$fork")
echo ""
continue
fi
dirty=$(git status --porcelain 2>/dev/null | wc -l)
if [[ "$dirty" -gt 0 ]]; then
echo -e " ${RED}FAIL: working tree has $dirty uncommitted change(s)${NC}"
echo -e " Commit or stash first, or use --force"
@@ -146,7 +219,38 @@ for fork in "${TARGET_FORKS[@]}"; do
# Determine upstream branch and ref
upstream_branch="master"
if [[ "$NO_FETCH" -eq 1 ]]; then
if [[ -n "$PIN_TO_TAG" ]]; then
# --to=<X.Y.Z>: pin upstream_ref to refs/tags/<tag>, skipping the
# default master/main logic entirely. Try refs/tags/<tag> first,
# then refs/tags/v<tag> (some upstreams prefix tags with `v`).
# Resolve via `git rev-parse` so we end up with a concrete SHA-bearing ref.
if [[ "$NO_FETCH" -eq 0 ]]; then
if ! (cd "$fork_dir" && git fetch upstream --quiet --tags 2>&1); then
echo -e " ${RED}FAIL: git fetch upstream --tags failed (network error?)${NC}"
echo -e " ${YELLOW}Use --no-fetch if upstream refs are already cached${NC}"
FAIL_COUNT=$((FAIL_COUNT + 1))
FAILED_FORKS+=("$fork")
echo ""
continue
fi
fi
pinned_ref=""
for candidate in "refs/tags/${PIN_TO_TAG}" "refs/tags/v${PIN_TO_TAG}"; do
if (cd "$fork_dir" && git rev-parse --verify "$candidate" >/dev/null 2>&1); then
pinned_ref="$candidate"
break
fi
done
if [[ -z "$pinned_ref" ]]; then
echo -e " ${RED}FAIL: --to=${PIN_TO_TAG}: neither refs/tags/${PIN_TO_TAG}${NC}"
echo -e " ${RED} nor refs/tags/v${PIN_TO_TAG} exists after fetch${NC}"
FAIL_COUNT=$((FAIL_COUNT + 1))
FAILED_FORKS+=("$fork")
echo ""
continue
fi
upstream_ref="$pinned_ref"
elif [[ "$NO_FETCH" -eq 1 ]]; then
# Use cached upstream refs — verify they exist
if ! (cd "$fork_dir" && git rev-parse --verify upstream/master >/dev/null 2>&1); then
if (cd "$fork_dir" && git rev-parse --verify upstream/main >/dev/null 2>&1); then
@@ -159,6 +263,7 @@ for fork in "${TARGET_FORKS[@]}"; do
continue
fi
fi
upstream_ref="upstream/$upstream_branch"
else
if ! (cd "$fork_dir" && git fetch upstream --quiet 2>&1); then
echo -e " ${RED}FAIL: git fetch upstream failed (network error?)${NC}"
@@ -179,9 +284,8 @@ for fork in "${TARGET_FORKS[@]}"; do
continue
fi
fi
upstream_ref="upstream/$upstream_branch"
fi
upstream_ref="upstream/$upstream_branch"
old_sha=$(cd "$fork_dir" && git rev-parse --short HEAD)
upstream_sha=$(cd "$fork_dir" && git rev-parse --short "$upstream_ref")
recipes/wip/AGENTS.md
+8 -2
View File
@@ -51,8 +51,8 @@ recipes/wip/
│ └── xwayland/ # XWayland (partially patched)
├── services/
│ └── seatd/ # Seat daemon recipe (service category, runtime trust still open)
├── kde/ # 9 KDE app recipes
│ ├── kde-dolphin/ # File manager (needs kio)
├── kde/ # HISTORICAL: 9 pre-KF6 KDE app recipes (not the active KDE surface)
│ ├── kde-dolphin/ # File manager (needs kio — kf6-kio now built under local/recipes/kde/)
│ ├── kdenlive/ # Video editor (needs MLT)
│ ├── krita/ # Painting (needs Qt + OpenGL)
│ ├── kdevelop/ # IDE (needs Qt + kio)
@@ -64,6 +64,12 @@ recipes/wip/
└── drivers/ # WIP driver ports (planned)
```
**Note:** the recipes in `wip/kde/` are pre-KF6-era experiments. The ACTIVE KDE
surface is `local/recipes/kde/` (40+ KF6 frameworks, KWin, Plasma, SDDM,
konsole) plus `config/redbear-full.toml`. Do not treat `wip/kde/` recipes as
the KDE path; they remain only as reference material until upstream promotes
them (see "OWNERSHIP RULE FOR UPSTREAM WIP" above).
## WHERE TO LOOK
| Task | Location |
scripts/run_mini1.sh
+1 -1
View File
@@ -11,7 +11,7 @@ esac
# virtio-gl, native CPU, net boost
qemu-system-x86_64 -m 8G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-mini.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio
qemu-system-x86_64 -machine type=q35,accel=kvm -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -m 8G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-mini.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio
#qemu-system-x86_64 -m 12G -smp 8 -device qemu-xhci -net nic,model=virtio -net user --enable-kvm -cpu host -display gtk,gl=on -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2/x64/OVMF_CODE.4m.fd -drive file=/home/kellito/Builds/RedBear-OS/build/x86_64/redbear-full.iso,format=raw -device virtio-gpu-pci -enable-kvm -serial mon:stdio