diff --git a/AGENTS.md b/AGENTS.md index 1d52ca357f..bdd5aa8e4f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -115,7 +115,7 @@ any package, or any target artifact with anything else. This is absolute: |---|---| | `cargo check` / `cargo build` / `cargo clippy` in any source tree | Bypasses the cross-compile target, sysroot, feature flags, and workspace orchestration. Produces wrong-target or mis-configured output and misleading "it compiles" claims. | | `cargo test --target x86_64-unknown-redox` (or any target) in a source tree | Same problem — compiles outside the cookbook's sysroot/prefix. | -| `./target/release/repo cook ` **as a substitute for a build** | Diagnostic/diagnostic-only. Outside build-redbear.sh it runs without prefix-staleness detection, source-fingerprint tracking, the concurrency lock, branch gates, or the stash-and-restore guard. The cookbook itself warns: *"repo invoked outside build-redbear.sh"*. | +| `./target/release/repo cook ` **as a substitute for a build** | Diagnostic/diagnostic-only. Outside build-redbear.sh it runs without prefix-staleness detection, source-fingerprint tracking, the concurrency lock, the branch gate, or the dirty-source gate. The cookbook itself warns: *"repo invoked outside build-redbear.sh"*. | | `make r.` / `make cr.` **as a substitute for a build** | Same — a thin wrapper over `repo cook`, with the same missing guarantees. | | `make all` / `make live` directly | build-redbear.sh wraps these with the required preflight (prefix rebuild, version sync, config parsing, protected-fetch policy). Calling them directly skips that preflight. | @@ -135,15 +135,22 @@ produces a Red Bear target artifact. - **Prefix staleness detection + rebuild** — the #1 cause of "undefined reference" link errors after relibc/kernel/base fork changes is a stale prefix; only build-redbear.sh detects and rebuilds it before any recipe compiles. -- **Stash-and-restore** — dirty fork sources (`local/sources/*`) are stashed before the - build and popped on EXIT, so fingerprints reflect committed HEAD. **Uncommitted fork - changes are NOT built** — commit to the `submodule/` branch first. +- **Dirty-source gate** — the build refuses if any fork (`local/sources/*`) has + uncommitted edits, so fingerprints reflect committed HEAD. **Uncommitted fork changes + are NOT built** — commit to the `submodule/` branch first, or pass `--allow-dirty` + to deliberately cook the working tree AS-IS. (The build never stashes your tree; see + `local/docs/BUILD-SYSTEM.md` §6.) - **Branch topology gate** — every fork worktree must be on its canonical `submodule/` branch; the build refuses otherwise. - **Source-fingerprint tracking + content-hash cache** — correct incremental rebuilds. - **`.config` parsing + protected-fetch policy + version sync + pre-cooking.** - Sets `REDBEAR_CANONICAL_BUILD=1` so the cookbook knows it is inside the pipeline. +**Full reference:** `local/docs/BUILD-SYSTEM.md` is the canonical build-system doc +(flags, pipeline stages, offline/release, caching, versioning). `build-redbear.sh +--help` lists the flags; the common knobs are flags (`--allow-dirty`, +`-j/--jobs`, `--release`, …), rarer escape hatches remain `REDBEAR_*` env vars. + **Correct per-component verification flow (the ONLY flow):** 1. Make your edits. 2. Commit to the right branch: fork code → `submodule/` in `local/sources/`; diff --git a/README.md b/README.md index 247622d9b1..8b8c1a8dea 100644 --- a/README.md +++ b/README.md @@ -89,8 +89,9 @@ cd RedBear-OS git clone https://vasilito:${REDBEAR_GITEA_TOKEN}@gitea.redbearos.org/vasilito/RedBear-OS.git # Canonical build entry point -./local/scripts/build-redbear.sh redbear-mini # Text-only target -./local/scripts/build-redbear.sh redbear-full # Desktop-capable target +./local/scripts/build-redbear.sh redbear-mini # Text-only target +./local/scripts/build-redbear.sh redbear-full # Desktop-capable target +./local/scripts/build-redbear.sh -j 8 --allow-dirty redbear-mini # flags: see --help # Boot in QEMU make qemu @@ -99,7 +100,9 @@ make qemu > `local/scripts/build-redbear.sh` is the **only supported build entry point**. It handles > `.config` parsing, prefix staleness detection, protected-recipe authorization, pre-cooking > critical packages, and source fingerprint tracking. Direct `make` invocations bypass these -> gates. See [`AGENTS.md` § Build Commands](./AGENTS.md) for details. +> gates. Run `build-redbear.sh --help` for flags. Full reference: +> [`local/docs/BUILD-SYSTEM.md`](./local/docs/BUILD-SYSTEM.md); see also +> [`AGENTS.md` § Build Commands](./AGENTS.md). ### Config Targets diff --git a/docs/06-BUILD-SYSTEM-SETUP.md b/docs/06-BUILD-SYSTEM-SETUP.md index de247064ba..b7f0a41eff 100644 --- a/docs/06-BUILD-SYSTEM-SETUP.md +++ b/docs/06-BUILD-SYSTEM-SETUP.md @@ -1,5 +1,9 @@ # 06 — Build System Setup Guide +> **Canonical build-system reference:** `local/docs/BUILD-SYSTEM.md` (entry point, +> flags, pipeline stages, offline/release, caching, versioning). This file covers +> host prerequisites and general setup mechanics. +> > **Status note (2026-04-15):** This file explains the mechanics of building the repository, but it > is not the canonical source for repository ownership policy or current execution order. For the > current repository model, use `README.md`, `AGENTS.md`, and diff --git a/local/docs/BUILD-SYSTEM-INVARIANTS.md b/local/docs/BUILD-SYSTEM-INVARIANTS.md index b436bb8cfb..297c4de4ce 100644 --- a/local/docs/BUILD-SYSTEM-INVARIANTS.md +++ b/local/docs/BUILD-SYSTEM-INVARIANTS.md @@ -1,5 +1,9 @@ # Build System Invariants +> Canonical overview: **[`BUILD-SYSTEM.md`](BUILD-SYSTEM.md)** (entry point, flags, +> pipeline stages). This document is the deep-dive on authoritative surfaces and +> propagation order. + ## Purpose This document defines the non-negotiable build-system invariants for Red Bear OS. diff --git a/local/docs/BUILD-SYSTEM.md b/local/docs/BUILD-SYSTEM.md new file mode 100644 index 0000000000..f79e77dc8e --- /dev/null +++ b/local/docs/BUILD-SYSTEM.md @@ -0,0 +1,159 @@ +# Red Bear OS — Build System (canonical reference) + +This is the single authoritative description of how Red Bear OS is built. Deeper +dives live in companion docs and are linked where relevant: +`BUILD-SYSTEM-INVARIANTS.md` (surfaces & propagation order), +`SCRIPT-BEHAVIOR-MATRIX.md` (per-script roles), `PATCH-GOVERNANCE.md`, +`BUILD-CACHE-PLAN.md`, `RELEASE-BUMP-WORKFLOW.md`, and host setup in +`../../docs/06-BUILD-SYSTEM-SETUP.md`. + +--- + +## 1. The one entry point + +``` +./local/scripts/build-redbear.sh [OPTIONS] [CONFIG] +``` + +`build-redbear.sh` is the ONLY supported way to build. It sets +`REDBEAR_CANONICAL_BUILD=1` and orchestrates every stage below. Do not invoke +`repo cook`, `make live`, or `cargo build --target …` directly as a build +substitute — they run outside the gates and produce off-pipeline artifacts +(diagnostic use only). + +`CONFIG` is one of: + +| Config | What it builds | +|--------|----------------| +| `redbear-full` | Desktop/graphics target (Wayland/KDE/Qt/Mesa). **Default.** | +| `redbear-mini` | Text-only console/recovery target | +| `redbear-grub` | Text-only with the GRUB boot manager | +| `redbear-bare` | Kernel + 7-daemon initfs + login shell (minimum bootable) | + +The running script prints its own version banner (magenta) at start, e.g. +`══════ build-redbear.sh v1.3 ══════`. `--version` prints full provenance +(script version, OS version from the git branch, repo commit, host, arch). + +## 2. Options (flags) + +Flags are the canonical interface. Colored output is automatic on a TTY and +suppressed when piped or when `NO_COLOR` is set. + +**Common:** + +| Flag | Effect | +|------|--------| +| `-j, --jobs N` | Parallel build jobs (default: `nproc`) | +| `--upstream` | Allow upstream recipe source refresh (goes online) | +| `--no-cache` | Force a clean rebuild, discarding cached packages | +| `--release VER` | Build from the immutable release archive `VER` | +| `--allow-dirty` | Build with uncommitted fork edits — **cooks the working tree AS-IS** | +| `--keep-build-state` | Keep the diagnostic state dir after exit | +| `-h, --help` / `-V, --version` | Help / version | + +**Advanced escape hatches** stay as environment variables (rare; emergency/CI): +`REDBEAR_ALLOW_CONCURRENT`, `REDBEAR_ALLOW_WRONG_BRANCH`, +`REDBEAR_SKIP_PREFIX_REBUILD`, `REDBEAR_FORCE_TOOLCHAIN_RECOOK`, +`REDBEAR_SKIP_FORK_VERIFY`, `REDBEAR_SKIP_DRIFT_CHECK`, +`REDBEAR_SKIP_FUNCTION_CHECK`, `REDBEAR_SKIP_COLLISION_CHECK`, +`REDBEAR_SKIP_PATCH_CONTENT_CHECK`, `REDBEAR_STRICT_DURABILITY`, +`REDBEAR_STRICT_METADATA`, `REDBEAR_NO_VALIDATE`, `REDBEAR_COLLISIONS_WARN`. +The common flags' old env vars (`REDBEAR_ALLOW_DIRTY`, `REDBEAR_KEEP_BUILD_STATE`, +`REDBEAR_ALLOW_UPSTREAM`, `REDBEAR_RELEASE`, `JOBS`) are still honored as +**deprecated fallbacks**; a flag always overrides the environment. + +## 3. What a build does, in order + +1. **Concurrency lock** — a `flock` refuses a second concurrent build + (`REDBEAR_ALLOW_CONCURRENT=1` overrides). +2. **Cookbook binary** — rebuilds `target/release/repo` if any `src/*.rs` + changed (content fingerprint, not mtime). +3. **Overlay integrity** — verifies `recipes/ → local/recipes/` symlinks; repairs + via `guard-recipes.sh --fix` if needed. +4. **Local-over-WIP** — `recipes/wip/*` that shadow a `local/recipes/` package + are replaced with symlinks to the local version. +5. **Dirty-source gate** — refuses to build if any `local/sources/` has + uncommitted edits, unless `--allow-dirty` (then it cooks the tree as-is). The + build does **not** stash your tree — see §6. +6. **Branch gate** — each fork must be on its `submodule/` branch. +7. **Staleness invalidation** — if a fork changed, invalidate the right pkgars. + relibc/base/etc. only relink the static initfs-critical binaries (base, + redoxfs, userutils, bootstrap); the dynamically-linked desktop is untouched + (shared `libc.so`, the glibc model). Host fstools are rebuilt if + installer/redoxfs changed. +8. **Prefix sysroot** — if relibc is newer than the prefix `libc.a`, the derived + prefix markers are removed and `make prefix` genuinely re-cooks relibc into + the sysroot (a no-op is reported honestly, never as success). kernel/base-only + staleness does not touch the prefix. +9. **Preflight** — `build-preflight.sh`: repairs broken recipe symlinks, fork + version/drift/function verification, collision detection, source-tree + validation. The fork-function check treats inert std-trait methods + (`fmt`/`eq`/`clone`/…) as warnings; intentional divergences are listed in each + fork's `.verify-fork-functions.exclude`. +10. **Version sync check** — `sync-versions.sh --check` (see §7). +11. **Pre-cook** — cooks a small ordered set of critical packages (`CI=1`, + offline unless `--upstream`). +12. **`make live`** — the cookbook builds the config's package graph and the + installer assembles the image. +13. **Critical-package gate** — for `redbear-full`, fails loudly if mesa/qt/… + pkgars are missing (cookbook returns 0 even when recipes fail). +14. **Fingerprints** — fork source fingerprints are recorded **only after** a + successful build, so a failed build never marks forks as "built". + +On failure the EXIT trap dumps diagnostics (last log lines, orphaned cook +processes, and per-recipe state **scoped to this build** so a mini build never +lists leftover graphical packages) and preserves the state dir under +`--keep-build-state`. + +## 4. Offline / release mode + +Cooking is offline by default (`COOKBOOK_OFFLINE=true`, `REPO_OFFLINE=1`). +`--upstream` goes online for non-protected recipes. `--release VER` builds from +`sources/redbear-/` immutable archives and forces offline; new releases are +created only via `provision-release.sh`. A stray `REDBEAR_RELEASE=` line in +`.config` silently switches the whole build off the local forks — the script +never auto-sets it. See `RELEASE-BUMP-WORKFLOW.md`. + +## 5. Caching & staleness + +Per-recipe caching is BLAKE3 content-hash based (`dep_hashes.toml` + +`source_hash.txt`), not mtime. `--no-cache` wipes `repo/` and recipe `target/` +dirs. Cargo path-dep crates (syscall/libredox/redox-scheme) have no pkgar, so +`build-redbear.sh` tracks their source fingerprints and forces a userspace +relink when they change. See `BUILD-CACHE-PLAN.md`. + +## 6. No working-tree stashing (design) + +The build **never stashes your working tree.** An earlier version stashed dirty +forks and restored them on exit, but modern git's `stash push` does not print the +stash SHA on stdout, so the stash was recorded incorrectly and never restored — +stranding operator WIP in `git stash list` on every dirty build. The machinery +was removed entirely. Today: a clean tree cooks committed HEAD; `--allow-dirty` +cooks the tree as-is. A read-only startup advisory reports any leftover +`redbear-build-*` stashes from the old code (recover them from +`git stash list`; see `../recovered-stashes/README.md`). + +## 7. Forks & versioning + +`local/sources//` (the 9 submodule forks) take precedence over +`recipes//`. Version policy (`sync-versions.sh`): + +- **Cat 1** — in-house crates under `local/recipes/*/source/`: `version = ` + (e.g. `0.3.1`). +- **Cat 2** — upstream forks under `local/sources/*/`: + `version = +rb` (e.g. `0.9.0+rb0.3.1`; `+rb` build + metadata, never `-rb`). +- **Vendored upstream** under `local/recipes/` (e.g. `brush`, a Cargo workspace): + keeps its **upstream** crate versions (internal `^` requirements). Mark it with + a `.vendored-upstream` file in the recipe dir so `sync-versions.sh` skips it — + do **not** stamp it with the branch version. `zbus`/`tlc` are excluded by name. + +`sync-versions.sh --check` is a build-time gate; `sync-versions.sh` applies the +sync. See `PATCH-GOVERNANCE.md` for the durability rules (source trees are +disposable; only `local/` survives). + +## 8. Self-versioning of this script + +`BUILD_REDBEAR_VERSION` in `build-redbear.sh` starts at `1.0` and is auto-bumped +(minor) by a `pre-commit` git hook (`bump-build-version.sh`) whenever the script +is committed. Install hooks with `./local/scripts/install-git-hooks.sh`. diff --git a/local/docs/SCRIPT-BEHAVIOR-MATRIX.md b/local/docs/SCRIPT-BEHAVIOR-MATRIX.md index 39a77d0028..6a199720e9 100644 --- a/local/docs/SCRIPT-BEHAVIOR-MATRIX.md +++ b/local/docs/SCRIPT-BEHAVIOR-MATRIX.md @@ -25,7 +25,7 @@ The goal is to remove guesswork from the sync/fetch/apply/build workflow. |---|---|---|---| | `local/scripts/provision-release.sh` | Refresh top-level upstream repo state | fetches upstream, reports conflict risk, rebases repo commits. Under v6.0 the "release fork reapplication" step is no longer needed because `local/sources/`, `local/recipes/`, and `local/patches//` already live in the main repo (Rule 1 + Rule 2). | does not automatically solve every subsystem release fork conflict; does not by itself make upstream WIP recipes safe shipping inputs | | `local/scripts/apply-patches.sh` | **LEGACY / ARCHIVED** — historical overlay only | under v5.x, applied build-system patches and relinked recipe patch symlinks; under v6.0 this is a no-op for in-tree components (Rule 1 direct edits) and is replaced by `cookbook_apply_patches` for big external projects (Rule 2). See `local/AGENTS.md`. | do not invoke during a v6.0 build. The `local/scripts/build-redbear.sh ` canonical entry point never calls it. | -| `local/scripts/build-redbear.sh` | **Canonical build entry point** for Red Bear profiles | under v6.0 it does NOT call `apply-patches.sh` — the release fork is already in `local/`. It enforces: (1) local-over-WIP recipe priority, (2) overlay integrity verification, (3) submodule dirty-state stash, (4) firmware presence warning, (5) profile validation, (6) cookbook build if needed, (7) image build. `--upstream` triggers explicit source immutable archived for non-protected recipes. | does not guarantee every nested upstream source tree is fresh; does not replace explicit subsystem/runtime validation | +| `local/scripts/build-redbear.sh` | **Canonical build entry point** for Red Bear profiles | under v6.0 it does NOT call `apply-patches.sh` — the release fork is already in `local/`. It enforces: (1) local-over-WIP recipe priority, (2) overlay integrity verification, (3) dirty-source gate (refuses uncommitted fork edits unless `--allow-dirty`; it does NOT stash the tree), (4) firmware presence warning, (5) profile validation, (6) cookbook build if needed, (7) image build. Canonical reference: `BUILD-SYSTEM.md`. `--upstream` triggers explicit source immutable archived for non-protected recipes. | does not guarantee every nested upstream source tree is fresh; does not replace explicit subsystem/runtime validation | | `scripts/fetch-all-sources.sh` | Fetch mainline recipe source inputs for builds | downloads mainline/upstream recipe sources, reports status/preflight, and supports config-scoped fetches while leaving local release fork in place | does not mean fetched upstream WIP source is the durable shipping source of truth | | `local/scripts/fetch-sources.sh` | Fetch mainline recipe sources for browsing and patching | when passed `--upstream`, fetches `recipes/*` source trees so the upstream-managed side is locally available for reading, editing, and patch preparation | does not decide whether upstream should replace the local release fork | | `local/scripts/build-redbear-wifictl-redox.sh` | Build `redbear-wifictl` for the Redox target with the repo toolchain | prepends `prefix/x86_64-unknown-redox/sysroot/bin` to `PATH` and runs `cargo build --target x86_64-unknown-redox` in the `redbear-wifictl` crate | does not prove runtime Wi-Fi behavior; only closes the target-build environment gap for this crate |