diff --git a/AGENTS.md b/AGENTS.md index f14a39e1eb..f49144f1ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,6 +88,11 @@ redox-master/ echo 'PODMAN_BUILD?=0' > .config # Native build (no container) echo 'PODMAN_BUILD?=1' > .config # Podman container build +# CRITICAL: .config MUST NOT contain REDBEAR_RELEASE=... in development mode. +# If it does, the build system re-extracts sources from the immutable release +# archive and ignores the local forks. Remove it if present: +grep -v REDBEAR_RELEASE .config > .config.tmp && mv .config.tmp .config + # Build Red Bear OS # Supported compile targets: # redbear-full desktop/graphics target (harddrive.img or live ISO) @@ -95,18 +100,30 @@ echo 'PODMAN_BUILD?=1' > .config # Podman container build # redbear-grub text-only with GRUB boot manager (live ISO) # Desktop/graphics target: redbear-full # Text-only targets: redbear-mini, redbear-grub -make all CONFIG_NAME=redbear-full # Desktop/graphics target → harddrive.img +./local/scripts/build-redbear.sh redbear-mini # Recommended (dev mode, local forks) +./local/scripts/build-redbear.sh --upstream redbear-mini # With online fetch (fast iteration) make all CONFIG_NAME=redbear-mini # Text-only target → harddrive.img +make all CONFIG_NAME=redbear-full # Desktop/graphics target → harddrive.img make live CONFIG_NAME=redbear-full # Full desktop live ISO make live CONFIG_NAME=redbear-mini # Text-only mini live ISO make live CONFIG_NAME=redbear-grub # Text-only mini live ISO with GRUB CI=1 make all CONFIG_NAME=redbear-mini # CI mode (disables TUI, for non-interactive) +# IMPORTANT: For local fork development, use: +# export REDBEAR_ALLOW_PROTECTED_FETCH=1 # base/kernel/relibc are protected recipes +# ./local/scripts/build-redbear.sh --upstream redbear-mini +# Without --upstream, the build is in offline mode and re-extracts from +# the immutable release archive instead of using the local forks. + # Run make qemu # Boot in QEMU make qemu QEMUFLAGS="-m 4G" # With more RAM make live # Build live ISO for real bare metal +# QEMU with network access (required for testing): +qemu-system-x86_64 -cdrom build/x86_64/redbear-mini.iso \ + -m 1024 -netdev user,id=n0 -device e1000,netdev=n0 + # Single recipe ./target/release/repo cook recipes/libs/mesa # Build one recipe ./target/release/repo fetch recipes/core/kernel # Fetch source only @@ -118,6 +135,26 @@ make clean # Remove build artifacts make distclean # Remove sources + artifacts ``` +### Local forks vs overlay patches + +As of 2026-06, the following core components use **local forks** (in +`local/sources//`) instead of `upstream + overlay patches`: + +- `relibc`, `kernel`, `bootloader`, `installer`, `redoxfs`, `userutils` +- **`base`** — uses `path = "..."` in `recipes/core/base/recipe.toml` to point + at the local fork + +The local fork approach was chosen because: + +1. The overlay patch chain accumulated drift from upstream, causing + repeated build failures (broken patches, version mismatches, etc). +2. The local forks are the durable source of truth, with version pins and + fixes already applied. +3. Iterating on a local fork is faster than maintaining dozens of patches. + +For details on how to import upstream commits into a local fork, see +`local/AGENTS.md` § "LOCAL FORK MODEL". + ## BUILD FLOW ``` @@ -140,7 +177,7 @@ make all - **Recipe format**: TOML with `[source]` + `[build]` + optional `[package]` - **Build templates**: `cargo`, `meson`, `cmake`, `make`, `configure`, `custom` - **WIP recipes**: Must start with `#TODO` comment explaining what's missing -- **Custom configs**: Name with `my-` prefix (git-ignored by convention) +- **Custom configs**: Name with `redbear-*` prefix (tracked in git). The legacy `my-*` prefix is deprecated and git-ignored. - **CI**: GitLab CI (`.gitlab-ci.yml`) at root + per-recipe; some have GitHub Actions - **Syscall ABI**: Unstable intentionally. Stability via `libredox` and `relibc` - **Drivers**: ALL userspace daemons via scheme system. No kernel-space drivers (except serio) @@ -288,17 +325,36 @@ does NOT strip timestamps — they should be removed from the patch file directl ### Protected Recipes Core recipes (`base`, `kernel`, `relibc`, `bootloader`, etc.) and any recipe carrying -Red Bear patches are **protected** and cannot be re-fetched online. Protected recipes -always use offline/archived sources from `sources/redbear-/tarballs/`. +Red Bear patches are **protected** and cannot be re-fetched online. -Use the `--allow-protected` flag (or set `REDBEAR_ALLOW_PROTECTED_FETCH=1`) to override -protection (e.g., for debugging or when archives are missing). +As of 2026-06, most protected recipes use **local forks** in +`local/sources//` rather than archives. The cookbook falls back to +the immutable release archive (`sources/redbear-/tarballs/`) only when +the local fork is missing. + +For the protected recipes, the build needs the source to exist. In development +mode with local forks, the cookbook copies the fork to `recipes//source/`. +In release mode, the source is extracted from the archive. + +**For development builds with local forks, use:** ```bash -repo --allow-protected fetch base -repo --allow-protected cook base +export REDBEAR_ALLOW_PROTECTED_FETCH=1 +./local/scripts/build-redbear.sh --upstream redbear-mini ``` +Or, equivalently: + +```bash +REDBEAR_ALLOW_PROTECTED_FETCH=1 ./local/scripts/build-redbear.sh --upstream redbear-mini +``` + +Without `REDBEAR_ALLOW_PROTECTED_FETCH=1`, the cookbook refuses to fetch the +protected recipe's source and the build fails. + +Without `--upstream`, the build defaults to offline mode and tries to extract +from the release archive (which may be stale or missing patches). + The full protected recipe list (in `src/cook/fetch.rs:redbear_protected_recipe()`) includes: all core system recipes, all Red Bear custom recipes, all patched Qt/Wayland/ KDE recipes, and all recipes carrying Red Bear patches (Qt, DRM, Mesa, Wayland, D-Bus, @@ -316,9 +372,22 @@ To allow online fetching for non-protected development recipes: COOKBOOK_OFFLINE=false make all CONFIG_NAME=redbear-full ``` +Or use the `--upstream` flag of `build-redbear.sh`: +```bash +./local/scripts/build-redbear.sh --upstream redbear-mini +``` + In release mode (`REDBEAR_RELEASE=0.1.0`), online fetching is **completely disabled** even with `COOKBOOK_OFFLINE=false`. Sources are immutable in release mode. +**For development with local forks** (see "LOCAL FORK MODEL" in `local/AGENTS.md`), +the recommended invocation is: +```bash +unset REDBEAR_RELEASE # CRITICAL: must not be set in dev +export REDBEAR_ALLOW_PROTECTED_FETCH=1 +./local/scripts/build-redbear.sh --upstream redbear-mini +``` + ### Workspace Pollution Prevention Before every fetch and build, the cookbook tool removes orphaned `Cargo.toml` @@ -342,7 +411,8 @@ When reordering patches: remove the source tree, re-fetch, and rebuild to verify **POLICY: Never gitignore critical build infrastructure, regardless of file size.** The Gitea server is our own — we have unlimited space. Do not put `local/patches/*`, -`local/recipes/*`, or any other durable Red Bear code in `.gitignore` because of file size. +`local/recipes/*`, `local/sources/*`, or any other durable Red Bear code in +`.gitignore` because of file size. **Historical context (resolved):** The original `local/patches/base/redox.patch` was a 100MB+ consolidated mega-patch that was placed in `.gitignore`. After it was lost, base @@ -350,15 +420,22 @@ recipe could not be built from a clean checkout. The mega-patch approach was abandoned in favor of ~100 individual `P*.patch` files (totaling 2.4MB) that are all committed to git. -**Current state:** All `local/patches/base/P*.patch` files are tracked in git. -The dangling `recipes/core/base/redox.patch` symlink (the old mega-patch -shortcut) has been removed because the base recipe uses individual `P*.patch` -entries directly, not the mega-patch. +**Current state (2026-06):** +- All `local/patches/base/P*.patch` files are tracked in git. +- The dangling `recipes/core/base/redox.patch` symlink (the old mega-patch + shortcut) has been removed. +- For components that need extensive changes (base, relibc, kernel), we now + use **local forks** in `local/sources//` instead of accumulating + many patches. See "LOCAL FORK MODEL" in `local/AGENTS.md`. **Future policy:** If a single patch ever grows beyond what Git LFS would comfortably handle, split it into multiple smaller patches. Do not put it in `.gitignore`. Do not store it in chunks that need reassembly. +**For local forks:** The full source tree of a fork can be large (e.g., +Redox kernel is ~30MB, relibc is ~10MB). Local forks in `local/sources/` are +git-tracked and pushed to gitea. The `.gitignore` MUST NOT exclude them. + ### Patch Governance See `local/docs/PATCH-GOVERNANCE.md` for the full patch governance rules. diff --git a/README.md b/README.md index 87583762cd..ba8ad6c8a4 100644 --- a/README.md +++ b/README.md @@ -41,20 +41,23 @@ See the [Redox Build Guide](https://doc.redox-os.org/book/podman-build.html) for git clone https://gitea.redbearos.org/vasilito/RedBear-OS.git cd RedBear-OS -# Build and run the desktop target in QEMU -./scripts/run.sh --build +# Recommended: use the Red Bear wrapper +./local/scripts/build-redbear.sh redbear-mini # Text-only target +./local/scripts/build-redbear.sh redbear-full # Desktop-capable target -# Build a live ISO for bare metal -./scripts/build-iso.sh redbear-full - -# Build the text-only recovery target -./scripts/run.sh --build --config redbear-mini +# Boot in QEMU with the resulting image +make qemu ``` +> **Build script:** `local/scripts/build-redbear.sh` is the canonical entry point. Bare +> `make all` works but bypasses the `.config` checking and `REDBEAR_ALLOW_PROTECTED_FETCH=1` +> gates that `build-redbear.sh` enforces. See `AGENTS.md` § Build Commands for full details. + ### Public Scripts | Script | Purpose | |--------|---------| +| `local/scripts/build-redbear.sh` | **Canonical** build wrapper for redbear-mini/full/grub | | `scripts/run.sh` | Build and run in QEMU (`-b` to build, `-c ` for target) | | `scripts/build-iso.sh` | Build a live ISO for bare-metal boot | | `scripts/build-all-isos.sh` | Build all live ISO targets | diff --git a/docs/05-KDE-PLASMA-ON-REDOX.md b/docs/05-KDE-PLASMA-ON-REDOX.md index 42c577742d..550a613bbd 100644 --- a/docs/05-KDE-PLASMA-ON-REDOX.md +++ b/docs/05-KDE-PLASMA-ON-REDOX.md @@ -27,7 +27,7 @@ | KF6 | All 32/32 built (some still blocked by QML gate) | | `config/redbear-full.toml` | Present with KDE session launcher | | `kwin`, `plasma-workspace`, `plasma-desktop` | Recipes exist; build/runtime trust is still incomplete and some recipe/source TODO markers remain | -| `kirigami` | Stub-only package for dependency resolution | +| `kirigami` | Build blocked at the QML gate (recipe is honest; no fake/fallback package shipped) | | `kf6-kio` | Heavy blocked by QML gate-based build recipe | | `kf6-kcmutils` | Stripped widget-only build recipe | | `libxcvt` | Now builds as a real package; no longer needs to stay in the KWin stub bucket | diff --git a/docs/06-BUILD-SYSTEM-SETUP.md b/docs/06-BUILD-SYSTEM-SETUP.md index 7a7b0cec56..98ab3e45e9 100644 --- a/docs/06-BUILD-SYSTEM-SETUP.md +++ b/docs/06-BUILD-SYSTEM-SETUP.md @@ -110,13 +110,26 @@ desktop-capable target. ## Building -### Full Build (Desktop) +### Quick Build (Recommended) ```bash -make all +# Use the Red Bear wrapper (auto-handles .config, REDBEAR_ALLOW_PROTECTED_FETCH, etc.) +./local/scripts/build-redbear.sh redbear-mini # Text-only console target +./local/scripts/build-redbear.sh redbear-full # Desktop-capable target +./local/scripts/build-redbear.sh redbear-grub # Text-only + GRUB boot manager ``` -This produces the image for the selected target, such as `build/x86_64/harddrive.img`. +These produce images such as `build/x86_64/harddrive.img` or `build/x86_64/redbear-mini.iso`. + +### Bare `make all` (Legacy / Advanced) + +```bash +make all CONFIG_NAME=redbear-mini +``` + +Bare `make all` works but bypasses the policy gates (`.config` checking, +`REDBEAR_ALLOW_PROTECTED_FETCH=1`, etc.) that `build-redbear.sh` enforces. Prefer the wrapper +unless you specifically need to bypass those gates. ### Export External Toolchain diff --git a/local/AGENTS.md b/local/AGENTS.md index 5b6f9f3e2f..53f257df0a 100644 --- a/local/AGENTS.md +++ b/local/AGENTS.md @@ -193,6 +193,133 @@ When Red Bear maintains a local recipe and upstream contains a package with the - Do not switch back to upstream WIP for active Red Bear builds - Re-evaluate only when upstream package exits WIP and becomes a normal maintained package +## LOCAL RECIPE SOURCE IMMUTABILITY + +**`local/recipes//source/` is unconditionally immutable** as of `cb8b093564` (2026-06-18). + +### Why + +Internal Red Bear subprojects (`tlc`, `redbear-*`, `redbear-greeter`, `cub`, `redbear-sessiond`, +etc.) live under `local/recipes/*/source/`. They have **no upstream apart from our gitea** — +they are committed to `https://gitea.redbearos.org/vasilito/redbear-os.git` and nowhere else. + +If a `local/recipes/*/source/` is destroyed, **it cannot be recovered from any public source**. + +### The guarantee + +The cookbook's `is_local_overlay()` check (in `src/bin/repo.rs` and `src/cook/fetch.rs`) returns +`true` for any path matching `/local/recipes/`. The destructive paths in the build system +(`unfetch`, `git reset --hard`, `git clean -ffdx`, source-wipe) all check this guard and +**refuse to operate on local overlays unconditionally**. No env var, no flag, no Makefile +target can override this. + +`REDBEAR_ALLOW_LOCAL_UNFETCH=1` was previously a kill switch for this guard. It was removed +in `cb8b093564` and is now dead code (its caller returns `false`). + +`make distclean-nuclear` previously bypassed the guard. It is now a no-op for local recipes +and behaves identically to `make distclean`. + +### What this means for operators + +- `make distclean` is safe for local recipes (only upstream recipes are removed). +- `make c.` only removes `target/`, never `source/`. Always safe. +- `make u.` (unfetch) refuses any local recipe. +- `repo unfetch ` refuses with a clear error message. +- `make distclean-nuclear` is now a synonym for `make distclean` (the dangerous variant was removed). + +### Recovery + +If a `local/recipes/*/source/` is destroyed despite these guards, recovery is only possible from: +- git history (if the source was tracked in the fork's local git repo) +- another operator's working copy +- a backup tarball (if one was taken) + +There is no automated recovery path. This is intentional. + +## LOCAL FORK MODEL (CORE COMPONENTS) + +As of 2026-06, the following core components are built from **local forks** in +`local/sources//` rather than from `upstream + overlay patches`: + +| Component | Local fork path | Recipe | +|-------------|------------------------------------------|-------------------------------------------------------| +| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (git URL) | +| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (git URL) | +| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` | +| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` | +| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` | +| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` | +| **base** | **`local/sources/base/`** (path source) | **`recipes/core/base/recipe.toml` (path = ...) ** | + +### Why local forks for these components? + +1. **Critical bug fixes** — these components had multiple broken patches that drifted + from upstream and accumulated alignment issues. +2. **Faster build iteration** — no need to fetch and apply dozens of patches on every build. +3. **Atomic changes** — a single commit captures a logical change set, not a + fragmented series of patches. +4. **Local recipes are already in fork form** — `local/recipes/` was already a fork + pattern; extending this to `local/sources/` is consistent. + +### Overlay patches are still permitted + +Overlay patches are still allowed for: + +- **Importing new upstream commits** into the local fork (apply upstream as a patch + on the fork, then commit to the fork). +- **Smaller or experimental changes** that don't justify a full fork commit. +- **New drivers or subsystems** that don't need to track upstream at all. + +When in doubt: if the change is likely to interact with upstream in the future, +use a patch. If the change is self-contained and Red Bear-specific, prefer a fork +commit. + +### Importing upstream commits into a local fork + +```bash +# Example: import upstream relibc into local/sources/relibc +cd local/sources/relibc +git remote add upstream https://gitlab.redox-os.org/redox-os/relibc.git +git fetch upstream +# Cherry-pick or merge: +git cherry-pick +# Or merge a branch: +git merge upstream/master --no-ff +# Resolve conflicts, then commit +git push +``` + +The local fork's `Cargo.toml` may contain hardcoded absolute paths +(`/home/kellito/Builds/RedBear-OS/...`) from a previous build environment. +**When creating or importing a local fork, make all `path = "..."` references +RELATIVE to the fork's own location.** The build system copies the fork to +`recipes//source/` before building, so absolute paths break. + +Recommended pattern: `path = "../../../local/recipes/"` or +`path = "../../sources//"`. + +### Build mode requirements for local forks + +Building a local fork requires the cookbook to fetch the source. In +development mode this works automatically, but the build system defaults to +`REPO_OFFLINE=1` which prevents fetching. + +For development builds, use: + +```bash +# Either set the env var +unset REDBEAR_RELEASE # CRITICAL: do not set this in dev +export REDBEAR_ALLOW_PROTECTED_FETCH=1 # needed for base, kernel, relibc +./local/scripts/build-redbear.sh --upstream redbear-mini + +# Or use the --upstream flag (recommended) +./local/scripts/build-redbear.sh --upstream redbear-mini +``` + +The `.config` file MUST NOT contain `REDBEAR_RELEASE=...` in development mode +(otherwise the build system re-extracts sources from the immutable release +archive and ignores the local fork). + ```bash # Automated sync (preferred): ./local/scripts/check-upstream-releases.sh # Check for new Redox snapshots (read-only) diff --git a/local/docs/BUILD-TOOLS-PORTING-PLAN.md b/local/docs/BUILD-TOOLS-PORTING-PLAN.md index 387841c2c3..93324a6b4b 100644 --- a/local/docs/BUILD-TOOLS-PORTING-PLAN.md +++ b/local/docs/BUILD-TOOLS-PORTING-PLAN.md @@ -52,7 +52,7 @@ Of 47 build-tool recipes in the codebase: | Core utils | coreutils (Rust), findutils (Rust), ripgrep, gnu-grep, sed | | File tools | patch, grep, sed | | Archives | bzip2, xz, zstd, lz4 | -| Scripting | python312, lua54 | +| Scripting | python314, lua54 | | Build systems | gnu-make, cmake 4.0.3, autoconf, automake, pkg-config | | Compilers (cross) | gcc13, llvm21, rust | | VCS | git (v2.13.1, old) | @@ -117,7 +117,7 @@ Full `cub build` functionality requires native compilation capability. ``` Level 0: Already available ├── bash, zsh, sed, grep, coreutils, findutils, patch, diffutils (in full) - ├── python312, lua54 + ├── python314, lua54 ├── bzip2, xz, zstd, lz4 └── pkg-config @@ -133,7 +133,7 @@ Level 2: Build systems (need Level 0-1) ├── autoconf ← already production ├── automake ← already production ├── libtool ← already builds (needs testing) - ├── meson ← needs: python312 + standalone script + ├── meson ← needs: python314 + standalone script └── ninja ← needs: cmake or python configure.py Level 3: Native compilers (need Level 0-2 + cross-compiler bootstrap) @@ -182,7 +182,7 @@ The same multi-stage approach works for native compilation. |------|--------|-------------|-------| | **Bootstrap `bison`** | 1 day | Phase 1 | Cross-compile bison on host, install as bootstrap. Then attempt native build. | | **Bootstrap `flex`** | 1 day | bison bootstrap | Same pattern: cross-compile → install → native build attempt. | -| **Get `meson` working** | 1 day | python312 | Create standalone meson script (the TODO in the recipe). python312 already works. | +| **Get `meson` working** | 1 day | python314 | Create standalone meson script (the TODO in the recipe). python314 already works. | | **Get `ninja` working** | 1 day | cmake or python | ninja builds with cmake (which works) or configure.py (python). | | **Validate `libtool`** | 1 day | Phase 1 | libtool builds but not tested. Run test suite, fix issues. | @@ -311,7 +311,7 @@ toolchain binaries. Not truly self-hosting. Updates require rebuild + repackage. | flex/bison circular dependency | High | Medium | Use cross-compiled bootstrap binaries. Standard practice in toolchain bootstrapping. | | GCC native build is too large (memory/disk) | Medium | Medium | GCC is ~500MB source, ~2GB build. Red Bear OS images are 1.5-4GB. May need larger images or swap. | | Make jobserver (`make -jN`) blocked by mkfifo | High | Low | Single-threaded `make` still works — just slower. Acceptable for initial porting. | -| Python312 module loading issues | Low | Medium | Dynamic loading of C modules works for main python312. May need fixes for specific modules meson uses. | +| Python314 module loading issues | Low | Medium | Dynamic loading of C modules works for main python314. May need fixes for specific modules meson uses. | | LLVM native build too resource-intensive | Medium | High | LLVM is ~3GB source, ~20GB build. May need to build on host and install as pre-built pkgar. | ## Resource Estimates