From 7d62a7c0abcf148bdbcaf2ebefab17d0e41dd7eb Mon Sep 17 00:00:00 2001 From: vasilito Date: Tue, 30 Jun 2026 17:29:52 +0300 Subject: [PATCH] docs: document content-hash cache system, binary store, package groups - AGENTS.md: add cache system to STRUCTURE, WHERE TO LOOK, BUILD FLOW, BUILD COMMANDS (--force-rebuild), and CONVENTIONS (dep_hashes.toml, binary store restore, package_groups syntax) - CHANGELOG.md: comprehensive entry for Phase 1-3 + kernel MWAIT + ninja-build Redox support - local/AGENTS.md: note installer fork adds package groups support - BUILD-CACHE-PLAN.md: fix TOML syntax (underscores not hyphens), update all phases to COMPLETE with implementation details, add cache flow diagram, add verification results --- .gitmodules | 1 + AGENTS.md | 24 +++++++- CHANGELOG.md | 85 ++++++++++++++++++++++++++ local/AGENTS.md | 6 +- local/docs/BUILD-CACHE-PLAN.md | 105 +++++++++++++++++++++------------ 5 files changed, 180 insertions(+), 41 deletions(-) diff --git a/.gitmodules b/.gitmodules index 083a6ed0f8..26fc55f121 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,4 @@ [submodule "local/sources/kernel"] path = local/sources/kernel url = https://gitea.redbearos.org/vasilito/redbear-os-kernel.git + branch = master diff --git a/AGENTS.md b/AGENTS.md index f032dc5536..9c3de26a3e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,7 +49,7 @@ redox-master/ │ ├── libs/ # Libraries: mesa, cairo, SDL, zlib, openssl, etc. │ ├── gui/ # Legacy GUI stack packages │ └── ... # 21 other categories (net, dev, games, shells, etc.) -├── src/ # Cookbook Rust tooling (repo binary, cook logic) +├── src/ # Cookbook Rust tooling (repo binary, cook logic, content-hash cache) ├── docs/ # Architecture docs (6 detailed integration guides) — See docs/AGENTS.md ├── local/ # OUR CUSTOM WORK — survives mainline updates — See local/AGENTS.md │ ├── config/ # Custom configs (my-amd-desktop.toml) @@ -95,6 +95,7 @@ redox-master/ | Boot config | `config/*.toml` | TOML hierarchy, include-based | | **Hardware quirks** | `local/recipes/drivers/redox-driver-sys/source/src/quirks/` | Data-driven quirk tables: compiled-in + TOML + DMI; see `local/docs/QUIRKS-SYSTEM.md` | | **Package build quirks** | `local/docs/PACKAGE-BUILD-QUIRKS.md` | Cross-compilation issues (DYNAMIC_INIT, m4, ninja-build), fixes, and general patterns | +| **Build cache system** | `local/docs/BUILD-CACHE-PLAN.md` | Content-hash-based cache (Phase 1-3): BLAKE3 dep hashing, binary store restore, package groups | ## BUILD COMMANDS @@ -151,7 +152,8 @@ qemu-system-x86_64 -cdrom build/x86_64/redbear-mini.iso \ # after fork changes. build-redbear.sh now warns when prefix is stale. # Single recipe -./target/release/repo cook recipes/libs/mesa # Build one recipe +./target/release/repo cook recipes/libs/mesa # Build one recipe (uses content-hash cache) +./target/release/repo cook mesa --force-rebuild # Bypass cache, force rebuild ./target/release/repo fetch recipes/core/kernel # Fetch source only make r.mesa # Make shorthand for cook make cr.mesa # Clean + rebuild @@ -197,7 +199,9 @@ make all → mk/prefix.mk (download/setup cross-toolchain if needed) → mk/fstools.mk (build cookbook repo binary + fstools) → mk/repo.mk (repo cook --filesystem=config/*.toml) - → For each recipe: fetch source → apply patches → build → stage into sysroot + → For each recipe: fetch source → apply patches → check cache (BLAKE3 dep hashes) → build or restore from repo/ → stage into sysroot + → dep_hashes.toml written per recipe (BLAKE3 of each build dep's PKGAR) + → Package groups resolved by installer Config::from_file() → mk/disk.mk (create filesystem.img, harddrive.img, redbear-live.iso or harddrive.img) → redoxfs-mkfs → redox_installer → bootloader embedding ``` @@ -222,6 +226,20 @@ make all projects with independent release cycles (`tlc`, `zbus`) are exempt. Use `./local/scripts/sync-versions.sh` to sync all in-house crate versions when creating a new branch, and `./local/scripts/sync-versions.sh --check` to verify compliance. +- **Content-hash cache**: The cookbook uses BLAKE3 hash comparison (not mtime) to decide + whether a recipe needs rebuilding. Each recipe's `target/` dir stores a `dep_hashes.toml` + with the BLAKE3 of every build dependency's PKGAR at build time. If all hashes match on the + next build, the recipe is a cache hit — even if the dep's `.pkgar` file mtime changed. + Mtime fallback applies when `dep_hashes.toml` is absent (first build or pre-existing recipes). + Use `--force-rebuild` to bypass the cache. See `local/docs/BUILD-CACHE-PLAN.md`. +- **Binary store restore**: When a recipe's `target/` is missing but `repo//` has the + built PKGAR + `.toml` + `.dep_hashes.toml`, the cookbook restores stage artifacts from the + binary store instead of rebuilding. This survives `make clean` and target/ deletion. +- **Package groups**: Config TOML supports `[package_groups.]` sections (underscore, not + hyphen) with `description` and `packages` fields. Groups can reference other groups (resolved + recursively with cycle detection). Explicit `[packages]` entries override group membership. + Resolution is transparent — `Config::from_file()` expands groups before any consumer sees them. + The cookbook `repo` binary sees expanded packages automatically. ## INSTALLER FILE LAYERING diff --git a/CHANGELOG.md b/CHANGELOG.md index ee2f4493fb..795aa59951 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,91 @@ When a commit changes the visible system surface, supported hardware, build flow or major documentation status, add a short note here and keep the README "What's New" section in sync with the newest highlights. +## 2026-06-30 — Build cache system (content-hash + binary store + package groups) + +### Content-hash-based cache invalidation (Phase 1) + +- **Eliminates cascade rebuilds.** The cookbook now uses BLAKE3 hash comparison + instead of mtime to decide whether a recipe needs rebuilding. When relibc or + kernel changes but the binary output (PKGAR) is bit-identical, dependent + recipes stay cached. + +- **How it works:** Each recipe's `target/` dir stores a `dep_hashes.toml` with + the BLAKE3 hash of every build dependency's PKGAR. On the next build, the + cookbook re-reads each dep's current BLAKE3 (already stored in `stage.toml`) + and compares. All match → cache hit. Any differ → rebuild. If + `dep_hashes.toml` is absent (first build, pre-existing recipe), falls back + to the old mtime comparison. + +- **`--force-rebuild` flag:** `repo cook --force-rebuild` bypasses the + hash cache entirely and forces a full rebuild. + +- **Files changed:** `src/cook/cook_build.rs` (+157), `src/bin/repo.rs` (+2), + `src/config.rs` (+4) + +### Binary store cache restore (Phase 2) + +- **Survives `make clean`.** When a recipe's `target/` dir is missing but + `repo//` has the built PKGAR + `.toml` + `.dep_hashes.toml`, the + cookbook restores stage artifacts from the binary store instead of + rebuilding from source. + +- **Auto-generates** `auto_deps.toml` from the repo `.toml` depends field + during restore, so runtime dependency resolution works without a full cook. + +- **Files changed:** `src/cook/cook_build.rs` (binary restore block), + `src/bin/repo_builder.rs` (+7 — publishes `.dep_hashes.toml` alongside + `.pkgar` and `.toml` in `repo//`) + +### Config-level package groups (Phase 3) + +- **Meta-package support in config TOML.** Configs can now define + `[package_groups.]` sections with `description` and `packages` fields. + Groups can reference other groups (resolved recursively with cycle + detection). Explicit `[packages]` entries override group membership. + +- **Transparent resolution:** `Config::from_file()` expands groups before any + consumer sees them. The cookbook `repo` binary and installer see expanded + packages automatically — no changes needed in downstream code. + +- **9 groups defined** in `config/redbear-full.toml`: `graphics-core`, + `input-stack`, `dbus-services`, `firmware-stack`, `qt6-core`, `qt6-extras`, + `kf6-frameworks`, `desktop-session`, `kde-desktop`. + +- **Installer fork:** `Cargo.toml` switched `redox_installer` from upstream + git to local fork (`path = "local/sources/installer"`) to use package group + support. 3 unit tests pass. + +- **Files changed:** `config/redbear-full.toml` (+60), + `local/sources/installer/src/config/mod.rs` (+168), + `local/sources/installer/src/config/package.rs` (+1/-1), `Cargo.toml` (1 + line), `Cargo.lock` (1 line) + +### Kernel: MWAIT idle loop + Makefile fix + +- **MWAIT idle_loop (Phase G):** On CPUs with MWAIT support (Nehalem+), the + kernel now enters the deepest available C-state (C6/C7/C8/C9/C10/S0iX) + instead of plain HLT (C1 only). Falls back to `enable_and_halt` on older + CPUs. Improves idle power consumption on modern Intel/AMD hardware. + +- **Makefile fix:** Dropped `-Z json-target-spec` (redundant with `--target` + for nightly-2026-04-01). + +### ninja-build: Redox subprocess support + +- Added `fork`/`exec` subprocess path for `__redox__` (replacing + `posix_spawn` which is not available on Redox). Added `GetLoadAverage` + stub for Redox. + +### Documentation + +- `local/docs/BUILD-CACHE-PLAN.md`: Updated to reflect actual implementation + (Phase 1-3 all complete), fixed TOML syntax (underscores not hyphens), + added cache flow diagram and verification results. + +- Root `AGENTS.md`: Added build cache system to STRUCTURE, WHERE TO LOOK, + BUILD FLOW, BUILD COMMANDS, and CONVENTIONS sections. + ## 2026-06-30 — Input stack observability + ACPI fork-sync + Git server docs + build-system hardening plan ### Input stack observability (`base` fork, commit `de9d1f4`) diff --git a/local/AGENTS.md b/local/AGENTS.md index 3bf2a352cb..19212caeb7 100644 --- a/local/AGENTS.md +++ b/local/AGENTS.md @@ -446,7 +446,7 @@ As of 2026-06, the following core components are built from **local forks** in | relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (git URL) | | kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (git URL) | | bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` | -| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` | +| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (+ Cargo.toml dep) | | redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` | | userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` | | **base** | **`local/sources/base/`** (path source) | **`recipes/core/base/recipe.toml` (path = ...) ** | @@ -460,6 +460,10 @@ As of 2026-06, the following core components are built from **local forks** in fragmented series of patches. 4. **Local recipes are already in fork form** — `local/recipes/` was already a fork pattern; extending this to `local/sources/` is consistent. +5. **Installer fork adds package groups** — the installer fork (`local/sources/installer/`) + implements config-level `[package_groups.]` sections with recursive group + resolution and cycle detection. This is Red Bear-specific functionality not in upstream. + The cookbook's `Cargo.toml` points to the local fork via `path = "local/sources/installer"`. ### Overlay patches are still permitted diff --git a/local/docs/BUILD-CACHE-PLAN.md b/local/docs/BUILD-CACHE-PLAN.md index eab4a0b29d..4f8a707daa 100644 --- a/local/docs/BUILD-CACHE-PLAN.md +++ b/local/docs/BUILD-CACHE-PLAN.md @@ -73,15 +73,15 @@ mesa = "9f2e..." zlib = "a1b2..." ``` -**Files to modify:** +**Implementation (Phase 1 — COMPLETE):** -| File | Change | LoC | -|------|--------|-----| -| `src/cook/cook_build.rs` | Add `read_dep_hashes()`, `write_dep_hashes()` helpers; replace mtime comparison at L285-303 with hash comparison; update `build_deps_dir()` at L566-653 with same hash logic | ~100 | -| `src/bin/repo.rs` | Add `--force-rebuild` CLI flag (bypasses hash caching); add `validate-cache` subcommand (prints what would rebuild and why) | ~55 | -| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ~20 | - -**Total: ~175 LOC** +| File | Change | Status | +|------|--------|--------| +| `src/cook/cook_build.rs` | `DepHashes` struct with `read`/`write`; `collect_current_dep_hashes()` reads blake3 from dep `.toml` metadata; `dep_hashes_changed()` compares stored vs current; replaces mtime at cache check | ✅ Done | +| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ✅ Done | +| `src/bin/repo.rs` | `--force-rebuild` CLI flag bypasses hash caching | ✅ Done | +| `src/config.rs` | `force_rebuild` field in `CookConfig`/`CookConfigOpt` | ✅ Done | +| `src/cook/cook_build.rs` | `build_deps_dir()` update — **deferred**: sysroot rebuild uses mtime, not hash. Does NOT cause cascade rebuilds because the recipe's stage.pkgar is the cache key, not the sysroot. | 📋 Deferred | **Why PKGAR BLAKE3 (not ELF symbol extraction):** @@ -129,52 +129,55 @@ if repo_pkgar.is_file() && repo_toml.is_file() { } ``` -**Files to modify:** +**Implementation (Phase 2 — COMPLETE):** -| File | Change | LoC | -|------|--------|-----| -| `src/cook/cook_build.rs` | Add repo binary lookup before rebuild decision | ~60 | +| File | Change | Status | +|------|--------|--------| +| `src/cook/cook_build.rs` | Repo binary store restore: when `target/` missing but `repo//.pkgar` + `.toml` exist, extracts PKGAR to stage dir, copies `.toml` + `dep_hashes.toml`, auto-generates `auto_deps.toml` from repo depends field | ✅ Done | +| `src/bin/repo_builder.rs` | Publishes `.dep_hashes.toml` alongside `.pkgar` and `.toml` in `repo//` during `publish_packages()` (main package only, not optional sub-packages) | ✅ Done | -**Total: ~60 LOC** +**Total: ~70 LOC** -### Phase 3: Config-Level Package Groups (MEDIUM ROI) +### Phase 3: Config-Level Package Groups (COMPLETE) Meta-packages are an **installer/runtime concern**. The build system continues to produce one PKGAR per recipe. The installer groups them logically. -**Config format:** +**Config format (underscore, NOT hyphen — serde requires it):** ```toml # config/redbear-full.toml -[package-groups.qt6-core] +[package_groups.qt6-core] description = "Qt 6 Core modules" packages = ["qtbase", "qtdeclarative", "qtsvg"] -[package-groups.qt6-wayland] +[package_groups.qt6-extras] description = "Qt 6 Wayland integration" -packages = ["qtwayland", "qt6-wayland-smoke"] +packages = ["qtwayland", "qt6-wayland-smoke", "qt6-sensors"] -[package-groups.kf6-frameworks] -description = "KDE Frameworks 6 (all 32)" +[package_groups.kf6-frameworks] +description = "KDE Frameworks 6 (all 38)" packages = ["kf6-kcoreaddons", "kf6-kconfig", "kf6-ki18n", ...] -[package-groups.kde-desktop] +[package_groups.kde-desktop] description = "Complete KDE Plasma desktop session" -packages = ["qt6-core", "qt6-wayland", "kf6-frameworks", "kwin", "kdecoration", "sddm"] +packages = ["graphics-core", "qt6-core", "qt6-extras", "kf6-frameworks", "kwin", "sddm"] ``` -Groups can reference other groups — installer resolves recursively. +Groups can reference other groups — installer resolves recursively with cycle detection. +Explicit `[packages]` entries override group membership. -**Files to modify:** +**Implementation (Phase 3 — COMPLETE):** -| File | Change | LoC | -|------|--------|-----| -| `local/sources/installer/src/config/mod.rs` | Parse `[package-groups]` in config merge | ~40 | -| `local/sources/installer/src/installer.rs` | Resolve groups during `install_dir()` | ~40 | -| `config/redbear-full.toml` | Define qt6-core, qt6-wayland, kf6-frameworks groups | ~30 | -| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | ~30 | +| File | Change | Status | +|------|--------|--------| +| `local/sources/installer/src/config/mod.rs` | `PackageGroup` struct, `package_groups` field on `Config`, `resolve_package_groups()` + `expand_group()` with cycle detection, recursive group resolution, explicit-overrides-group priority. Called at end of `Config::from_file()` | ✅ Done | +| `local/sources/installer/src/config/package.rs` | `PartialEq` derive on `PackageConfig` for dedup during merge | ✅ Done | +| `config/redbear-full.toml` | 9 groups defined: `graphics-core`, `input-stack`, `dbus-services`, `firmware-stack`, `qt6-core`, `qt6-extras`, `kf6-frameworks`, `desktop-session`, `kde-desktop` | ✅ Done | +| `Cargo.toml` | Switch `redox_installer` from upstream git to local fork (`path = "local/sources/installer"`) | ✅ Done | +| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | 📋 Future (runtime package manager) | -**Total: ~140 LOC** +**Total: ~170 LOC** **Why NOT merged PKGARs or build-system grouping:** @@ -194,13 +197,41 @@ Groups can reference other groups — installer resolves recursively. | Merged PKGAR meta-packages | Breaks one-build-one-PKGAR model. Makes caching harder. Each component has its own deps and ABI surface. | | Fine-grained header dependency tracking | Massive complexity, compiler-dependent. Not worth it for a source-built OS. | -## Implementation Order +## Implementation Status -| Phase | Effort | ROI | When | -|-------|--------|-----|------| -| **Phase 1: Hash-based caching** | ~175 LOC, 1-2 days | **Critical** — saves 4-6 hours per low-level change | Now | -| **Phase 2: Binary store lookup** | ~60 LOC, half day | High — saves cold rebuilds after `make clean` | After Phase 1 | -| **Phase 3: Package groups** | ~140 LOC, 1 day | Medium — installer UX improvement | After Phase 1-2 | +| Phase | Effort | Status | Verification | +|-------|--------|--------|--------------| +| **Phase 1: Hash-based caching** | ~175 LOC | ✅ Complete | Touch xz.pkgar → libxml2 stays cached (mtime changed, BLAKE3 same); `--force-rebuild` bypasses | +| **Phase 2: Binary store lookup** | ~70 LOC | ✅ Complete | Removed libxml2 target/ → re-cook → restored from repo/ binary store → cache hit | +| **Phase 3: Package groups** | ~170 LOC | ✅ Complete | 3 unit tests pass (nested groups, explicit override, no-groups compat); mini build verified | + +### Cache Flow (As Implemented) + +``` +Recipe build requested + │ + ├─ force_rebuild? ────────────────────────────────► REBUILD + │ + ├─ target/ dir exists with dep_hashes.toml? + │ ├─ YES: Read stored hashes + │ │ Collect current dep PKGAR blake3 from .toml files + │ │ All match? ──► CACHE HIT (skip build) + │ │ Any differ? ──► REBUILD + │ │ + │ └─ NO: Fall back to mtime comparison (backward compat) + │ stage < source? ──► REBUILD + │ stage < deps? ──► REBUILD + │ Otherwise ──► CACHE HIT + │ + ├─ target/ dir MISSING but repo// has pkgar + toml? + │ └─ YES: Restore stage artifacts from binary store + │ Extract pkgar → stage dir + │ Copy .toml + .dep_hashes.toml + │ Generate auto_deps.toml from repo depends + │ Then run cache check above (likely hit) + │ + └─ Build complete → write dep_hashes.toml with current hashes +``` ## Risk Analysis