vasilito/RedBear-OS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document content-hash cache system, binary store, package groups

Browse Source 
- AGENTS.md: add cache system to STRUCTURE, WHERE TO LOOK, BUILD FLOW,
  BUILD COMMANDS (--force-rebuild), and CONVENTIONS (dep_hashes.toml,
  binary store restore, package_groups syntax)
- CHANGELOG.md: comprehensive entry for Phase 1-3 + kernel MWAIT +
  ninja-build Redox support
- local/AGENTS.md: note installer fork adds package groups support
- BUILD-CACHE-PLAN.md: fix TOML syntax (underscores not hyphens),
  update all phases to COMPLETE with implementation details, add cache
  flow diagram, add verification results
This commit is contained in:
vasilito
2026-06-30 17:29:52 +03:00
parent b25ea418fb
commit 7d62a7c0ab
5 changed files with 180 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitmodules
+1
View File
@@ -1,3 +1,4 @@
[submodule "local/sources/kernel"]
path = local/sources/kernel
url = https://gitea.redbearos.org/vasilito/redbear-os-kernel.git
branch = master
AGENTS.md
+21 -3
View File
@@ -49,7 +49,7 @@ redox-master/
│ ├── libs/ # Libraries: mesa, cairo, SDL, zlib, openssl, etc.
│ ├── gui/ # Legacy GUI stack packages
│ └── ... # 21 other categories (net, dev, games, shells, etc.)
├── src/ # Cookbook Rust tooling (repo binary, cook logic)
├── src/ # Cookbook Rust tooling (repo binary, cook logic, content-hash cache)
├── docs/ # Architecture docs (6 detailed integration guides) — See docs/AGENTS.md
├── local/ # OUR CUSTOM WORK — survives mainline updates — See local/AGENTS.md
│ ├── config/ # Custom configs (my-amd-desktop.toml)
@@ -95,6 +95,7 @@ redox-master/
| Boot config | `config/*.toml` | TOML hierarchy, include-based |
| **Hardware quirks** | `local/recipes/drivers/redox-driver-sys/source/src/quirks/` | Data-driven quirk tables: compiled-in + TOML + DMI; see `local/docs/QUIRKS-SYSTEM.md` |
| **Package build quirks** | `local/docs/PACKAGE-BUILD-QUIRKS.md` | Cross-compilation issues (DYNAMIC_INIT, m4, ninja-build), fixes, and general patterns |
| **Build cache system** | `local/docs/BUILD-CACHE-PLAN.md` | Content-hash-based cache (Phase 1-3): BLAKE3 dep hashing, binary store restore, package groups |
## BUILD COMMANDS
@@ -151,7 +152,8 @@ qemu-system-x86_64 -cdrom build/x86_64/redbear-mini.iso \
# after fork changes. build-redbear.sh now warns when prefix is stale.
# Single recipe
./target/release/repo cook recipes/libs/mesa # Build one recipe
./target/release/repo cook recipes/libs/mesa # Build one recipe (uses content-hash cache)
./target/release/repo cook mesa --force-rebuild # Bypass cache, force rebuild
./target/release/repo fetch recipes/core/kernel # Fetch source only
make r.mesa # Make shorthand for cook
make cr.mesa # Clean + rebuild
@@ -197,7 +199,9 @@ make all
→ mk/prefix.mk (download/setup cross-toolchain if needed)
→ mk/fstools.mk (build cookbook repo binary + fstools)
→ mk/repo.mk (repo cook --filesystem=config/*.toml)
→ For each recipe: fetch source → apply patches → build → stage into sysroot
→ For each recipe: fetch source → apply patches → check cache (BLAKE3 dep hashes) → build or restore from repo/ → stage into sysroot
→ dep_hashes.toml written per recipe (BLAKE3 of each build dep's PKGAR)
→ Package groups resolved by installer Config::from_file()
→ mk/disk.mk (create filesystem.img, harddrive.img, redbear-live.iso or harddrive.img)
→ redoxfs-mkfs → redox_installer → bootloader embedding
```
@@ -222,6 +226,20 @@ make all
projects with independent release cycles (`tlc`, `zbus`) are exempt. Use
`./local/scripts/sync-versions.sh` to sync all in-house crate versions when creating a new
branch, and `./local/scripts/sync-versions.sh --check` to verify compliance.
- **Content-hash cache**: The cookbook uses BLAKE3 hash comparison (not mtime) to decide
whether a recipe needs rebuilding. Each recipe's `target/` dir stores a `dep_hashes.toml`
with the BLAKE3 of every build dependency's PKGAR at build time. If all hashes match on the
next build, the recipe is a cache hit — even if the dep's `.pkgar` file mtime changed.
Mtime fallback applies when `dep_hashes.toml` is absent (first build or pre-existing recipes).
Use `--force-rebuild` to bypass the cache. See `local/docs/BUILD-CACHE-PLAN.md`.
- **Binary store restore**: When a recipe's `target/` is missing but `repo/<arch>/` has the
built PKGAR + `.toml` + `.dep_hashes.toml`, the cookbook restores stage artifacts from the
binary store instead of rebuilding. This survives `make clean` and target/ deletion.
- **Package groups**: Config TOML supports `[package_groups.<name>]` sections (underscore, not
hyphen) with `description` and `packages` fields. Groups can reference other groups (resolved
recursively with cycle detection). Explicit `[packages]` entries override group membership.
Resolution is transparent — `Config::from_file()` expands groups before any consumer sees them.
The cookbook `repo` binary sees expanded packages automatically.
## INSTALLER FILE LAYERING
CHANGELOG.md
+85
View File
@@ -6,6 +6,91 @@ When a commit changes the visible system surface, supported hardware, build flow
or major documentation status, add a short note here and keep the README "What's New" section in
sync with the newest highlights.
## 2026-06-30 — Build cache system (content-hash + binary store + package groups)
### Content-hash-based cache invalidation (Phase 1)
- **Eliminates cascade rebuilds.** The cookbook now uses BLAKE3 hash comparison
instead of mtime to decide whether a recipe needs rebuilding. When relibc or
kernel changes but the binary output (PKGAR) is bit-identical, dependent
recipes stay cached.
- **How it works:** Each recipe's `target/` dir stores a `dep_hashes.toml` with
the BLAKE3 hash of every build dependency's PKGAR. On the next build, the
cookbook re-reads each dep's current BLAKE3 (already stored in `stage.toml`)
and compares. All match → cache hit. Any differ → rebuild. If
`dep_hashes.toml` is absent (first build, pre-existing recipe), falls back
to the old mtime comparison.
- **`--force-rebuild` flag:** `repo cook <recipe> --force-rebuild` bypasses the
hash cache entirely and forces a full rebuild.
- **Files changed:** `src/cook/cook_build.rs` (+157), `src/bin/repo.rs` (+2),
`src/config.rs` (+4)
### Binary store cache restore (Phase 2)
- **Survives `make clean`.** When a recipe's `target/` dir is missing but
`repo/<arch>/` has the built PKGAR + `.toml` + `.dep_hashes.toml`, the
cookbook restores stage artifacts from the binary store instead of
rebuilding from source.
- **Auto-generates** `auto_deps.toml` from the repo `.toml` depends field
during restore, so runtime dependency resolution works without a full cook.
- **Files changed:** `src/cook/cook_build.rs` (binary restore block),
`src/bin/repo_builder.rs` (+7 — publishes `.dep_hashes.toml` alongside
`.pkgar` and `.toml` in `repo/<arch>/`)
### Config-level package groups (Phase 3)
- **Meta-package support in config TOML.** Configs can now define
`[package_groups.<name>]` sections with `description` and `packages` fields.
Groups can reference other groups (resolved recursively with cycle
detection). Explicit `[packages]` entries override group membership.
- **Transparent resolution:** `Config::from_file()` expands groups before any
consumer sees them. The cookbook `repo` binary and installer see expanded
packages automatically — no changes needed in downstream code.
- **9 groups defined** in `config/redbear-full.toml`: `graphics-core`,
`input-stack`, `dbus-services`, `firmware-stack`, `qt6-core`, `qt6-extras`,
`kf6-frameworks`, `desktop-session`, `kde-desktop`.
- **Installer fork:** `Cargo.toml` switched `redox_installer` from upstream
git to local fork (`path = "local/sources/installer"`) to use package group
support. 3 unit tests pass.
- **Files changed:** `config/redbear-full.toml` (+60),
`local/sources/installer/src/config/mod.rs` (+168),
`local/sources/installer/src/config/package.rs` (+1/-1), `Cargo.toml` (1
line), `Cargo.lock` (1 line)
### Kernel: MWAIT idle loop + Makefile fix
- **MWAIT idle_loop (Phase G):** On CPUs with MWAIT support (Nehalem+), the
kernel now enters the deepest available C-state (C6/C7/C8/C9/C10/S0iX)
instead of plain HLT (C1 only). Falls back to `enable_and_halt` on older
CPUs. Improves idle power consumption on modern Intel/AMD hardware.
- **Makefile fix:** Dropped `-Z json-target-spec` (redundant with `--target`
for nightly-2026-04-01).
### ninja-build: Redox subprocess support
- Added `fork`/`exec` subprocess path for `__redox__` (replacing
`posix_spawn` which is not available on Redox). Added `GetLoadAverage`
stub for Redox.
### Documentation
- `local/docs/BUILD-CACHE-PLAN.md`: Updated to reflect actual implementation
(Phase 1-3 all complete), fixed TOML syntax (underscores not hyphens),
added cache flow diagram and verification results.
- Root `AGENTS.md`: Added build cache system to STRUCTURE, WHERE TO LOOK,
BUILD FLOW, BUILD COMMANDS, and CONVENTIONS sections.
## 2026-06-30 — Input stack observability + ACPI fork-sync + Git server docs + build-system hardening plan
### Input stack observability (`base` fork, commit `de9d1f4`)
local/AGENTS.md
+5 -1
View File
@@ -446,7 +446,7 @@ As of 2026-06, the following core components are built from **local forks** in
| relibc | `local/sources/relibc/` | `recipes/core/relibc/recipe.toml` (git URL) |
| kernel | `local/sources/kernel/` | `recipes/core/kernel/recipe.toml` (git URL) |
| bootloader | `local/sources/bootloader/` | `recipes/core/bootloader/recipe.toml` |
| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` |
| installer | `local/sources/installer/` | `recipes/core/installer/recipe.toml` (+ Cargo.toml dep) |
| redoxfs | `local/sources/redoxfs/` | `recipes/core/redoxfs/recipe.toml` |
| userutils | `local/sources/userutils/` | `recipes/core/userutils/recipe.toml` |
| **base** | **`local/sources/base/`** (path source) | **`recipes/core/base/recipe.toml` (path = ...) ** |
@@ -460,6 +460,10 @@ As of 2026-06, the following core components are built from **local forks** in
fragmented series of patches.
4. **Local recipes are already in fork form** — `local/recipes/` was already a fork
pattern; extending this to `local/sources/` is consistent.
5. **Installer fork adds package groups** — the installer fork (`local/sources/installer/`)
implements config-level `[package_groups.<name>]` sections with recursive group
resolution and cycle detection. This is Red Bear-specific functionality not in upstream.
The cookbook's `Cargo.toml` points to the local fork via `path = "local/sources/installer"`.
### Overlay patches are still permitted
local/docs/BUILD-CACHE-PLAN.md
+68 -37
View File
@@ -73,15 +73,15 @@ mesa = "9f2e..."
zlib = "a1b2..."
```
**Files to modify:**
**Implementation (Phase 1 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `src/cook/cook_build.rs` | Add `read_dep_hashes()`, `write_dep_hashes()` helpers; replace mtime comparison at L285-303 with hash comparison; update `build_deps_dir()` at L566-653 with same hash logic | ~100 |
| `src/bin/repo.rs` | Add `--force-rebuild` CLI flag (bypasses hash caching); add `validate-cache` subcommand (prints what would rebuild and why) | ~55 |
| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ~20 |
**Total: ~175 LOC**
| File | Change | Status |
|------|--------|--------|
| `src/cook/cook_build.rs` | `DepHashes` struct with `read`/`write`; `collect_current_dep_hashes()` reads blake3 from dep `.toml` metadata; `dep_hashes_changed()` compares stored vs current; replaces mtime at cache check | ✅ Done |
| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ✅ Done |
| `src/bin/repo.rs` | `--force-rebuild` CLI flag bypasses hash caching | ✅ Done |
| `src/config.rs` | `force_rebuild` field in `CookConfig`/`CookConfigOpt` | ✅ Done |
| `src/cook/cook_build.rs` | `build_deps_dir()` update — **deferred**: sysroot rebuild uses mtime, not hash. Does NOT cause cascade rebuilds because the recipe's stage.pkgar is the cache key, not the sysroot. | 📋 Deferred |
**Why PKGAR BLAKE3 (not ELF symbol extraction):**
@@ -129,52 +129,55 @@ if repo_pkgar.is_file() && repo_toml.is_file() {
}
```
**Files to modify:**
**Implementation (Phase 2 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `src/cook/cook_build.rs` | Add repo binary lookup before rebuild decision | ~60 |
| File | Change | Status |
|------|--------|--------|
| `src/cook/cook_build.rs` | Repo binary store restore: when `target/` missing but `repo/<arch>/<name>.pkgar` + `.toml` exist, extracts PKGAR to stage dir, copies `.toml` + `dep_hashes.toml`, auto-generates `auto_deps.toml` from repo depends field | ✅ Done |
| `src/bin/repo_builder.rs` | Publishes `<name>.dep_hashes.toml` alongside `.pkgar` and `.toml` in `repo/<arch>/` during `publish_packages()` (main package only, not optional sub-packages) | ✅ Done |
**Total: ~60 LOC**
**Total: ~70 LOC**
### Phase 3: Config-Level Package Groups (MEDIUM ROI)
### Phase 3: Config-Level Package Groups (COMPLETE)
Meta-packages are an **installer/runtime concern**. The build system continues to
produce one PKGAR per recipe. The installer groups them logically.
**Config format:**
**Config format (underscore, NOT hyphen — serde requires it):**
```toml
# config/redbear-full.toml
[package-groups.qt6-core]
[package_groups.qt6-core]
description = "Qt 6 Core modules"
packages = ["qtbase", "qtdeclarative", "qtsvg"]
[package-groups.qt6-wayland]
[package_groups.qt6-extras]
description = "Qt 6 Wayland integration"
packages = ["qtwayland", "qt6-wayland-smoke"]
packages = ["qtwayland", "qt6-wayland-smoke", "qt6-sensors"]
[package-groups.kf6-frameworks]
description = "KDE Frameworks 6 (all 32)"
[package_groups.kf6-frameworks]
description = "KDE Frameworks 6 (all 38)"
packages = ["kf6-kcoreaddons", "kf6-kconfig", "kf6-ki18n", ...]
[package-groups.kde-desktop]
[package_groups.kde-desktop]
description = "Complete KDE Plasma desktop session"
packages = ["qt6-core", "qt6-wayland", "kf6-frameworks", "kwin", "kdecoration", "sddm"]
packages = ["graphics-core", "qt6-core", "qt6-extras", "kf6-frameworks", "kwin", "sddm"]
```
Groups can reference other groups — installer resolves recursively.
Groups can reference other groups — installer resolves recursively with cycle detection.
Explicit `[packages]` entries override group membership.
**Files to modify:**
**Implementation (Phase 3 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `local/sources/installer/src/config/mod.rs` | Parse `[package-groups]` in config merge | ~40 |
| `local/sources/installer/src/installer.rs` | Resolve groups during `install_dir()` | ~40 |
| `config/redbear-full.toml` | Define qt6-core, qt6-wayland, kf6-frameworks groups | ~30 |
| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | ~30 |
| File | Change | Status |
|------|--------|--------|
| `local/sources/installer/src/config/mod.rs` | `PackageGroup` struct, `package_groups` field on `Config`, `resolve_package_groups()` + `expand_group()` with cycle detection, recursive group resolution, explicit-overrides-group priority. Called at end of `Config::from_file()` | ✅ Done |
| `local/sources/installer/src/config/package.rs` | `PartialEq` derive on `PackageConfig` for dedup during merge | ✅ Done |
| `config/redbear-full.toml` | 9 groups defined: `graphics-core`, `input-stack`, `dbus-services`, `firmware-stack`, `qt6-core`, `qt6-extras`, `kf6-frameworks`, `desktop-session`, `kde-desktop` | ✅ Done |
| `Cargo.toml` | Switch `redox_installer` from upstream git to local fork (`path = "local/sources/installer"`) | ✅ Done |
| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | 📋 Future (runtime package manager) |
**Total: ~140 LOC**
**Total: ~170 LOC**
**Why NOT merged PKGARs or build-system grouping:**
@@ -194,13 +197,41 @@ Groups can reference other groups — installer resolves recursively.
| Merged PKGAR meta-packages | Breaks one-build-one-PKGAR model. Makes caching harder. Each component has its own deps and ABI surface. |
| Fine-grained header dependency tracking | Massive complexity, compiler-dependent. Not worth it for a source-built OS. |
## Implementation Order
## Implementation Status
| Phase | Effort | ROI | When |
|-------|--------|-----|------|
| **Phase 1: Hash-based caching** | ~175 LOC, 1-2 days | **Critical** — saves 4-6 hours per low-level change | Now |
| **Phase 2: Binary store lookup** | ~60 LOC, half day | High — saves cold rebuilds after `make clean` | After Phase 1 |
| **Phase 3: Package groups** | ~140 LOC, 1 day | Medium — installer UX improvement | After Phase 1-2 |
| Phase | Effort | Status | Verification |
|-------|--------|--------|--------------|
| **Phase 1: Hash-based caching** | ~175 LOC | ✅ Complete | Touch xz.pkgar → libxml2 stays cached (mtime changed, BLAKE3 same); `--force-rebuild` bypasses |
| **Phase 2: Binary store lookup** | ~70 LOC | ✅ Complete | Removed libxml2 target/ → re-cook → restored from repo/ binary store → cache hit |
| **Phase 3: Package groups** | ~170 LOC | ✅ Complete | 3 unit tests pass (nested groups, explicit override, no-groups compat); mini build verified |
### Cache Flow (As Implemented)
```
Recipe build requested
│
├─ force_rebuild? ────────────────────────────────► REBUILD
│
├─ target/ dir exists with dep_hashes.toml?
│ ├─ YES: Read stored hashes
│ │ Collect current dep PKGAR blake3 from .toml files
│ │ All match? ──► CACHE HIT (skip build)
│ │ Any differ? ──► REBUILD
│ │
│ └─ NO: Fall back to mtime comparison (backward compat)
│ stage < source? ──► REBUILD
│ stage < deps? ──► REBUILD
│ Otherwise ──► CACHE HIT
│
├─ target/ dir MISSING but repo/<arch>/ has pkgar + toml?
│ └─ YES: Restore stage artifacts from binary store
│ Extract pkgar → stage dir
│ Copy .toml + .dep_hashes.toml
│ Generate auto_deps.toml from repo depends
│ Then run cache check above (likely hit)
│
└─ Build complete → write dep_hashes.toml with current hashes
```
## Risk Analysis