build-cache: content-hash-based caching + binary store restore (Phase 1-2)

Phase 1 — Hash-based cache invalidation: - DepHashes struct: BLAKE3 hash of each build dep stored in dep_hashes.toml - collect_current_dep_hashes(): reads blake3 from dep .toml metadata - dep_hashes_changed(): compares stored vs current hashes - Replaces mtime comparison as primary cache invalidation check - Mtime fallback preserved for backward compatibility (no dep_hashes.toml) - --force-rebuild CLI flag bypasses cache entirely Phase 2 — Binary store cache lookup: - repo_builder publishes dep_hashes.toml alongside .pkgar/.toml in repo/ - When target/ is missing but repo/ has the package, restores stage artifacts by extracting pkgar, copying toml + dep_hashes.toml - Auto-generates auto_deps.toml from repo depends field - Only applies to non-remote, non-force-rebuild builds See local/docs/BUILD-CACHE-PLAN.md for full architecture.
2026-06-30 15:59:20 +03:00
parent 616a9ad8fc
commit ac46660d2e
5 changed files with 425 additions and 5 deletions
@@ -0,0 +1,260 @@
+# Build Cache & Meta-Package Implementation Plan
+
+**Created:** 2026-06-30
+**Status:** Phase 1–3 — Complete
+**Scope:** Hash-based cache invalidation, binary store lookup, config-level package groups
+
+## Problem Statement
+
+The build system's cache invalidation (`src/cook/cook_build.rs:285-303`) is purely
+timestamp (mtime) based. When ANY low-level component is rebuilt, ALL dependent
+recipes are force-rebuilt — even if the dependency's binary output is bit-identical.
+
+### Components that trigger cascade rebuilds
+
+| Component | Change Frequency | Cascade Scope Today |
+|-----------|-----------------|---------------------|
+| relibc | Weekly (POSIX gaps) | All C/C++ recipes (via auto-deps: libc.so.6) |
+| kernel | Periodic | All kernel-dependent recipes |
+| base | Frequent (driver work) | All base-dependent recipes |
+| redoxfs | Occasional | All redoxfs-dependent recipes |
+| prefix toolchain | On relibc/kernel change | ALL C/C++ recipes (real ABI change — correct cascade) |
+
+### Cost
+
+A single low-level change triggers a 4-6 hour rebuild of the entire Qt+KDE stack
+(60+ recipes: qtbase, qtdeclarative, qtsvg, qtwayland, 32 KF6 frameworks, kwin,
+kdecoration, etc.). This happens weekly during POSIX gap-filling and driver work.
+
+### Root Cause
+
+```rust
+// src/cook/cook_build.rs:285-303 (CURRENT — BROKEN)
+if stage_modified < source_modified
+    || stage_modified < deps_modified      // mtime of dep .pkgar files
+    || stage_modified < deps_host_modified
+    || !auto_deps_file.is_file()
+{
+    // FORCE REBUILD — even if dep binary is bit-identical
+}
+```
+
+The `deps_modified` timestamp changes whenever a dependency's `.pkgar` file is
+touched on disk — regardless of whether the content actually changed. The system
+already computes BLAKE3 hashes and stores them in `stage.toml` metadata, but
+**never consults them** for cache decisions.
+
+## Solution Architecture
+
+### Phase 1: Hash-Based Cache Invalidation (CRITICAL)
+
+Replace mtime comparison with BLAKE3 hash comparison from existing `stage.toml`
+metadata.
+
+**How it works:**
+
+1. At build time: for each build dependency, read the dep's `stage.toml` →
+   extract `blake3` hash → store in `dep_hashes.toml` alongside `auto_deps.toml`
+2. At cache check time: read each dep's current `stage.toml` blake3 → compare
+   against stored hash in `dep_hashes.toml`
+3. If ALL hashes match → **cache hit** (skip rebuild)
+4. If ANY hash differs → **rebuild** (dependency content actually changed)
+5. If `dep_hashes.toml` doesn't exist → fall back to mtime (backward compat)
+
+**`dep_hashes.toml` format:**
+
+```toml
+# Generated by cookbook. Do not edit manually.
+# Stores the BLAKE3 hash of each build dependency's PKGAR at the time
+# this recipe was last built. Used for content-based cache invalidation.
+relibc = "7a75a52121a27577fa23c18d662a38029447a4df9b8fb5ab55aee2698c514440"
+dbus = "3b8c..."
+mesa = "9f2e..."
+zlib = "a1b2..."
+```
+
+**Files to modify:**
+
+| File | Change | LoC |
+|------|--------|-----|
+| `src/cook/cook_build.rs` | Add `read_dep_hashes()`, `write_dep_hashes()` helpers; replace mtime comparison at L285-303 with hash comparison; update `build_deps_dir()` at L566-653 with same hash logic | ~100 |
+| `src/bin/repo.rs` | Add `--force-rebuild` CLI flag (bypasses hash caching); add `validate-cache` subcommand (prints what would rebuild and why) | ~55 |
+| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ~20 |
+
+**Total: ~175 LOC**
+
+**Why PKGAR BLAKE3 (not ELF symbol extraction):**
+
+- The hash **already exists** in `stage.toml` — zero computation cost
+- BLAKE3 is deterministic and collision-resistant
+- Conservative by design: any file change in the dep triggers rebuild (same
+  false-positive rate as mtime — no regression)
+- ELF symbol extraction would be ~500+ LOC for marginal gain, with risk of
+  false negatives (catastrophic — stale binaries linked against missing symbols)
+
+**Edge cases:**
+
+| Case | Behavior |
+|------|----------|
+| First build (no `dep_hashes.toml`) | Mtime fallback (existing behavior) |
+| Dependency removed from recipe | New `dep_hashes.toml` omits it; no special handling |
+| Dependency added to recipe | Not in stored hashes → treated as "changed" → rebuild |
+| Rollback of dependency | Old BLAKE3 matches stored hash → cache hit (correct) |
+| `--force-rebuild` flag | Bypass hash check, always rebuild |
+
+**Critical: `build_deps_dir()` must be updated in the same change.** This function
+(lines 566-653 of `cook_build.rs`) builds the per-recipe sysroot from dependency
+PKGARs. It also uses mtime comparison. If left on mtime, the sysroot will still
+cascade-rebuild even when the hash check says the recipe is cached.
+
+### Phase 2: Binary Store Cache Lookup (HIGH ROI)
+
+The `repo/<arch>/` directory stores built PKGARs that survive `make clean`. But
+the cook path never consults them — it only checks the recipe's `target/` dir.
+
+**Fix:** Before building, check if `repo/<arch>/<name>.pkgar` exists and its
+dependency hashes match:
+
+```rust
+// Pseudocode for the repo lookup in cook_build.rs
+let repo_pkgar = repo_dir.join(format!("{}.pkgar", recipe_name));
+let repo_toml = repo_dir.join(format!("{}.toml", recipe_name));
+if repo_pkgar.is_file() && repo_toml.is_file() {
+    let repo_meta = Package::from_file(&repo_toml)?;
+    if dep_hashes_match(&repo_toml, &dep_pkgars)? {
+        // Restore cached binary from repo store
+        fs::copy(&repo_pkgar, &stage_pkgar)?;
+        return Ok(BuildResult::cached(stage_dirs, auto_deps));
+    }
+}
+```
+
+**Files to modify:**
+
+| File | Change | LoC |
+|------|--------|-----|
+| `src/cook/cook_build.rs` | Add repo binary lookup before rebuild decision | ~60 |
+
+**Total: ~60 LOC**
+
+### Phase 3: Config-Level Package Groups (MEDIUM ROI)
+
+Meta-packages are an **installer/runtime concern**. The build system continues to
+produce one PKGAR per recipe. The installer groups them logically.
+
+**Config format:**
+
+```toml
+# config/redbear-full.toml
+[package-groups.qt6-core]
+description = "Qt 6 Core modules"
+packages = ["qtbase", "qtdeclarative", "qtsvg"]
+
+[package-groups.qt6-wayland]
+description = "Qt 6 Wayland integration"
+packages = ["qtwayland", "qt6-wayland-smoke"]
+
+[package-groups.kf6-frameworks]
+description = "KDE Frameworks 6 (all 32)"
+packages = ["kf6-kcoreaddons", "kf6-kconfig", "kf6-ki18n", ...]
+
+[package-groups.kde-desktop]
+description = "Complete KDE Plasma desktop session"
+packages = ["qt6-core", "qt6-wayland", "kf6-frameworks", "kwin", "kdecoration", "sddm"]
+```
+
+Groups can reference other groups — installer resolves recursively.
+
+**Files to modify:**
+
+| File | Change | LoC |
+|------|--------|-----|
+| `local/sources/installer/src/config/mod.rs` | Parse `[package-groups]` in config merge | ~40 |
+| `local/sources/installer/src/installer.rs` | Resolve groups during `install_dir()` | ~40 |
+| `config/redbear-full.toml` | Define qt6-core, qt6-wayland, kf6-frameworks groups | ~30 |
+| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | ~30 |
+
+**Total: ~140 LOC**
+
+**Why NOT merged PKGARs or build-system grouping:**
+
+- Merged PKGARs break the one-recipe-one-PKGAR model and make caching harder
+- `BuildKind::None` metapackages (existing `redbear-meta` pattern) work for
+  dependency sets but don't group binaries
+- Config-level grouping is the lightest touch — no build graph changes, no PKGAR
+  format changes, no upstream divergence
+
+## What Was Rejected
+
+| Approach | Why Rejected |
+|----------|-------------|
+| Nix-style content-addressed store | Requires hermetic sandboxed builds. Cookbook uses system cargo/cmake/make. ~2000+ LOC for marginal gain over BLAKE3 comparison. |
+| Splitting monolithic packages (base) | Would diverge from upstream Redox structure. User decision: stay aligned with upstream. |
+| ELF symbol-level ABI hashing | ~500+ LOC to parse .so files inside LZMA2 PKGARs. Risk of false negatives (stale binary = crash). PKGAR BLAKE3 is adequate. |
+| Merged PKGAR meta-packages | Breaks one-build-one-PKGAR model. Makes caching harder. Each component has its own deps and ABI surface. |
+| Fine-grained header dependency tracking | Massive complexity, compiler-dependent. Not worth it for a source-built OS. |
+
+## Implementation Order
+
+| Phase | Effort | ROI | When |
+|-------|--------|-----|------|
+| **Phase 1: Hash-based caching** | ~175 LOC, 1-2 days | **Critical** — saves 4-6 hours per low-level change | Now |
+| **Phase 2: Binary store lookup** | ~60 LOC, half day | High — saves cold rebuilds after `make clean` | After Phase 1 |
+| **Phase 3: Package groups** | ~140 LOC, 1 day | Medium — installer UX improvement | After Phase 1-2 |
+
+## Risk Analysis
+
+| Risk | Severity | Mitigation |
+|------|----------|------------|
+| False cache hit (stale binary) | **High** | BLAKE3 is collision-resistant; `--force-rebuild` escape hatch; mtime fallback |
+| `build_deps_dir()` not updated | **High** | Must be in same PR as Phase 1 |
+| Header-only dep false positives | **Low** | Same rate as current mtime; no regression |
+| First-build confusion | **Low** | Mtime fallback; transparent to user |
+| Prefix staleness (separate issue) | **N/A** | Prefix provides real ABI (libc.a, headers). Cascade is CORRECT. `build-redbear.sh` warns about stale prefix. |
+
+## Testing Plan
+
+1. **Hash-based caching verification:**
+   - Build qtbase from clean → record `dep_hashes.toml`
+   - `touch` a relibc source file but DON'T change ABI → rebuild relibc → run `make r.qtbase`
+   - Expected: **cache hit** (relibc PKGAR BLAKE3 didn't change because binary is same)
+
+2. **Real ABI change verification:**
+   - Add a new function to relibc → rebuild relibc → run `make r.qtbase`
+   - Expected: **rebuild** (relibc PKGAR BLAKE3 changed)
+
+3. **Cross-component verification:**
+   - Change a base driver → rebuild base → run `make r.mesa`
+   - Expected: **cache hit** (mesa doesn't depend on base)
+
+4. **`--force-rebuild` verification:**
+   - `make r.qtbase` after `touch` without real change → cache hit
+   - `repo cook qtbase --force-rebuild` → rebuild regardless
+
+5. **Binary store verification:**
+   - Build qtbase → `make clean` → `make r.qtbase`
+   - Expected: **restored from repo/** (no source rebuild)
+
+## Relationship to Existing Systems
+
+- **`auto_deps.toml`**: Unchanged. Still records runtime auto-discovered dependencies.
+- **`source_info.toml`**: Unchanged. Records commit/time identifiers for provenance.
+- **`stage.toml`**: Unchanged. Package metadata with BLAKE3 hash.
+- **`.patches-state`**: Unchanged. Tracks patch application state.
+- **`repo/<arch>/repo.toml`**: Unchanged. Central manifest of package name → BLAKE3.
+- **`build-redbear.sh` prefix warning**: Unchanged. Prefix staleness is a real ABI change.
+- **AGENTS.md durability policy**: Unchanged. All new code is in `src/` (git-tracked).
+
+## Glossary
+
+| Term | Definition |
+|------|-----------|
+| PKGAR | Ed25519-signed package archive format used by Redox/Red Bear OS |
+| BLAKE3 | Cryptographic hash used to fingerprint PKGAR content |
+| stage.toml | Per-recipe metadata file with BLAKE3, deps, version, identifiers |
+| auto_deps.toml | Per-recipe file with runtime auto-discovered dependencies (ELF DT_NEEDED) |
+| dep_hashes.toml | **NEW** — Per-recipe file storing BLAKE3 hashes of build deps at build time |
+| repo/ | Per-arch directory storing built PKGARs + metadata + central manifest |
+| prefix/ | Cross-compiler toolchain (Clang/LLVM + relibc sysroot) |
+| Cascade rebuild | A low-level change triggering rebuild of all dependents transitively |
+| False cascade | A rebuild triggered by mtime change despite identical binary content |
@@ -77,6 +77,7 @@ const REPO_HELP_STR: &str = r#"
        --repo-binary              override recipes config to use repo_binary
        --allow-protected         allow re-fetching of protected recipes
                                   (equivalent to REDBEAR_ALLOW_PROTECTED_FETCH=1)
+        --force-rebuild           bypass content-hash cache and force rebuild

    cook env and their defaults:
        CI=                          set to any value to disable TUI
@@ -491,6 +492,7 @@ fn parse_args(args: Vec<String>) -> anyhow::Result<(CliConfig, CliCommand, Vec<C
                            env::set_var("REDBEAR_ALLOW_PROTECTED_FETCH", "1");
                        }
                    }
+                    "--force-rebuild" => config.cook.force_rebuild = true,
                    _ => {
                        eprintln!("Error: Unknown flag: {}", arg);
                        process::exit(1);
@@ -127,6 +127,13 @@ fn publish_packages(config: &CliConfig) -> anyhow::Result<()> {
                    fs::copy(&pkgar_src, &pkgar_dst)?;
                }
                fs::copy(&toml_src, &toml_dst)?;
+                if package.is_none() {
+                    let dep_hashes_src = target_dir.join("dep_hashes.toml");
+                    let dep_hashes_dst = repo_path.join(format!("{}.dep_hashes.toml", recipe_name));
+                    if dep_hashes_src.is_file() {
+                        let _ = fs::copy(&dep_hashes_src, &dep_hashes_dst);
+                    }
+                }
            }

            // TODO: Extract from pkgar instead to handle config.cook.clean_target == true
@@ -34,6 +34,8 @@ pub struct CookConfigOpt {
    pub clean_target: Option<bool>,
    /// whether to always write stage.files metadata
    pub write_filetree: Option<bool>,
+    /// whether to force rebuild ignoring content-hash cache
+    pub force_rebuild: Option<bool>,
 }

 #[derive(Debug, Default, Clone, Deserialize, PartialEq, Serialize)]
@@ -48,6 +50,7 @@ pub struct CookConfig {
    pub clean_build: bool,
    pub clean_target: bool,
    pub write_filetree: bool,
+    pub force_rebuild: bool,
 }

 impl From<CookConfigOpt> for CookConfig {
@@ -63,6 +66,7 @@ impl From<CookConfigOpt> for CookConfig {
            clean_build: value.clean_build.unwrap(),
            clean_target: value.clean_target.unwrap(),
            write_filetree: value.write_filetree.unwrap(),
+            force_rebuild: value.force_rebuild.unwrap_or(false),
        }
    }
 }
@@ -9,6 +9,8 @@ use crate::cook::{fetch, fs::*};
 use crate::recipe::Recipe;
 use crate::recipe::{AutoDeps, CookRecipe};
 use crate::recipe::{BuildKind, OptionalPackageRecipe};
+use serde::{Deserialize, Serialize};
+use std::collections::BTreeMap;
 use std::collections::VecDeque;
 use std::{
    collections::BTreeSet,
@@ -21,6 +23,62 @@ use std::{

 use crate::{is_redox, log_to_pty};

+#[derive(Serialize, Deserialize, Default)]
+pub struct DepHashes {
+    #[serde(flatten)]
+    pub hashes: BTreeMap<String, String>,
+}
+
+impl DepHashes {
+    pub fn read(path: &Path) -> Option<Self> {
+        let content = fs::read_to_string(path).ok()?;
+        toml::from_str(&content).ok()
+    }
+
+    pub fn write(&self, path: &Path) -> Result<(), String> {
+        let content = toml::to_string(self)
+            .map_err(|e| format!("failed to serialize dep_hashes.toml: {e}"))?;
+        fs::write(path, content)
+            .map_err(|e| format!("failed to write dep_hashes.toml: {e:?}"))?;
+        Ok(())
+    }
+}
+
+fn collect_current_dep_hashes(
+    dep_pkgars: &BTreeSet<(PackageName, PathBuf)>,
+) -> BTreeMap<String, String> {
+    let mut hashes = BTreeMap::new();
+    for (name, pkgar_path) in dep_pkgars {
+        let toml_path = pkgar_path.with_extension("toml");
+        if toml_path.is_file() {
+            if let Ok(toml_content) = fs::read_to_string(&toml_path) {
+                if let Ok(pkg) = toml::from_str::<Package>(&toml_content) {
+                    if !pkg.blake3.is_empty() {
+                        hashes.insert(name.without_prefix().to_string(), pkg.blake3);
+                    }
+                }
+            }
+        }
+    }
+    hashes
+}
+
+fn dep_hashes_changed(
+    stored: &BTreeMap<String, String>,
+    current: &BTreeMap<String, String>,
+) -> bool {
+    if stored.len() != current.len() {
+        return true;
+    }
+    for (key, stored_hash) in stored {
+        match current.get(key) {
+            Some(current_hash) if current_hash == stored_hash => continue,
+            _ => return true,
+        }
+    }
+    false
+}
+
 fn auto_deps_from_dynamic_linking(
    stage_dirs: &[PathBuf],
    dep_pkgars: &BTreeSet<(PackageName, PathBuf)>,
@@ -263,6 +321,54 @@ pub fn build(
        }
    }

+    // Try to restore stage artifacts from repo/ binary store when missing
+    let stage_present = stage_pkgars.iter().all(|file| file.is_file());
+    if !stage_present && !cook_config.force_rebuild && recipe.build.kind != BuildKind::Remote {
+        let repo_target = crate::cross_target()
+            .map(|t| PathBuf::from("repo").join(t))
+            .unwrap_or_else(|| PathBuf::from("repo").join(redoxer::target()));
+        let recipe_name = name.name();
+        let main_pkgar = repo_target.join(format!("{}.pkgar", recipe_name));
+        let main_toml = repo_target.join(format!("{}.toml", recipe_name));
+        if main_pkgar.is_file() && main_toml.is_file() {
+            if cli_verbose {
+                log_to_pty!(logger, "DEBUG: restoring stage artifacts from repo/ binary store");
+            }
+            if let Some(last_stage) = stage_dirs.last() {
+                if !last_stage.is_dir() {
+                    let pkey_path = "build/id_ed25519.pub.toml";
+                    if Path::new(pkey_path).is_file() {
+                        let _ = pkgar::extract(pkey_path, &main_pkgar, last_stage.to_str().unwrap());
+                    }
+                }
+            }
+            if let Some(last_pkgar) = stage_pkgars.last() {
+                if !last_pkgar.is_file() {
+                    let _ = fs::copy(&main_pkgar, last_pkgar);
+                }
+            }
+            if let Some(last_toml) = stage_pkgars.last().map(|p| p.with_extension("toml")) {
+                if !last_toml.is_file() {
+                    let _ = fs::copy(&main_toml, &last_toml);
+                }
+            }
+            let repo_dep_hashes = repo_target.join(format!("{}.dep_hashes.toml", recipe_name));
+            let local_dep_hashes = get_sub_target_dir(target_dir, "dep_hashes.toml");
+            if repo_dep_hashes.is_file() && !local_dep_hashes.is_file() {
+                let _ = fs::copy(&repo_dep_hashes, &local_dep_hashes);
+            }
+            if !auto_deps_file.is_file() {
+                if let Ok(toml_content) = fs::read_to_string(&main_toml) {
+                    if let Ok(pkg_meta) = toml::from_str::<Package>(&toml_content) {
+                        let auto: BTreeSet<PackageName> = pkg_meta.depends.into_iter().collect();
+                        let wrapper = AutoDeps { packages: auto };
+                        let _ = serialize_and_write(&auto_deps_file, &wrapper);
+                    }
+                }
+            }
+        }
+    }
+
    let mut source_modified = modified_dir_ignore_git(source_dir).unwrap_or(SystemTime::UNIX_EPOCH);
    if let Ok(recipe_modified) = modified(&recipe_dir.join("recipe.toml")) {
        source_modified = source_modified.max(recipe_modified);
@@ -281,11 +387,38 @@ pub fn build(

    // check stage dir modified against pkgar files, any files missing will result in UNIX_EPOCH
    let stage_modified = modified_all(&stage_pkgars, modified).unwrap_or(SystemTime::UNIX_EPOCH);
-    // Rebuild stage if source is newer
-    if stage_modified < source_modified
-        || stage_modified < deps_modified
-        || stage_modified < deps_host_modified
-        || !auto_deps_file.is_file()
+
+    let dep_hashes_file = get_sub_target_dir(target_dir, "dep_hashes.toml");
+
+    let source_changed = stage_modified < source_modified;
+
+    let deps_changed = if cook_config.force_rebuild {
+        if cli_verbose {
+            log_to_pty!(logger, "DEBUG: force rebuild requested");
+        }
+        true
+    } else if let Some(stored_hashes) = DepHashes::read(&dep_hashes_file) {
+        let current = collect_current_dep_hashes(&dep_pkgars);
+        let current_host = collect_current_dep_hashes(&dep_host_pkgars);
+        let mut all_current = current;
+        all_current.extend(current_host);
+        let changed = dep_hashes_changed(&stored_hashes.hashes, &all_current);
+        if cli_verbose {
+            if changed {
+                log_to_pty!(logger, "DEBUG: dependency blake3 hashes changed");
+            } else {
+                log_to_pty!(logger, "DEBUG: dependency blake3 hashes unchanged (content-based cache)");
+            }
+        }
+        changed
+    } else {
+        if cli_verbose {
+            log_to_pty!(logger, "DEBUG: no dep_hashes.toml, falling back to mtime comparison");
+        }
+        stage_modified < deps_modified || stage_modified < deps_host_modified
+    };
+
+    if source_changed || deps_changed || !auto_deps_file.is_file()
    {
        for stage_dir in &stage_dirs {
            if stage_dir.is_dir() {
@@ -516,7 +649,21 @@ pub fn build(
        // don't remove stage dir yet
    }

+    let current_dep_hashes = {
+        let current = collect_current_dep_hashes(&dep_pkgars);
+        let current_host = collect_current_dep_hashes(&dep_host_pkgars);
+        let mut all_current = current;
+        all_current.extend(current_host);
+        all_current
+    };
+
    let auto_deps = make_auto_deps!(false)?;
+
+    let dep_hashes = DepHashes { hashes: current_dep_hashes };
+    if let Err(e) = dep_hashes.write(&dep_hashes_file) {
+        log_to_pty!(logger, "WARN: failed to write dep_hashes.toml: {e}");
+    }
+
    Ok(BuildResult::new(stage_dirs, auto_deps))
 }