vasilito/RedBear-OS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document content-hash cache system, binary store, package groups

Browse Source 
- AGENTS.md: add cache system to STRUCTURE, WHERE TO LOOK, BUILD FLOW,
  BUILD COMMANDS (--force-rebuild), and CONVENTIONS (dep_hashes.toml,
  binary store restore, package_groups syntax)
- CHANGELOG.md: comprehensive entry for Phase 1-3 + kernel MWAIT +
  ninja-build Redox support
- local/AGENTS.md: note installer fork adds package groups support
- BUILD-CACHE-PLAN.md: fix TOML syntax (underscores not hyphens),
  update all phases to COMPLETE with implementation details, add cache
  flow diagram, add verification results
This commit is contained in:
vasilito
2026-06-30 17:29:52 +03:00
parent b25ea418fb
commit 7d62a7c0ab
5 changed files with 180 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
local/docs/BUILD-CACHE-PLAN.md
+68 -37
View File
@@ -73,15 +73,15 @@ mesa = "9f2e..."
zlib = "a1b2..."
```
**Files to modify:**
**Implementation (Phase 1 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `src/cook/cook_build.rs` | Add `read_dep_hashes()`, `write_dep_hashes()` helpers; replace mtime comparison at L285-303 with hash comparison; update `build_deps_dir()` at L566-653 with same hash logic | ~100 |
| `src/bin/repo.rs` | Add `--force-rebuild` CLI flag (bypasses hash caching); add `validate-cache` subcommand (prints what would rebuild and why) | ~55 |
| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ~20 |
**Total: ~175 LOC**
| File | Change | Status |
|------|--------|--------|
| `src/cook/cook_build.rs` | `DepHashes` struct with `read`/`write`; `collect_current_dep_hashes()` reads blake3 from dep `.toml` metadata; `dep_hashes_changed()` compares stored vs current; replaces mtime at cache check | ✅ Done |
| `src/cook/cook_build.rs` | Mtime fallback when `dep_hashes.toml` absent | ✅ Done |
| `src/bin/repo.rs` | `--force-rebuild` CLI flag bypasses hash caching | ✅ Done |
| `src/config.rs` | `force_rebuild` field in `CookConfig`/`CookConfigOpt` | ✅ Done |
| `src/cook/cook_build.rs` | `build_deps_dir()` update — **deferred**: sysroot rebuild uses mtime, not hash. Does NOT cause cascade rebuilds because the recipe's stage.pkgar is the cache key, not the sysroot. | 📋 Deferred |
**Why PKGAR BLAKE3 (not ELF symbol extraction):**
@@ -129,52 +129,55 @@ if repo_pkgar.is_file() && repo_toml.is_file() {
}
```
**Files to modify:**
**Implementation (Phase 2 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `src/cook/cook_build.rs` | Add repo binary lookup before rebuild decision | ~60 |
| File | Change | Status |
|------|--------|--------|
| `src/cook/cook_build.rs` | Repo binary store restore: when `target/` missing but `repo/<arch>/<name>.pkgar` + `.toml` exist, extracts PKGAR to stage dir, copies `.toml` + `dep_hashes.toml`, auto-generates `auto_deps.toml` from repo depends field | ✅ Done |
| `src/bin/repo_builder.rs` | Publishes `<name>.dep_hashes.toml` alongside `.pkgar` and `.toml` in `repo/<arch>/` during `publish_packages()` (main package only, not optional sub-packages) | ✅ Done |
**Total: ~60 LOC**
**Total: ~70 LOC**
### Phase 3: Config-Level Package Groups (MEDIUM ROI)
### Phase 3: Config-Level Package Groups (COMPLETE)
Meta-packages are an **installer/runtime concern**. The build system continues to
produce one PKGAR per recipe. The installer groups them logically.
**Config format:**
**Config format (underscore, NOT hyphen — serde requires it):**
```toml
# config/redbear-full.toml
[package-groups.qt6-core]
[package_groups.qt6-core]
description = "Qt 6 Core modules"
packages = ["qtbase", "qtdeclarative", "qtsvg"]
[package-groups.qt6-wayland]
[package_groups.qt6-extras]
description = "Qt 6 Wayland integration"
packages = ["qtwayland", "qt6-wayland-smoke"]
packages = ["qtwayland", "qt6-wayland-smoke", "qt6-sensors"]
[package-groups.kf6-frameworks]
description = "KDE Frameworks 6 (all 32)"
[package_groups.kf6-frameworks]
description = "KDE Frameworks 6 (all 38)"
packages = ["kf6-kcoreaddons", "kf6-kconfig", "kf6-ki18n", ...]
[package-groups.kde-desktop]
[package_groups.kde-desktop]
description = "Complete KDE Plasma desktop session"
packages = ["qt6-core", "qt6-wayland", "kf6-frameworks", "kwin", "kdecoration", "sddm"]
packages = ["graphics-core", "qt6-core", "qt6-extras", "kf6-frameworks", "kwin", "sddm"]
```
Groups can reference other groups — installer resolves recursively.
Groups can reference other groups — installer resolves recursively with cycle detection.
Explicit `[packages]` entries override group membership.
**Files to modify:**
**Implementation (Phase 3 — COMPLETE):**
| File | Change | LoC |
|------|--------|-----|
| `local/sources/installer/src/config/mod.rs` | Parse `[package-groups]` in config merge | ~40 |
| `local/sources/installer/src/installer.rs` | Resolve groups during `install_dir()` | ~40 |
| `config/redbear-full.toml` | Define qt6-core, qt6-wayland, kf6-frameworks groups | ~30 |
| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | ~30 |
| File | Change | Status |
|------|--------|--------|
| `local/sources/installer/src/config/mod.rs` | `PackageGroup` struct, `package_groups` field on `Config`, `resolve_package_groups()` + `expand_group()` with cycle detection, recursive group resolution, explicit-overrides-group priority. Called at end of `Config::from_file()` | ✅ Done |
| `local/sources/installer/src/config/package.rs` | `PartialEq` derive on `PackageConfig` for dedup during merge | ✅ Done |
| `config/redbear-full.toml` | 9 groups defined: `graphics-core`, `input-stack`, `dbus-services`, `firmware-stack`, `qt6-core`, `qt6-extras`, `kf6-frameworks`, `desktop-session`, `kde-desktop` | ✅ Done |
| `Cargo.toml` | Switch `redox_installer` from upstream git to local fork (`path = "local/sources/installer"`) | ✅ Done |
| `pkg` crate (runtime) | `pkg install qt6-core` resolves group | 📋 Future (runtime package manager) |
**Total: ~140 LOC**
**Total: ~170 LOC**
**Why NOT merged PKGARs or build-system grouping:**
@@ -194,13 +197,41 @@ Groups can reference other groups — installer resolves recursively.
| Merged PKGAR meta-packages | Breaks one-build-one-PKGAR model. Makes caching harder. Each component has its own deps and ABI surface. |
| Fine-grained header dependency tracking | Massive complexity, compiler-dependent. Not worth it for a source-built OS. |
## Implementation Order
## Implementation Status
| Phase | Effort | ROI | When |
|-------|--------|-----|------|
| **Phase 1: Hash-based caching** | ~175 LOC, 1-2 days | **Critical** — saves 4-6 hours per low-level change | Now |
| **Phase 2: Binary store lookup** | ~60 LOC, half day | High — saves cold rebuilds after `make clean` | After Phase 1 |
| **Phase 3: Package groups** | ~140 LOC, 1 day | Medium — installer UX improvement | After Phase 1-2 |
| Phase | Effort | Status | Verification |
|-------|--------|--------|--------------|
| **Phase 1: Hash-based caching** | ~175 LOC | ✅ Complete | Touch xz.pkgar → libxml2 stays cached (mtime changed, BLAKE3 same); `--force-rebuild` bypasses |
| **Phase 2: Binary store lookup** | ~70 LOC | ✅ Complete | Removed libxml2 target/ → re-cook → restored from repo/ binary store → cache hit |
| **Phase 3: Package groups** | ~170 LOC | ✅ Complete | 3 unit tests pass (nested groups, explicit override, no-groups compat); mini build verified |
### Cache Flow (As Implemented)
```
Recipe build requested
│
├─ force_rebuild? ────────────────────────────────► REBUILD
│
├─ target/ dir exists with dep_hashes.toml?
│ ├─ YES: Read stored hashes
│ │ Collect current dep PKGAR blake3 from .toml files
│ │ All match? ──► CACHE HIT (skip build)
│ │ Any differ? ──► REBUILD
│ │
│ └─ NO: Fall back to mtime comparison (backward compat)
│ stage < source? ──► REBUILD
│ stage < deps? ──► REBUILD
│ Otherwise ──► CACHE HIT
│
├─ target/ dir MISSING but repo/<arch>/ has pkgar + toml?
│ └─ YES: Restore stage artifacts from binary store
│ Extract pkgar → stage dir
│ Copy .toml + .dep_hashes.toml
│ Generate auto_deps.toml from repo depends
│ Then run cache check above (likely hit)
│
└─ Build complete → write dep_hashes.toml with current hashes
```
## Risk Analysis