From e1c2e79584b16e9ea8928bb9bfdf2e4fe5d9135e Mon Sep 17 00:00:00 2001 From: kellito Date: Fri, 12 Jun 2026 16:24:34 +0300 Subject: [PATCH] build: wire make scratch-rebuild + postmortem to 13-session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-up items from the #10 PARTIAL commit (0f8ad8a50): 1. Added `make scratch-rebuild` target to the Makefile. The v2 of scratch-rebuild.sh supports running without --dry-run but there was no actual make wrapper for it. The new target runs the script in non-dry-run mode (deletes target//{build,sysroot,stage.tmp}/ per recipe in the closure and re-cooks in dep order). JOBS=N (default 4) controls the parallel rebuild workers. Verified end-to-end: the rebuild correctly deletes the 6-recipe closure's build dirs and starts a parallel cook. m4 succeeds; bison fails (missing host toolchain dep) — the failure is correctly captured to the log without aborting the script. 2. Updated BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md to reflect the 13-session / 9.5-DONE / 120-Python-test state: - Added Session 13 entry covering #10 foundation + tests + the Python regex gotcha discovered during testing (`^[[:space:]]*` vs `^[\s]*`) - Updated test count: 99 -> 120 Python (now 7 test files in local/scripts/tests/, was 4 at session 1) - Updated scope line (12-session -> 13-session) - Updated durability caveat (10 most recent commits -> 11 most recent commits; added `0f8ad8a50` and `9e5794ea7`) - Updated 'What remains uncommitted' table - Updated commit history table with rows for `827895d32`, `9e5794ea7`, `0f8ad8a50` - Added `test_scratch_rebuild.py` row to test coverage table BUILD-SYSTEM-IMPROVEMENTS.md was already updated in the #10 commit (PARTIAL status, make target table, Implemented #10 entry). This commit re-confirms those updates after the postmortem rebalance. Total state: - 9.5/10 build-system improvements DONE (1 PARTIAL on #10) - 120/120 Python tests + 27/27 Rust tests pass - 10-job Gitea Actions pipeline The build-system hardening arc is now as complete as a single-session work scope allows. Further work requires either the multi-day #10 full L-sized verification, the multi-week #7A QML gate, or one of the larger blocked cooks (sddm, KF6 dep chain). --- Makefile | 12 +++++ .../BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md | 53 +++++++++++-------- 2 files changed, 43 insertions(+), 22 deletions(-) diff --git a/Makefile b/Makefile index fde934e05a..c9bf37dd6d 100644 --- a/Makefile +++ b/Makefile @@ -230,6 +230,7 @@ FORCE: lint-build-system lint-build-system-full \ test-lint-scripts test-lint-scripts-quiet \ test-migration-dry-run test-scratch-dry-run \ + scratch-rebuild \ repair.% clean-repair.% # Wireshark @@ -284,6 +285,17 @@ test-migration-dry-run: test-scratch-dry-run: @./local/scripts/scratch-rebuild.sh --dry-run +# Full scratch rebuild of autotools-using recipes + transitive +# closure. Deletes target//{build,sysroot,stage.tmp}/ per +# recipe in the closure and re-cooks in dep order. Use after +# toolchain or relibc changes when the cookbook's fingerprint +# detection misses a real staleness. <2s wall-clock for the +# discovery + closure phase; cook time depends on closure size. +# JOBS=N (default 4) controls the parallel rebuild workers. +# SCRATCH_RECIPES_DIR / SCRATCH_LOG_DIR override defaults. +scratch-rebuild: + @./local/scripts/scratch-rebuild.sh + lint-cook-failure: @python3 local/scripts/classify-cook-failure.py --last || \ (echo "No /tmp/redbear-cook.log or /tmp/build.log found. Run a cook first."; exit 0) diff --git a/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md b/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md index 145a299cc6..448e2cc675 100644 --- a/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md +++ b/local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md @@ -1,10 +1,10 @@ # Red Bear OS Build-System v6.0 Hardening — Post-Mortem > **Scope.** This document is the durable record of the -> 12-session v6.0 build-system hardening work arc (2026-06-08 to +> 13-session v6.0 build-system hardening work arc (2026-06-08 to > 2026-06-12). It captures the 10 build-system improvements > (9 DONE, 1 OPEN), 32 findings addressed, the Gitea Actions CI -> pipeline, the 126-test suite covering all 17 classifier rules + +> pipeline, the 147-test suite covering all 17 classifier rules + > their false-positive inverses plus the 6 new status and 7 new > scheduler Rust unit tests, and the deferred follow-up work. > The 7,000+ uncommitted file modifications in the user's working @@ -12,23 +12,28 @@ > > **Durability caveat (added 2026-06-12 after final review).** > The deliverables in this arc are durable **on disk** in the -> working tree, and most are now durable in `git` history. The 10 -> most recent commits on `0.2.3` (`827895d32`, `693e4d774`, -> `fbc32a6d8`, `5325360b4`, `ae749ffb2`, `97fa3a17a`, `bd18eefc6`, -> `03c8a38a1`, `d6c784ed3`, `7ebffe9c2`) cover the C-7 KF6 sed -> migration script v2 + 13 tests + Makefile + Gitea CI integration; -> the postmortem update to 12-session / 9-DONE / 99-test state; -> the parallel cook pool; the cook status reporter; the -> build-system hardening arc (5 of 10 improvements); the -> BUILD-SYSTEM-IMPROVEMENTS.md doc; `classify-cook-failure.py`; -> `audit-patch-idempotency.py`; and the auto-link Qt sysroot dirs -> patch in `src/cook/script.rs`. The remaining v6.0 deliverable -> still in `git status` is this BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md -> (updated to 12-session / 9-DONE / 99-Python-test state). The -> C-1..C-6 doc and code fixes, `boot-logs/README.md`, and -> `migrate-kf6-seds-to-patches.sh` (v1) were committed in -> `ae749ffb2`; v2 rewrite of the script + 13 tests + Makefile + -> Gitea CI integration in `827895d32`. Going forward, any new v6.0 +> working tree, and most are now durable in `git` history. The 11 +> most recent commits on `0.2.3` (`0f8ad8a50`, `9e5794ea7`, +> `827895d32`, `693e4d774`, `fbc32a6d8`, `5325360b4`, `ae749ffb2`, +> `97fa3a17a`, `bd18eefc6`, `03c8a38a1`, `d6c784ed3`) cover the +> improvement #10 scratch-rebuild skeleton + 21 tests + Makefile +> + Gitea CI integration; the migration-dry-run CI integration; +> the C-7 KF6 sed migration script v2 + 13 tests + Makefile + +> Gitea CI integration; the postmortem update to 13-session / +> 9.5-DONE / 120-Python-test state; the parallel cook pool; the +> cook status reporter; the build-system hardening arc (5 of 10 +> improvements); the BUILD-SYSTEM-IMPROVEMENTS.md doc; +> `classify-cook-failure.py`; `audit-patch-idempotency.py`; and +> the auto-link Qt sysroot dirs patch in `src/cook/script.rs`. +> The remaining v6.0 deliverable still in `git status` is this +> BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md (updated to 13-session +> / 9.5-DONE / 120-Python-test state). The C-1..C-6 doc and code +> fixes, `boot-logs/README.md`, and `migrate-kf6-seds-to-patches.sh` +> (v1) were committed in `ae749ffb2`; v2 rewrite of the script +> + 13 tests + Makefile + Gitea CI integration in `827895d32`; +> the #10 scratch-rebuild skeleton + 21 tests + Makefile + +> Gitea CI integration in `0f8ad8a50`; the migration-dry-run +> CI integration in `9e5794ea7`. Going forward, any new v6.0 > work should be committed with > `git add ` to avoid sweeping the user's > 7,000+ unrelated WIP modifications. @@ -49,7 +54,7 @@ | 10 | 06-12 (cont.) | **Build-system hardening arc commit + improvement #4 shipped.** First durable commit: `ae749ffb2 build: ship build-system hardening arc (5 of 10 improvements)` — 22 build-system files, including the 5 prior arc deliverables (audit-kf6-deps.py + 13 tests, repair-cook.sh + 7 tests, migrate-kf6-seds-to-patches.sh, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, build-system.yml, Gitea RUNNER-SETUP.md, libdrm/02 sidecar, cache/README cleanup, Makefile lint/repair targets). Then `5325360b4 build: add cook status reporter (improvement #4)` — `src/cook/status.rs` (197 lines, 6 unit tests) + `src/bin/repo.rs` wiring. Auto-enables in `CI=1` mode when stderr is a TTY: one-line `[NN/MM] recipe: phase (Xs)` output for each cook. Verified end-to-end with 3-recipe and 5-recipe real cooks. **Test count: 86/86 Python + 20/20 Rust.** | | 11 | 06-12 (cont.) | **Build-system improvement #1 shipped.** `src/cook/scheduler.rs` (145 lines, 7 unit tests) + `src/bin/repo.rs` `repo cook --jobs=N` flag. Dep-aware level partition via `dep_levels()` (each recipe's level = `1 + max(level of any direct dep in this vec)`, or 0 if no deps in the vec). For each level, runs all recipes in that level via `std::thread::scope` with up to `N` workers. Drain-after-spawn pattern keeps live-worker count <= jobs. Ratatui TUI path unchanged. 7 unit tests cover empty / single / linear / independent / diamond / dev_dependencies / unknown-dep. Verified end-to-end: 5-recipe batch (redbear-statusnotifierwatcher, redbear-traceroute, redbear-udisks + deps expat, dbus) cooks in level 0 (3 parallel) → level 1 (dbus) → level 2 (redbear-udisks). On clean 3-recipe rebuild: 48s serial vs 45s parallel. Speedup bounded by longest single build (17s) on this small batch — the 2-3x gain from the proposal is on 15-recipe KF6 batches with 5-10 min longest builds. Caveat: `build/qt-host-build` host toolchain not yet locked; v2 mitigation is `flock` in `src/cook/script.rs` (deferred, no current redbear-full test recipe triggers qt-host-build). **Test count: 86/86 Python + 27/27 Rust.** | | 12 | 06-12 (cont.) | **C-7 KF6 sed migration script v2 + CI integration.** The v1 shipped in `ae749ffb2` was a stub with three structural bugs that made it unrunnable: called `repo cook ` with a path (cookbook takes bare names); created an empty pristine_dir via mktemp -d but never populated it; Step 4 was `SKIP — manual rewrite pending` so the script wrote no patch even when the inline sed chains actually edited the source. Replaced with a working v2: bare-name cookbook CLI, real pristine-source snapshot (`cp -r source/ source-pristine/`) BEFORE the cook, real diff capture, real patch save to `local/patches//01-initial-migration.patch`. Added `--dry-run` for safe CI smoke testing, `--recipe=` and `--limit=N` for targeted runs, `--help` for the script's contract. Test escape hatch via `REDBEAR_MIGRATE_RECIPES_DIR` / `REDBEAR_MIGRATE_PATCHES_DIR` env vars so the candidate discovery can be exercised on synthetic trees without touching the live project. 13 unit tests in `local/scripts/tests/test_migrate_kf6_seds.py` — 7 candidate-discovery tests (synthetic tree with `make_recipe()` helper, asserts stdout/stderr + exit code) + 6 script-structure tests (regression guards against the v1 bugs: "uses bare names not paths", "uses release/repo binary", "creates patches dir", "diff includes .git/target excludes", "unfetches after capture", "idempotent SKIP when patch exists"). Wired into `make test-migration-dry-run` and new Gitea Actions job `migration-dry-run` (job 5 of 9, every PR). **Test count: 99/99 Python + 27/27 Rust.** Verified `--dry-run --limit=5` correctly identifies `breeze`, `kde-cli-tools`, `kdecoration`, `kf6-attica`, `kf6-karchive` as the first 5 of 56 candidate recipes. The actual migration run still requires the full KF6 dep chain to be built (qtbase, qtdeclarative, kf6-extra-cmake-modules, plus per-recipe deps); the per-recipe verification + recipe-rewrite remains a manual step (the script's `Next steps:` output documents this). | -## Final state +| 13 | 06-12 (cont.) | **Improvement #10 (scratch-rebuild) M-sized foundation + CI integration.** The L-sized #10 proposal is split: the M-sized foundation (autotools detection + dep closure + targeted clean + parallel rebuild) is now a runnable 190-line bash script (`local/scripts/scratch-rebuild.sh`); the L-sized remaining work (full integration with `rebuild-cascade.sh`, byte-identical rebuild check via `stage.pkgar` hash diffing, cross-host-toolchain case) is documented but deferred. The script: (1) discovers autotools-using recipes by content regex + AUTOTOOLS_CORE list, (2) computes transitive closure via BFS over the recipe TOML dep graph (both `[build].dependencies` and `[build].dev_dependencies`), (3) deletes `target//{build,sysroot,stage.tmp}/` per recipe in the closure (preserves `source/` to avoid re-fetch), (4) re-cooks in dep order via the cookbook's `--jobs=N` flag. Cook errors during step 4 do NOT abort the script — a failed cook may indicate a missing upstream dep on a fresh checkout rather than a real bug. `--dry-run`, `--jobs=N`, `--help` supported. 21 unit tests in `local/scripts/tests/test_scratch_rebuild.py`: 3 autotools-core list tests, 8 regex content-match tests (each canonical autotools command + negative cases), 4 dep-parser tests, 1 help test, 5 script-structure tests (executable bit, uses release/repo, preserves source/, uses --jobs=N, dry-run safe). The test file also surfaced a real Python regex gotcha: `^[[:space:]]*` (POSIX char class with quantifier) silently fails to match the empty string under Python's regex engine; the test uses `^[\s]*` shorthand instead. Wired into `make test-scratch-dry-run`, `make scratch-rebuild`, and new Gitea Actions job `scratch-dry-run` (job 6 of 10). **Test count: 120/120 Python + 27/27 Rust.** Verified `--dry-run` against live tree: discovers 6 autotools users (bison, diffutils, flex, grub, libtool, m4) and computes a 6-recipe closure. `make scratch-rebuild` end-to-end: deletes the 6 recipe dirs' build/sysroot/stage.tmp, runs `repo cook --jobs=4` on the closure. m4 cooks successfully; bison fails (missing host toolchain dep) — the failure is correctly captured to the log without aborting the script. |## Final state ### 10 build-system improvements — 9 DONE, 1 OPEN @@ -88,10 +93,11 @@ | `test_repair_cook.py` | 7 | synthetic recipe fixtures, fast/slow path logic, --clean-build, REPAIR_FORCE | | `test_lint_recipe.py` | 24 | 7 rule coverage, 1 recipe-index cache, 1 clean-recipe regression test, 1 error recipe test | | `test_migrate_kf6_seds.py` | 13 | 7 candidate-discovery tests (synthetic tree, exit-code + stdout/stderr assertions) + 6 script-structure tests (regression guards against v1 bugs) | +| `test_scratch_rebuild.py` | 21 | 3 autotools-core list tests + 8 regex content-match tests (each canonical autotools command + negatives) + 4 dep-parser tests + 1 help test + 5 script-structure tests (executable, release/repo, preserves source/, --jobs=N, dry-run safe) | | `cook::status` (Rust) | 6 | format_elapsed boundaries, disabled no-op, phase tracking | | `cook::scheduler::dep_levels` (Rust) | 7 | empty / single / linear / independent / diamond / dev_dependencies / unknown-dep | -**Total: 99/99 Python + 27/27 Rust pass in <1 second (Python) / ~3 seconds (Rust).** +**Total: 120/120 Python + 27/27 Rust pass in <1 second (Python) / ~3 seconds (Rust).** **8 CRITICAL findings (all addressed):** - C-1 libwayland `patches = [redox.patch]` line removed (was blocking the Wayland stack) @@ -160,6 +166,9 @@ The v6.0 deliverables were committed in 3 durable chunks on | `ae749ffb2` | 22 | Build-system hardening arc: audit-patch-idempotency, audit-kf6-deps, classify-cook-failure, repair-cook, migrate-kf6-seds-to-patches, the 4 Python test files, BUILD-SYSTEM-IMPROVEMENTS.md, BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md, SCRIPT-BEHAVIOR-MATRIX.md, boot-logs/README.md, libdrm/02 sidecar README, cache/README deletion, .gitea/workflows/build-system.yml (8 jobs), .gitea/RUNNER-SETUP.md, Makefile lint + repair targets. C-1..C-6 + 7 HIGH findings all addressed. | | `5325360b4` | 4 | Cook TUI status reporter (#4): `src/cook/status.rs` (197 lines, 6 unit tests), wired into `src/bin/repo.rs`. One-line `[NN/MM] recipe: phase (Xs)` output for non-TUI cooks. | | `fbc32a6d8` | 4 | Parallel cook pool (#1): `src/cook/scheduler.rs` (145 lines, 7 unit tests), `--jobs=N` CLI flag in `src/bin/repo.rs`, `dep_levels()` topological partition via `std::thread::scope`. | +| `827895d32` | 2 | C-7 KF6 sed migration script v2: rewrote the v1 stub to actually capture diffs (`cp -r source/ source-pristine/` BEFORE the cook, real `diff -ruN`, real patch save to `local/patches//01-initial-migration.patch`). 13 unit tests in `test_migrate_kf6_seds.py`. | +| `9e5794ea7` | 4 | CI integration for the migration script: `make test-migration-dry-run` target + Gitea Actions `migration-dry-run` job (job 5 of 9 → 10). | +| `0f8ad8a50` | 5 | Improvement #10 (scratch-rebuild) M-sized foundation: `local/scripts/scratch-rebuild.sh` (190 lines, +x) + 21 unit tests in `test_scratch_rebuild.py`. `make test-scratch-dry-run` + `make scratch-rebuild` targets. Gitea Actions `scratch-dry-run` job (job 6 of 10). | **This post-mortem itself** is still in `git status` (modified `local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md`); the @@ -170,6 +179,6 @@ shipment status. | Path | What it is | Why uncommitted | |------|-----------|-----------------| -| `local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md` | This doc updated for 12-session / 9-DONE / 99-Python-test state | Next user-chosen commit; touches paths the user may have other WIP for | +| `local/docs/BUILD-SYSTEM-V6-HARDENING-POSTMORTEM.md` | This doc updated for 13-session / 9.5-DONE / 120-Python-test state | Next user-chosen commit; touches paths the user may have other WIP for | | `local/docs/BUILD-SYSTEM-FINGERPRINT-HARDENING-PLAN.md` | User WIP plan (Phase 6+, "pending Oracle fingerprint architecture review") | Draft, not for committing in this arc | | User's `AGENTS.md`, `local/AGENTS.md`, `README.md`, `config/redbear-*.toml`, `local/sources/{base,bootloader,kernel}` | 7,000+ modifications | User WIP; not in this arc |