I tried resetting the build dir whenever rsync reported changed files, to catch
stale GENERATED build files after a recipe edit -- the xcb-util-image case where
the staged source said `SUBDIRS = image` but build/Makefile still said
`SUBDIRS = image test`, so a correct fix looked like it had no effect.
That signal is wrong. --delete restores the pristine tree, undoing the previous
build's seds, so rsync reports changes on EVERY run. Measured on a plain
kirigami rebuild: 6 "changed" files that were only the sed undo, and the build
dir was wiped. Shipping it would have forced a full rebuild of every recipe on
every build.
Reverted, with the reasoning recorded at the call site so it is not attempted
again. Stale generated files are the job of the content-hash invalidation the
cookbook already performs against the PRISTINE source; the staged copy is not a
valid basis for that comparison.
Verified after revert: kirigami is cached again rather than rebuilding.
Three separate defects, one per class:
1. script.rs passed autoreconf only -I${COOKBOOK_HOST_SYSROOT}/share/aclocal, so
macros installed by a DEPENDENCY into the per-recipe sysroot were invisible.
util-macros was built and declared as a dependency, yet configure still said
configure.ac:10: error: must install xorg-macros 1.16.0 or later
Now also searches ${COOKBOOK_SYSROOT}/usr/share/aclocal and exports
ACLOCAL_PATH. Fixed generically -- every autotools recipe benefits.
2. Upstream builds test/test_xcb_image_shm unconditionally (Makefile.am:15
SUBDIRS = image test) and it needs SysV shared memory, which Redox lacks.
The library itself does not: configure only probes sys/shm.h and adapts. So
dropping test/ removes a test binary, not functionality.
3. libxcb is a STATIC archive here, so its libXau/libXdmcp dependencies do not
propagate to consumers as a shared library's DT_NEEDED would:
xcb_auth.c: undefined reference to `XauGetBestAuthByAddr'
Named them via LIBS, not LDFLAGS: autoconf emits LDFLAGS BEFORE the objects
and LIBS AFTER, and a static archive only satisfies symbols referenced by
objects preceding it. With -lXau in LDFLAGS the link still failed while
libXau.a sat in the sysroot. Same ordering rule as -lgcc in
redox-toolchain.cmake and -licudata in plasma-workspace -- third occurrence.
Verified: cook xcb-util-image - successful, libxcb-image.a staged.
fetch_offline hashed source.tar BEFORE checking it exists, so a missing tarball
surfaced as
repo: failed to fetch: Opening file for blake3 failed at
"recipes/x11/libice/source.tar": No such file (os error 2)
which names neither the recipe nor the remedy. That message sent me to work
around the symptom by hand twice -- libogg/libvorbis, then four newly promoted
X11 recipes -- because it points at hashing rather than at a missing download.
Now it reports the recipe, the upstream URL and the exact curl command.
Also drops libice/libsm from redbear-full. They are X11 session-management
libraries used only by ksmserver, which sits inside if(WITH_X11) and is not
built, and they need xorg util-macros, which no recipe in the tree provides.
Including them would have meant porting a build dependency for code that is
never compiled.
A recipe whose script exits 0 but installs nothing currently reports success,
and the damage lands far from the cause. qtspeech produced a 136-byte package
because upstream `return()`s with only a NOTICE when Qt6::Multimedia is absent;
that surfaced much later as an unresolved symbol in a consumer, not as a
qtspeech failure.
Zero regular files staged is unambiguous -- every real package installs at
least one file -- so this needs no per-recipe size threshold and cannot
false-positive on a small-but-valid package. The error names the likely cause
(a configure step soft-bailing on a missing optional dependency) rather than
just reporting emptiness. Metapackages that genuinely stage nothing opt out by
mentioning COOKBOOK_ALLOW_EMPTY_PACKAGE in their script.
This is the same failure shape as the libudev/libpciaccess API-surface risk: a
gap that reports success locally and only becomes visible as a link error in
something downstream.
redbear-greeter's manifest has
redbear-login-protocol = { path = "../../redbear-login-protocol/source" }
Cargo resolves that relative to the manifest, so out-of-tree staging pointed it
at <recipe>/target/redbear-login-protocol/source:
failed to read .../target/redbear-login-protocol/source/Cargo.toml
No such file or directory (os error 2)
Rewriting the manifest would silently change dependency paths behind the
recipe's back. Leaving these in tree is the honest option: they are cargo
recipes that do not sed their source, so the mutation risk staging exists to
prevent does not apply to them.
Two systemic fixes for failure classes this session kept re-hitting.
1. COOK_JOBS default 4 -> 1. Recipes are not isolated: redbear_qt_ensure_dep_
sysroots repairs OTHER recipes' sysroots, so one cook can relink the include/
or lib/ directory another is compiling against. The tell was a failure that
MOVED between source files across runs (kirigami died in KirigamiTemplates on
build 59, KirigamiPrivateplugin on 61) with a sibling's rm -f/ln -sf
interleaved in the log. Making the relink atomic closed one window; the
sharing itself remains unsound. Compilation is still parallel -- each recipe
keeps the full -j budget -- only the number of recipes in flight drops to 1.
An intermittent build costs more than a serial one.
2. Auto-reset build dirs whose CMakeCache records a different source path.
CMake stores CMAKE_HOME_DIRECTORY and hard-aborts when the source moves:
CMake Error: The source ".../source-staged/..." does not match the source
... used to generate cache
Out-of-tree staging moved every tracked recipe's source and stranded 66 build
dirs. Two rounds of clearing them by hand were not a fix: the next source-path
change strands them again, and the abort surfaces in a downstream recipe
rather than at the cause. cook now detects the mismatch and resets the dir.
Two defects in the out-of-tree staging from 221a91cba4/a3d9240d62, both found
by actually running it rather than by inspection.
1. Submodules were staged. `git ls-files --error-unmatch` succeeds on a
gitlink, so relibc looked like an ordinary tracked tree and was copied
wholesale, losing the nested-repo context cargo needs -- crt0, crti, crtn
and ld_so all failed to build. Fork submodules are already covered by the
stricter REFUSING-TO-BUILD dirty gate, so skipping them loses no protection.
2. `rm -rf` before the copy made the staged path vanish mid-build. Qt bakes its
COOKBOOK_SOURCE into installed cmake files (QtSeparateDebugInfo.cmake
try_compile()s ${QT_SOURCE_TREE}/config.tests/...), so once that path was the
staged tree, rebuilding qtbase pulled it out from under consumers:
CMake Error: The source ".../qtbase/.../source-staged/config.tests/
binary_for_strip/CMakeLists.txt" does not ...
(qtsvg and qtshadertools). Now rsync -a --delete refreshes in place: same
pristine result, but the directory never stops existing. Verified with 4000
probes across a full kirigami rebuild -- 0 observations of it missing.
Staging is confirmed working: a 26-package run staged 10 trees, skipped relibc,
and left 0 dirty tracked source files (previously 22-29).
The out-of-tree staging added in 221a91cba4 never actually ran. Recipes are
reached through the overlay path `recipes/<cat>/<name>`, a symlink into
`local/recipes/...`, and `git ls-files --error-unmatch` refuses any pathspec
that traverses a symlink ("pathspec is behind a symbolic link"). The check
therefore reported every tracked vendored tree as untracked, staging was
skipped, and builds went on mutating git-tracked sources -- 22 dirty source
files after a 42-package run.
Canonicalize the path before both the git query and the copy. Verified: the
check fails via the symlink, succeeds via the resolved path, and cooking
kirigami now logs
[out-of-tree] staged tracked source -> .../kirigami/target/.../source-staged
with the recipe's sed rewriting the staged copy instead of the tracked tree.
libcanberra: plasma-workspace calls find_package(Canberra) TYPE REQUIRED and
includes <canberra.h> from four translation units with no HAVE_CANBERRA guard,
one of them libnotificationmanager (the core notification library). It cannot
be made optional, so it is ported for real. Built shared: it dlopens its
backend through libltdl, the libtool recipe stages libltdl.so only (no .a), so
a static build left every consumer with an undefined lt_dlopenext.
Audio is SILENT. Upstream 0.30 offers alsa/oss/pulse/gstreamer/null and none
speak Redox; the pulse driver needs PulseAudio's client library, which PipeWire
does not provide. The null driver is real upstream code, so the API/ABI is
genuine and all consumers work -- but nothing reaches the speakers until a
native /scheme/audio driver exists.
KX11Extras et al: gate the five X11-only KWindowSystem headers behind the
HAVE_X11 that plasma-workspace already defines. X11 is compiled OUT, not added.
gate-kx11extras.py brace-matches each block and keeps any trailing else outside
the guard, since wrapping a whole if/else-if chain yields invalid C++.
check-recipe-escapes.py: a single backslash in a TOML multi-line string is a
TOML escape, so a sed written as \bFoo\b parses to <BS>Foo<BS> and silently
matches nothing -- no TOML error, no sed error. This class bit the tree three
times. Now gated in preflight; it also found openssh, which used an invalid \$
and could not be parsed by any compliant parser.
cook_build.rs: build git-tracked vendored sources OUT OF TREE. Recipe scripts
rewrite their source, so builds were mutating version-controlled files: a
no-op sed became indistinguishable from a working one, and the integrity gate
fired so often that clearing it stopped being protective. Uses
cp -a --reflink=auto, not cp -al: sed -i is link-safe but `cp -f` and `>`
truncate in place and would write through a hard link into the tracked
original. Content hashing still runs against the pristine tree, so hashes now
describe committed state. Escape hatch: REDBEAR_IN_TREE_BUILD=1.
uutils / onig_sys
oniguruma guards its alloca.h include on the autoconf macro:
#if defined(HAVE_ALLOCA_H)
# include <alloca.h>
#endif
onig_sys is compiled by the `cc` crate, so no configure ever runs and
nothing defines it. Under GCC 14+ the resulting implicit declaration is
an error, so uutils failed at
regint.h:271: error: implicit declaration of function 'alloca'
relibc does ship <alloca.h>, so declare it in the cookbook's C flags.
This asserts a fact about the sysroot rather than silencing a warning,
and is what a cross-build environment is expected to supply for sources
that have no configure step. Packages whose own configure defines it to
1 are unaffected -- identical redefinitions are not diagnosed.
netutils
Declares `libredox = "0.1"`, a version string, so cargo resolved
libredox and its transitive redox_syscall from crates.io instead of the
forks. With the forks now at 0.1.19 / 0.9.1 the crates.io side is not in
the offline cache and the build died with
failed to download `redox_syscall v0.9.0`
Caused by: attempting to make an HTTP request, but --offline was
specified
local/AGENTS.md 'Local fork dependency rule (ABSOLUTE)' prohibits
version strings for crates that have a local fork, for exactly this
reason -- a second copy of the crate in the graph gives mismatched
types, and offline builds cannot resolve it at all. [patch.crates-io]
now redirects the direct and transitive resolutions onto the forks.
Both cook clean.
Three GCC 16 gaps found by the redbear-full build.
-fhardened (seatd and other meson recipes, 24 errors)
-fhardened is a host-glibc hardening bundle x86_64-unknown-redox cannot
implement, but GCC accepts it on the command line, so meson's
cc.has_argument('-fhardened') probe answers YES and meson adds it to
every compile. GCC then refuses it for real:
cc1: error: '-fhardened' not supported for this target [-Werror]
Note the tag is [-Werror], not [-Werror=hardened] -- it is an
unconditional warning, so -Wno-hardened does not silence it (tried, and
it did not). The flag has to be cancelled instead; -fno-hardened does
that cleanly and the cookbook's flags are appended after the project's
own, so it wins. Nothing is weakened: the compiler is telling us the
option is inert on this target.
termcap (two independent defects, both latent until now)
1. Makefile.in hardcoded 'CFLAGS = -g' and has no @CFLAGS@ substitution
at all, so nothing configure resolved ever reached the compiler --
including the toolchain's default C dialect. termcap is pre-ANSI
code, so being compiled as GCC 16's default C23 failed at once with
'too many arguments to function malloc; expected 0, have 1'.
Substituting @CFLAGS@ is what a normal autotools Makefile.in does.
2. With CFLAGS flowing, the remaining errors were implicit declarations
of strlen/memcpy/exit/write, which GCC 14+ makes errors regardless of
-std. Cause: autoconf 2.70+ removed AC_HEADER_STDC, so STDC_HEADERS
is never defined and termcap.c/tparam.c took their pre-ANSI branch,
declaring 'char *malloc ();' instead of including <stdlib.h>.
Autoconf's guidance on dropping AC_HEADER_STDC is to assume those
headers exist, which holds for every target Red Bear builds.
pam-redbear
Cargo.lock missed by the earlier sweep; regenerated for the 0.3.2 fork
versions.
termcap and seatd now cook clean.
The unique-temp-name fix was treating symptoms. The actual defect is
structural: run_parallel_cook documents the invariant
'each recipe builds in its own target/stage/sysroot'
and that invariant is false. dep_levels() keys on the package NAME, so a
recipe's optional-package variants -- llvm-native, llvm-native.dev,
llvm-native.runtime -- are distinct units with no dependency between
them. They therefore land in the same level and cook concurrently while
sharing one target/<triple>/ directory, because build/, sysroot/ and the
stage* dirs are all derived from the recipe directory, not the package
name.
create_dir_clean() on the shared build/ then wipes a sibling's tree
mid-compile:
ninja: error: failed recompaction: No such file or directory
which is the error the earlier parallel builds kept producing and which
unique temp names could never have fixed -- build/ is a real, shared,
destructively re-created directory, not a staging path. It is also why
llvm-native cooks clean in isolation but failed in every parallel build,
and why that failure was misread as a porting gap serious enough to
exclude the package from the config.
Make the recipe DIRECTORY the unit of mutual exclusion. Distinct recipes
still cook fully in parallel; only siblings serialise, and that costs
almost nothing because one cook already populates every stage dir for
the recipe -- the siblings that follow find their work done and return
cached. It also stops LLVM being built three times over.
Residual, not addressed here: recipes joined by [source] same_as (e.g.
llvm-native -> llvm21) share a source tree. That is read-only during
cook, but a concurrent fetch/patch of the shared source would be a
separate race.
Concurrency
-----------
Package variants of one recipe (llvm-native, llvm-native.dev,
llvm-native.runtime) share a single target/<triple>/ directory and are
cooked concurrently by the cook_jobs thread pool, but every staging path
under it had a fixed name -- stage.tmp and sysroot.tmp. The variants
therefore raced: one renamed or removed the shared directory while a
sibling was still writing into it.
Canonicalize stage dir failed at '.../stage.tmp': No such file or directory
Renaming failed from '.../sysroot.tmp' to '.../sysroot': File exists
llvm-native cooks clean on its own -- verified, zero errors, all three
variants published -- but failed in every parallel build for this
reason. It was never a compile error, which is what the earlier
'C++/pthread header gaps' note in the config misattributed.
Both stage.tmp sites and the sysroot site now go through one
unique_tmp_dir() helper. A pid alone cannot disambiguate (the pool is
threads in one process), so a process-wide counter supplies uniqueness.
Config
------
Restore gcc-native, llvm-native and rust-native to redbear-full.
gcc-native was set to "ignore" and the other two were commented out as
'not needed for greeter proof'. Both forms are forbidden by AGENTS.md
ABSOLUTE RULE -- NEVER DELETE, NEVER IGNORE, NEVER COMMENT OUT, and by
local/AGENTS.md 'No build-excluded until ported packages': a package
that does not build gets ported, not excluded. llvm-native is required,
not optional -- it supplies the host LLVM dev tree that libclc and
Mesa's iris/radeonsi CLC path need.
Two related races on recipes/<r>/target/<target>/sysroot, hit by
llvm-native during a -j 20 redbear-full build:
repo: failed to build: Renaming failed from '.../sysroot.tmp' to
'.../sysroot': File exists (os error 17)
llvm-native, llvm-native.dev and llvm-native.runtime are package
variants of one recipe and share that target directory. The cook_jobs
pool runs them as concurrent threads in a single process, so:
1. Shared staging path. Every variant used the same '<sysroot>.tmp', and
create_dir_clean() at the start of the staging block would wipe a
sibling's half-populated tree while it was still extracting pkgars
into it. Staging is now per-call, named from an atomic counter (a pid
would not disambiguate -- the pool is threads, not processes).
2. TOCTOU on the publish. The '!deps_dir.is_dir()' guard is checked long
before the rename, so all variants can pass it and then race; rename(2)
returns EEXIST/ENOTEMPTY once a winner has published a non-empty
directory, and the losers failed the whole recipe. Losing is harmless
-- every variant populates from the same recipe's dep set, so the
winner's tree is equivalent -- so the losers now discard their staging
and use it. The destination is deliberately not removed first, since a
concurrent cook may already be reading it.
GCC 13.2.0 defaulted to gnu17 (__STDC_VERSION__ 201710L); GCC 16.1.0
defaults to gnu23 (202311L). The recipe tree is C17-era code, and C23
turns an empty parameter list from 'unspecified arguments' into 'no
arguments', which is a hard error against a real prototype.
Pin the cookbook's default C dialect to gnu17. This states the dialect
these sources were written against rather than suppressing a diagnostic,
and matches how the distributions handled the same GCC 14/15 transition.
Packages migrate to C23 as they are touched; the pin is dropped when the
tree is clean. A per-recipe -std= still wins, being appended after. C++
is deliberately NOT pinned -- kwin needs C++23, which is the entire point
of the GCC 16 upgrade.
Fix a real cookbook bug this exposed: CMAKE_CXX_FLAGS was built from
CFLAGS, so the C dialect flag reached the C++ compiler and g++ reported
"'-std=gnu17' is valid for C/ObjC but not for C++" on every file. The
meson path already keeps c_args/cpp_args apart; CMake now matches.
CPPFLAGS still reaches both, which is what carries the sysroot includes.
First opportunistic C23 fix: libiconv's lib/loop_wchar.h declared
'extern size_t mbrtowc ();', conflicting with relibc's four-argument
prototype. That declaration exists only for platforms whose <wchar.h>
does not declare mbrtowc -- per its own comment, BeOS, which lacks
mbstate_t and #defines it -- and the very next line already tests
'#ifdef mbstate_t'. Moving it inside that guard keeps it where it is
needed and drops it where a real prototype is in scope.
libiconv now cooks clean against GCC 16.1.0.
Path-source recipes symlink recipes/<comp>/source at the local fork. The
guard used Path::exists(), which follows symlinks, so two cases silently
failed and then aborted the build with EEXIST:
* a DANGLING symlink (left behind after the checkout moved) reports
exists()==false, so nothing was removed;
* a live symlink TO a directory reports true, but remove_dir_all()
refuses to operate on a symlink and the error was swallowed by .ok().
Every core fork (relibc, kernel, base, bootloader, installer, redoxfs,
userutils) was dangling for this reason and no build could start.
force_symlink() classifies the entry with symlink_metadata() (which does
not follow the final component) and removes it correctly. It also writes
a RELATIVE link: recipe source links are committed build state that must
survive the checkout being moved or cloned to another prefix, and an
absolute link bakes in one machine's layout.
fetch_make_symlink() now repairs a same_as link whose target no longer
matches the recipe, instead of keeping it forever.
Real data from a running build: 6 recipes cooking concurrently but only ONE
actual compiler process, load 2.2/16. Recipes almost never hit their compile
phase simultaneously — most cook-time is single-threaded configure/link/IO — so
the strict per_make = jobs/concurrency division starved the one recipe that was
compiling (mesa pinned to -j2 while 14 cores idled).
Give each concurrent cook the full -j; the active compiler now uses the
whole machine and the OS scheduler absorbs the rare overlap. Worst-case
parallelism is bounded by cook_jobs * jobs, so build-redbear.sh now defaults
COOKBOOK_COOK_JOBS to 4 (was JOBS/2) to cap simultaneous heavy C++ compiles for
RAM safety. A shared make jobserver would cap total jobs precisely while keeping
full per-recipe -j — noted as the proper long-term fix.
The cook orchestrator built recipes strictly serially — COOKBOOK_MAKE_JOBS only
parallelized within a single recipe's make, so cores sat idle during every
recipe's configure/link/small-file phases. src/cook/scheduler.rs (dep_levels)
was written for this but never wired in (the module wasn't even declared, so it
never compiled).
- Declare the scheduler module (src/cook.rs) so dep_levels + its 7 tests build.
- New COOKBOOK_COOK_JOBS setting (config.rs, default 1 = serial): how many
recipes to cook concurrently.
- run_parallel_cook (repo.rs): partition recipes into topological dep-levels
(same level = no inter-deps = safe concurrent), process levels in order behind
a barrier, and within a level pull recipes from an atomic work-index across
scoped worker threads. The make-job budget is divided per level so
a lone heavyweight recipe (e.g. qtbase, alone at its level) still gets the full
-j, while crowded levels share it — no oversubscription, no slowdown for
serial-level recipes. Publishing stays a single batched step. Failure/nonstop
semantics preserved; only engaged for with cook_jobs>1 and >1 recipe.
- Safe because each recipe builds in its own target/stage/sysroot and a recipe's
sysroot is assembled from its dependencies' (lower-level, already-complete)
stage pkgars — read-only for concurrent siblings.
- package.rs: serialize packaging behind a global PACKAGE_LOCK. package() shares
process-global state (the lazily-created pkgar signing key at
build/id_ed25519.toml, name/metadata construction) that raced under concurrent
cooks (corrupt key, mangled names like 'x.dev.dev'); packaging is cheap, so
serializing it keeps the compile-parallelism win. Also make the package-name
construction fail gracefully (Err, not unwrap-panic) so a bad name fails one
recipe instead of crashing the whole scoped-thread build.
Validated: all-cached tree (COOK_JOBS=4) and a real concurrent build of 3
independent recipes (COOK_JOBS=3) both succeed and publish correctly.
Un-defer xwayland in config/redbear-full.toml (it is a kwin build dependency).
Getting it to build required:
- cook_build script.rs: cookbook_configure now defeats autotools maintainer-mode
regeneration. Release tarballs ship a complete build system, but extraction
timestamp skew makes make re-run the version-pinned aclocal-<N>/automake-<N>
the tarball was generated with, which the host (a different automake) lacks
('aclocal-1.NN: command not found'). Force generated files newer than their
sources (staggered mtimes) and no-op ACLOCAL/AUTOMAKE/AUTOCONF/AUTOHEADER as a
safety net. This unblocked the whole X11 autotools stack (libxcb, libx11,
libxfont2, libxkbfile, ...) with no per-recipe edits.
- xwayland recipe: meson -> custom template so it can (1) overlay the host
wayland-scanner onto the sysroot path meson resolves (the sysroot scanner is a
Redox binary, unrunnable on the build host; it is a build-time-only generator
emitting arch-independent C), and (2) unset COOKBOOK_DYNAMIC so the cross
pkg-config resolves --static, pulling the X11 libs' Requires.private closure
(xcb -> xau) — those libs are static-only on Redox (libtool has no Redox
shared support), so the closure was otherwise dropped and the link failed on
XauGetBestAuthByAddr/XauDisposeAuth.
- os/access.c: include <sys/utsname.h> on __redox__ (Redox lacks SIOCGIFCONF so
the uname()-based DefineSelf is compiled; relibc provides uname/struct utsname).
Produces a proper Redox /usr/bin/Xwayland ELF.
Root cause of kf6-kwayland's 'PlasmaWaylandProtocols >= 1.21.0 not found': the
plasma-wayland-protocols recipe (a custom data-only recipe) HARD-CODED
set(PACKAGE_VERSION "1.16.0") in the hand-written ConfigVersionCmake heredoc,
while the package + source are 1.21.0. Bumped it to 1.21.0.
Build-system fix (src/cook/cook_build.rs): the !check_source early-return served
the cached stage for dependency cooks WITHOUT honoring --force-rebuild — it fired
before the force_rebuild handling, so a stale binary-store artifact could never
be regenerated. Guard it with !cook_config.force_rebuild so --force-rebuild always
takes effect (repo binary rebuilt).
kf6-kwayland: declare redbear-input-headers (fakeinput.cpp needs <linux/input.h>).
The content-hash cache covered recipe-level PKGARs and the recipe's own
source tree, but not Cargo 'path =' dependency sources inside a crate —
the exact pattern the local fork model mandates (redox_syscall, libredox,
redox-driver-*, pcid_interface). Editing a path-dep's source left the
dependent recipe cached and stale. Observed live during the
driver-manager QEMU gate: 'cook driver-manager - cached' while
redox-driver-core sources had changed; the markers never reached the ISO.
Fix:
- New cook::cargo_path_deps module: resolves path deps from Cargo.toml
([dependencies], dev/build deps, target-specific deps, [patch.*]
tables, workspace members) recursively with cycle protection, and
hashes each resolved source tree (sorted relpath+content walk,
excluding .git/target — same policy as SourceContentHash).
- DepHashes gains a cargo_path_deps table (serde default for backward
compatibility; a missing field invalidates once, then re-syncs).
- The rebuild decision now ORs recipe-PKGAR changes with cargo path-dep
tree changes; the post-build write records the new map.
Acceptance: cook -> cached; content edit in redox-driver-core ->
'DEBUG: cargo path-dep source hashes changed' + rebuild; revert ->
rebuild once -> cached. 48 tests pass (43 cookbook lib + 5 new).
Boxing the Command(Command, ExitStatus) variant (204 bytes) and
Pkgar(pkgar::Error) variant (272 bytes) reduces the Error enum from
272+ bytes to ~80 bytes, eliminating all 70 result_large_err warnings.
Changes:
- Error::Command(Command, ExitStatus) → Command(Box<(Command, ExitStatus)>)
- Error::Pkgar(pkgar::Error) → Pkgar(Box<pkgar::Error>)
- Updated 2 construction sites in fs.rs (Command)
- Updated 1 construction site in package.rs (Pkgar)
- Updated 1 From conversion in lib.rs (Pkgar)
- Updated Display match in lib.rs (Command destructuring)
This is a heap allocation on error paths only — zero cost on the happy path.
Box<T> auto-implements Display/Debug when inner type does, so formatting
is unchanged.
Verification: cargo check ✅, cargo test --lib ✅ 38/38.
Clippy: 74 → 4 warnings (70 eliminated).
Q1 from assessment Part 6.2 — closes the largest robustness gap (Mechanism #2):
corrupt cached pkgar that passes mtime check was previously not detected
until runtime.
Publish side (repo_builder.rs):
- Accumulates BLAKE3 hash of each published file (pkgar, toml, dep_hashes,
auto_deps) into a BinaryStoreManifest
- Writes <recipe>.manifest.toml alongside published artifacts in repo/
Restore side (cook_build.rs):
- BinaryStoreManifest struct with read() returning Result<Option<Self>, String>
(same pattern as DepHashes: Ok(None) for missing = backward compat,
Err for corrupt TOML = loud WARN + skip)
- After restoring from binary store, verifies each file's BLAKE3 against
the manifest. On mismatch or missing file: WARN + all_restored=false
(forces rebuild). Missing manifest = backward compatible (older cookbook
published without one).
Helper (fs.rs):
- compute_file_blake3_hex(): 64KB chunked BLAKE3 hash, avoids loading
entire pkgar into memory
Tests: 3 new (roundtrip, missing-file Ok(None), corrupt-TOML Err).
Total: 38/38 pass. cargo check clean. No new clippy warnings.
Assessment doc Part 6.2/6.3/7 updated: Q1-Q6 all marked resolved.
Auto-correction table updated: transient network failure (Q2 retry) and
corrupt cached pkgar (Q1 BLAKE3 manifest) moved from 'does NOT auto-correct'
to 'auto-corrects'.
Tier 3 code improvements (from BUILD-SYSTEM-ASSESSMENT-2026-07-18 Part 6.2):
- Q2: Add run_command_with_retry() helper with exponential backoff (1s, 2s, 4s).
Wrap download_wget and git clone with 3 attempts. Git clone closure cleans
partial tmp dir between retries (git refuses non-empty clone targets).
- Q3: Preserve auto_deps.toml alongside dep_hashes.toml in binary store publish
(repo_builder.rs). Restore path prefers the preserved copy, falling back to
declared-depends-only reconstruction only when the preserved copy is absent
(e.g. published by an older cookbook). Preserves full ELF dynamic dep graph.
- Q4: DepHashes::read now returns Result<Option<Self>, String> discriminating
'missing' (legitimate first build, fall back to mtime) from 'corrupt' (loud
WARN + force rebuild). Closes a silent-swallow gap where a corrupt
dep_hashes.toml produced an incorrect mtime-fallback rebuild.
- Q6: Standardize error reporting in repo.rs nonstop path to {:#} format,
matching the pattern established in the earlier Tier 1 message commit.
Tier 1 message fixes (remaining items from assessment Part 5.3/5.4):
- M26: build-redbear.sh:569 '30-60 minutes' -> config-dependent range
- A6/A8: Makefile container messages add remediation hint
- B3: mk/config.mk sccache hint adds install guidance
- H2: mk/qemu.mk Unsupported ARCH lists supported values
- mk/redbear.mk comment 'fully-patched' -> 'versioned'
Verification: cargo check 0 errors, cargo test --lib 35/35 passed,
cargo clippy 0 new warnings vs baseline, make -n live + validate clean.
Phase 1 remediation of the build-system assessment:
- Makefile: wire 'validate' target into build/live/reimage flows; surface
lint-config and init-service validators as a first-class gate.
- mk/disk.mk: add 'validate' target running lint-config, init-service
validator, and file-ownership validator; suppress noisy unmount warnings
that masked real failures.
- mk/redbear.mk: add source-fingerprint tracking so integrate-redbear.sh
re-runs when local/recipes, local/Assets, or local/firmware change.
- src/cook/cook_build.rs: atomic dep_hashes.toml write (tmp + rename) to
prevent torn-write cache corruption; binary-store restore now checks
dep_hashes before silent restore; fix production bug in
collect_files_recursive that silently dropped subdirectories whose
name matched an exclude pattern.
- src/cook/fetch.rs, src/cook/fetch_repo.rs: harden atomic patch
application and protected-recipe gating.
- AGENTS.md, local/AGENTS.md, local/docs/COLLISION-DETECTION-STATUS.md:
sync collision-detection status to 'implemented (Phase 15.0)' now that
CollisionTracker is wired across all four installer layers
(installer submodule pointer tracked separately in 13cc6fb0c3).
Verified: cargo check (0 errors), cargo test --lib (35/35 passed),
cargo clippy (0 new warnings vs baseline), make -n validate, make -n live.
The repo_builder binary had a guard at line 79-81:
if std::env::var("COOKBOOK_CROSS_TARGET").is_ok_and(|x| !x.is_empty()) {
return Ok(());
}
This silently returned early when COOKBOOK_CROSS_TARGET was set —
which is ALWAYS the case during a Red Bear build (we cross-compile
for x86_64-unknown-redox from the host).
Net effect: pkgar files were NEVER published to repo/<target>/ during
the canonical build path, leaving the binary store effectively empty
across builds. With no cache, every recipe was forced to re-cook on
every build, even when its source and inputs had not changed.
Removed the guard so repo_builder always publishes. The per-target
subdirectory layout (repo/<target>/<pkg>.pkgar) is correct because
repo_builder uses redoxer::target() which resolves to the TARGET env
var, matching the lookup path used by cook_build.rs::restore_from_repo().
Previously, same_as recipes (ncursesw uses same_as = ../ncurses) would
fail with 'cannot guess version' because the cookbook only falls back
to Cargo.toml extraction, not the target's recipe.toml.
Now when source is same_as, read the target's recipe.toml directly and
use its [package].version field. Falls back to Cargo.toml probe if no
version is set in the target recipe.
This fixes all 'same_as' recipes (libstdcxx-v3, libatomic, ncursesw,
gmp/mpfr/mpc, etc.) without requiring explicit [package] on each.
netdb: git source with no rev/branch — added explicit version 0.1.0.
Reverted the 'guess_version -> 0.1.0' cookbook fallback — masking
missing versions is not a fix, proper recipe metadata is.
When recipe has no [package].version, no parseable tar URL version,
no git rev/branch version, no Cargo.toml version, and no directory-name
version — fall back to '0.1.0' instead of failing with 'cannot guess
version'. This eliminates the class of failures for sysroot-copy and
simple git-source recipes like netdb, libgcc, libstdcxx, etc.
Patches moved to legacy-superseded-2026-07-12/ because:
kernel/P4-s3-suspend-resume.patch
ALREADY APPLIED in kernel commit 51fdae08. This patch only added
Cargo.toml's acpi_ext dep, which is now in the fork.
kernel/redbear-consolidated.patch
APPLIED via 51fdae08. The cargo dep, build.rs, Makefile, and new
src/asm/x86_64/s3_wakeup.asm + src/arch/x86_shared/sleep.rs files
all made it in. Patch is now redundant.
userutils/P5-redbear-branding.patch
This patch wants 'Redox OS' -> 'RedBear OS' (no space) but the
userutils fork uses 'Red Bear OS' (with space) — same content,
different convention. The fork IS branded; this patch is a no-op.
relibc/P3-dns-resolver-hardening.patch
Source file structure has changed too far for patch(1) to apply.
Fork rebase (upgrade-forks.sh relibc) needed. Marked Phase 2.4 work.
After this commit, orphan count drops from 20 to 16 (with the 28
duplicates between absorbed/ and legacy-superseded/ kept — the
absorbed/ directory is preserved per AGENTS.md 'NEVER DELETE' rule).
Phase 0 stop-the-bleeding fixes:
1. ROOT COOKBOOK VERSION DRIFT (G2)
Cargo.toml was at 0.2.5 while branch is 0.3.1. Now matches.
Added inline comment explaining the sync-versions.sh invariant
to prevent the next fork bump from re-introducing drift.
2. SILENT TODO FALLBACK (G3)
src/cook/package.rs:199 used .unwrap_or("TODO".into()) which
emitted literal "TODO" into pkgar metadata when a recipe had
no parseable version. Now fails fast with a precise error
message that names the offending recipe and suggests a fix.
Replaces silent metadata lie with actionable diagnostic.
3. CARGO.LOCK DRIFT (G1)
After 0.3.0 -> 0.3.1 fork version sync, four Cargo.lock files
were regenerated to match the new +rb0.3.1 suffix:
- root Cargo.lock
- local/sources/libredox/Cargo.lock
- local/sources/userutils/Cargo.lock
- local/sources/base/Cargo.lock (also accumulated bytemuck
and bytemuck_derive patch bumps as a side effect)
- local/sources/bootloader/Cargo.lock
- local/sources/installer/Cargo.lock
Used 'cargo generate-lockfile' per fork with their path-dep
+ [patch.crates-io] config preserved. Verified each lockfile
now contains the correct Cat 2 fork versions matching their
Cargo.toml at +rb0.3.1.
4. BOOTLOADER VERSION OVERSIGHT
local/sources/bootloader/Cargo.toml was still at
1.0.0+rb0.3.0 after sync-versions.sh --check passed for
everything else. Manual fix applied. Also regenerated its
Cargo.lock to match.
5. COLLISION DETECTION HONESTY (G9)
AGENTS.md and local/AGENTS.md claimed a CollisionTracker
module existed in src/cook/collision.rs and was wired
through the installer at runtime. A whole-tree search
confirmed NO such code exists anywhere. Removed the false
promises and linked to local/docs/COLLISION-DETECTION-STATUS.md
which documents the actual current state (lint-config-only
init-service detection works; general package-vs-config
detection does NOT).
Verified:
- bash -n on all touched scripts: clean
- cargo check --bin repo: clean (redbear_cookbook v0.3.1 now)
- sync-versions.sh --check: clean (75 Cat 1 + 10 Cat 2)
- verify-fork-versions.sh: 1 pre-existing FAIL (bootloader
fork genuinely diverges from upstream tag 1.0.0 — out of
scope for Phase 0; documented for Phase 1 upgrade-forks work)
Implements Improvement 2 from local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md.
Replaces the mtime-only source invalidation with a BLAKE3 content hash
of every file in the source tree (plus recipe.toml and patches). The hash
catches content modifications that preserve mtime (git checkout, git
bisect, cp -a, rsync -a) and silently leave the cache stale.
Design choices:
- Hash written to target/<arch>/source_hash.txt after each successful
build, atomically (tmp + rename) to survive concurrent builds.
- Two-tier check: mtime fast-path (skip hash compute when mtime says
'changed'); hash verification (when mtime says 'unchanged', still
compute hash to catch mtime-preserving edits). The hash compute cost
is one pass over the source tree per build — acceptable for the
correctness gain.
- Symlinks are NOT followed in collect_files_recursive (avoids cycles
and surprises across the sysroot).
- Unreadable files are hashed as <unreadable> sentinel so a permission
fix later changes the hash and triggers a rebuild (correct
behavior).
- Patch filenames are hashed alongside contents so reordering or
renaming patches invalidates the cache (order matters for atomic apply).
- Devices, sockets, .git/, target/, *.swp, *.tmp are excluded.
Smoke-tested: hash is deterministic and different source dirs produce
different hashes. Cookbook compiles cleanly. Integration with the
existing source_changed decision is via OR (mtime OR hash triggers
rebuild).
Implements 7 of 8 improvements from local/docs/BUILD-SYSTEM-IMPROVEMENT-PROPOSAL.md:
Improvement 1+5: trap-based stash-and-restore for ALL 9 fork sources
- Replaces single-relibc stash with a loop over the canonical fork
inventory (relibc, kernel, base, bootloader, installer, redoxfs,
libredox, syscall, userutils)
- Records each stash SHA in a label-keyed map for safe round-tripping
- Pop stashes in LIFO order matching user-visible stash pop convention
- Idempotent: re-entry during the same build does not double-stash
- Reports (does not silently swallow) real git errors during stash push
- Restored on EXIT regardless of success/failure/signal
Improvement 3: cookbook binary freshness check
- Replaces existence-only check with BLAKE3-of-src/.rs fingerprint
- Hash written to target/release/.cookbook-src-fingerprint
- Rebuild triggers on any source file content change, not just mtime
- Survives git operations that change content without touching mtime
Improvement 4: strict-by-default uncommitted-edit gate
- Refuses to build with uncommitted edits in any fork source
- Catches the 'works on my machine' bug class where pkgar fingerprints
silently mismatch the committed HEAD
- Escape hatch: REDBEAR_ALLOW_DIRTY=1 for emergency CI use
- apply-durable-source-edits.py auto-enables strict mode when
REDBEAR_BUILD_PHASE is set (set by build-redbear.sh)
Improvement 6: failure-cleanup trap with diagnostics
- EXIT trap captures last 30 lines of each cookbook log
- Lists orphaned cook/cargo processes (often root cause of follow-on
build failures via held locks)
- Reports per-recipe build state (complete vs incomplete)
- Preserves diagnostics in REDBEAR_BUILD_STATE_DIR (mktemp -d)
- REDBEAR_KEEP_BUILD_STATE=1 retains the state dir after exit
Improvement 7: pre-cook offline/upstream consistency
- Pre-cook now follows the same offline policy as the main build phase
- REDBEAR_ALLOW_UPSTREAM=1 → offline=false
- default / REDBEAR_RELEASE → offline=true
- Pre-cook logs captured into REDBEAR_BUILD_LOGS_DIR for diagnostics
- Sets REDBEAR_BUILD_PHASE=pre-cook so downstream code can distinguish
Improvement 8: cookbook protection against out-of-band invocations
- src/bin/repo.rs emits a warning when invoked without
REDBEAR_CANONICAL_BUILD=1 in the environment
- The warning lists what is bypassed (cache invalidation, prefix
rebuild, fingerprint tracking, dirty gate)
- Non-fatal: CI scripts that manage their own checks are unaffected
Smoke-tested: ./local/scripts/build-redbear.sh redbear-mini now exits 1
when relibc/kernel have uncommitted edits, and proceeds when
REDBEAR_ALLOW_DIRTY=1 is set.
When a git-sourced recipe's source/ directory exists but has no
embedded .git (e.g., a cleanup pass removed .git directories from
build-cache sources, or the dir was extracted from an archive without
.git), the cookbook previously hard-bailed with
'{:?} is not a git repository, but recipe indicated git source'
This required manual intervention: the operator had to find the
broken source/ dir, rm -rf it, and re-run the build. With many
local recipes that use git URLs and embedded .git directories as
build caches (e.g., local/recipes/dev/ninja-build, local/recipes/kde/*),
this happened easily.
Fix: detect the missing .git, wipe the source dir, and re-clone from
the recipe's git URL. The fresh-clone logic is extracted to a new
reclone_git_source() helper used by both the initial-fetch path and
the self-heal path. After the self-heal, the source/ has a valid
.git and the rest of the fetch flow continues normally.
Tested by: deleting local/recipes/dev/ninja-build/source/.git (the
exact regression that triggered this fix) and running
./local/scripts/build-redbear.sh --upstream redbear-mini
which now self-heals instead of hard-failing.
C1 (Critical): Binary store restore now iterates ALL stage directories
instead of only the first, matching how cook_creates handles multi-stage.
C2 (Critical): All binary store restore errors are now logged via
log_to_pty! instead of being silently discarded with let _ =.
H1 (High): dep_hashes.toml keys now use full PackageName (including
host: prefix) via name.to_string() instead of name.without_prefix(),
preventing host/target key collisions.
H2 (High): Patch file mtimes are now included in source_modified
calculation, so editing a patch correctly triggers a rebuild.
H4 (High): All to_str().unwrap() calls replaced with safe alternatives
(to_string_lossy, direct PathBuf refs) to prevent panics on non-UTF8
paths.
H5 (High): auto_deps.toml reconstruction now logs a warning that it
may be incomplete (does not include ELF-discovered dynamic linking deps).
M1 (Medium): dep_hashes.toml is now written atomically via write-to-tmp
+ fs::rename, preventing corrupted/partial files on crash.
M3 (Medium): Missing source dir now triggers rebuild (SystemTime::now()
fallback) instead of being masked as no-change via UNIX_EPOCH.
Phase 1 — Hash-based cache invalidation:
- DepHashes struct: BLAKE3 hash of each build dep stored in dep_hashes.toml
- collect_current_dep_hashes(): reads blake3 from dep .toml metadata
- dep_hashes_changed(): compares stored vs current hashes
- Replaces mtime comparison as primary cache invalidation check
- Mtime fallback preserved for backward compatibility (no dep_hashes.toml)
- --force-rebuild CLI flag bypasses cache entirely
Phase 2 — Binary store cache lookup:
- repo_builder publishes dep_hashes.toml alongside .pkgar/.toml in repo/
- When target/ is missing but repo/ has the package, restores stage
artifacts by extracting pkgar, copying toml + dep_hashes.toml
- Auto-generates auto_deps.toml from repo depends field
- Only applies to non-remote, non-force-rebuild builds
See local/docs/BUILD-CACHE-PLAN.md for full architecture.
- Add x11proto to redbear-full.toml package list
- libxau recipe updated with x11proto dependency and custom build script
- Fixes libxau build failure: 'Package xproto was not found'
TLC (Twilight Commander) was missing from both ISO configs. Added
tlc = {} to [packages] in redbear-mini.toml and redbear-full.toml.
Created missing symlink: recipes/tui/tlc -> ../../local/recipes/tui/tlc.
When the cookbook copies a Path source into recipes/<name>/source, the
relative 'path = ...' references in [workspace.dependencies] are resolved
relative to the COPY location (recipes/<name>/source/), not the original
fork location (local/sources/<name>/). Since these directories are at
different depths from repo root (3 levels vs 4 levels), no single relative
path can resolve correctly in both locations.
Fix: for Path sources that point inside local/sources/ or local/recipes/,
use symlink instead of copy. The symlink preserves the original location
so the workspace root remains the fork's native directory, and relative
paths resolve consistently.
Also fixes symlink target bug: use canonicalize() to convert the path to
absolute form before symlinking. The previous code used the relative path
as the symlink target, which was resolved relative to the symlink's parent
directory (not where the symlink was created), producing broken symlinks.
vasilito