cook: give each parallel recipe the full -j budget (was dividing it)
redbear-ci / check (push) Waiting to run

Real data from a running build: 6 recipes cooking concurrently but only ONE
actual compiler process, load 2.2/16. Recipes almost never hit their compile
phase simultaneously — most cook-time is single-threaded configure/link/IO — so
the strict per_make = jobs/concurrency division starved the one recipe that was
compiling (mesa pinned to -j2 while 14 cores idled).

Give each concurrent cook the full -j; the active compiler now uses the
whole machine and the OS scheduler absorbs the rare overlap. Worst-case
parallelism is bounded by cook_jobs * jobs, so build-redbear.sh now defaults
COOKBOOK_COOK_JOBS to 4 (was JOBS/2) to cap simultaneous heavy C++ compiles for
RAM safety. A shared make jobserver would cap total jobs precisely while keeping
full per-recipe -j — noted as the proper long-term fix.
This commit is contained in:
2026-08-02 08:16:03 +03:00
parent f8d3a30a23
commit b20f45817a
2 changed files with 22 additions and 16 deletions
+10 -12
View File
@@ -9,7 +9,7 @@ PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# REDBEAR_VERSION (which tracks the OS release derived from the git branch).
# Starts at 1.0 and is bumped AUTOMATICALLY on every change by the pre-commit
# git hook (local/scripts/bump-build-version.sh); do not edit the minor by hand.
BUILD_REDBEAR_VERSION="1.10"
BUILD_REDBEAR_VERSION="1.11"
# ── Colorized output ──────────────────────────────────
# Enabled only on a TTY with NO_COLOR unset, so redirected build logs and CI
@@ -302,17 +302,15 @@ fi
CONFIG="redbear-full"
JOBS="${JOBS:-$(nproc)}"
# Dep-level parallel recipe cooking: cook this many independent (same-level)
# recipes at once. The make-job budget (JOBS) is divided across the concurrent
# cooks per level, so total compile parallelism stays ~JOBS (RAM stays bounded to
# roughly a serial -j${JOBS} build) while the extra recipe-concurrency fills the
# cores left idle during configure/link/small-file phases.
#
# Default to JOBS/2: this keeps per-recipe make jobs >= 2 (so a busy level still
# uses the whole -j${JOBS} budget), while giving enough recipe-concurrency to
# saturate the machine. Going higher (e.g. JOBS/1) would floor per-recipe jobs to
# 1 and leave part of the budget unused on large levels. Override with
# COOKBOOK_COOK_JOBS=1 to force serial, or any value to tune.
COOK_JOBS_DEFAULT=$(( JOBS / 2 )); [ "$COOK_JOBS_DEFAULT" -lt 1 ] && COOK_JOBS_DEFAULT=1
# recipes at once. Each concurrent cook gets the FULL -j${JOBS} make budget (the
# orchestrator no longer divides it) — recipes rarely hit their compile phase at
# the same moment, so the active compiler should use the whole machine while the
# others sit in single-threaded configure/link/IO. COOK_JOBS then bounds the
# worst case: at most COOK_JOBS * JOBS make jobs if every concurrent recipe
# compiled at once (rare). Default to 4 — enough to keep a compiler busy and fill
# gaps, while capping simultaneous heavy C++ compiles so RAM stays safe. Raise it
# on a big-RAM box; COOKBOOK_COOK_JOBS=1 forces serial.
COOK_JOBS_DEFAULT=4; [ "$JOBS" -lt "$COOK_JOBS_DEFAULT" ] && COOK_JOBS_DEFAULT="$JOBS"
export COOKBOOK_COOK_JOBS="${COOKBOOK_COOK_JOBS:-$COOK_JOBS_DEFAULT}"
APPLY_PATCHES="${APPLY_PATCHES:-1}"
NO_CACHE=0
+12 -4
View File
@@ -403,11 +403,19 @@ fn run_parallel_cook(
continue;
}
// Concurrency for this level, and the per-recipe make budget. A level
// with one recipe gets the full `total_make`; crowded levels share it so
// total parallelism stays ~`total_make` (no oversubscription).
// Give each concurrently-cooked recipe the FULL make budget rather than
// dividing it across the level. Recipes almost never hit their compile
// phase simultaneously — most cook-time is single-threaded configure/
// link/IO — so a strict `total_make / concurrency` division starves the
// one recipe that IS compiling (observed: a lone compiler pinned to -j2
// while 14 cores sat idle at load 2/16). Full -j lets the active
// compiler use the whole machine; the OS scheduler absorbs the rare
// overlap. Worst-case parallelism is bounded by `cook_jobs * total_make`,
// so keep COOKBOOK_COOK_JOBS modest if simultaneous heavy C++ compiles
// pressure RAM. (A shared make jobserver would cap total jobs precisely
// while still allowing full per-recipe -j — the proper long-term fix.)
let concurrency = cook_jobs.min(level_recipes.len());
let per_make = (total_make / concurrency).max(1);
let per_make = total_make;
let mut level_config = config.clone();
level_config.cook.jobs = per_make;