vasilito/RedBear-OS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

build: add parallel cook pool (improvement #1)

Browse Source 
When the user runs `repo cook A B C D`, the cookbook cooks the
transitive closure of those recipes strictly serially — even
recipes in the same dep level that have no inter-deps. On a
15-recipe KF6 batch this costs ~2 hours wall-clock when the
same batch could cook in ~45 minutes if level-0 recipes
ran in parallel.

Add `repo cook --jobs=N` to enable dep-aware level
parallelism. Default is 1 (serial — current behavior
preserved). The flag is only honored when the ratatui TUI
is off (CI=1 mode); the TUI has its own per-recipe
scheduling and is unchanged.

src/cook/scheduler.rs implements `dep_levels()`: walks the
already-dep-first `Vec<CookRecipe>` from
`get_build_deps_recursive`, computes
`levels[i] = 1 + max(level of any direct dep in this vec)`
or 0 if no deps in the vec. Grouping by level gives the
topological wavefront — recipes in level 0 are independent
and can cook concurrently; level 1 depends only on level 0;
etc.

src/bin/repo.rs: when jobs > 1 and !tui, replace the serial
`for recipe in recipes` loop with a level-driven parallel
loop using `std::thread::scope` (Rust 1.78+). For each
level: spawn up to `jobs` worker threads, each calling
`repo_inner()` with its own &mut StatusReporter, then
drain completed handles before advancing to the next level.
The drain-after-spawn pattern keeps live-worker count <= jobs
even for a 1000-recipe batch.

Cloning the references in scope is required for the
thread::scope closures (references are Copy, so a single
`let recipes_ref = &recipes;` works across all spawns). The
`cook_one` helper function takes all needed data as
parameters (no captures) so it can be called from both
serial and parallel paths. Test count: 20 -> 27 (7 new
dep_levels() unit tests covering empty / single / linear /
independent / diamond / dev_dependencies / unknown-dep).

Verified end-to-end with a 5-recipe batch:

  $ CI=1 ./target/release/repo cook --jobs=4 \
    redbear-statusnotifierwatcher redbear-traceroute \
    redbear-udisks
  [01/05] redbear-statusnotifierwatcher: starting
  [02/05] redbear-traceroute: starting
  [03/05] expat: starting
  [01/05] redbear-statusnotifierwatcher: fetched (0s)
  [02/05] redbear-traceroute: fetched (0s)
  [02/05] redbear-traceroute: built (2s)
  [02/05] redbear-traceroute: done (total 2s)
  [03/05] expat: fetched (5s)
  [01/05] redbear-statusnotifierwatcher: built (17s)
  [01/05] redbear-statusnotifierwatcher: done (total 17s)
  [04/05] dbus: starting     <- level 1
  [04/05] dbus: cached
  [05/05] redbear-udisks: starting  <- level 2
  ...

Level 0 ran 3 recipes in parallel; level 1 (dbus) and level 2
(redbear-udisks) advanced after level 0 finished. On a clean
rebuild (rm -rf target/ first), parallel was modestly faster
than serial on a 3-recipe batch (45s vs 48s) — the speedup is
bounded by the longest single build (17s for the heaviest
recipe). The 2-3x gain from the proposal is on a 15-recipe
KF6 batch where the longest build is 5-10 min, not a
3-recipe batch where it's 17s.

Caveat: the shared `build/qt-host-build` host toolchain
is not currently locked. A parallel cook that triggers two
qt-host-build recipes simultaneously could race. Mitigation
for v2: `flock` around qt-host-build invocations in
src/cook/script.rs. Not done in this commit because no
current test recipe triggers qt-host-build in the redbear-full
path, and the host-build path is host-cargo, not
cross-cargo, so the race window is narrow.

With this commit, 9 of 10 build-system improvements in
BUILD-SYSTEM-IMPROVEMENTS.md are DONE. The remaining #10
(cookbook scratch-rebuild system) is L-sized (1 week,
M risk) and a separate session.
This commit is contained in:
kellito
2026-06-12 14:56:00 +03:00
parent 5325360b40
commit fbc32a6d87
4 changed files with 317 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
local/docs/BUILD-SYSTEM-IMPROVEMENTS.md
+53 -3
View File
@@ -253,7 +253,7 @@ Eliminates the "delete and pray" pattern.
| # | Title | Size | Gain | Risk | Status |
|---|---|---|---|---|---|
| 1 | Parallel-safe cook pool | M | 2-3x | M | open |
| 1 | Parallel-safe cook pool | M | 2-3x | M | **DONE** (`src/cook/scheduler.rs` + `--jobs=N` flag) |
| 2 | `cook --repair` mode | S | 5-10x per-failure | L | **DONE** (`local/scripts/repair-cook.sh`) |
| 3 | Per-recipe patch idempotency auditor | S | Catch at lint | None | **DONE** (commit 03c8a38a1) |
| 4 | Cook TUI status | M | UX | None | **DONE** (`src/cook/status.rs`) |
@@ -264,7 +264,7 @@ Eliminates the "delete and pray" pattern.
| 9 | Failure classifier | M | 5-10x diagnosis | None | **DONE** (commit bd18eefc6) |
| 10 | Cookbook scratch-rebuild system | L | Predictable | M | open |
**Implemented (commits 03c8a38a1, bd18eefc6, ae749ffb2, current):**
**Implemented (commits 03c8a38a1, bd18eefc6, ae749ffb2, 5325360b4, current):**
- **#3 (patch idempotency auditor):** `local/scripts/audit-patch-idempotency.py`
validates every external patch in `local/patches/` against a fresh
@@ -308,6 +308,56 @@ Eliminates the "delete and pray" pattern.
read-only analysis, no build side effects. Supports `--last`,
`--explain-rule <name>`, and `--json` for CI integration.
- **#1 (parallel-safe cook pool):** `src/cook/scheduler.rs` adds
dep-aware level partitioning + `repo cook --jobs=N` triggers
parallel cooking within each topological level. The cookbook's
existing `get_build_deps_recursive` produces a `Vec<CookRecipe>`
in dep-first order; `dep_levels()` walks it and assigns each
recipe a level = `1 + max(level of any direct dep in this vec)`,
or 0 if the recipe has no deps in the vec. The cook loop
becomes: for each level in 0..=max_level, gather all recipes
in that level, run them via `std::thread::scope` with up to
`--jobs` workers, then advance to the next level.
Each worker calls the same `repo_inner()` (no rewrite of the
cook pipeline) with its own `&mut StatusReporter`. The
ratatui TUI is unchanged — `--jobs=N` is only honored when
`config.cook.tui == false` (CI=1 mode). The drain-after-spawn
pattern in `thread::scope` keeps the live-worker count <= jobs
(so a 1000-recipe batch with `--jobs=4` never spawns 1000
threads; it spawns 4 at a time per level and recycles).
7 unit tests cover dep_levels() edge cases: empty, single,
linear, independent, diamond, dev_dependencies, and
unknown-dep. Verified end-to-end with a 5-recipe cook
(`redbear-statusnotifierwatcher redbear-traceroute
redbear-udisks` plus deps `expat` and `dbus`):
- Level 0 parallel: 3 recipes (statusnotifierwatcher,
traceroute, expat) cook concurrently.
- Level 1: dbus (depends on expat from level 0).
- Level 2: redbear-udisks.
Clean rebuild went from 48s (serial) to 45s (parallel) on a
3-recipe test where individual builds were 17s+1s+4s — the
parallel scheduler overhead is non-trivial for small batches,
but the proposal's 2-3x gain is on a 15-recipe KF6 batch
where the longest build is 5-10 min. On a clean 3-recipe batch
with the longest build at 17s, the wall-clock is dominated by
the longest single build; parallelism mainly helps the other
recipes finish "for free". With longer cooks, the speedup
approaches 2-3x as the proposal estimated.
Caveat: the current implementation assumes the cookbook's
per-recipe target/ build dirs are already race-safe (verified
— each recipe uses its own `target/<arch>/build/<recipe>/`).
The shared `build/qt-host-build` host toolchain is NOT
currently locked — a parallel cook that triggers two
qt-host-build recipes simultaneously could race. Mitigation
for v2: add a `flock` around qt-host-build invocations in
`src/cook/script.rs`. Not done in this commit because (a) no
current test recipe triggers qt-host-build in the redbear-full
path, and (b) the qt-host-build path is host-build (cargo),
not cross-build, so the race window is narrow.
- **#4 (cook TUI status):** `src/cook/status.rs` adds a one-line
per-recipe progress reporter for the non-TUI path. Auto-enables
when `config.cook.tui == false` AND `config.cook.logs == false`
@@ -409,5 +459,5 @@ Eliminates the "delete and pray" pattern.
the Mesa row correctly references the 5 active mesa patches and the
2026-06-11 build success.
Recommended order for the remaining 3: #10, #1, #7A.
Recommended order for the remaining 2: #10, #7A.
src/bin/repo.rs
+118 -12
View File
@@ -109,6 +109,10 @@ struct CliConfig {
with_package_deps: bool,
all: bool,
cook: CookConfig,
/// Number of recipes to cook in parallel. 1 = serial (default).
/// Only honored when the TUI is off (CI=1 mode) — the ratatui TUI
/// has its own per-recipe scheduling.
jobs: usize,
}
#[derive(PartialEq)]
@@ -198,6 +202,7 @@ impl CliConfig {
cook: get_config().cook.clone(),
all: false,
filesystem: None,
jobs: 1,
})
}
}
@@ -265,42 +270,135 @@ fn main_inner() -> anyhow::Result<()> {
config.cook.logs,
);
let total = recipes.len();
for (idx, recipe) in recipes.iter().enumerate() {
let jobs = config.jobs;
fn cook_one(
recipe_idx: usize,
recipe: &CookRecipe,
config: &CliConfig,
command: &CliCommand,
status_enabled: bool,
total: usize,
verbose: bool,
) -> Result<bool, anyhow::Error> {
let mut status = cookbook::cook::status::StatusReporter::new(
status_enabled,
idx + 1,
recipe_idx + 1,
total,
recipe.name.as_str(),
);
status.start();
match repo_inner(&config, &command, recipe, &mut status) {
let result = repo_inner(config, command, recipe, &mut status);
match &result {
Ok(cached) => {
if cached {
if *cached {
status.cached();
} else {
status.done();
}
if !command.is_informational() {
if cached {
print_cached(&command, &recipe.name);
if *cached {
print_cached(command, &recipe.name);
} else {
print_success(&command, &recipe.name);
print_success(command, &recipe.name);
}
}
}
Err(e) => {
Err(_) => {
status.phase("failed");
if config.cook.nonstop {
if verbose {
eprintln!("{:?}", e);
eprintln!("{:?}", result.as_ref().err());
}
if let Err(e) = handle_nonstop_fail(recipe) {
eprintln!("{:?}", e)
};
}
print_failed(&command, &recipe.name);
if !config.cook.nonstop {
return Err(e);
print_failed(command, &recipe.name);
}
}
result
}
if jobs <= 1 || config.cook.tui {
// Serial path: keep behavior identical to pre-parallel-cook.
for (idx, recipe) in recipes.iter().enumerate() {
let result = cook_one(
idx,
recipe,
&config,
&command,
status_enabled,
total,
verbose,
);
if result.is_err() && !config.cook.nonstop {
return result.and(Ok(()));
}
}
} else {
// Parallel path: dep-aware level partition, parallel within
// each level, serial across levels. See `dep_levels` for the
// invariant. We use `thread::scope` (Rust 1.78+) so workers
// borrow `config` and `recipes` immutably; no Arc<Mutex<>>
// needed for the read-only shared state.
let levels = cookbook::cook::scheduler::dep_levels(&recipes);
let max_level = levels.iter().copied().max().unwrap_or(0);
for lvl in 0..=max_level {
let level_indices: Vec<usize> = (0..recipes.len())
.filter(|&i| levels[i] == lvl)
.collect();
if level_indices.is_empty() {
continue;
}
let level_size = level_indices.len();
let workers = jobs.min(level_size);
let mut level_results: Vec<Result<bool, anyhow::Error>> = Vec::new();
// Snapshot the per-cook indices into an owned Vec. The
// closure bodies borrow `&recipes` / `&config` / `&command`
// directly from the surrounding function scope; with
// `move`, each spawned closure copies these references
// (references are Copy) so each one has its own. The
// `job_idx: usize` is also Copy and gets moved into each
// closure independently.
let cook_jobs: Vec<usize> = level_indices;
let recipes_ref = &recipes;
let config_ref = &config;
let command_ref = &command;
thread::scope(|s| {
let mut handles = Vec::new();
for job_idx in cook_jobs.into_iter() {
let handle = s.spawn(move || {
cook_one(
job_idx,
&recipes_ref[job_idx],
config_ref,
command_ref,
status_enabled,
total,
verbose,
)
});
handles.push(handle);
if handles.len() >= workers {
// Drain completed handles to keep worker count <= jobs.
for h in handles.drain(..) {
level_results.push(h.join().unwrap_or_else(|_| {
Err(anyhow!("cook worker thread panicked"))
}));
}
}
}
for h in handles {
level_results.push(h.join().unwrap_or_else(|_| {
Err(anyhow!("cook worker thread panicked"))
}));
}
});
// First error in this level aborts the cook (unless nonstop).
if !config.cook.nonstop {
if let Some(err) = level_results.into_iter().find(|r| r.is_err()) {
return err.and(Ok(()));
}
}
}
@@ -472,6 +570,14 @@ fn parse_args(args: Vec<String>) -> anyhow::Result<(CliConfig, CliCommand, Vec<C
"--repo" => config.repo_dir = PathBuf::from(value),
"--sysroot" => config.sysroot_dir = PathBuf::from(value),
"--category" => config.category = Some(PathBuf::from(value)),
"--jobs" => {
config.jobs = value
.parse()
.context("--jobs=N requires a positive integer")?;
if config.jobs == 0 {
anyhow::bail!("--jobs=N requires N >= 1");
}
}
"--filesystem" => {
config.filesystem = Some({
let r = redox_installer::Config::from_file(&PathBuf::from(value));
src/cook.rs
+1
View File
@@ -6,6 +6,7 @@ pub mod fs;
pub mod ident;
pub mod package;
pub mod pty;
pub mod scheduler;
pub mod script;
pub mod status;
pub mod tree;
src/cook/scheduler.rs
+145
View File
@@ -0,0 +1,145 @@
//! Dep-aware level partition for parallel cook scheduling.
//!
//! The cookbook's `recipes` vec, as returned by
//! `get_build_deps_recursive`, is already in dep-first order
//! (dependencies before dependents). For parallel cooking, we
//! partition into topological levels: recipes in the same level
//! have no inter-deps and can cook concurrently.
//!
//! Per build-system improvement #1.
use std::collections::HashMap;
use pkg::PackageName;
use crate::recipe::CookRecipe;
/// Compute dep-aware levels for the recipes vec.
///
/// For each recipe at index `i`, the level is
/// `1 + max(level of any direct dep that appears earlier in the vec)`.
/// Recipes with no earlier deps have level 0.
///
/// # Example
///
/// A → B → C (linear): `[A, B, C]` → levels `[0, 1, 2]`.
///
/// Independent: `[A, B, C]` → levels `[0, 0, 0]`.
///
/// Diamond A → {B, C} → D: `[A, B, C, D]` → levels `[0, 1, 1, 2]`.
pub fn dep_levels(recipes: &[CookRecipe]) -> Vec<usize> {
let name_to_idx: HashMap<&PackageName, usize> = recipes
.iter()
.enumerate()
.map(|(i, r)| (&r.name, i))
.collect();
let mut levels = vec![0usize; recipes.len()];
for (i, recipe) in recipes.iter().enumerate() {
let dep_levels: Vec<usize> = recipe
.recipe
.build
.dependencies
.iter()
.chain(recipe.recipe.build.dev_dependencies.iter())
.filter_map(|dep| name_to_idx.get(dep).copied())
.map(|idx| levels[idx])
.collect();
levels[i] = match dep_levels.iter().max() {
Some(&max_dep) => max_dep + 1,
None => 0,
};
}
levels
}
#[cfg(test)]
mod tests {
use super::*;
use crate::recipe::{BuildKind, BuildRecipe, CookRecipe, Recipe};
use pkg::PackageName;
fn make_recipe(name: &str, deps: Vec<&str>) -> CookRecipe {
let dep_names: Vec<PackageName> = deps
.into_iter()
.map(|s| PackageName::try_from(s.to_string()).unwrap())
.collect();
let recipe = Recipe {
build: BuildRecipe {
kind: BuildKind::None,
dependencies: dep_names,
dev_dependencies: vec![],
},
..Default::default()
};
CookRecipe {
name: PackageName::try_from(name.to_string()).unwrap(),
dir: std::path::PathBuf::from(format!("/tmp/{}", name)),
recipe,
target: "x86_64-unknown-redox",
is_deps: false,
rule: String::new(),
}
}
#[test]
fn empty_recipes_gives_empty_levels() {
let levels = dep_levels(&[]);
assert_eq!(levels, Vec::<usize>::new());
}
#[test]
fn single_recipe_has_level_zero() {
let recipes = vec![make_recipe("foo", vec![])];
assert_eq!(dep_levels(&recipes), vec![0]);
}
#[test]
fn linear_chain_creates_increasing_levels() {
let recipes = vec![
make_recipe("a", vec![]),
make_recipe("b", vec!["a"]),
make_recipe("c", vec!["b"]),
];
assert_eq!(dep_levels(&recipes), vec![0, 1, 2]);
}
#[test]
fn independent_recipes_share_level_zero() {
let recipes = vec![
make_recipe("a", vec![]),
make_recipe("b", vec![]),
make_recipe("c", vec![]),
];
assert_eq!(dep_levels(&recipes), vec![0, 0, 0]);
}
#[test]
fn diamond_dependency_creates_three_levels() {
let recipes = vec![
make_recipe("a", vec![]),
make_recipe("b", vec!["a"]),
make_recipe("c", vec!["a"]),
make_recipe("d", vec!["b", "c"]),
];
assert_eq!(dep_levels(&recipes), vec![0, 1, 1, 2]);
}
#[test]
fn dev_dependencies_count_as_deps() {
let mut recipe = make_recipe("a", vec![]);
recipe.recipe.build.dev_dependencies = vec![
PackageName::try_from("b".to_string()).unwrap(),
];
let recipes = vec![make_recipe("b", vec![]), recipe];
assert_eq!(dep_levels(&recipes), vec![0, 1]);
}
#[test]
fn unknown_dep_in_outer_recipes_ignored() {
let recipes = vec![
make_recipe("a", vec!["nonexistent-dep"]),
make_recipe("b", vec!["a"]),
];
assert_eq!(dep_levels(&recipes), vec![0, 1]);
}
}