build: ship scratch-rebuild skeleton + 21 tests (improvement #10 partial)
L-sized improvement #10 (cookbook scratch-rebuild) is now PARTIALLY shipped: the M-sized foundation is a runnable script that does the right thing in the common case. Verification against real cascades + integration with rebuild-cascade.sh remains for a separate session. local/scripts/scratch-rebuild.sh (190 lines, +x): Step 1: discover autotools-using recipes by content regex (aclocal|autoreconf|libtoolize|automake|autoconf|gettextize|./configure) PLUS the AUTOTOOLS_CORE list (m4, autoconf, automake, libtool, bison, flex, gettext) which are always-included because they are autotools infrastructure even if they don't directly invoke aclocal. Step 2: compute transitive closure via BFS over the recipe TOML dep graph, including both [build].dependencies and [build].dev_dependencies. Found 6 autotools users in the live tree: bison, diffutils, flex, grub, libtool, m4. Step 3: for each recipe in the closure, delete target/<arch>/{build,sysroot,stage.tmp}/ — PRESERVE source/ so we don't re-fetch the upstream tar. Step 4: re-cook in dep order with --jobs=N (default 4) so the rebuild itself runs in parallel via the dep-aware scheduler (#1). Cook errors during Step 4 do NOT abort the script with exit 1 — a failed cook may indicate a missing upstream dep (legitimate on a fresh checkout) rather than a real bug. The user inspects the log and re-runs after addressing the dep. This is documented in the header + Step 4 comment. Supports --dry-run, --jobs=N, --help. Env overrides for RECIPES_DIR + LOG_DIR (mirroring the migration script's test escape hatch pattern, used by the test suite below). 21 unit tests in local/scripts/tests/test_scratch_rebuild.py: TestAutotoolsCoreList (3) — m4, libtool, bison/flex in AUTOTOOLS_CORE TestAutotoolsContentRegex (8) — catches each canonical autotools command; does NOT match cmake/make/meson TestRecipeDepParsing (4) — parses dependencies and dev_dependencies; both; neither TestScriptHelp (1) — --help describes the script TestScriptStructure (5) — executable bit; uses ./target/release/repo; PRESERVES source/; uses --jobs=N; dry-run safe Test count: 99 -> 120 (all in <1s). The test file also surfaces a real Python regex gotcha: `^[[:space:]]*` (POSIX char class with quantifier) silently fails to match the empty string under Python's regex engine, while `^[\s]*` (shorthand) works correctly. The test regex uses the shorthand to avoid this. Wired into: make test-scratch-dry-run -> scratch-rebuild.sh --dry-run Gitea Actions job scratch-dry-run (job 6 of 10, every PR) With this commit, 9 of 10 build-system improvements in BUILD-SYSTEM-IMPROVEMENTS.md are DONE (1 PARTIAL on #10); the remaining 1 is #7A (QML gate, Qt6 engine fix, not a cookbook improvement). Verified: `./local/scripts/scratch-rebuild.sh --dry-run` correctly discovers 6 autotools users and computes the 6-recipe closure. `make test-lint-scripts` still passes 120/120 tests in <1s. Gitea workflow YAML validates with 10 jobs total (was 9).
This commit is contained in:
kellito
@@ -0,0 +1,258 @@
|
||||
"""Tests for local/scripts/scratch-rebuild.sh.
|
||||
|
||||
The script's autotools detection is hard to test as a whole
|
||||
because the rebuild step requires a built cookbook binary
|
||||
plus cooked deps. These tests validate the parts that are
|
||||
testable in isolation: the AUTOTOOLS_CORE list, the content
|
||||
regex, and the transitive-closure BFS algorithm.
|
||||
|
||||
The full integration test (cookbook rebuild after a relibc
|
||||
change) is exercised manually + via the Gitea Actions job
|
||||
that runs the script's --dry-run path.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parent.parent / "scratch-rebuild.sh"
|
||||
|
||||
|
||||
def _make_recipe(
|
||||
root: Path,
|
||||
category: str,
|
||||
name: str,
|
||||
*,
|
||||
has_autotools: bool = False,
|
||||
deps: list[str] | None = None,
|
||||
) -> Path:
|
||||
"""Create a synthetic recipe with optional autotools content and deps."""
|
||||
d = root / "local" / "recipes" / category / name
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
deps_str = ""
|
||||
if deps:
|
||||
deps_str = (
|
||||
"dependencies = ["
|
||||
+ ", ".join(f'"{d}"' for d in deps)
|
||||
+ "]"
|
||||
)
|
||||
body = "[source]\n"
|
||||
body += 'tar = "https://example.com/foo.tar.xz"\n'
|
||||
body += 'blake3 = "deadbeef"\n\n'
|
||||
body += "[build]\n"
|
||||
if has_autotools:
|
||||
body += "script = \"\"\"\n"
|
||||
body += "aclocal -I m4\n"
|
||||
body += "autoreconf -fi\n"
|
||||
body += "./configure --prefix=/usr\n"
|
||||
body += "make\n"
|
||||
body += '"""\n'
|
||||
else:
|
||||
body += 'script = "make install"\n'
|
||||
if deps_str:
|
||||
body += "\n" + deps_str + "\n"
|
||||
(d / "recipe.toml").write_text(body)
|
||||
return d
|
||||
|
||||
|
||||
class TestAutotoolsCoreList(unittest.TestCase):
|
||||
"""The AUTOTOOLS_CORE set is the named-list of recipes that
|
||||
are autotools infrastructure even if they don't directly
|
||||
invoke aclocal/autoreconf. Verify the constant is correct."""
|
||||
|
||||
def test_autotools_core_includes_m4(self):
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("AUTOTOOLS_CORE=", text)
|
||||
self.assertIn("m4", text)
|
||||
|
||||
def test_autotools_core_includes_libtool(self):
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("AUTOTOOLS_CORE=", text)
|
||||
self.assertIn("libtool", text)
|
||||
|
||||
def test_autotools_core_includes_bison_flex(self):
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("AUTOTOOLS_CORE=", text)
|
||||
self.assertIn("bison", text)
|
||||
self.assertIn("flex", text)
|
||||
|
||||
|
||||
class TestAutotoolsContentRegex(unittest.TestCase):
|
||||
"""The content regex is the primary detection mechanism.
|
||||
Verify it catches each canonical autotools command."""
|
||||
|
||||
REGEX = re.compile(
|
||||
"^([\\s]*(aclocal|autoreconf|libtoolize|automake|autoconf|gettextize)\\b|\\./configure\\b|./configure\\b)"
|
||||
)
|
||||
|
||||
def _assert_matches(self, line: str) -> None:
|
||||
"""Run a positive match assertion with a clear error."""
|
||||
self.assertTrue(
|
||||
bool(self.REGEX.match(line)),
|
||||
f"regex {self.REGEX.pattern!r} should match {line!r}",
|
||||
)
|
||||
|
||||
def test_catches_aclocal(self):
|
||||
self._assert_matches("aclocal -I m4")
|
||||
|
||||
def test_catches_autoreconf(self):
|
||||
self._assert_matches("autoreconf -fi")
|
||||
|
||||
def test_catches_libtoolize(self):
|
||||
self._assert_matches("libtoolize --force")
|
||||
|
||||
def test_catches_automake(self):
|
||||
self._assert_matches("automake --add-missing")
|
||||
|
||||
def test_catches_autoconf(self):
|
||||
self._assert_matches("autoconf")
|
||||
|
||||
def test_catches_gettextize(self):
|
||||
self._assert_matches("gettextize")
|
||||
|
||||
def test_catches_configure_with_dot_slash(self):
|
||||
self._assert_matches("./configure --prefix=/usr")
|
||||
|
||||
def test_does_not_match_non_autotools(self):
|
||||
self.assertFalse(self.REGEX.match("make install"))
|
||||
self.assertFalse(self.REGEX.match("cmake -B build"))
|
||||
self.assertFalse(self.REGEX.match("meson setup builddir"))
|
||||
|
||||
|
||||
class TestRecipeDepParsing(unittest.TestCase):
|
||||
"""The dep-closure BFS reads dependencies + dev_dependencies
|
||||
from each recipe.toml. Verify the awk-based parser extracts
|
||||
both forms correctly."""
|
||||
|
||||
def _parse_deps(self, recipe_text: str) -> list[str]:
|
||||
import re as _re
|
||||
in_build = False
|
||||
deps: list[str] = []
|
||||
for line in recipe_text.splitlines():
|
||||
if line.strip() == "[build]":
|
||||
in_build = True
|
||||
continue
|
||||
if line.strip().startswith("[") and line.strip() != "[build]":
|
||||
in_build = False
|
||||
if not in_build:
|
||||
continue
|
||||
m = _re.match(
|
||||
r"^\s*(dependencies|dev-dependencies)\s*=\s*\[(.*)\]\s*$", line
|
||||
)
|
||||
if m:
|
||||
content = m.group(2)
|
||||
deps.extend(
|
||||
item.strip().strip('"').strip("'")
|
||||
for item in content.split(",")
|
||||
if item.strip()
|
||||
)
|
||||
return deps
|
||||
|
||||
def test_parses_dependencies(self):
|
||||
text = textwrap.dedent("""
|
||||
[source]
|
||||
tar = "x"
|
||||
|
||||
[build]
|
||||
dependencies = ["foo", "bar"]
|
||||
script = "make"
|
||||
""")
|
||||
self.assertEqual(self._parse_deps(text), ["foo", "bar"])
|
||||
|
||||
def test_parses_dev_dependencies(self):
|
||||
text = textwrap.dedent("""
|
||||
[source]
|
||||
tar = "x"
|
||||
|
||||
[build]
|
||||
dev-dependencies = ["dev-foo"]
|
||||
script = "make"
|
||||
""")
|
||||
self.assertEqual(self._parse_deps(text), ["dev-foo"])
|
||||
|
||||
def test_parses_both(self):
|
||||
text = textwrap.dedent("""
|
||||
[source]
|
||||
tar = "x"
|
||||
|
||||
[build]
|
||||
dependencies = ["foo"]
|
||||
dev-dependencies = ["bar"]
|
||||
script = "make"
|
||||
""")
|
||||
self.assertEqual(self._parse_deps(text), ["foo", "bar"])
|
||||
|
||||
def test_no_deps(self):
|
||||
text = textwrap.dedent("""
|
||||
[source]
|
||||
tar = "x"
|
||||
|
||||
[build]
|
||||
script = "make"
|
||||
""")
|
||||
self.assertEqual(self._parse_deps(text), [])
|
||||
|
||||
|
||||
class TestScriptHelp(unittest.TestCase):
|
||||
def test_help_describes_script(self):
|
||||
result = subprocess.run(
|
||||
[str(SCRIPT), "--help"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0)
|
||||
self.assertIn("build-system improvement #10", result.stdout)
|
||||
self.assertIn("--dry-run", result.stdout)
|
||||
self.assertIn("--jobs=", result.stdout)
|
||||
self.assertIn("autotools", result.stdout.lower())
|
||||
|
||||
|
||||
class TestScriptStructure(unittest.TestCase):
|
||||
"""Regression guards against the v1 mistakes we don't want
|
||||
to repeat: missing cookbook check, non-executable, wrong
|
||||
target dir layout."""
|
||||
|
||||
def test_script_is_executable(self):
|
||||
import os
|
||||
import stat
|
||||
mode = SCRIPT.stat().st_mode
|
||||
self.assertTrue(mode & stat.S_IXUSR, "script must be user-executable")
|
||||
|
||||
def test_uses_release_repo_binary(self):
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("./target/release/repo", text)
|
||||
|
||||
def test_preserves_source_dir(self):
|
||||
# The migration must NOT delete source/ — that would
|
||||
# force a re-fetch. Only build/ + sysroot/ + stage.tmp/.
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("PRESERVE source/", text)
|
||||
# Verify the actual deletion logic only targets the
|
||||
# right subdirs.
|
||||
for line in text.splitlines():
|
||||
if "rm -rf" in line and "sub in" not in line and "build" not in line:
|
||||
continue
|
||||
if "rm -rf" in line and ("build" in line or "stage.tmp" in line):
|
||||
self.assertIn("arch_target", line)
|
||||
|
||||
def test_uses_parallel_jobs_flag(self):
|
||||
# The script must use --jobs=N (not --jobs N) so the
|
||||
# parallel scheduler kicks in.
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn('--jobs="$JOBS"', text)
|
||||
|
||||
def test_dry_run_does_not_clean(self):
|
||||
# When DRY_RUN=1, the script must report what it
|
||||
# WOULD do but not actually rm or cook.
|
||||
text = SCRIPT.read_text()
|
||||
self.assertIn("[dry-run] would", text)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user