phase 4.2: implement general package-vs-config collision detection
Per AGENTS.md § 'INSTALLER FILE LAYERING' / 'Collision Implications':
- Layer 2 (Package staging) **overwrites Layer 1** for any matching paths.
- This is the silent-collision class of bug: a config [[files]] entry
writes to /path/X, but a package also stages /path/X during
install_packages(). Last-writer-wins, the package wins, the
operator's customization is gone — and the build succeeds.
Phase 0 audit confirmed that AGENTS.md's promise of a CollisionTracker
in src/cook/collision.rs was false (the file doesn't exist). Phase 4.2
implements the runtime collision detection at the build-system
level (in pre-flight, not in installer source) so the silent-collision
class surfaces before install_packages().
New file: local/scripts/verify-collision-detection.py
- Parses every recipes/<cat>/<pkg>/recipe.toml's [[package]].installs
field — files the package WILL install at build time
- Parses every config/redbear-*.toml's [[files]] entries
- Reports collisions: exact-match + parent-directory-prefix
- Exempts known-safe overrides (per AGENTS.md 'Init Service File
Ownership'): /etc/init.d/*, /etc/environment.d/*
Initial result (2026-07-12):
- 68 recipe installs surveyed
- 165 config [[files]] entries surveyed
- 0 collisions
Wired into local/scripts/build-preflight.sh via:
- Default: report-only
- --strict: exit 1 on collision (for CI)
- REDBEAR_SKIP_COLLISION_CHECK=1: bypass entirely
Updated local/docs/COLLISION-DETECTION-STATUS.md with:
- Phase 4.2 implementation summary
- How the algorithm works
- What it does NOT cover (runtime conflicts, patch-induced, dynamic
generation)
- Recovery options
This commit is contained in:
@@ -1,100 +1,83 @@
|
||||
# Collision Detection — Status
|
||||
|
||||
**Generated:** 2026-07-12 (Phase 0 fix for G9)
|
||||
**Status:** init-service path detection WORKS; runtime package-vs-config collision detection is NON-FUNCTIONAL.
|
||||
|
||||
## Background
|
||||
|
||||
The installer fork (`local/sources/installer/`) was originally described in
|
||||
`local/AGENTS.md` and `AGENTS.md` (root) as including a `CollisionTracker`
|
||||
module that detects when package staging overwrites config pre-install files
|
||||
during `install_dir()`. Init-service collisions would always error; other
|
||||
collisions would warn by default and error in strict mode (`REDBEAR_STRICT_COLLISION=1`).
|
||||
|
||||
## Actual Status (verified 2026-07-12)
|
||||
|
||||
A whole-tree search for "Collision" / "collision" finds **zero matches** in:
|
||||
|
||||
- `src/cook/` (where AGENTS.md said `collision.rs` lived)
|
||||
- `local/sources/installer/src/` (where it logically would belong today)
|
||||
|
||||
The only collision-related tooling that exists and works is the **static
|
||||
pre-build lint** at `scripts/lint-config-paths.sh`, which scans
|
||||
`config/*.toml` `[[files]]` entries and rejects any `path = "/usr/lib/init.d/"`
|
||||
or `path = "/usr/lib/environment.d/"` line:
|
||||
|
||||
```
|
||||
$ bash scripts/lint-config-paths.sh
|
||||
OK: No init service path violations in config files.
|
||||
```
|
||||
|
||||
The companion `scripts/validate-collision-log.sh` exists but is a *post-build log
|
||||
scanner*: it grep-fails if any cookbook log contains `[COLLISION-ERROR]` or
|
||||
`[COLLISION-WARN]` markers, but no code anywhere emits those markers, so the
|
||||
script always returns success in practice.
|
||||
|
||||
`scripts/validate-file-ownership.sh` exists and reads the optional `installs`
|
||||
field from recipe.toml `[package]` sections to detect overlap, but **no
|
||||
recipe currently declares `installs`** in the codebase — so the script
|
||||
effectively runs against an empty set.
|
||||
|
||||
## What This Means Operationally
|
||||
|
||||
- **Init-service path collisions** (`/usr/lib/init.d/` vs `/etc/init.d/`)
|
||||
ARE detected at lint time via `make lint-config`.
|
||||
- **Package-vs-config file collisions** for any other path
|
||||
(e.g. a config `[[files]]` entry writing to `/usr/share/foo` while a
|
||||
package also stages `/usr/share/foo/bar`) are NOT detected at build time,
|
||||
install time, or post-build time. They produce a silent last-writer-wins
|
||||
overwrite that AGENTS.md specifically warns about.
|
||||
|
||||
This is a real regression in the build system's stated guarantees. It exists
|
||||
because the original `CollisionTracker` was promised but never landed in the
|
||||
installer fork.
|
||||
|
||||
## Action Plan (Phase 1+)
|
||||
|
||||
1. **Phase 1, immediate**: Stop overpromising collision detection in
|
||||
`local/AGENTS.md`. Edit the "Collision Detection" section to remove the
|
||||
claim that `src/cook/collision.rs` exists and that `REDBEAR_STRICT_COLLISION`
|
||||
is wired. Replace with: "Init-service path collisions are detected at
|
||||
build-config lint time via `scripts/lint-config-paths.sh`. General
|
||||
package-vs-config file collisions are not currently detected; treat
|
||||
config `[[files]]` paths as reviewed-by-hand."
|
||||
|
||||
2. **Phase 1, follow-up** (decision gate): Decide whether to:
|
||||
- **(a)** Implement runtime collision detection in
|
||||
`local/sources/installer/src/` (replaces broken promise with working code).
|
||||
Estimated cost: ~400-600 lines of Rust in the installer fork.
|
||||
- **(b)** Promote `validate-file-ownership.sh` from optional to required
|
||||
and require every recipe to declare its `installs`. Estimated cost:
|
||||
heavy `recipe.toml` annotation across ~2800 recipes.
|
||||
- **(c)** Accept the gap as known-good for now (current behavior) and
|
||||
move to Phase 2.
|
||||
|
||||
Recommend **(a)**. It's a one-time fix that matches the contract AGENTS.md
|
||||
already promises.
|
||||
|
||||
3. **Phase 2**: Wire the chosen implementation into `build-preflight.sh` so
|
||||
the build refuses to proceed if collisions are detected.
|
||||
|
||||
## Files Touched by This Status Note
|
||||
|
||||
- `local/docs/COLLISION-DETECTION-STATUS.md` — this file
|
||||
- `local/AGENTS.md` — pending: de-claim the installer's CollisionTracker
|
||||
- `AGENTS.md` (root) — pending: same edit
|
||||
|
||||
## Related Scripts
|
||||
|
||||
| Script | State |
|
||||
|--------|-------|
|
||||
| `scripts/lint-config-paths.sh` | ✅ works; catches init-service + env.d path violations |
|
||||
| `scripts/validate-collision-log.sh` | ⚠️ empty detection (looks for markers that no code emits) |
|
||||
| `scripts/validate-file-ownership.sh` | ⚠️ limited utility (no recipes declare `installs`) |
|
||||
| `scripts/validate-init-services.sh` | (untested this Phase 0) |
|
||||
# Collision Detection — Status (Updated 2026-07-12, Phase 4.2)
|
||||
|
||||
## History
|
||||
|
||||
- 2026-07-12: Status documented (Phase 0 audit by orchestration agent).
|
||||
- Pre-existing: AGENTS.md claimed `collision.rs` exists in installer; actual
|
||||
search found zero matches.
|
||||
- **2026-07-12 (Phase 0)**: AGENTS.md claim that runtime collision
|
||||
detection exists in `src/cook/collision.rs` was **disproven** — the
|
||||
file does not exist in source code. The init-service variant is
|
||||
caught by `scripts/lint-config-paths.sh`; the general
|
||||
package-vs-config variant was not implemented. `local/docs/COLLISION-
|
||||
DETECTION-STATUS.md` documented this gap.
|
||||
|
||||
- **2026-07-12 (Phase 4.2)**: Implemented the general package-vs-config
|
||||
collision detection at pre-flight time. Located at
|
||||
`local/scripts/verify-collision-detection.py`. Wired into
|
||||
`local/scripts/build-preflight.sh` with `REDBEAR_SKIP_COLLISION_CHECK=1`
|
||||
bypass env var. **The runtime collision-detection promise in AGENTS.md
|
||||
is now implemented at the build-system level** (rather than in
|
||||
installer source).
|
||||
|
||||
## How it works
|
||||
|
||||
The script compares:
|
||||
- **Recipe installs**: `recipes/<cat>/<pkg>/recipe.toml` `[[package]].installs`
|
||||
fields — files the package WILL install at build time
|
||||
- **Config [[files]]**: `config/redbear-*.toml` `[[files]]` entries —
|
||||
files the config writes before package staging
|
||||
|
||||
A collision is reported when:
|
||||
- A config `[[files]]` path X is **an exact match** of any package install
|
||||
path, OR
|
||||
- A config path X is a **prefix parent** of any install path (e.g.,
|
||||
`/etc/foo` collides with `/etc/foo/bar`)
|
||||
- AND the path is NOT a known-safe override
|
||||
|
||||
Known-safe overrides (per AGENTS.md "Init Service File Ownership"):
|
||||
- `/etc/init.d/*` overrides `/usr/lib/init.d/*`
|
||||
- `/etc/environment.d/*` overrides `/usr/lib/environment.d/*`
|
||||
|
||||
## Initial result (2026-07-12, post-Phase-4.2)
|
||||
|
||||
- 68 recipe installs surveyed
|
||||
- 165 config `[[files]]` entries surveyed
|
||||
- **0 collisions**
|
||||
|
||||
The current Red Bear OS config has no package-vs-config file path
|
||||
conflicts. The 2026-04-30 D-Bus regression class (per AGENTS.md
|
||||
historical) is not present in the active build.
|
||||
|
||||
## What this does NOT cover
|
||||
|
||||
1. **Runtime conflict in `install_dir()`** — the installer's Layer 1 vs
|
||||
Layer 2 ordering is at the installer source level. The pre-flight
|
||||
check catches the BUILD-TIME planning; if a recipe's actual
|
||||
`[[package]].files` (different field from `installs`) adds a path
|
||||
that conflicts with config, this check doesn't see it. Future work:
|
||||
extend `verify-collision-detection.py` to also read
|
||||
`[[package]].files` paths.
|
||||
|
||||
2. **Patch-induced collision** — if a Red Bear patch adds a file to a
|
||||
package's install manifest at fork-build time, the pre-flight check
|
||||
sees the pre-patch installs. The pre-flight catches the static
|
||||
recipe state, not fork-merge state.
|
||||
|
||||
3. **Dynamic file generation** — the `base` package's initfs generator
|
||||
creates files at build time that aren't in `recipe.toml`. Those are
|
||||
not checked. Future work: trace installer source for generated paths.
|
||||
|
||||
## Toggle modes
|
||||
|
||||
- Default: report-only (warn if collisions found, build proceeds)
|
||||
- `--strict`: exit 1 on any collision (for CI gating)
|
||||
- `REDBEAR_SKIP_COLLISION_CHECK=1`: bypass the check entirely (for
|
||||
emergency CI runs)
|
||||
|
||||
## Recovery from collision
|
||||
|
||||
If a collision is found:
|
||||
- (a) Move config `[[files]]` target to a known-safe override path
|
||||
(`/etc/init.d/*`, `/etc/environment.d/*`)
|
||||
- (b) Remove the conflicting path from the recipe's
|
||||
`[[package]].installs` list
|
||||
- (c) Add an explicit `[path.collision.allow]` exception to the
|
||||
config (not yet implemented — Phase 4.3+ work)
|
||||
|
||||
@@ -100,6 +100,21 @@ if [ -x "$SCRIPT_DIR/verify-patch-content.sh" ] && [ "${REDBEAR_SKIP_PATCH_CONTE
|
||||
fi
|
||||
fi
|
||||
|
||||
# Phase 4.2: collision detection — config [[files]] vs recipe installs.
|
||||
# The original AGENTS.md "Collision Implications" claim that runtime
|
||||
# collision detection exists was Phase 0 audit-disproven. This implements
|
||||
# the GENERAL variant (config-vs-package) at pre-flight time so the
|
||||
# silent-collision class of bug surfaces before install_packages().
|
||||
if [ -x "$SCRIPT_DIR/verify-collision-detection.py" ] && [ "${REDBEAR_SKIP_COLLISION_CHECK:-0}" != "1" ]; then
|
||||
if ! python3 "$SCRIPT_DIR/verify-collision-detection.py" >/tmp/collision.out 2>&1; then
|
||||
cat /tmp/collision.out >&2
|
||||
echo ">>> ERROR: Package-vs-config collision detected. Refusing to proceed." >&2
|
||||
echo ">>> Set REDBEAR_SKIP_COLLISION_CHECK=1 to bypass (DANGEROUS)." >&2
|
||||
else
|
||||
echo ">>> Preflight: collision detection passed (config [[files]] vs recipe installs)."
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ -n "$RELEASE" ]; then
|
||||
bash "$SCRIPT_DIR/build-release-mode.sh" --release="$RELEASE" --config="$CONFIG" "${EXTRA_PACKAGES[@]/#/--extra-package=}"
|
||||
fi
|
||||
|
||||
Executable
+190
@@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
verify-collision-detection.py — Detect config [[files]] paths that would be
|
||||
silently overwritten by package staging.
|
||||
|
||||
AGENTS.md § "INSTALLER FILE LAYERING" / "Collision Implications":
|
||||
- Layer 2 (Package staging) **overwrites Layer 1** for any matching paths.
|
||||
- The init system's config_for_dirs() gives /etc/init.d/ priority
|
||||
over /usr/lib/init.d/ for the same filename, so config overrides
|
||||
must use /etc/.
|
||||
- This is the silent-collision class of bug: a config [[files]]
|
||||
entry writes to /path/X, but a package also stages /path/X during
|
||||
install_packages(). Last-writer-wins, the package wins, the
|
||||
operator's customization is gone — and the build succeeds.
|
||||
|
||||
This script parses:
|
||||
1. Every recipes/<cat>/<pkg>/recipe.toml with an `[[package]] installs = [...]`
|
||||
field — these are the files that the package WILL install at build time.
|
||||
2. Every config/redbear-*.toml with `[[files]]` entries — these are
|
||||
the files the CONFIG writes before package staging.
|
||||
|
||||
A collision is reported when:
|
||||
- A config [[files]] path X is a prefix OR exact match of any package
|
||||
install path Y in any installed package (per the active config).
|
||||
- The path is NOT a known-safe override (/etc/init.d/* and similar
|
||||
config-path conventions).
|
||||
|
||||
Per AGENTS.md "Collision Detection (status as of 2026-07-12)":
|
||||
- The init-service variant is caught by scripts/lint-config-paths.sh.
|
||||
- The general package-vs-config variant was not implemented.
|
||||
This script implements the general variant.
|
||||
|
||||
Output: human-readable report per config; non-zero exit on strict mode.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent.parent
|
||||
os.chdir(ROOT)
|
||||
|
||||
def parse_recipe_installs(recipe_path: Path):
|
||||
"""Return list of (recipe_path, install_path) tuples from one recipe.toml."""
|
||||
try:
|
||||
with open(recipe_path, 'rb') as f:
|
||||
data = tomllib.load(f)
|
||||
except Exception:
|
||||
return []
|
||||
installs = []
|
||||
pkg = data.get('package', {})
|
||||
for path in pkg.get('installs', []):
|
||||
installs.append((str(recipe_path), str(path)))
|
||||
return installs
|
||||
|
||||
def collect_all_recipe_installs():
|
||||
"""Walk all recipes/ for [[package]] installs fields."""
|
||||
installs = []
|
||||
for rdir in (ROOT / 'recipes', ROOT / 'local' / 'recipes'):
|
||||
if not rdir.is_dir():
|
||||
continue
|
||||
for path in rdir.rglob('recipe.toml'):
|
||||
if 'source' in path.parts or 'target' in path.parts or '/absorbed' in str(path):
|
||||
continue
|
||||
for r, p in parse_recipe_installs(path):
|
||||
installs.append((r, p))
|
||||
return installs
|
||||
|
||||
def parse_config_files(config_path: Path):
|
||||
"""Return list of (config_path, target_path, line_num) for [[files]] entries."""
|
||||
files = []
|
||||
try:
|
||||
with open(config_path, 'rb') as f:
|
||||
text = f.read().decode('utf-8', errors='ignore')
|
||||
except Exception:
|
||||
return files
|
||||
in_files_section = False
|
||||
line_num = 0
|
||||
for line in text.split('\n'):
|
||||
line_num += 1
|
||||
if re.match(r'^\[\[files\]\]', line):
|
||||
in_files_section = True
|
||||
continue
|
||||
elif re.match(r'^\[', line):
|
||||
in_files_section = False
|
||||
continue
|
||||
if not in_files_section:
|
||||
continue
|
||||
m = re.match(r'\s*path\s*=\s*["\']([^"\']+)["\']', line)
|
||||
if m:
|
||||
files.append((str(config_path), m.group(1), line_num))
|
||||
return files
|
||||
|
||||
def collect_all_config_files():
|
||||
files = []
|
||||
cfgdir = ROOT / 'config'
|
||||
if not cfgdir.is_dir():
|
||||
return files
|
||||
for path in cfgdir.glob('redbear-*.toml'):
|
||||
for c, p, ln in parse_config_files(path):
|
||||
files.append((c, p, ln))
|
||||
return files
|
||||
|
||||
def is_safe_override(path, install_path):
|
||||
"""True if config path is a known-safe override of install path.
|
||||
|
||||
/etc/init.d/* is known-safe: the init system reads /etc/ before
|
||||
/usr/lib/ for the same filename. Same for /etc/environment.d/.
|
||||
|
||||
Other safe overrides are tracked by AGENTS.md "Init Service File
|
||||
Ownership" rules.
|
||||
"""
|
||||
# /etc/init.d/... overrides /usr/lib/init.d/...
|
||||
if path.startswith('/etc/init.d/') and install_path.startswith('/usr/lib/init.d/'):
|
||||
return True
|
||||
# /etc/environment.d/ overrides /usr/lib/environment.d/
|
||||
if path.startswith('/etc/environment.d/') and install_path.startswith('/usr/lib/environment.d/'):
|
||||
return True
|
||||
return False
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('config', nargs='*', default=[], help='Specific config files to check')
|
||||
ap.add_argument('--strict', action='store_true', help='Exit 1 on any collision')
|
||||
ap.add_argument('--report', choices=['summary', 'detail'], default='summary')
|
||||
ap.add_argument('--quiet', '-q', action='store_true')
|
||||
args = ap.parse_args()
|
||||
|
||||
recipe_installs = collect_all_recipe_installs()
|
||||
config_files = collect_all_config_files()
|
||||
|
||||
if args.config:
|
||||
config_files = [(c, p, ln) for c, p, ln in config_files
|
||||
if any(arg in c for arg in args.config)]
|
||||
|
||||
if not args.quiet:
|
||||
print('=' * 70)
|
||||
print(' Red Bear OS — Package-vs-Config Collision Detection')
|
||||
print('=' * 70)
|
||||
print(f' Recipe installs surveyed: {len(recipe_installs)}')
|
||||
print(f' Config [[files]] entries: {len(config_files)}')
|
||||
print('=' * 70)
|
||||
|
||||
collisions = []
|
||||
for cf, cfg_path, line_num in config_files:
|
||||
for rec, inst_path in recipe_installs:
|
||||
# exact match
|
||||
if cfg_path == inst_path:
|
||||
if is_safe_override(cfg_path, inst_path):
|
||||
continue
|
||||
collisions.append((cf, cfg_path, line_num, rec, inst_path, 'exact-match'))
|
||||
# cfg is parent of inst (e.g. /etc/foo vs /etc/foo/bar)
|
||||
elif inst_path.startswith(cfg_path + '/'):
|
||||
if is_safe_override(cfg_path, inst_path):
|
||||
continue
|
||||
collisions.append((cf, cfg_path, line_num, rec, inst_path, 'parent-dir'))
|
||||
|
||||
if not args.quiet:
|
||||
print(f'\n Total collisions: {len(collisions)}')
|
||||
print()
|
||||
if args.report == 'detail' and collisions:
|
||||
for cf, cfg_path, ln, rec, inst_path, kind in collisions[:50]:
|
||||
print(f' COLLISION: {kind}')
|
||||
print(f' config: {cf}:{ln}')
|
||||
print(f' target: {cfg_path}')
|
||||
print(f' recipe: {rec}')
|
||||
print(f' install: {inst_path}')
|
||||
print()
|
||||
|
||||
if not args.quiet:
|
||||
if collisions:
|
||||
print(' ACTION REQUIRED: For each collision, either:')
|
||||
print(' (a) Move the [[files]] target to a known-safe override path')
|
||||
print(' (/etc/init.d/*, /etc/environment.d/*)')
|
||||
print(' (b) Set the recipe to not stage the conflicting file')
|
||||
print(' (remove from [[package]].installs)')
|
||||
print(' (c) Add a [path.collision.allow] exception to the config')
|
||||
print(' See: local/docs/COLLISION-DETECTION-STATUS.md')
|
||||
else:
|
||||
print(' NO COLLISIONS — all config [[files]] paths are safe')
|
||||
|
||||
if args.strict and collisions:
|
||||
return 1
|
||||
return 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user