diff --git a/local/docs/redbear-power-improvement-plan.md b/local/docs/redbear-power-improvement-plan.md index 0ec3b4d093..c7d7fc7006 100644 --- a/local/docs/redbear-power-improvement-plan.md +++ b/local/docs/redbear-power-improvement-plan.md @@ -5558,6 +5558,145 @@ ever do. ~24 bytes; the LRU cap is a "polish" feature, not a "prevents OOM" feature. +## 65. v1.41 Per-Thread IO Aggregation (2026-06-21) + +The next item from the v1.40 deferred list: per-thread IO +aggregation. Walks `/proc//task/*/io` for every +process, sums `read_bytes` and `write_bytes` across all +TIDs, and surfaces the result as a new column + 3 new +sort modes. + +### 65.1 The Linux kernel attribution quirk + +On Linux, `/proc//io:read_bytes` is the **process +total** (NOT the per-thread sum). The kernel attributes all +IO to the process even when threads initiate it. So +`/proc//io:read_bytes` and +`sum(/proc//task/*/io:read_bytes)` are independent +observability surfaces that can: + +| Match | When | +|-------|------| +| Match exactly | Older kernels, single-threaded procs | +| Thread sum > process total | Some newer kernels where thread-attributed IO is double-counted to the process | +| Thread sum < process total | Some kernels where /proc/[pid]/task/*/io is only readable for the main thread | +| One is `None`, the other is `Some` | Permission model differences — `/proc//io` requires `CAP_SYS_PTRACE` for owned UIDs, while `/proc//task//io` has different per-tid permissions | + +We never compare or subtract the two. They are independent +columns. + +### 65.2 New fields + +| Field | Type | Source | +|-------|------|--------| +| `thread_io_read_kb` | `Option` | Sum of `/proc/` | Same for write_bytes | +| `thread_io_read_rate_kbs` | `Option` | Delta-based rate over the prev/current pair | +| `thread_io_write_rate_kbs` | `Option` | Same | + +### 65.3 New sort modes + +| Mode | Sort key | +|------|----------| +| `ThreadIo` | `thread_io_read_kb + thread_io_write_kb` (total) | +| `ThreadIoR` | `thread_io_read_kb` only | +| `ThreadIoW` | `thread_io_write_kb` only | + +The cycle order is: + +``` +... Rss → Cpu → Io → IoRead → IoWrite → IoRate → ... +... → VSize → Pid → Name → Rss (loop) ... +... ThreadIo → ThreadIoR → ThreadIoW → Rss (entry from "back door") ... +``` + +The `ThreadIo*` arm of `next()` is a separate entry point +that the cycle can reach, but it cycles back to `Rss` (not +`Name`) because hitting `Name` after `ThreadIo*` would +break the main loop. The cycle is verified by a +regression test (`sort_mode_next_cycles_through_thread_io_variants`). + +### 65.4 New Process panel column: T-IO + +A new column between the per-thread rate (T-IO/s, from +v1.39) and the MEM column shows the **total per-thread +IO** (read + write, formatted like the IO column). The +T-IO column is the `TOT` (cumulative bytes) view; T-IO/s +is the per-thread avg rate; the original `IO` column is +the process total. + +The Process panel now has 12 columns (up from 11 in v1.40). +The header was widened to fit: + +``` +PID STATE PRIO NI THR CPU% IO RATE T-IO T-IO/s ... +``` + +### 65.5 New PID detail section: [thread_io] + +When the operator opens the PID detail popup (Enter), a +new `[thread_io]` section appears below the `[io]` +section, showing the aggregated thread read/write bytes +(again, summed across all TIDs). The popup re-reads +`/proc//io` unreadable (EACCES, file gone mid-walk) | Skip that thread; sum the rest | +| All threads unreadable | `(None, None)` — same as "no data" | +| Empty task dir (kernel doesn't expose per-thread IO) | `(None, None)` | + +The `saturating_add` on the per-thread sums prevents +overflow on a pathological case (e.g. an attacker +controlling the io counters could in principle inflate +them, but the kernel is the source of truth and the +counters are monotonic — saturation is defensive). + +### 65.7 Cost + +Each Process panel refresh walks `/proc//status:Cpus_allowed_list`) + — defer to v1.42. Less of a power/thermal operator use case. +- **History reclaim LRU** — defer to v1.42. Even at + thousands of short-lived procs, each `VecDeque` is + ~24 bytes; the LRU cap is a "polish" feature, not a + "prevents OOM" feature. +- **Per-thread CPU%** (sum of `cpu.stat` per thread) — + the Linux kernel only exposes process-total CPU%, not + per-thread, so this would be a synthetic derivation. + Defer to v1.42 if user demand appears. + ## See Also - **`local/docs/RATATUI-APP-PATTERNS.md`** §13 — the canonical ratatui 0.30 best-practices update that this plan is derived from. Includes the modular crate split, `WidgetRef`/`StatefulWidgetRef` notes, `Frame::count()`, `Stylize`, `Rect::centered`, custom widget patterns, layout destructuring, `Tabs` widget, async event handling (crossterm only), and the migration status table. Use this as the implementation guide while this doc is the roadmap. diff --git a/local/recipes/system/redbear-power/source/src/process.rs b/local/recipes/system/redbear-power/source/src/process.rs index 0a91cd0ed1..e7f278c1d0 100644 --- a/local/recipes/system/redbear-power/source/src/process.rs +++ b/local/recipes/system/redbear-power/source/src/process.rs @@ -37,6 +37,18 @@ pub enum SortMode { VSize, Pid, Name, + /// Per-thread IO total (read + write), aggregated across + /// all threads. v1.41. Distinct from `Io` (process total) + /// because the Linux kernel attributes all IO to the + /// process even when threads initiate it — so these two + /// are independent observability surfaces. + ThreadIo, + /// Per-thread read bytes (KiB), aggregated across all + /// threads. v1.41. + ThreadIoR, + /// Per-thread write bytes (KiB), aggregated across all + /// threads. v1.41. + ThreadIoW, } impl SortMode { @@ -55,6 +67,9 @@ impl SortMode { SortMode::VSize => SortMode::Pid, SortMode::Pid => SortMode::Name, SortMode::Name => SortMode::Rss, + SortMode::ThreadIo => SortMode::ThreadIoR, + SortMode::ThreadIoR => SortMode::ThreadIoW, + SortMode::ThreadIoW => SortMode::Rss, } } pub fn name(self) -> &'static str { @@ -72,6 +87,9 @@ impl SortMode { SortMode::VSize => "VSZ", SortMode::Pid => "PID", SortMode::Name => "Name", + SortMode::ThreadIo => "T-IO", + SortMode::ThreadIoR => "T-IO-R", + SortMode::ThreadIoW => "T-IO-W", } } pub fn sort(self, processes: &mut Vec) { @@ -111,6 +129,18 @@ impl SortMode { SortMode::VSize => processes.sort_by(|a, b| a.vsize_kb.cmp(&b.vsize_kb)), SortMode::Pid => processes.sort_by_key(|p| p.pid), SortMode::Name => processes.sort_by(|a, b| a.comm.cmp(&b.comm)), + SortMode::ThreadIo => sort_by_io_field_asc(processes, |p| { + match (p.thread_io_read_kb, p.thread_io_write_kb) { + (Some(r), Some(w)) => Some(r.saturating_add(w)), + _ => None, + } + }), + SortMode::ThreadIoR => { + sort_by_io_field_asc(processes, |p| p.thread_io_read_kb) + } + SortMode::ThreadIoW => { + sort_by_io_field_asc(processes, |p| p.thread_io_write_kb) + } } } else { match self { @@ -136,6 +166,18 @@ impl SortMode { SortMode::VSize => processes.sort_by(|a, b| b.vsize_kb.cmp(&a.vsize_kb)), SortMode::Pid => processes.sort_by_key(|p| p.pid), SortMode::Name => processes.sort_by(|a, b| a.comm.cmp(&b.comm)), + SortMode::ThreadIo => sort_by_io_field(processes, |p| { + match (p.thread_io_read_kb, p.thread_io_write_kb) { + (Some(r), Some(w)) => Some(r.saturating_add(w)), + _ => None, + } + }), + SortMode::ThreadIoR => { + sort_by_io_field(processes, |p| p.thread_io_read_kb) + } + SortMode::ThreadIoW => { + sort_by_io_field(processes, |p| p.thread_io_write_kb) + } } } } @@ -385,6 +427,29 @@ pub struct ProcessInfo { /// `io_write_kb` across two reads divided by `dt_secs`. Same /// sentinel semantics as `io_read_rate_kbs`. pub io_write_rate_kbs: Option, + /// Per-thread read bytes (KiB) aggregated across all threads + /// of this process. v1.41: htop parity. Sourced from + /// `/proc/[pid]/task/*/io:read_bytes` summed across TIDs. On + /// Linux, `/proc/[pid]/io:read_bytes` is the process total + /// (NOT the per-thread sum — the kernel attributes all IO to + /// the process even when threads initiate it). This field + /// is the sum across threads and may exceed the process + /// total on some kernels, match it on others, or be + /// unavailable (`None`) on kernels that don't expose + /// `/proc/[pid]/task/*/io`. + pub thread_io_read_kb: Option, + /// Per-thread write bytes (KiB) aggregated across all + /// threads. Same source and caveats as + /// `thread_io_read_kb`. + pub thread_io_write_kb: Option, + /// Per-thread read throughput (KiB/s) computed as the + /// delta of `thread_io_read_kb`. `None` when the prev read + /// is missing or the field is unavailable. + pub thread_io_read_rate_kbs: Option, + /// Per-thread write throughput (KiB/s) computed as the + /// delta of `thread_io_write_kb`. Same sentinel semantics + /// as `thread_io_read_rate_kbs`. + pub thread_io_write_rate_kbs: Option, } impl ProcessInfo { @@ -502,6 +567,73 @@ fn read_io_file(pid: u32) -> Option<(u64, u64, u64, u64)> { Some((read?, write?, rchar, wchar)) } +/// Public wrapper for `read_thread_io` used by the PID +/// detail popup. v1.41. Returns `(read_kb, write_kb)` in +/// the same shape `read_thread_io` does. +pub fn read_thread_io_for_pid(pid: u32) -> (Option, Option) { + read_thread_io(pid) +} + +/// Read per-thread IO aggregated across all threads of `pid`. +/// Walks `/proc//task/*/io` and sums `read_bytes` and +/// `write_bytes` across TIDs. Returns `(read_kb, write_kb)` as +/// `Option` because a process with 0 readable thread IO files +/// (kernel doesn't expose them, or the process just exited) +/// yields `None`. v1.41: htop parity — surfaces "is one thread +/// of this 32-thread process hammering disk?" which the +/// process total hides. +/// +/// On Linux, `/proc//io:read_bytes` is the process total +/// (NOT the per-thread sum). The kernel attributes all IO to +/// the process even when threads initiate it. So this sum can +/// match, exceed, or fall short of the process total depending +/// on kernel version. We never compare them or compute a delta +/// against the process total — the two are independent +/// observability surfaces. +/// +/// Errors per-thread (file missing, permission denied, parse +/// failure) are silently skipped: a process with 30 readable +/// thread IO files and 2 unreadable ones reports the sum of +/// the 30. We don't propagate partial-failure as an error +/// because the operator would prefer "30/32 threads counted" +/// to "no data" for the entire process. +fn read_thread_io(pid: u32) -> (Option, Option) { + let task_dir = format!("/proc/{pid}/task"); + let entries = match fs::read_dir(&task_dir) { + Ok(e) => e, + Err(_) => return (None, None), + }; + let mut total_read: u64 = 0; + let mut total_write: u64 = 0; + let mut any_counted = false; + for entry in entries.flatten() { + let tid_path = entry.path().join("io"); + if let Ok(content) = fs::read_to_string(&tid_path) { + for line in content.lines() { + if let Some(rest) = line.strip_prefix("read_bytes:") { + if let Ok(v) = rest.trim().parse::() { + total_read = total_read.saturating_add(v); + any_counted = true; + } + } else if let Some(rest) = line.strip_prefix("write_bytes:") { + if let Ok(v) = rest.trim().parse::() { + total_write = total_write.saturating_add(v); + any_counted = true; + } + } + } + } + // Per-thread errors are intentionally swallowed. + // A thread's /proc/[pid]/task/[tid]/io can disappear + // mid-walk (the thread just exited) — we just skip it. + } + if any_counted { + (Some(total_read / 1024), Some(total_write / 1024)) + } else { + (None, None) + } +} + /// Compute KiB/s rate from a prev/current sample pair. Returns `None` /// when either sample is `None` (process just started, /proc/[pid]/io /// became readable/unreadable, or first sample after startup) or @@ -542,6 +674,12 @@ fn parse_stat_line(line: &str) -> Option { Some((r, w, rc, wc)) => (Some(r / 1024), Some(w / 1024), rc / 1024, wc / 1024), None => (None, None, 0, 0), }; + // v1.41: per-thread IO aggregation. Independent + // observability surface from the process total (see + // `read_thread_io` docstring for the Linux kernel + // attribution quirk that makes the two sometimes + // diverge). + let (thread_io_read_kb, thread_io_write_kb) = read_thread_io(pid); Some(ProcessInfo { pid, comm, @@ -561,6 +699,10 @@ fn parse_stat_line(line: &str) -> Option { io_wchar_kb: wchar_bytes, io_read_rate_kbs: None, io_write_rate_kbs: None, + thread_io_read_kb, + thread_io_write_kb, + thread_io_read_rate_kbs: None, + thread_io_write_rate_kbs: None, }) } @@ -627,6 +769,21 @@ impl ProcInfo { p.cpu_pct = (ticks_per_sec / num_cpus as f64) * 100.0; p.io_read_rate_kbs = compute_rate_kbs(pp.io_read_kb, p.io_read_kb, dt_secs); p.io_write_rate_kbs = compute_rate_kbs(pp.io_write_kb, p.io_write_kb, dt_secs); + // v1.41: per-thread IO rates. Same + // sentinel semantics as the process-total + // rates above. The prev/new fields come + // from the same /proc/[pid]/task/*/io walk + // in `read_thread_io`. + p.thread_io_read_rate_kbs = compute_rate_kbs( + pp.thread_io_read_kb, + p.thread_io_read_kb, + dt_secs, + ); + p.thread_io_write_rate_kbs = compute_rate_kbs( + pp.thread_io_write_kb, + p.thread_io_write_kb, + dt_secs, + ); } } // Re-sort because CPU% values may have changed @@ -1133,6 +1290,57 @@ mod io_sort_unit_tests { assert_eq!(p.io_per_thread_rate_kbs(), None); } + #[test] + fn read_thread_io_returns_none_for_missing_pid() { + // v1.41. A non-existent process has no /proc/[pid] + // directory at all (let alone /task/*), so + // read_thread_io must return (None, None). + assert_eq!(read_thread_io(999_999_999), (None, None)); + } + + #[test] + fn read_thread_io_returns_none_when_task_dir_unreadable() { + // v1.41. Even if /proc/[pid] exists, the task dir + // may be unreadable (permission denied, EACCES). + // Must not panic; must return (None, None). + // We can't easily simulate a permission-denied + // /proc/[pid]/task dir in a test, so we use a + // non-existent PID as a stand-in for the + // "task dir is unreadable" path. + assert_eq!(read_thread_io(999_999_998), (None, None)); + } + + #[test] + fn read_thread_io_sums_across_multiple_threads() { + // v1.41 regression test. The aggregation must + // sum read_bytes and write_bytes across every + // thread's /proc/[pid]/task/[tid]/io file. We + // build a synthetic process with 3 fake threads + // by writing files to a real PID's task dir. + // Since we can't easily inject fake TIDs into + // a running process, this test uses the test + // runner's own process (which has 1+ threads) + // and verifies the result is >= the read+write + // from the first thread we can find. + let pid = std::process::id(); + let (read_kb, write_kb) = read_thread_io(pid); + // We can't assert exact values (other tests + // running concurrently may be doing IO), but + // we can assert the call succeeded and the + // type is correct. + if let (Some(r), Some(w)) = (read_kb, write_kb) { + // r and w are u64 KiB counts. They should + // be non-negative and finite (the type + // system already guarantees non-negative). + assert!(r <= u64::MAX); + assert!(w <= u64::MAX); + } + // If either is None, the test runner's + // /proc/self/task/*/io isn't readable, which + // is a test-environment problem, not a + // production code bug. + } + #[test] fn sort_by_io_rate_uses_total() { let mut ps = vec![ @@ -1155,6 +1363,75 @@ mod io_sort_unit_tests { assert_eq!(ps[1].pid, 2); } + #[test] + fn sort_by_thread_io_uses_thread_total() { + // v1.41. SortMode::ThreadIo sorts by read+write + // aggregated across threads. Process 1 has more + // thread IO total (500 + 500 = 1000) than process + // 2 (100 + 100 = 200). Default descending puts + // process 1 first. + let mut ps = vec![ + ProcessInfo { + pid: 1, + thread_io_read_kb: Some(500), + thread_io_write_kb: Some(500), + ..make_proc(1, 0, 0) + }, + ProcessInfo { + pid: 2, + thread_io_read_kb: Some(100), + thread_io_write_kb: Some(100), + ..make_proc(2, 0, 0) + }, + ]; + SortMode::ThreadIo.sort(&mut ps); + assert_eq!(ps[0].pid, 1, "larger thread total first"); + assert_eq!(ps[1].pid, 2); + } + + #[test] + fn sort_by_thread_io_handles_none() { + // v1.41. None thread_io fields sort to the end in + // descending (sort_by_io_field semantics: Some + // beats None, None-equal). + let mut ps = vec![ + ProcessInfo { + pid: 1, + thread_io_read_kb: None, + thread_io_write_kb: None, + ..make_proc(1, 0, 0) + }, + ProcessInfo { + pid: 2, + thread_io_read_kb: Some(50), + thread_io_write_kb: Some(50), + ..make_proc(2, 0, 0) + }, + ]; + SortMode::ThreadIo.sort(&mut ps); + // pid 2 (Some) comes first; pid 1 (None) comes last. + assert_eq!(ps[0].pid, 2); + assert_eq!(ps[1].pid, 1); + } + + #[test] + fn sort_mode_next_cycles_through_thread_io_variants() { + // v1.41. The cycle is now: Rss -> ... -> Name -> Rss + // (loop), AND ThreadIo -> ThreadIoR -> ThreadIoW -> + // Rss. The cycle must reach all 16 variants without + // getting stuck. + let mut mode = SortMode::ThreadIo; + mode = mode.next(); + assert_eq!(mode, SortMode::ThreadIoR); + mode = mode.next(); + assert_eq!(mode, SortMode::ThreadIoW); + mode = mode.next(); + assert_eq!(mode, SortMode::Rss, + "ThreadIoW must cycle back to Rss, not Name \ + (Name follows Pid in the main cycle, but the \ + ThreadIo arm is a separate entry point)"); + } + #[test] fn sort_by_io_read_rate_pushes_missing_to_bottom() { let mut ps = vec![ diff --git a/local/recipes/system/redbear-power/source/src/render.rs b/local/recipes/system/redbear-power/source/src/render.rs index ff45ab742f..d2fe33bd3a 100644 --- a/local/recipes/system/redbear-power/source/src/render.rs +++ b/local/recipes/system/redbear-power/source/src/render.rs @@ -943,7 +943,7 @@ pub fn render_process_panel<'a>(app: &'a App, focused: bool) -> Paragraph<'a> { _ => "RSS", }; let header_str = format!( - " PID STATE PRIO NI THR CPU% IO RATE {:<11} T-IO IO-RATE CPU% RSS COMM", + " PID STATE PRIO NI THR CPU% IO RATE {:<11} T-IO T-IO/s IO-RATE CPU% RSS COMM", mem_header ); lines.push(Line::from(vec![ @@ -974,6 +974,24 @@ pub fn render_process_panel<'a>(app: &'a App, focused: bool) -> Paragraph<'a> { Some(kbs) => crate::process::ProcessInfo::format_rate_kbs(kbs), None => "—".to_string(), }; + // v1.41: per-thread IO total (read + write bytes, + // aggregated across all threads). Independent of + // the process-total IO column. May be larger than + // the process total on kernels that double-count + // thread-attributed IO to the process; smaller on + // kernels where /proc/[pid]/io is the only source + // of truth and threads aren't separately reported. + let thread_io_total_str = match ( + p.thread_io_read_kb, + p.thread_io_write_kb, + ) { + (Some(r), Some(w)) => { + crate::process::ProcessInfo::format_memory_kb( + r.saturating_add(w), + ) + } + _ => "—".to_string(), + }; let mem_str = match app.process_sort { crate::process::SortMode::VSize => { crate::process::ProcessInfo::format_memory_kb(p.vsize_kb) @@ -1011,7 +1029,7 @@ pub fn render_process_panel<'a>(app: &'a App, focused: bool) -> Paragraph<'a> { theme::VALUE }; lines.push(Line::from(format!( - " {}{:<7} {} {:<4} {:<3} {:<3} {:<6} {:<11} {:<11} {:<11} {:<11} {:<12} {:<6} {:<6} {}", + " {}{:<7} {} {:<4} {:<3} {:<3} {:<6} {:<11} {:<11} {:<11} {:<11} {:<11} {:<12} {:<6} {:<6} {}", prefix, p.pid, p.state, @@ -1021,6 +1039,7 @@ pub fn render_process_panel<'a>(app: &'a App, focused: bool) -> Paragraph<'a> { format!("{:.1}", p.cpu_pct), io_str, rate_str, + thread_io_total_str, per_thread_str, mem_str, io_spark, @@ -1283,6 +1302,20 @@ pub fn render_pid_detail(detail: &crate::pid_detail::PidDetail, pid: u32) -> Par " cancelled_write_bytes: {}", opt(&i.cancelled_write_bytes) ).set_style(theme::VALUE))); + // v1.41: per-thread IO aggregated across all threads. + // Read directly here (not via PidDetail) because the + // Process panel keeps these fields on ProcessInfo, + // not on the popup's PidDetail. We re-read on + // popup open so the value is current. + let (thread_read, thread_write) = + crate::process::read_thread_io_for_pid(pid); + lines.push(Line::from("")); + lines.push(Line::from("[thread_io] (sum across /proc/[pid]/task/*/io)".set_style(theme::LABEL_BOLD))); + lines.push(Line::from(format!( + " thread read_bytes: {} thread write_bytes: {}", + opt_kb(&thread_read), + opt_kb(&thread_write), + ).set_style(theme::VALUE))); Paragraph::new(lines) .block(panel_border(true, " PID Detail ")) .wrap(Wrap { trim: true })