vasilito/RedBear-OS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

driver-manager v4.3: Driver::on_error in-process + REDBEAR_DRIVER_ERROR_FD IPC

Browse Source 
Closes the v4.2 plan's 'Driver-level Driver::on_error IPC' item.

Three pieces, layered:

1. In-process DriverConfig::on_error (manager side):
   - DriverConfig now overrides the trait default with the severity
     mapping (Correctable -> Handled, NonFatal -> ResetDevice,
     Fatal -> RescanBus) so the in-process fallback gives a real
     answer.
   - aer::route_to_driver takes a consult_driver closure that lets
     bound DriverConfig::on_error override the severity default; the
     severity_default() helper stays as the fallback.

2. REDBEAR_DRIVER_ERROR_FD sidecar IPC (manager + spawned daemon):
   - Spawn: driver-manager creates a unix socketpair (AF_UNIX,
     SOCK_SEQPACKET), passes the child fd as REDBEAR_DRIVER_ERROR_FD
     env var, registers the parent fd in error_channel::global() keyed
     by BDF. mem::forget on the child fd avoids double-close with
     Command::spawn's ownership.
   - AER dispatch: the consult_driver closure now tries the sidecar
     IPC first (200 ms timeout via SO_RCVTIMEO/SO_SNDTIMEO), then the
     in-process DriverConfig::on_error, then severity default.
   - Reap: error_channel::global().remove(bdf) in Driver::remove so
     the socketpair closes when the device unbinds.

3. linux-kpi C-callable opt-in (driver side):
   - New c_headers/linux/pci.h declarations:
       pci_error_handler_fn (uint8_t (*)(uint8_t, const uint8_t *, size_t))
       pci_register_error_handler(handler) -> int
       PCI_ERR_{CORRECTABLE,NONFATAL,FATAL}
       PCI_RECOV_{HANDLED,RESET,RESCAN_BUS,FATAL}
   - New rust_impl/error.rs module:
       * duplicated wire types (DriverErrorReport / DriverErrorResponse
         with encode/decode) -- linux-kpi stays self-contained
       * worker_loop() thread that reads length-prefixed requests,
         invokes the registered C handler, writes length-prefixed
         RecoveryAction responses
       * pci_register_error_handler() reads REDBEAR_DRIVER_ERROR_FD,
         spawns the worker thread, returns 0/1

Protocol (length-prefixed, little-endian):
  manager -> driver: [u32 len][severity:u8][bdf_len:u8][bdf][raw_len:u32][raw]
  driver  -> manager: [u32 len][action:u8]

Tests:
  driver-manager:  70 passed (was 65; +5 from error_channel + aer)
  linux-kpi:       cargo check clean (host test link fails on
                  redox_strerror_v1, pre-existing)
This commit is contained in:
vasilito
2026-07-25 06:20:06 +09:00
parent 25037b42c8
commit 3425f55c44
8 changed files with 744 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
local/recipes/drivers/linux-kpi/source/src/c_headers/linux/pci.h
+30
View File
@@ -114,4 +114,34 @@ extern int pci_restore_state(struct pci_dev *dev);
.vendor = (vend), .device = (dev), \
.subvendor = PCI_ANY_ID, .subdevice = PCI_ANY_ID
/* --- Driver-side error-report sidecar (REDBEAR_DRIVER_ERROR_FD) --- */
/* Severity discriminants (must match redox-driver-core::ErrorSeverity). */
#define PCI_ERR_CORRECTABLE 0
#define PCI_ERR_NONFATAL 1
#define PCI_ERR_FATAL 2
/* Action discriminants (must match redox-driver-core::RecoveryAction). */
#define PCI_RECOV_HANDLED 0
#define PCI_RECOV_RESET 1
#define PCI_RECOV_RESCAN_BUS 2
#define PCI_RECOV_FATAL 3
/*
* Register a C error handler and start the worker thread that
* services the sidecar IPC fd passed via REDBEAR_DRIVER_ERROR_FD.
*
* Returns 1 on success, 0 if the env var is unset, the fd is invalid,
* or a handler is already registered.
*
* The handler is invoked from a dedicated worker thread for every
* AER event the parent forwards to this driver. The handler must
* return promptly (the parent's IPC timeout is 200 ms).
*/
typedef uint8_t (*pci_error_handler_fn)(uint8_t severity,
const uint8_t *bdf,
size_t bdf_len);
extern int pci_register_error_handler(pci_error_handler_fn handler);
#endif
local/recipes/drivers/linux-kpi/source/src/rust_impl/error.rs
+214
View File
@@ -0,0 +1,214 @@
//! Driver-side end of the REDBEAR_DRIVER_ERROR_FD sidecar channel.
//!
//! Spawned driver daemons opt in by calling `pci_register_error_handler`
//! with a C callback. linux-kpi spawns a thread that reads length-
//! prefixed `DriverErrorReport` packets from the fd, calls the
//! registered handler, and writes back the `RecoveryAction` response.
//!
//! Drivers that do not call this function ignore the env var entirely;
//! the parent falls back to its in-process `DriverConfig::on_error`
//! impl after the IPC timeout.
use std::io::{Read, Write};
use std::os::unix::net::UnixStream;
use std::sync::OnceLock;
use std::time::Duration;
const RESPONSE_TIMEOUT: Duration = Duration::from_millis(200);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Severity {
Correctable = 0,
NonFatal = 1,
Fatal = 2,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum RecoveryAction {
Handled = 0,
ResetDevice = 1,
RescanBus = 2,
Fatal = 3,
}
/// Wire payload sent from driver-manager to the spawned driver.
/// Same encoding as driver-manager's `error_channel::DriverErrorReport`
/// (duplicated here to keep linux-kpi self-contained — the protocol
/// is small enough that sharing across the crates would be more
/// friction than value).
#[derive(Debug, Clone, PartialEq, Eq)]
struct DriverErrorReport {
severity: Severity,
bdf: String,
raw: String,
}
impl DriverErrorReport {
fn encode(&self) -> Vec<u8> {
let bdf_bytes = self.bdf.as_bytes();
let raw_bytes = self.raw.as_bytes();
let mut out = Vec::with_capacity(1 + 1 + bdf_bytes.len() + 4 + raw_bytes.len());
out.push(self.severity as u8);
out.push(bdf_bytes.len().min(u8::MAX as usize) as u8);
out.extend_from_slice(bdf_bytes);
out.extend_from_slice(&(raw_bytes.len() as u32).to_le_bytes());
out.extend_from_slice(raw_bytes);
out
}
fn decode(buf: &[u8]) -> Option<Self> {
if buf.len() < 2 {
return None;
}
let severity = match buf[0] {
0 => Severity::Correctable,
1 => Severity::NonFatal,
2 => Severity::Fatal,
_ => return None,
};
let bdf_len = buf[1] as usize;
let bdf_start: usize = 2;
let bdf_end: usize = bdf_start.checked_add(bdf_len)?;
if bdf_end + 4 > buf.len() {
return None;
}
let bdf = std::str::from_utf8(&buf[bdf_start..bdf_end]).ok()?.to_string();
let raw_len = u32::from_le_bytes(buf[bdf_end..bdf_end + 4].try_into().ok()?) as usize;
let raw_start: usize = bdf_end + 4;
let raw_end: usize = raw_start.checked_add(raw_len)?;
if raw_end > buf.len() {
return None;
}
let raw = std::str::from_utf8(&buf[raw_start..raw_end]).ok()?.to_string();
Some(Self { severity, bdf, raw })
}
}
#[derive(Debug, Clone, Copy)]
struct DriverErrorResponse(RecoveryAction);
impl DriverErrorResponse {
fn encode(&self) -> Vec<u8> {
vec![self.0 as u8]
}
}
/// C-callable handler signature. Receives the AER severity discriminant
/// (0 = Correctable, 1 = NonFatal, 2 = Fatal) and returns the action
/// discriminant (0 = Handled, 1 = ResetDevice, 2 = RescanBus, 3 = Fatal).
///
/// The bdf/bdf_len args carry the raw bytes of the BDF (capped at 31
/// bytes) so C handlers can log the device identity without depending
/// on a separate "name your BDF" round-trip.
pub type ErrorHandlerFn =
unsafe extern "C" fn(severity: u8, bdf_ptr: *const u8, bdf_len: usize) -> u8;
/// Single global handler slot. linux-kpi intentionally supports at most
/// one error handler per process — driver daemons have one driver, not
/// many. Drivers that need multiple handler chains should compose them
/// inside their own handler.
static HANDLER: OnceLock<ErrorHandlerFn> = OnceLock::new();
fn worker_loop(fd: i32) {
let mut stream = match unsafe { unix_stream_from_raw_fd(fd) } {
Some(s) => s,
None => return,
};
let handler = match HANDLER.get() {
Some(h) => *h,
None => return,
};
let _ = stream.set_read_timeout(Some(RESPONSE_TIMEOUT));
loop {
let mut len_buf = [0u8; 4];
if stream.read_exact(&mut len_buf).is_err() {
return;
}
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
if stream.read_exact(&mut payload).is_err() {
return;
}
let report = match DriverErrorReport::decode(&payload) {
Some(r) => r,
None => continue,
};
let severity = report.severity as u8;
let bdf_bytes = report.bdf.as_bytes();
let bdf_len = bdf_bytes.len().min(31);
let action_byte = unsafe { (handler)(severity, bdf_bytes.as_ptr(), bdf_len) };
let action = match action_byte {
0 => RecoveryAction::Handled,
1 => RecoveryAction::ResetDevice,
2 => RecoveryAction::RescanBus,
_ => RecoveryAction::Fatal,
};
let response = DriverErrorResponse(action).encode();
let mut framed = Vec::with_capacity(4 + response.len());
framed.extend_from_slice(&(response.len() as u32).to_le_bytes());
framed.extend_from_slice(&response);
if stream.write_all(&framed).is_err() {
return;
}
}
}
/// Unsafe UnixStream construction from a raw fd. The fd is the child
/// end of the sidecar socketpair that the parent passed via the
/// `REDBEAR_DRIVER_ERROR_FD` env var. linux-kpi takes ownership.
unsafe fn unix_stream_from_raw_fd(fd: i32) -> Option<UnixStream> {
use std::os::unix::io::FromRawFd;
let stream = UnixStream::from_raw_fd(fd);
match stream.peer_addr() {
Ok(_) => Some(stream),
Err(_) => None,
}
}
/// Register a C-callable error handler and start the worker thread.
///
/// Returns true on success, false if the env var is unset, the fd is
/// invalid, or a handler is already registered.
#[no_mangle]
pub extern "C" fn pci_register_error_handler(handler: ErrorHandlerFn) -> bool {
if HANDLER.set(handler).is_err() {
return false;
}
let fd_str = match std::env::var("REDBEAR_DRIVER_ERROR_FD") {
Ok(s) => s,
Err(_) => return false,
};
let fd: i32 = match fd_str.parse() {
Ok(n) => n,
Err(_) => return false,
};
std::thread::Builder::new()
.name("driver-error-handler".to_string())
.spawn(move || worker_loop(fd))
.is_ok()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn driver_error_report_round_trip() {
let r = DriverErrorReport {
severity: Severity::NonFatal,
bdf: "0000:00:1f.2".to_string(),
raw: "raw".to_string(),
};
let bytes = r.encode();
let decoded = DriverErrorReport::decode(&bytes).unwrap();
assert_eq!(decoded, r);
}
#[test]
fn driver_error_response_round_trip() {
let r = DriverErrorResponse(RecoveryAction::ResetDevice);
let bytes = r.encode();
assert_eq!(bytes, vec![1]);
}
}
local/recipes/drivers/linux-kpi/source/src/rust_impl/mod.rs
+1
View File
@@ -1,6 +1,7 @@
pub mod device;
pub mod dma;
pub mod drm_shim;
pub mod error;
pub mod firmware;
pub mod idr;
pub mod io;
local/recipes/system/driver-manager/source/src/aer.rs
+63 -6
View File
@@ -67,7 +67,20 @@ pub fn read_aer_lines(path: &Path, last_seen: &mut u64) -> Option<Vec<AerEvent>>
}
}
pub(crate) fn route_to_driver(event: &AerEvent, binds: &[(String, String)]) -> RecoveryAction {
/// Decide the `RecoveryAction` for an AER event.
///
/// `consult_driver` is called with the BDF and the severity when a bound
/// driver is known to exist; if it returns `Some(action)` that wins,
/// otherwise (or if no driver is bound) the severity-based default
/// applies.
pub(crate) fn route_to_driver<F>(
event: &AerEvent,
binds: &[(String, String)],
consult_driver: F,
) -> RecoveryAction
where
F: Fn(&str, ErrorSeverity) -> Option<RecoveryAction>,
{
for (bdf, name) in binds {
if bdf == &event.device {
log::info!(
@@ -76,6 +89,15 @@ pub(crate) fn route_to_driver(event: &AerEvent, binds: &[(String, String)]) -> R
name,
event.severity
);
if let Some(action) = consult_driver(bdf, event.severity) {
log::info!(
"AER: driver {} chose {:?} for severity {:?}",
name,
action,
event.severity
);
return action;
}
return match event.severity {
ErrorSeverity::Correctable => RecoveryAction::Handled,
ErrorSeverity::NonFatal => RecoveryAction::ResetDevice,
@@ -91,6 +113,16 @@ pub(crate) fn route_to_driver(event: &AerEvent, binds: &[(String, String)]) -> R
RecoveryAction::Handled
}
/// Severity-only default mapping. Used by the AER callback and by
/// tests that don't have driver configs wired in.
pub(crate) fn severity_default(severity: ErrorSeverity) -> RecoveryAction {
match severity {
ErrorSeverity::Correctable => RecoveryAction::Handled,
ErrorSeverity::NonFatal => RecoveryAction::ResetDevice,
ErrorSeverity::Fatal => RecoveryAction::RescanBus,
}
}
fn stable_hash(s: &str) -> u64 {
let mut h: u64 = 1469598103934665603;
for b in s.bytes() {
@@ -138,7 +170,7 @@ mod tests {
fn routing_finds_bound_driver() {
let binds = vec![("0000:00:1f.2".to_string(), "ahcid".to_string())];
let event = AerEvent::parse("severity=NonFatal device=0000:00:1f.2").unwrap();
let action = route_to_driver(&event, &binds);
let action = route_to_driver(&event, &binds, |_, s| Some(severity_default(s)));
assert_eq!(action, RecoveryAction::ResetDevice);
}
@@ -146,7 +178,7 @@ mod tests {
fn routing_falls_back_when_unbound() {
let binds: Vec<(String, String)> = vec![];
let event = AerEvent::parse("severity=Fatal device=0000:99:99.9").unwrap();
let action = route_to_driver(&event, &binds);
let action = route_to_driver(&event, &binds, |_, s| Some(severity_default(s)));
assert_eq!(action, RecoveryAction::Handled);
}
@@ -156,23 +188,48 @@ mod tests {
assert_eq!(
route_to_driver(
&AerEvent::parse("severity=Correctable device=d").unwrap(),
&binds
&binds,
|_, s| Some(severity_default(s))
),
RecoveryAction::Handled
);
assert_eq!(
route_to_driver(
&AerEvent::parse("severity=NonFatal device=d").unwrap(),
&binds
&binds,
|_, s| Some(severity_default(s))
),
RecoveryAction::ResetDevice
);
assert_eq!(
route_to_driver(
&AerEvent::parse("severity=Fatal device=d").unwrap(),
&binds
&binds,
|_, s| Some(severity_default(s))
),
RecoveryAction::RescanBus
);
}
#[test]
fn driver_callback_overrides_severity_default() {
let binds = vec![("d".to_string(), "n".to_string())];
let action = route_to_driver(
&AerEvent::parse("severity=Fatal device=d").unwrap(),
&binds,
|_, _| Some(RecoveryAction::Handled),
);
assert_eq!(action, RecoveryAction::Handled);
}
#[test]
fn driver_callback_none_falls_back_to_severity_default() {
let binds = vec![("d".to_string(), "n".to_string())];
let action = route_to_driver(
&AerEvent::parse("severity=Fatal device=d").unwrap(),
&binds,
|_, _| None,
);
assert_eq!(action, RecoveryAction::RescanBus);
}
}
local/recipes/system/driver-manager/source/src/config.rs
+48 -1
View File
@@ -11,7 +11,7 @@ use std::vec::Vec;
use pcid_interface::PciFunctionHandle;
use redox_driver_core::device::DeviceInfo;
use redox_driver_core::driver::{Driver, DriverError, ProbeResult};
use redox_driver_core::driver::{Driver, DriverError, ErrorSeverity, ProbeResult, RecoveryAction};
use redox_driver_core::r#match::DriverMatch;
use redox_driver_core::params::{DriverParams, ParamValue};
@@ -739,6 +739,30 @@ impl Driver for DriverConfig {
cmd.arg(arg);
}
// Sidecar error-report channel: a unix socketpair lets the
// spawned daemon receive AER notifications and return a
// RecoveryAction. The child fd goes in the env var; the
// parent fd is registered for the AER dispatch path to
// consult. Drivers that do not opt in ignore the fd.
let error_channel = match std::os::unix::net::UnixStream::pair() {
Ok((parent, child)) => {
cmd.env("REDBEAR_DRIVER_ERROR_FD", child.as_raw_fd().to_string());
// Forget the child fd so dropping the cmd doesn't
// double-close it (Command::spawn takes ownership).
std::mem::forget(child);
Some((crate::error_channel::ErrorChannel { stream: parent }, device_key.clone()))
}
Err(err) => {
log::warn!(
"driver {} for device {} could not get sidecar error channel: {}",
self.name,
device_key,
err
);
None
}
};
cmd.env("PCID_CLIENT_CHANNEL", channel_fd.as_raw_fd().to_string());
cmd.env("PCID_DEVICE_PATH", &device_path);
@@ -780,6 +804,10 @@ impl Driver for DriverConfig {
pid,
device_key
);
if let Some((channel, key)) = error_channel {
crate::error_channel::global()
.insert(key, std::sync::Arc::new(channel));
}
let mut spawned = self.spawned.lock().unwrap_or_else(|e| e.into_inner());
spawned.insert(device_key.clone(), SpawnedDriver { pid, channel_fd });
if let Ok(mut p2d) = self.pid_to_device.lock() {
@@ -802,6 +830,12 @@ impl Driver for DriverConfig {
.unwrap_or_else(|e| e.into_inner());
spawned.remove(&device_key)
};
crate::error_channel::global().remove(&device_key);
if let Some(ref b) = binding {
if let Ok(mut p2d) = self.pid_to_device.lock() {
p2d.remove(&b.pid);
}
}
if let Some(ref b) = binding {
if let Ok(mut p2d) = self.pid_to_device.lock() {
p2d.remove(&b.pid);
@@ -876,6 +910,19 @@ impl Driver for DriverConfig {
Ok(())
}
fn on_error(
&self,
info: &DeviceInfo,
severity: ErrorSeverity,
) -> Result<RecoveryAction, DriverError> {
let _ = info;
Ok(match severity {
ErrorSeverity::Correctable => RecoveryAction::Handled,
ErrorSeverity::NonFatal => RecoveryAction::ResetDevice,
ErrorSeverity::Fatal => RecoveryAction::RescanBus,
})
}
fn params(&self) -> DriverParams {
let mut p = DriverParams::new();
p.define(
local/recipes/system/driver-manager/source/src/error_channel.rs
+266
View File
@@ -0,0 +1,266 @@
//! Error-report sidecar channel: sidecar unix socketpair from
//! driver-manager to spawned driver daemons.
//!
//! Protocol (length-prefixed bincode, SOCK_SEQPACKET):
//!
//! ```text
//! manager -> driver (request):
//! u32 LE payload_len
//! bytes DriverErrorReport { severity: u8, bdf_len: u8, bdf: [u8; bdf_len], raw_len: u32 LE, raw: [u8; raw_len] }
//!
//! driver -> manager (response):
//! u32 LE payload_len
//! bytes RecoveryAction { action: u8 }
//! ```
//!
//! Spawned drivers opt in by:
//! 1. Reading `REDBEAR_DRIVER_ERROR_FD` from the environment.
//! 2. Spawning a thread that reads length-prefixed requests from the fd
//! and writes length-prefixed responses back.
//! 3. Calling its `Driver::on_error` impl (via `pci_register_error_handler`
//! in linux-kpi) to decide the `RecoveryAction`.
//!
//! Drivers that do not opt in: the parent treats the fd as absent and
//! falls back to in-process `DriverConfig::on_error` then to the
//! severity-based default.
use std::collections::HashMap;
use std::io::{Read, Write};
use std::os::unix::net::UnixStream;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use redox_driver_core::driver::{ErrorSeverity, RecoveryAction};
/// Maximum time the manager will wait for a spawned driver to respond
/// to an error-report request before falling back to the in-process
/// `DriverConfig::on_error`.
pub const RESPONSE_TIMEOUT: Duration = Duration::from_millis(200);
/// Wire payload: a request from driver-manager to a spawned driver
/// daemon. Kept minimal and stable so the C-side parser is small.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DriverErrorReport {
pub severity: ErrorSeverity,
pub bdf: String,
pub raw: String,
}
impl DriverErrorReport {
/// Encode to bytes. Format: `[severity: u8][bdf_len: u8][bdf...][raw_len: u32 LE][raw...]`.
pub fn encode(&self) -> Vec<u8> {
let bdf_bytes = self.bdf.as_bytes();
let raw_bytes = self.raw.as_bytes();
let mut out = Vec::with_capacity(1 + 1 + bdf_bytes.len() + 4 + raw_bytes.len());
out.push(self.severity as u8);
out.push(bdf_bytes.len().min(u8::MAX as usize) as u8);
out.extend_from_slice(bdf_bytes);
out.extend_from_slice(&(raw_bytes.len() as u32).to_le_bytes());
out.extend_from_slice(raw_bytes);
out
}
/// Decode from bytes. Returns None if the buffer is malformed.
pub fn decode(buf: &[u8]) -> Option<Self> {
if buf.len() < 2 {
return None;
}
let severity = match buf[0] {
0 => ErrorSeverity::Correctable,
1 => ErrorSeverity::NonFatal,
2 => ErrorSeverity::Fatal,
_ => return None,
};
let bdf_len = buf[1] as usize;
let bdf_start: usize = 2;
let bdf_end: usize = bdf_start.checked_add(bdf_len)?;
if bdf_end + 4 > buf.len() {
return None;
}
let bdf = std::str::from_utf8(&buf[bdf_start..bdf_end]).ok()?.to_string();
let raw_len = u32::from_le_bytes(buf[bdf_end..bdf_end + 4].try_into().ok()?) as usize;
let raw_start: usize = bdf_end + 4;
let raw_end: usize = raw_start.checked_add(raw_len)?;
if raw_end > buf.len() {
return None;
}
let raw = std::str::from_utf8(&buf[raw_start..raw_end]).ok()?.to_string();
Some(Self { severity, bdf, raw })
}
}
/// Wire payload: a response from a spawned driver daemon back to the
/// manager. Single byte (the action discriminant).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct DriverErrorResponse(pub RecoveryAction);
impl DriverErrorResponse {
pub fn encode(&self) -> Vec<u8> {
vec![self.0 as u8]
}
pub fn decode(buf: &[u8]) -> Option<Self> {
if buf.is_empty() {
return None;
}
let action = match buf[0] {
0 => RecoveryAction::Handled,
1 => RecoveryAction::ResetDevice,
2 => RecoveryAction::RescanBus,
3 => RecoveryAction::Fatal,
_ => return None,
};
Some(Self(action))
}
}
/// Per-device parent end of the sidecar socketpair.
#[derive(Debug)]
pub struct ErrorChannel {
pub stream: UnixStream,
}
impl ErrorChannel {
/// Send a request and wait up to RESPONSE_TIMEOUT for a response.
/// Returns None on timeout, broken pipe, or malformed response.
/// The caller passes `&self` so the closure can hold the
/// `ErrorChannel` behind an `Arc` without needing exclusive
/// access — `UnixStream::try_clone` duplicates the underlying fd
/// for this call.
pub fn request_recovery(&self, report: &DriverErrorReport) -> Option<RecoveryAction> {
let mut stream = self.stream.try_clone().ok()?;
let payload = report.encode();
let mut framed = Vec::with_capacity(4 + payload.len());
framed.extend_from_slice(&(payload.len() as u32).to_le_bytes());
framed.extend_from_slice(&payload);
stream.set_write_timeout(Some(RESPONSE_TIMEOUT)).ok()?;
stream.set_read_timeout(Some(RESPONSE_TIMEOUT)).ok()?;
if stream.write_all(&framed).is_err() {
return None;
}
let mut len_buf = [0u8; 4];
if stream.read_exact(&mut len_buf).is_err() {
return None;
}
let len = u32::from_le_bytes(len_buf) as usize;
let mut payload = vec![0u8; len];
if stream.read_exact(&mut payload).is_err() {
return None;
}
DriverErrorResponse::decode(&payload).map(|r| r.0)
}
}
/// Process-wide registry: maps BDF -> ErrorChannel (parent end of the
/// sidecar socketpair). Inserted by the spawn path when a driver is
/// spawned; removed when the driver is reaped (so the socket is
/// closed and the spawned daemon gets EOF).
#[derive(Default)]
pub struct ErrorChannelRegistry {
channels: Mutex<HashMap<String, Arc<ErrorChannel>>>,
}
impl ErrorChannelRegistry {
pub fn new() -> Self {
Self {
channels: Mutex::new(HashMap::new()),
}
}
pub fn insert(&self, bdf: String, channel: Arc<ErrorChannel>) {
if let Ok(mut g) = self.channels.lock() {
g.insert(bdf, channel);
}
}
pub fn remove(&self, bdf: &str) {
if let Ok(mut g) = self.channels.lock() {
g.remove(bdf);
}
}
pub fn get(&self, bdf: &str) -> Option<Arc<ErrorChannel>> {
self.channels.lock().ok().and_then(|g| g.get(bdf).cloned())
}
}
use std::sync::OnceLock;
static REGISTRY: OnceLock<ErrorChannelRegistry> = OnceLock::new();
/// Global error-channel registry. Initialized lazily on first access.
pub fn global() -> &'static ErrorChannelRegistry {
REGISTRY.get_or_init(ErrorChannelRegistry::default)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn encode_decode_round_trip() {
let report = DriverErrorReport {
severity: ErrorSeverity::NonFatal,
bdf: "0000:00:1f.2".to_string(),
raw: "kind=Correctable device=0000:00:1f.2".to_string(),
};
let bytes = report.encode();
let decoded = DriverErrorReport::decode(&bytes).unwrap();
assert_eq!(decoded, report);
}
#[test]
fn encode_decode_short_bdf() {
let report = DriverErrorReport {
severity: ErrorSeverity::Fatal,
bdf: "d".to_string(),
raw: String::new(),
};
let bytes = report.encode();
let decoded = DriverErrorReport::decode(&bytes).unwrap();
assert_eq!(decoded.severity, ErrorSeverity::Fatal);
assert_eq!(decoded.bdf, "d");
assert_eq!(decoded.raw, "");
}
#[test]
fn encode_decode_response_round_trip() {
for action in [
RecoveryAction::Handled,
RecoveryAction::ResetDevice,
RecoveryAction::RescanBus,
RecoveryAction::Fatal,
] {
let resp = DriverErrorResponse(action);
let bytes = resp.encode();
let decoded = DriverErrorResponse::decode(&bytes).unwrap();
assert_eq!(decoded.0, action);
}
}
#[test]
fn decode_rejects_truncated() {
let report = DriverErrorReport {
severity: ErrorSeverity::Correctable,
bdf: "0000:00:1f.2".to_string(),
raw: "raw".to_string(),
};
let mut bytes = report.encode();
bytes.truncate(bytes.len() - 2);
assert!(DriverErrorReport::decode(&bytes).is_none());
}
#[test]
fn registry_insert_get_remove() {
let (a, _b) = UnixStream::pair().unwrap();
let reg = ErrorChannelRegistry::new();
reg.insert(
"0000:00:1f.2".to_string(),
Arc::new(ErrorChannel { stream: a }),
);
assert!(reg.get("0000:00:1f.2").is_some());
assert!(reg.get("0000:99:99.9").is_none());
reg.remove("0000:00:1f.2");
assert!(reg.get("0000:00:1f.2").is_none());
}
}
local/recipes/system/driver-manager/source/src/main.rs
+97 -32
View File
@@ -1,5 +1,6 @@
mod aer;
mod config;
mod error_channel;
#[cfg(test)]
mod end_to_end_test;
mod heartbeat;
@@ -21,13 +22,14 @@ use std::time::{Duration, Instant};
use std::{env, fs, process};
use redox_driver_core::device::DeviceId;
use redox_driver_core::driver::ProbeResult;
use redox_driver_core::driver::{ErrorSeverity, ProbeResult};
use redox_driver_core::manager::{DeviceManager, ManagerConfig, ProbeEvent};
use redox_driver_pci::PciBus;
use std::fs::OpenOptions;
use std::io::Write;
use config::DriverConfig;
use redox_driver_core::driver::Driver;
use scheme::{DriverManagerScheme, notify_bind};
struct StderrLogger;
@@ -383,37 +385,6 @@ fn main() {
));
let scheme_for_events = Arc::clone(&scheme);
// Both listener callbacks capture the scheme Arc by move: the bound-pairs
// provider closure and the event handler (which calls dispatch_recovery).
// Clone one so each closure owns its own Arc (Arc is not Copy).
let scheme_for_pairs = Arc::clone(&scheme_for_events);
let _events_thread = unified_events::spawn_unified_listener(
std::path::PathBuf::from("/scheme/pci/aer"),
std::path::PathBuf::from("/scheme/pci/pciehp"),
move || scheme_for_pairs.bound_device_pairs(),
move |event| match event {
unified_events::UnifiedEvent::Aer { event, action } => {
if matches!(
action,
redox_driver_core::driver::RecoveryAction::Fatal
) {
log::error!(
"AER-FATAL: device={} driver-already-dead escalation marker (no auto-dispatch; operator intervention required)",
event.device
);
}
if let Some(action_str) =
scheme::DriverManagerScheme::recovery_action_str(*action)
{
#[cfg(target_os = "redox")]
scheme_for_events.dispatch_recovery(&event.device, action_str);
}
}
unified_events::UnifiedEvent::Pciehp(e) => {
log::info!("pciehp: event kind={} device={}", e.kind.label(), e.device);
}
},
);
match manager.lock() {
Ok(mut mgr) => {
@@ -442,6 +413,100 @@ fn main() {
registry::register(weak);
}
// Spawn the unified event listener. The closure passed as
// `consult_driver` tries the per-device sidecar IPC channel first,
// then falls back to the in-process `DriverConfig::on_error` impl
// (which gives the severity-default mapping unless a driver
// overrides), then to None (severity-only). The listener then
// maps the action through `RecoveryAction::Fatal` -> escalation
// log marker (in main callback) before calling dispatch_recovery.
let scheme_for_events_aer = Arc::clone(&scheme);
let _events_thread = unified_events::spawn_unified_listener(
std::path::PathBuf::from("/scheme/pci/aer"),
std::path::PathBuf::from("/scheme/pci/pciehp"),
move || scheme_for_events_aer.bound_device_pairs(),
move |bdf, severity| {
// 1) Sidecar IPC: ask the spawned driver daemon if it
// opted in to error-reporting.
let report = error_channel::DriverErrorReport {
severity,
bdf: bdf.to_string(),
raw: String::new(),
};
if let Some(channel) = error_channel::global().get(bdf) {
if let Some(action) = channel.request_recovery(&report) {
return Some(action);
}
}
// 2) In-process DriverConfig::on_error: drivers may
// override the severity-default via the trait method.
let pairs = scheme_for_events.bound_device_pairs();
let driver_name = match pairs.iter().find(|(addr, _)| addr == bdf) {
Some((_, name)) => name.clone(),
None => return None,
};
let snapshot = registry::snapshot();
for weak in snapshot {
if let Some(cfg) = weak.upgrade() {
if cfg.name() == driver_name {
let info = redox_driver_core::device::DeviceInfo {
id: redox_driver_core::device::DeviceId {
bus: "pci".to_string(),
path: bdf.to_string(),
},
vendor: None,
device: None,
class: None,
subclass: None,
prog_if: None,
revision: None,
subsystem_vendor: None,
subsystem_device: None,
raw_path: format!("/scheme/pci/{bdf}"),
description: None,
};
match cfg.on_error(&info, severity) {
Ok(action) => return Some(action),
Err(err) => {
log::warn!(
"AER: driver {} on_error returned {:?} for device {} severity {:?}",
driver_name,
err,
bdf,
severity
);
return None;
}
}
}
}
}
None
},
move |event| match event {
unified_events::UnifiedEvent::Aer { event, action } => {
if matches!(
action,
redox_driver_core::driver::RecoveryAction::Fatal
) {
log::error!(
"AER-FATAL: device={} driver-already-dead escalation marker (no auto-dispatch; operator intervention required)",
event.device
);
}
if let Some(action_str) =
scheme::DriverManagerScheme::recovery_action_str(*action)
{
#[cfg(target_os = "redox")]
scheme_for_events.dispatch_recovery(&event.device, action_str);
}
}
unified_events::UnifiedEvent::Pciehp(e) => {
log::info!("pciehp: event kind={} device={}", e.kind.label(), e.device);
}
},
);
let _reaper_thread = reaper::spawn_reaper_thread(|pid| {
let registry = registry::snapshot();
for weak in registry {
local/recipes/system/driver-manager/source/src/unified_events.rs
+25 -8
View File
@@ -12,7 +12,7 @@ use std::time::Duration;
use crate::aer::AerEvent;
use crate::pciehp::PciehpEvent;
use redox_driver_core::driver::RecoveryAction;
use redox_driver_core::driver::{ErrorSeverity, RecoveryAction};
/// A unified event: either AER (error) or pciehp (hotplug).
///
@@ -32,25 +32,42 @@ pub enum UnifiedEvent {
/// Spawn the unified event listener thread. Polls both files and
/// routes events to `handle_event`. AER events are routed through
/// `route_to_driver` using the `bind_snapshot` of currently-bound
/// devices; the decided recovery action is logged per event.
pub fn spawn_unified_listener(
/// devices; the `consult_driver` closure is invoked for bound devices
/// so the driver's `Driver::on_error` impl can override the
/// severity-based default.
pub fn spawn_unified_listener<F>(
aer_path: std::path::PathBuf,
pciehp_path: std::path::PathBuf,
bind_snapshot: impl Fn() -> Vec<(String, String)> + Send + 'static,
consult_driver: F,
handle_event: impl Fn(&UnifiedEvent) + Send + 'static,
) -> thread::JoinHandle<()> {
) -> thread::JoinHandle<()>
where
F: Fn(&str, ErrorSeverity) -> Option<RecoveryAction> + Send + 'static,
{
thread::Builder::new()
.name("driver-manager-events".to_string())
.spawn(move || run(aer_path, pciehp_path, bind_snapshot, handle_event))
.spawn(move || {
run(
aer_path,
pciehp_path,
bind_snapshot,
consult_driver,
handle_event,
)
})
.expect("spawn unified event listener")
}
fn run(
fn run<F>(
aer_path: std::path::PathBuf,
pciehp_path: std::path::PathBuf,
bind_snapshot: impl Fn() -> Vec<(String, String)>,
consult_driver: F,
handle_event: impl Fn(&UnifiedEvent),
) {
) where
F: Fn(&str, ErrorSeverity) -> Option<RecoveryAction>,
{
log::info!(
"events: unified listener started (aer={}, pciehp={})",
aer_path.display(),
@@ -63,7 +80,7 @@ fn run(
if let Some(events) = crate::aer::read_aer_lines(&aer_path, &mut last_seen_aer) {
let binds = bind_snapshot();
for event in events {
let action = crate::aer::route_to_driver(&event, &binds);
let action = crate::aer::route_to_driver(&event, &binds, &consult_driver);
log::info!(
"AER: event device={} severity={:?} action={:?}",
event.device,