RedBear-OS/docs/03-WAYLAND-ON-REDOX.md

03 — Wayland on Redox: Concrete Implementation Path

Goal

Get a working Wayland compositor on Redox OS that can run KDE Plasma applications.

For the current build/runtime truth summary of the desktop stack, use local/docs/DESKTOP-STACK-CURRENT-STATUS.md together with local/docs/QT6-PORT-STATUS.md. This file should now be read primarily as implementation history plus deeper Wayland-specific porting notes.

Current State

• config/wayland.toml exists — now uses the Red Bear runtime-service fragments and launches the smallvil path by default via orbital-wayland
• 21 Wayland recipes in recipes/wip/wayland/ — most untested
• libwayland 1.24.0 now rebuilds with a much smaller redox.patch; the P3 POSIX path (signalfd, timerfd, eventfd, open_memstream, MSG_CMSG_CLOEXEC, MSG_NOSIGNAL) is back on the native path, and the remaining patch is down to Redox-specific build quirks
• smallvil (Smithay) is still the recommended first runtime target because it is the smallest compositor path already present in-tree
• cosmic-comp builds but still has no working keyboard input; the remaining issue is runtime/input integration, not simply the absence of a libinput recipe
• libdrm builds with amdgpu and Intel enabled
• Mesa builds with EGL+GBM+GLES2 (software via LLVMpipe; hardware acceleration still requires kernel DMA-BUF)
• evdevd (scheme:evdev) provides Linux-compatible /dev/input/eventX interface
• udev-shim (scheme:udev) provides udev-like device enumeration
• seatd now builds for Redox and is wired into the KDE runtime config, but DRM-lease/runtime validation is still open
• redox-drm (scheme:drm) provides DRM/KMS with AMD + Intel GPU support

---

Status Correction (2026-04-14)

This document is partly historical. The repo already contains substantial P3/P4 work in-tree, but the downstream Wayland stack is still not free of compatibility patches.

What is actually true today:

• relibc now contains in-tree implementations for signalfd, timerfd, eventfd, open_memstream, MSG_CMSG_CLOEXEC, and MSG_NOSIGNAL support paths
• the relibc module wiring plus the consumer-visible signalfd / timerfd / eventfd / open_memstream export path were fixed in this repo pass
• libwayland/redox.patch has been reduced to residual Redox-specific build tweaks (wayland-scanner detection and F_DUPFD_CLOEXEC guard), so the old POSIX bypasses are no longer the main blocker
• evdevd, udev-shim, and redox-drm exist in-tree, but full compositor/runtime validation remains open
• seatd now also builds in-tree for Redox and is started by the KDE runtime config, but has not yet been validated end-to-end with the compositor/DRM path

Read the step-by-step sections below as design history plus implementation notes, not as an exact current-state checklist.

For the current Wayland runtime entrypoint in this repo, use:

• config/redbear-wayland.toml
• local/scripts/test-phase4-wayland-qemu.sh

Numbering note: the "Phase 4" in the script name above refers to the historical P0-P6 hardware-enablement sequence (see AGENTS.md). In the v2.0 desktop plan (local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md), Wayland compositor work falls under Phase 2. The scripts still work under their original names.

That path is the current smallvil-first Wayland validation target.

Current runtime evidence for that target:

• redbear-wayland builds to a bootable image
• the image boots to a real login prompt in QEMU/UEFI
• redbear-phase4-wayland-check runs in-guest and confirms the Wayland launch surface is present
• smallvil reaches xkbcommon initialization and selects the Redox EGL platform in the live guest
• the direct launcher local/scripts/test-phase4-wayland-qemu.sh now boots the built redbear-wayland disk image instead of re-entering the build pipeline

For the first runtime-validation pass, keep the runtime target intentionally small: smallvil as the compositor and no heavier secondary Wayland stack as the primary goal. The current redbear-wayland profile still inherits the broader desktop package set from desktop.toml, but Phase 4 validation in this repo is specifically about the orbital-wayland → smallvil path, not about treating every inherited desktop package as part of the Wayland validation surface.

---

Step 1: Fix relibc POSIX Gaps (1-2 weeks)

What to implement

Historically these were the 7 APIs that libwayland/redox.patch worked around. They now exist in-tree in relibc, and this repo pass restored the full build-side path for signalfd, timerfd, eventfd, open_memstream, and the related message flags. The remaining work is no longer basic POSIX availability, but runtime validation of the full Wayland stack.

1.1 signalfd / signalfd4

Files to create/modify in relibc:

src/header/signal/mod.rs        — add signalfd(), signalfd4()
src/header/signal/src.rs        — add SFD_CLOEXEC, SFD_NONBLOCK constants
src/header/signal/types.rs      — add signalfd_siginfo struct
src/platform/redox/mod.rs       — wire to kernel event scheme or userspace signal handler

Implementation approach:

// src/header/signal/mod.rs
pub fn signalfd(fd: c_int, mask: *const sigset_t, flags: c_int) -> c_int {
    // If fd == -1, create a new "signal FD" using event scheme
    // Register signal mask with the signal handling infrastructure
    // Return FD that becomes readable when signals arrive
    // Map to Redox: use event: scheme + signal userspace handler
}

Approximate effort: ~200 lines of Rust.

1.2 timerfd

Files to create in relibc:

src/header/sys_timerfd/mod.rs   — NEW: timerfd_create(), timerfd_settime(), timerfd_gettime()
src/header/sys_timerfd/types.rs — NEW: itimerspec, TFD_CLOEXEC, TFD_NONBLOCK, TFD_TIMER_ABSTIME
src/platform/redox/mod.rs       — wire to time: scheme

Implementation approach:

// src/header/sys_timerfd/mod.rs
pub fn timerfd_create(clockid: c_int, flags: c_int) -> c_int {
    // Create a timer FD using Redox time: scheme
    // Return FD that becomes readable when timer fires
    // Read returns uint64_t count of expirations
}

pub fn timerfd_settime(fd: c_int, flags: c_int, new: *const itimerspec, old: *mut itimerspec) -> c_int {
    // Arm/disarm timer
    // Use time: scheme for absolute/relative timers
}

Approximate effort: ~300 lines of Rust.

1.3 eventfd

Files to create in relibc:

src/header/sys_eventfd/mod.rs   — NEW: eventfd(), eventfd_read(), eventfd_write()
src/header/sys_eventfd/types.rs — EFD_CLOEXEC, EFD_NONBLOCK, EFD_SEMAPHORE

Implementation approach:

// Simplest of the three — just an atomic counter accessed via read/write
pub fn eventfd(initval: c_uint, flags: c_int) -> c_int {
    // Create a pipe-like FD backed by a shared atomic counter
    // read() blocks until counter > 0, returns counter, resets to 0
    // write() adds to counter
    // Use Redox pipe: scheme internally
}

Approximate effort: ~100 lines of Rust.

1.4 F_DUPFD_CLOEXEC

File to modify in relibc:

src/header/fcntl/mod.rs — add F_DUPFD_CLOEXEC constant (value 0x40 on Linux x86_64)
src/platform/redox/alloc.rs — handle F_DUPFD_CLOEXEC in fcntl()

// In fcntl handler:
pub const F_DUPFD_CLOEXEC: c_int = 0x406; // Linux value

// In fcntl() match:
F_DUPFD_CLOEXEC => {
    let new_fd = syscall::dup(fd, None)?;
    // Set CLOEXEC flag on new_fd
    // Return new_fd
}

Approximate effort: ~20 lines.

1.5 MSG_CMSG_CLOEXEC and MSG_NOSIGNAL

Files to modify in relibc:

src/header/sys_socket/mod.rs — add MSG_CMSG_CLOEXEC (0x40000000), MSG_NOSIGNAL (0x4000)
src/platform/redox/mod.rs    — handle in recvmsg/sendmsg

MSG_NOSIGNAL: suppress SIGPIPE on broken connection. On Redox, SIGPIPE handling is already userspace — just don't send the signal when this flag is set.

MSG_CMSG_CLOEXEC: set CLOEXEC on FDs received via SCM_RIGHTS. Apply the flag when processing ancillary data in recvmsg.

Approximate effort: ~50 lines.

1.6 open_memstream

File to modify in relibc:

src/header/stdio/mod.rs — add open_memstream()
src/header/stdio/src.rs  — implementation

pub fn open_memstream(bufp: *mut *mut c_char, sizep: *mut usize) -> *mut FILE {
    // Create a write-only stream that dynamically grows a buffer
    // On close or flush, update *bufp and *sizep
    // Can be implemented using a backing Vec<u8> and custom FILE vtable
}

Approximate effort: ~200 lines.

Verification

After implementing all 7 APIs:

1. Rebuild relibc: ./target/release/repo cook recipes/core/relibc
2. Rebuild libwayland without redox.patch — it should compile natively
3. Test: wayland-rs_simple_window runs without crashes

---

Step 2: evdev Input Daemon (4-6 weeks)

Architecture

┌──────────────────┐     ┌──────────────────────┐     ┌──────────────┐
│  libinput         │────→│  /dev/input/eventX    │────→│  evdevd      │
│  (ported)         │     │  (character devices)  │     │  (daemon)    │
└──────────────────┘     └──────────────────────┘     └──────┬───────┘
                                                              │
                                                     reads Redox schemes:
                                                     input:, scheme:irq

What to build

New daemon: evdevd (userspace, like all Redox drivers)

Create as a new recipe: recipes/core/evdevd/

Source structure:

evdevd/
├── Cargo.toml
├── src/
│   ├── main.rs          — daemon entry, scheme registration
│   ├── scheme.rs        — implements "evdev" scheme
│   ├── device.rs        — translates Redox events to input_event
│   └── ioctl.rs         — handles EVIOCG* ioctls

Key implementation:

// src/main.rs
fn main() {
    // 1. Open existing Redox input sources
    let keyboard = File::open("scheme:input/keyboard")?;
    let mouse = File::open("scheme:input/mouse")?;
    
    // 2. Create /dev/input symlinks (pointing to our scheme)
    // /dev/input/event0 → /scheme/evdev/keyboard
    // /dev/input/event1 → /scheme/evdev/mouse
    
    // 3. Register evdev scheme
    let scheme = File::create(":evdev")?;
    
    // 4. Event loop: read from Redox input schemes, translate, write to evdev clients
    loop {
        let redox_event = read_redox_event(&keyboard)?;
        let evdev_event = translate_to_input_event(redox_event);
        // Deliver to subscribed clients
    }
}

// src/ioctl.rs — implement evdev ioctls
fn handle_ioctl(fd: usize, request: usize, arg: usize) -> Result<usize> {
    match request {
        EVIOCGNAME => { /* write device name string to arg */ },
        EVIOCGBIT => { /* write supported event types bitmap to arg */ },
        EVIOCGABS => { /* write absinfo struct for absolute axes */ },
        EVIOCGRAB => { /* grab/exclusive access to device */ },
        EVIOCGPROP => { /* write device properties bitmap */ },
        _ => Err(syscall::Error::new(syscall::EINVAL)),
    }
}

Also needed: udev shim

Historical path note: the recipes/wip/... paths in the example steps below document the original upstream-oriented implementation path. Under the current Red Bear policy, upstream WIP is an input/reference, but the shipping/fixed version may live under local/recipes/ instead.

Create recipes/wip/wayland/udev-shim/ — a minimal udev implementation that:

• Enumerates /dev/input/event* devices
• Emits "add"/"remove" events via netlink-compatible socket
• Provides udev_device_get_property_value() for ID_INPUT_* properties

libinput needs this for hotplug. A minimal shim is ~500 lines of Rust.

Then port libinput:

Modify recipes/wip/wayland/libinput/ (currently missing — create it):

[source]
tar = "https://gitlab.freedesktop.org/wayland/libinput/-/archive/1.27.0/libinput-1.27.0.tar.gz"
patches = ["redox.patch"]

[build]
template = "meson"
dependencies = [
    "evdevd",
    "libffi",
    "libwayland",
    "udev-shim",
    "mtdev",        # touchpad multi-touch
    "libevdev",     # evdev wrapper library
]
mesonflags = [
    "-Ddocumentation=false",
    "-Dtests=false",
    "-Ddebug-gui=false",
]

Verification

1. Build and run evdevd
2. cat /dev/input/event0 shows keyboard events
3. Build libinput against evdevd
4. libinput list-devices shows keyboard and mouse

---

Step 3: DRM/KMS Scheme (8-12 weeks)

Architecture

┌──────────────┐    ┌───────────────────┐    ┌────────────────┐
│  libdrm       │───→│  scheme:drm/card0  │───→│  drmd (daemon) │
│  (ported)     │    │  DRM ioctls via    │    │  GPU driver    │
│               │    │  scheme protocol   │    │  userspace     │
└──────────────┘    └───────────────────┘    └───────┬────────┘
                                                      │
                                            scheme:memory + scheme:irq
                                                      │
                                                  Hardware (GPU)

What to build

New daemon: drmd (DRM daemon — starts with Intel support)

Create as: recipes/core/drmd/

Source structure:

drmd/
├── Cargo.toml
├── src/
│   ├── main.rs              — daemon entry, PCI enumeration
│   ├── scheme.rs            — registers "drm" scheme
│   ├── kms/
│   │   ├── mod.rs           — KMS object management
│   │   ├── crtc.rs          — CRTC implementation
│   │   ├── connector.rs     — connector (HDMI, DP, eDP)
│   │   ├── encoder.rs       — encoder
│   │   ├── plane.rs         — primary + cursor planes
│   │   └── framebuffer.rs   — framebuffer allocation
│   ├── gem/
│   │   └── mod.rs           — GEM buffer management
│   └── drivers/
│       ├── mod.rs           — driver trait
│       └── intel.rs         — Intel GPU driver (modesetting)

Core DRM scheme protocol:

// src/scheme.rs
// DRM scheme implements the same ioctls as Linux /dev/dri/card0
// but via Redox scheme read/write/packet protocol

enum DrmRequest {
    // Core
    GetVersion,
    GetCap { capability: u64 },
    
    // KMS
    ModeGetResources,
    ModeGetConnector { connector_id: u32 },
    ModeGetEncoder { encoder_id: u32 },
    ModeGetCrtc { crtc_id: u32 },
    ModeSetCrtc { crtc_id: u32, fb_id: u32, x: u32, y: u32, connectors: Vec<u32>, mode: ModeModeInfo },
    ModePageFlip { crtc_id: u32, fb_id: u32, flags: u32, user_data: u64 },
    ModeAtomicCommit { flags: u32, props: Vec<AtomicProp> },
    
    // GEM
    GemCreate { size: u64 },
    GemClose { handle: u32 },
    GemMmap { handle: u32 },
    
    // Prime/DMA-BUF
    PrimeHandleToFd { handle: u32, flags: u32 },
    PrimeFdToHandle { fd: i32 },
}

Intel driver (starting point):

// src/drivers/intel.rs
// Based on public Intel GPU documentation:
// https://01.org/linuxgraphics/documentation/hardware-specification-prm

pub struct IntelDriver {
    mmio: *mut u8,              // Memory-mapped I/O registers (via scheme:memory)
    gtt_size: usize,            // Graphics Translation Table size
    framebuffer: PhysAddr,      // Current scanout buffer
}

impl IntelDriver {
    pub fn new(pci_dev: &PciDev) -> Result<Self> {
        // Map MMIO registers via scheme:memory/physical
        let mmio = map_physical_memory(pci_dev.bar[0], pci_dev.bar_size[0])?;
        
        // Initialize GTT (Graphics Translation Table)
        // Set up display pipeline
        
        Ok(Self { mmio, gtt_size, framebuffer })
    }
    
    pub fn modeset(&self, mode: &ModeInfo) -> Result<()> {
        // 1. Allocate framebuffer in GTT
        // 2. Configure pipe (timing, PLL)
        // 3. Configure transcoder
        // 4. Configure port (HDMI/DP)
        // 5. Enable scanout from new framebuffer
        Ok(())
    }
    
    pub fn page_flip(&self, crtc: u32, fb: PhysAddr) -> Result<()> {
        // 1. Update GTT entry to point to new framebuffer
        // 2. Trigger page flip on next VBlank
        // 3. VBlank interrupt signals completion (via scheme:irq)
        Ok(())
    }
}

Verification

1. drmd registers scheme:drm/card0
2. Port modetest (from libdrm tests) — shows connector info and modes
3. modetest -M intel -s 0:1920x1080 sets a mode and shows test pattern

---

Step 4: Working Wayland Compositor (4-6 weeks after Steps 1-3)

Recommended: Smithay/smallvil first, then KWin

Why Smithay first:

• Pure Rust — no C++ toolchain issues
• Already has a Redox branch (https://github.com/jackpot51/smithay, branch redox)
• Smithay's input backend is pluggable — write a Redox-specific one
• Gets us a working compositor months before KWin is ported

What to modify in Smithay:

smithay/
├── src/backend/
│   ├── input/
│   │   └── redox.rs          — NEW: Redox input backend (reads evdev scheme)
│   ├── drm/
│   │   └── redox.rs          — NEW: Redox DRM backend (uses scheme:drm)
│   └── egl/
│       └── redox.rs          — NEW: Redox EGL display (uses Mesa)

Redox input backend:

// src/backend/input/redox.rs
pub struct RedoxInputBackend {
    devices: Vec<EvdevDevice>,  // opened from /dev/input/eventX
}

impl InputBackend for RedoxInputBackend {
    fn dispatch(&mut self) -> Vec<InputEvent> {
        // Read from all evdev devices via evdevd
        // Translate to Smithay's InternalEvent type
    }
}

Redox DRM backend:

// src/backend/drm/redox.rs
pub struct RedoxDrmBackend {
    drm_fd: File,  // opened from /scheme/drm/card0
}

impl DrmBackend for RedoxDrmBackend {
    fn create_surface(&self, size: Size) -> Surface {
        // Create framebuffer via DRM GEM
        // Set KMS mode via scheme:drm
    }
    
    fn page_flip(&self, surface: &Surface) -> Result<VBlank> {
        // DRM page flip via scheme
    }
}

Recipe to add/modify

# recipes/wip/wayland/smallvil/recipe.toml (modify existing)
[source]
git = "https://github.com/jackpot51/smithay"
branch = "redox"

[build]
template = "cargo"
dependencies = [
    "libffi",
    "libwayland",
    "libxkbcommon",
    "mesa",        # for EGL
    "libdrm",      # for DRM backend
    "evdevd",      # for input
    "seatd",       # for session management
]
cargopackages = ["smallvil"]

Verification

1. smallvil launches with DRM backend — takes over display
2. Keyboard and mouse work via evdevd
3. libcosmic-wayland_application renders a window on the compositor
4. Screenshot shows the window

---

Step 5: Enable cosmic-comp and Other Compositors

Once Steps 1-4 are done:

1. cosmic-comp: Uncomment libinput dependency in recipe, rebuild
2. wlroots: Build with libdrm + libinput + GBM
3. sway: Should work once wlroots builds
4. KWin: See local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md for the canonical desktop path; 05-KDE-PLASMA-ON-REDOX.md for historical rationale

---

Fastest Path Summary

Week 1-2:   Implement signalfd/timerfd/eventfd/etc in relibc
            → libwayland builds without patches

Week 3-8:   Build evdevd (input daemon) + udev shim
            → libinput works

Week 9-20:  Build drmd (DRM daemon) with Intel modesetting
            → libdrm works, modesetting functional

Week 21-26: Smithay Redox backends (input + DRM + EGL)
            → Working Wayland compositor with hardware display

Week 27+:   Port Qt, KDE Frameworks, Plasma Shell
            → KDE Plasma desktop (see doc 05)

Key insight: Steps 2 (evdev) and 3 (DRM) can run in parallel. With 2 developers, the Wayland compositor is achievable in ~6 months.