RedBear-OS/docs/03-WAYLAND-ON-REDOX.md

# 03 — Wayland on Redox: Concrete Implementation Path

## Goal

Get a working Wayland compositor on Redox OS that can run KDE Plasma applications.

For the current build/runtime truth summary of the desktop stack, use
`local/docs/DESKTOP-STACK-CURRENT-STATUS.md` together with `local/docs/QT6-PORT-STATUS.md`.
This file should now be read primarily as implementation history plus deeper Wayland-specific
porting notes.

## Current State

- `config/wayland.toml` exists — now uses the Red Bear runtime-service fragments and launches the
  `smallvil` path by default via `orbital-wayland`
- 21 Wayland recipes in `recipes/wip/wayland/` — most untested
- `libwayland` 1.24.0 now rebuilds with a much smaller `redox.patch`; the P3 POSIX path (`signalfd`, `timerfd`, `eventfd`, `open_memstream`, `MSG_CMSG_CLOEXEC`, `MSG_NOSIGNAL`) is back on the native path, and the remaining patch is down to Redox-specific build quirks
- `smallvil` (Smithay) is still the recommended first runtime target because it is the smallest
  compositor path already present in-tree
- `cosmic-comp` builds but still has no working keyboard input; the remaining issue is runtime/input integration, not simply the absence of a libinput recipe
- `libdrm` builds with amdgpu and Intel enabled
- Mesa builds with EGL+GBM+GLES2 (software via LLVMpipe; hardware acceleration still requires kernel DMA-BUF)
- **evdevd** (`scheme:evdev`) provides Linux-compatible `/dev/input/eventX` interface
- **udev-shim** (`scheme:udev`) provides udev-like device enumeration
- **seatd** now builds for Redox and is wired into the KDE runtime config, but DRM-lease/runtime validation is still open
- **redox-drm** (`scheme:drm`) provides DRM/KMS with AMD + Intel GPU support

---

## Status Correction (2026-04-14)

This document is partly historical. The repo already contains substantial P3/P4 work in-tree,
but the downstream Wayland stack is still not free of compatibility patches.

What is actually true today:
- relibc now contains in-tree implementations for `signalfd`, `timerfd`, `eventfd`, `open_memstream`, `MSG_CMSG_CLOEXEC`, and `MSG_NOSIGNAL` support paths
- the relibc module wiring plus the consumer-visible `signalfd` / `timerfd` / `eventfd` / `open_memstream` export path were fixed in this repo pass
- `libwayland/redox.patch` has been reduced to residual Redox-specific build tweaks (`wayland-scanner` detection and `F_DUPFD_CLOEXEC` guard), so the old POSIX bypasses are no longer the main blocker
- `evdevd`, `udev-shim`, and `redox-drm` exist in-tree, but full compositor/runtime validation remains open
- `seatd` now also builds in-tree for Redox and is started by the KDE runtime config, but has not yet been validated end-to-end with the compositor/DRM path

Read the step-by-step sections below as design history plus implementation notes, not as an exact
current-state checklist.

For the current Wayland runtime entrypoint in this repo, use:

- `config/redbear-wayland.toml`
- `local/scripts/test-phase4-wayland-qemu.sh`

> **Numbering note:** the "Phase 4" in the script name above refers to the historical P0-P6
> hardware-enablement sequence (see `AGENTS.md`). In the v2.0 desktop plan
> (`local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md`), Wayland compositor work falls under Phase 2.
> The scripts still work under their original names.

That path is the current smallvil-first Wayland validation target.

Current runtime evidence for that target:

- `redbear-wayland` builds to a bootable image
- the image boots to a real login prompt in QEMU/UEFI
- `redbear-phase4-wayland-check` runs in-guest and confirms the Wayland launch surface is present
- `smallvil` reaches xkbcommon initialization and selects the Redox EGL platform in the live guest
- the direct launcher `local/scripts/test-phase4-wayland-qemu.sh` now boots the built
  `redbear-wayland` disk image instead of re-entering the build pipeline

For the first runtime-validation pass, keep the **runtime target** intentionally small: `smallvil`
 as the compositor and no heavier secondary Wayland stack as the primary goal. The current
 `redbear-wayland` profile still inherits the broader desktop package set from `desktop.toml`, but
 Phase 4 validation in this repo is specifically about the `orbital-wayland` → `smallvil` path,
 not about treating every inherited desktop package as part of the Wayland validation surface.

---

## Step 1: Fix relibc POSIX Gaps (1-2 weeks)

### What to implement

Historically these were the 7 APIs that `libwayland/redox.patch` worked around. They now exist
in-tree in relibc, and this repo pass restored the full build-side path for `signalfd`, `timerfd`,
`eventfd`, `open_memstream`, and the related message flags. The remaining work is no longer basic
POSIX availability, but runtime validation of the full Wayland stack.

#### 1.1 `signalfd` / `signalfd4`

**Files to create/modify in relibc:**
```
src/header/signal/mod.rs        — add signalfd(), signalfd4()
src/header/signal/src.rs        — add SFD_CLOEXEC, SFD_NONBLOCK constants
src/header/signal/types.rs      — add signalfd_siginfo struct
src/platform/redox/mod.rs       — wire to kernel event scheme or userspace signal handler
```

**Implementation approach:**
```rust
// src/header/signal/mod.rs
pub fn signalfd(fd: c_int, mask: *const sigset_t, flags: c_int) -> c_int {
    // If fd == -1, create a new "signal FD" using event scheme
    // Register signal mask with the signal handling infrastructure
    // Return FD that becomes readable when signals arrive
    // Map to Redox: use event: scheme + signal userspace handler
}
```

**Approximate effort**: ~200 lines of Rust.

#### 1.2 `timerfd`

**Files to create in relibc:**
```
src/header/sys_timerfd/mod.rs   — NEW: timerfd_create(), timerfd_settime(), timerfd_gettime()
src/header/sys_timerfd/types.rs — NEW: itimerspec, TFD_CLOEXEC, TFD_NONBLOCK, TFD_TIMER_ABSTIME
src/platform/redox/mod.rs       — wire to time: scheme
```

**Implementation approach:**
```rust
// src/header/sys_timerfd/mod.rs
pub fn timerfd_create(clockid: c_int, flags: c_int) -> c_int {
    // Create a timer FD using Redox time: scheme
    // Return FD that becomes readable when timer fires
    // Read returns uint64_t count of expirations
}

pub fn timerfd_settime(fd: c_int, flags: c_int, new: *const itimerspec, old: *mut itimerspec) -> c_int {
    // Arm/disarm timer
    // Use time: scheme for absolute/relative timers
}
```

**Approximate effort**: ~300 lines of Rust.

#### 1.3 `eventfd`

**Files to create in relibc:**
```
src/header/sys_eventfd/mod.rs   — NEW: eventfd(), eventfd_read(), eventfd_write()
src/header/sys_eventfd/types.rs — EFD_CLOEXEC, EFD_NONBLOCK, EFD_SEMAPHORE
```

**Implementation approach:**
```rust
// Simplest of the three — just an atomic counter accessed via read/write
pub fn eventfd(initval: c_uint, flags: c_int) -> c_int {
    // Create a pipe-like FD backed by a shared atomic counter
    // read() blocks until counter > 0, returns counter, resets to 0
    // write() adds to counter
    // Use Redox pipe: scheme internally
}
```

**Approximate effort**: ~100 lines of Rust.

#### 1.4 `F_DUPFD_CLOEXEC`

**File to modify in relibc:**
```
src/header/fcntl/mod.rs — add F_DUPFD_CLOEXEC constant (value 0x40 on Linux x86_64)
src/platform/redox/alloc.rs — handle F_DUPFD_CLOEXEC in fcntl()
```

```rust
// In fcntl handler:
pub const F_DUPFD_CLOEXEC: c_int = 0x406; // Linux value

// In fcntl() match:
F_DUPFD_CLOEXEC => {
    let new_fd = syscall::dup(fd, None)?;
    // Set CLOEXEC flag on new_fd
    // Return new_fd
}
```

**Approximate effort**: ~20 lines.

#### 1.5 `MSG_CMSG_CLOEXEC` and `MSG_NOSIGNAL`

**Files to modify in relibc:**
```
src/header/sys_socket/mod.rs — add MSG_CMSG_CLOEXEC (0x40000000), MSG_NOSIGNAL (0x4000)
src/platform/redox/mod.rs    — handle in recvmsg/sendmsg
```

`MSG_NOSIGNAL`: suppress SIGPIPE on broken connection. On Redox, SIGPIPE handling
is already userspace — just don't send the signal when this flag is set.

`MSG_CMSG_CLOEXEC`: set CLOEXEC on FDs received via SCM_RIGHTS. Apply the flag
when processing ancillary data in recvmsg.

**Approximate effort**: ~50 lines.

#### 1.6 `open_memstream`

**File to modify in relibc:**
```
src/header/stdio/mod.rs — add open_memstream()
src/header/stdio/src.rs  — implementation
```

```rust
pub fn open_memstream(bufp: *mut *mut c_char, sizep: *mut usize) -> *mut FILE {
    // Create a write-only stream that dynamically grows a buffer
    // On close or flush, update *bufp and *sizep
    // Can be implemented using a backing Vec<u8> and custom FILE vtable
}
```

**Approximate effort**: ~200 lines.

### Verification

After implementing all 7 APIs:
1. Rebuild relibc: `./target/release/repo cook recipes/core/relibc`
2. Rebuild libwayland **without** `redox.patch` — it should compile natively
3. Test: `wayland-rs_simple_window` runs without crashes

---

## Step 2: evdev Input Daemon (4-6 weeks)

### Architecture

```
┌──────────────────┐     ┌──────────────────────┐     ┌──────────────┐
│  libinput         │────→│  /dev/input/eventX    │────→│  evdevd      │
│  (ported)         │     │  (character devices)  │     │  (daemon)    │
└──────────────────┘     └──────────────────────┘     └──────┬───────┘
                                                              │
                                                     reads Redox schemes:
                                                     input:, scheme:irq
```

### What to build

**New daemon: `evdevd`** (userspace, like all Redox drivers)

Create as a new recipe: `recipes/core/evdevd/`

**Source structure:**
```
evdevd/
├── Cargo.toml
├── src/
│   ├── main.rs          — daemon entry, scheme registration
│   ├── scheme.rs        — implements "evdev" scheme
│   ├── device.rs        — translates Redox events to input_event
│   └── ioctl.rs         — handles EVIOCG* ioctls
```

**Key implementation:**

```rust
// src/main.rs
fn main() {
    // 1. Open existing Redox input sources
    let keyboard = File::open("scheme:input/keyboard")?;
    let mouse = File::open("scheme:input/mouse")?;
    
    // 2. Create /dev/input symlinks (pointing to our scheme)
    // /dev/input/event0 → /scheme/evdev/keyboard
    // /dev/input/event1 → /scheme/evdev/mouse
    
    // 3. Register evdev scheme
    let scheme = File::create(":evdev")?;
    
    // 4. Event loop: read from Redox input schemes, translate, write to evdev clients
    loop {
        let redox_event = read_redox_event(&keyboard)?;
        let evdev_event = translate_to_input_event(redox_event);
        // Deliver to subscribed clients
    }
}
```

```rust
// src/ioctl.rs — implement evdev ioctls
fn handle_ioctl(fd: usize, request: usize, arg: usize) -> Result<usize> {
    match request {
        EVIOCGNAME => { /* write device name string to arg */ },
        EVIOCGBIT => { /* write supported event types bitmap to arg */ },
        EVIOCGABS => { /* write absinfo struct for absolute axes */ },
        EVIOCGRAB => { /* grab/exclusive access to device */ },
        EVIOCGPROP => { /* write device properties bitmap */ },
        _ => Err(syscall::Error::new(syscall::EINVAL)),
    }
}
```

**Also needed: udev shim**

> **Historical path note:** the `recipes/wip/...` paths in the example steps below document the
> original upstream-oriented implementation path. Under the current Red Bear policy, upstream WIP is
> an input/reference, but the shipping/fixed version may live under `local/recipes/` instead.

Create `recipes/wip/wayland/udev-shim/` — a minimal udev implementation that:
- Enumerates `/dev/input/event*` devices
- Emits "add"/"remove" events via netlink-compatible socket
- Provides `udev_device_get_property_value()` for `ID_INPUT_*` properties

libinput needs this for hotplug. A minimal shim is ~500 lines of Rust.

**Then port libinput:**

Modify `recipes/wip/wayland/libinput/` (currently missing — create it):
```toml
[source]
tar = "https://gitlab.freedesktop.org/wayland/libinput/-/archive/1.27.0/libinput-1.27.0.tar.gz"
patches = ["redox.patch"]

[build]
template = "meson"
dependencies = [
    "evdevd",
    "libffi",
    "libwayland",
    "udev-shim",
    "mtdev",        # touchpad multi-touch
    "libevdev",     # evdev wrapper library
]
mesonflags = [
    "-Ddocumentation=false",
    "-Dtests=false",
    "-Ddebug-gui=false",
]
```

### Verification

1. Build and run `evdevd`
2. `cat /dev/input/event0` shows keyboard events
3. Build libinput against evdevd
4. `libinput list-devices` shows keyboard and mouse

---

## Step 3: DRM/KMS Scheme (8-12 weeks)

### Architecture

```
┌──────────────┐    ┌───────────────────┐    ┌────────────────┐
│  libdrm       │───→│  scheme:drm/card0  │───→│  drmd (daemon) │
│  (ported)     │    │  DRM ioctls via    │    │  GPU driver    │
│               │    │  scheme protocol   │    │  userspace     │
└──────────────┘    └───────────────────┘    └───────┬────────┘
                                                      │
                                            scheme:memory + scheme:irq
                                                      │
                                                  Hardware (GPU)
```

### What to build

**New daemon: `drmd`** (DRM daemon — starts with Intel support)

Create as: `recipes/core/drmd/`

**Source structure:**
```
drmd/
├── Cargo.toml
├── src/
│   ├── main.rs              — daemon entry, PCI enumeration
│   ├── scheme.rs            — registers "drm" scheme
│   ├── kms/
│   │   ├── mod.rs           — KMS object management
│   │   ├── crtc.rs          — CRTC implementation
│   │   ├── connector.rs     — connector (HDMI, DP, eDP)
│   │   ├── encoder.rs       — encoder
│   │   ├── plane.rs         — primary + cursor planes
│   │   └── framebuffer.rs   — framebuffer allocation
│   ├── gem/
│   │   └── mod.rs           — GEM buffer management
│   └── drivers/
│       ├── mod.rs           — driver trait
│       └── intel.rs         — Intel GPU driver (modesetting)
```

**Core DRM scheme protocol:**

```rust
// src/scheme.rs
// DRM scheme implements the same ioctls as Linux /dev/dri/card0
// but via Redox scheme read/write/packet protocol

enum DrmRequest {
    // Core
    GetVersion,
    GetCap { capability: u64 },
    
    // KMS
    ModeGetResources,
    ModeGetConnector { connector_id: u32 },
    ModeGetEncoder { encoder_id: u32 },
    ModeGetCrtc { crtc_id: u32 },
    ModeSetCrtc { crtc_id: u32, fb_id: u32, x: u32, y: u32, connectors: Vec<u32>, mode: ModeModeInfo },
    ModePageFlip { crtc_id: u32, fb_id: u32, flags: u32, user_data: u64 },
    ModeAtomicCommit { flags: u32, props: Vec<AtomicProp> },
    
    // GEM
    GemCreate { size: u64 },
    GemClose { handle: u32 },
    GemMmap { handle: u32 },
    
    // Prime/DMA-BUF
    PrimeHandleToFd { handle: u32, flags: u32 },
    PrimeFdToHandle { fd: i32 },
}
```

**Intel driver (starting point):**

```rust
// src/drivers/intel.rs
// Based on public Intel GPU documentation:
// https://01.org/linuxgraphics/documentation/hardware-specification-prm

pub struct IntelDriver {
    mmio: *mut u8,              // Memory-mapped I/O registers (via scheme:memory)
    gtt_size: usize,            // Graphics Translation Table size
    framebuffer: PhysAddr,      // Current scanout buffer
}

impl IntelDriver {
    pub fn new(pci_dev: &PciDev) -> Result<Self> {
        // Map MMIO registers via scheme:memory/physical
        let mmio = map_physical_memory(pci_dev.bar[0], pci_dev.bar_size[0])?;
        
        // Initialize GTT (Graphics Translation Table)
        // Set up display pipeline
        
        Ok(Self { mmio, gtt_size, framebuffer })
    }
    
    pub fn modeset(&self, mode: &ModeInfo) -> Result<()> {
        // 1. Allocate framebuffer in GTT
        // 2. Configure pipe (timing, PLL)
        // 3. Configure transcoder
        // 4. Configure port (HDMI/DP)
        // 5. Enable scanout from new framebuffer
        Ok(())
    }
    
    pub fn page_flip(&self, crtc: u32, fb: PhysAddr) -> Result<()> {
        // 1. Update GTT entry to point to new framebuffer
        // 2. Trigger page flip on next VBlank
        // 3. VBlank interrupt signals completion (via scheme:irq)
        Ok(())
    }
}
```

### Verification

1. `drmd` registers `scheme:drm/card0`
2. Port `modetest` (from libdrm tests) — shows connector info and modes
3. `modetest -M intel -s 0:1920x1080` sets a mode and shows test pattern

---

## Step 4: Working Wayland Compositor (4-6 weeks after Steps 1-3)

### Recommended: Smithay/smallvil first, then KWin

**Why Smithay first:**
- Pure Rust — no C++ toolchain issues
- Already has a Redox branch (`https://github.com/jackpot51/smithay`, branch `redox`)
- Smithay's input backend is pluggable — write a Redox-specific one
- Gets us a working compositor months before KWin is ported

**What to modify in Smithay:**

```
smithay/
├── src/backend/
│   ├── input/
│   │   └── redox.rs          — NEW: Redox input backend (reads evdev scheme)
│   ├── drm/
│   │   └── redox.rs          — NEW: Redox DRM backend (uses scheme:drm)
│   └── egl/
│       └── redox.rs          — NEW: Redox EGL display (uses Mesa)
```

**Redox input backend:**
```rust
// src/backend/input/redox.rs
pub struct RedoxInputBackend {
    devices: Vec<EvdevDevice>,  // opened from /dev/input/eventX
}

impl InputBackend for RedoxInputBackend {
    fn dispatch(&mut self) -> Vec<InputEvent> {
        // Read from all evdev devices via evdevd
        // Translate to Smithay's InternalEvent type
    }
}
```

**Redox DRM backend:**
```rust
// src/backend/drm/redox.rs
pub struct RedoxDrmBackend {
    drm_fd: File,  // opened from /scheme/drm/card0
}

impl DrmBackend for RedoxDrmBackend {
    fn create_surface(&self, size: Size) -> Surface {
        // Create framebuffer via DRM GEM
        // Set KMS mode via scheme:drm
    }
    
    fn page_flip(&self, surface: &Surface) -> Result<VBlank> {
        // DRM page flip via scheme
    }
}
```

### Recipe to add/modify

```toml
# recipes/wip/wayland/smallvil/recipe.toml (modify existing)
[source]
git = "https://github.com/jackpot51/smithay"
branch = "redox"

[build]
template = "cargo"
dependencies = [
    "libffi",
    "libwayland",
    "libxkbcommon",
    "mesa",        # for EGL
    "libdrm",      # for DRM backend
    "evdevd",      # for input
    "seatd",       # for session management
]
cargopackages = ["smallvil"]
```

### Verification

1. `smallvil` launches with DRM backend — takes over display
2. Keyboard and mouse work via evdevd
3. `libcosmic-wayland_application` renders a window on the compositor
4. Screenshot shows the window

---

## Step 5: Enable cosmic-comp and Other Compositors

Once Steps 1-4 are done:

1. **cosmic-comp**: Uncomment libinput dependency in recipe, rebuild
2. **wlroots**: Build with libdrm + libinput + GBM
3. **sway**: Should work once wlroots builds
4. **KWin**: See `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` for the canonical desktop path; `05-KDE-PLASMA-ON-REDOX.md` for historical rationale

---

## Fastest Path Summary

```
Week 1-2:   Implement signalfd/timerfd/eventfd/etc in relibc
            → libwayland builds without patches

Week 3-8:   Build evdevd (input daemon) + udev shim
            → libinput works

Week 9-20:  Build drmd (DRM daemon) with Intel modesetting
            → libdrm works, modesetting functional

Week 21-26: Smithay Redox backends (input + DRM + EGL)
            → Working Wayland compositor with hardware display

Week 27+:   Port Qt, KDE Frameworks, Plasma Shell
            → KDE Plasma desktop (see doc 05)
```

**Key insight**: Steps 2 (evdev) and 3 (DRM) can run in parallel.
With 2 developers, the Wayland compositor is achievable in ~6 months.