# Red Bear OS: DMA-BUF Improvement Plan

**Date**: 2026-04-16
**Status**: historical DMA-BUF bring-up plan, partially superseded by the current DRM shared-core work. PRIME import/export and bounded private-CS groundwork are now implemented; real vendor render/fence completion is still blocked.

> **Planning authority note (2026-04-18):** this file remains the detailed DMA-BUF/PRIME reference,
> not the canonical GPU/DRM execution plan. Use `local/docs/DRM-MODERNIZATION-EXECUTION-PLAN.md`
> for current sequencing and acceptance criteria.
**Scope**: Cross-process GPU buffer sharing for hardware-accelerated KDE Plasma on Wayland

## Bottom Line

Redox kernel already has the three primitives needed for DMA-BUF-style cross-process buffer
sharing:

1. **`Provider::FmapBorrowed`** + **`Grant::borrow_fmap()`** — kernel mechanism for borrowing
   pages from a scheme into another process's address space, mapping the same physical frames
   (zero-copy). Source: `kernel/source/src/context/memory.rs:1157`, `memory.rs:1401`.

2. **`sendfd`** syscall — passes file descriptors between processes via scheme IPC. Both
   processes hold the same `Arc<LockedFileDescription>`. Source: `kernel/source/src/syscall/fs.rs:415`.

3. **`PhysBorrow`** in `scheme:memory` — maps physical addresses directly into process space
   (already used for GPU registers/BARs). Source: `kernel/source/src/scheme/memory.rs`.

No new kernel syscalls or scheme types are needed for v1. The work is entirely in userspace:
redox-drm scheme daemon, libdrm, and Mesa.

## Architecture Principle

**DMA-BUF is a sharing and lifetime contract, not a global allocator.**

Linux `dma_buf` is an exporter/importer contract. The exporter owns allocation and controls
lifetime. The importer gets shared access. Red Bear OS follows the same model:

- **Allocation stays with `redox-drm`** (the exporter). `DmaBuffer::allocate()` in `gem.rs`
  already allocates physically-contiguous system RAM.
- **Sharing uses scheme-backed fds + `sendfd`**. No synthetic fd numbers. No global registry.
- **Mapping uses `FmapBorrowed`**. The kernel maps the same physical pages into the importer's
  address space — zero-copy.

## Data Flow

```
Process A (GPU client, e.g. Mesa/radeonsi)
  1. open("/scheme/drm/card0")
  2. DRM_IOCTL_GEM_CREATE → allocate GPU buffer             ← EXISTS
  3. DRM_IOCTL_PRIME_HANDLE_TO_FD → get opaque export token     ← IMPLEMENTED
  4. open("/scheme/drm/card0/dmabuf/{token}") → get scheme fd   ← IMPLEMENTED
  5. sendfd(socket, fd) → pass fd to compositor                 ← KERNEL EXISTS

Process B (compositor, e.g. KWin)
  6. recvfd(socket) → receive the fd                            ← KERNEL EXISTS
  7. DRM_IOCTL_PRIME_FD_TO_HANDLE → import as local GEM         ← IMPLEMENTED
  7. mmap(fd, size) → kernel uses FmapBorrowed               ← KERNEL EXISTS
  8. Both processes see same physical pages                   ← ZERO-COPY
```

Steps 1-2 are already working. PRIME import/export is also now implemented in `redox-drm`, and
the shared DRM core now includes a bounded private command-submission surface for contract
hardening. Real vendor render/fence completion is still missing.

## Current State

### What Exists

| Component | Status | Detail |
|-----------|--------|--------|
| GEM_CREATE ioctl | ✅ Working | `DmaBuffer::allocate()` in `gem.rs`, physically contiguous system RAM |
| GEM_CLOSE ioctl | ✅ Working | Ownership tracking, reference counting, safe cleanup |
| GEM_MMAP ioctl | ✅ Working | Returns virtual address for mmap_prep |
| KMS/modesetting ioctls | ✅ Working | 16 KMS ioctls, CRTC/connector/encoder/plane |
| Kernel FmapBorrowed | ✅ Exists | `Provider::FmapBorrowed` at `memory.rs:1157`, `Grant::borrow_fmap()` at `memory.rs:1401` |
| Kernel sendfd | ✅ Exists | `SYS_SENDFD` at `syscall/fs.rs:415`, passes `Arc<LockedFileDescription>` |
| Kernel PhysBorrow | ✅ Exists | `scheme:memory` physical address mapping |
| libdrm `__redox__` | ✅ Full | Opens `/scheme/drm`, dispatches KMS + PRIME ioctls via `redox_fpath` |

### What Is Missing

| Component | Status | Impact |
|-----------|--------|--------|
| PRIME_HANDLE_TO_FD | ✅ Implemented | Opaque export tokens via prime_exports map |
| PRIME_FD_TO_HANDLE | ✅ Implemented | Token lookup via prime_exports, adds to owned_gems |
| libdrm PRIME/GEM dispatch | ✅ Implemented | __redox__ wrappers in drmPrimeHandleToFD/drmPrimeFDToHandle |
| Mesa Redox winsys | 🚧 Scaffolding | Stubs compile but do not render — blocked on GPU CS |
| GPU command submission | ⚠️ Bounded shared surface only | Shared private-CS contract exists; no real vendor render ioctl/ring programming yet |
| GPU fence/signaling | ❌ Not implemented | No GPU completion notification |

### What Was Cleaned Up (Historical Session)

The old fake PRIME implementation used synthetic fd numbers starting at 10,000 that were not real
kernel file descriptors. Other processes could not resolve them. Oracle caught this across 4
verification rounds. The cleanup and follow-up hardening:

- Removed `exported_dmafds` tracking from Handle struct
- Removed the old fake PRIME synthetic-fd bookkeeping. Imported GEM tracking still exists in the current honest PRIME path.
- Removed DMA-BUF methods from `GpuDriver` trait and AMD/Intel driver impls
- Removed `DmabufManager` from `GemManager`
- Removed `mod dmabuf` from `main.rs`
- Removed PRIME wire structs (`DrmPrimeHandleToFdWire`, `DrmPrimeFdToHandleWire`)
- PRIME handlers are now implemented in the current scheme path with non-guessable export tokens; only unsupported fake sync/render paths return `EOPNOTSUPP`.
- Removed all `#[allow(dead_code)]` from fake bookkeeping

## Phased Implementation

### v1: System RAM, Linear, Single GPU (Target: working PRIME)

**Current note:** the PRIME/export-import milestones in this section are largely complete in the
live `redox-drm` scheme. The remaining value of this document is the dependency narrative and the
later DMA-BUF / winsys / fencing roadmap, not the claim that PRIME is still unimplemented.

**Goal**: A compositor (KWin) can import a buffer rendered by a GPU client (Mesa) and display it.
All buffers in system RAM, linear layout, single GPU.

**Duration estimate**: 6-10 weeks (2 developers)

#### Step 1: Delete dead dmabuf.rs

Remove `local/recipes/gpu/redox-drm/source/src/dmabuf.rs`. It is dead code — `mod dmabuf` was
removed from `main.rs` but the file still exists.

**Effort**: trivial

#### Step 2: Implement PRIME export in redox-drm

When `PRIME_HANDLE_TO_FD` is called:

1. Look up the GEM handle in the calling fd's `owned_gems`
2. Validate ownership (same as GEM_MMAP check)
3. Generate an opaque export token and store `prime_exports[token] = gem_handle`
4. Return the token to the caller (NOT a scheme fd or GEM handle)

The client then opens `/scheme/drm/card0/dmabuf/{token}` to get a real scheme fd. The open
handler validates the token against `prime_exports`, creates a `NodeKind::DmaBuf` scheme handle,
and bumps the GEM export refcount. When that scheme fd is closed, the refcount is dropped.

Key design: export tokens are opaque, non-guessable identifiers, not synthetic fd numbers or raw GEM handles.
The `prime_exports` map resolves tokens to GEM handles. Tokens are cleaned up when the last
export ref for a GEM handle is dropped.

**Changes to `scheme.rs`**:
- Add `NodeKind::DmaBuf { gem_handle, export_token }` variant
- Add `prime_exports: BTreeMap<u32, GemHandle>` and `next_export_token: u32`
- `PRIME_HANDLE_TO_FD` handler: validate ownership → generate token → store in prime_exports → return token
- `PRIME_FD_TO_HANDLE` handler: receive token → look up in prime_exports → add GEM to caller's `owned_gems`
- `open()` handler: accept `"card0/dmabuf/{token}"` path → validate token → create DmaBuf node → bump export ref
- `mmap_prep()` handler: for DmaBuf nodes, return GEM physical address

**Changes to `driver.rs`**:
- No changes needed. GEM operations stay on the trait as-is. PRIME is a scheme-level concern,
  not a driver-level concern.

**Effort**: 1-2 weeks

#### Step 3: Add reference counting for shared GEM objects

When a GEM buffer is exported via PRIME, multiple scheme fds may reference it. The `close()` path
must only call `driver.gem_close()` when ALL references (original GEM + all exported fds) are gone.

**Changes**:
- Add `gem_refcounts: BTreeMap<GemHandle, usize>` to `DrmScheme`
- Increment on export, decrement on close of DmaBuf fd
- `gem_close()` checks refcount before calling driver

**Effort**: 3-5 days

#### Step 4: Validate with a two-process reproducer

Build a minimal test that:
1. Process A opens `/scheme/drm/card0`, creates a GEM buffer, writes a pattern
2. Process A exports via PRIME_HANDLE_TO_FD
3. Process A sends the fd to Process B via `sendfd` (or equivalent scheme IPC)
4. Process B receives the fd, imports via PRIME_FD_TO_HANDLE
5. Process B mmaps the imported handle and reads the pattern
6. Verify both processes see the same physical pages (same data, zero-copy)

This validates the full chain: redox-drm → scheme fd → sendfd → import → mmap → FmapBorrowed.

**Effort**: 1 week

#### Step 5: libdrm Redox PRIME/GEM dispatch

libdrm already has `__redox__` conditionals. Add dispatch for:
- `drmPrimeHandleToFD()` → send `PRIME_HANDLE_TO_FD` ioctl to `/scheme/drm`
- `drmPrimeFDToHandle()` → send `PRIME_FD_TO_HANDLE` ioctl
- `drmPrimeClose()` → close the exported/imported fd
- `drmGemHandleToPrimeFD()` / `drmPrimeFDToGemHandle()` — aliases for the above

The libdrm WIP recipe is at `recipes/wip/x11/libdrm/`. The `__redox__` handling already opens
`/scheme/drm` and has ioctl dispatch infrastructure. The gap is PRIME/GEM-specific ioctl codes.

**Effort**: 1-2 weeks

#### Step 6: Mesa Redox winsys (compile-time scaffolding)

Add `src/gallium/winsys/redox/` to Mesa that:
- Opens the DRM scheme
- Allocates GEM buffers via `GEM_CREATE`
- Exports them via `PRIME_HANDLE_TO_FD`
- Imports shared buffers via `PRIME_FD_TO_HANDLE`
- Maps them via `mmap` (which triggers `FmapBorrowed`)

Pattern: similar to `winsys/amdgpu/drm/` but using Redox scheme IPC. This is scaffolding — it
compiles but cannot render without GPU command submission (Step 8).

Split into:
- **6a**: Compile-time winsys structure, buffer allocation, PRIME export/import
- **6b**: Runtime buffer-sharing enablement (depends on step 4 validation)

**Effort**: 3-4 weeks

### v2: VRAM/GTT Placement, Tiling, Multi-GPU

**Goal**: Buffers can live in VRAM with GTT aperture access. Tiled/modifier support for
scanout-optimized layouts. Multi-GPU buffer sharing.

**Duration estimate**: 8-12 weeks (after v1)

- AMD GTT/VRAM placement via `amdgpu_gtt_mgr` / `amdgpu_vram_mgr` equivalents
- Intel GGTT/PPGTT population for imported buffers
- DRM format modifiers: `DRM_FORMAT_MOD_LINEAR` + vendor-specific tiling
- Multi-GPU: each GPU has its own `redox-drm` instance, PRIME between them
- This tier requires the AMD/Intel driver GTT programming that is currently partial

### v3: Fencing, Explicit Sync, Vulkan

**Goal**: GPU fence objects for render/scanout synchronization. Explicit sync protocol for
Wayland. Vulkan driver support.

**Duration estimate**: 12-16 weeks (after v2)

- `dma_fence` equivalent: kernel waitable event per page-flip or command submission
- `sync_file` equivalent: fd-backed fence that can be passed between processes
- Wayland `zwp_linux_explicit_synchronization_v1` protocol in compositor
- Vulkan `VK_KHR_external_memory` / `VK_KHR_external_semaphore` backed by DMA-BUF fds
- AMD: fence through ring buffer writeback + IRQ
- Intel: fence through seqno writeback + IRQ

## Dependency Graph

```
Step 1 (delete dmabuf.rs)
  → no dependency, do immediately

Step 2 (PRIME export/import in scheme)
  → depends on: nothing
  → enables: steps 3, 4, 5

Step 3 (refcount for shared GEM)
  → depends on: step 2
  → enables: step 4

Step 4 (two-process reproducer)
  → depends on: steps 2, 3
  → validates: the full chain works

Step 5 (libdrm dispatch)
  → depends on: step 2 (ioctl protocol defined)
  → can start in parallel with steps 3-4

Step 6 (Mesa winsys)
  → depends on: step 5 (libdrm API available)
  → 6a can start once step 2 protocol is defined
  → 6b should wait for step 4 validation
```

Steps 5 and 6a can proceed in parallel with steps 3-4 once step 2 is done.

## What This Does NOT Cover

This plan covers **cross-process buffer sharing** (the DMA-BUF/PRIME contract). It does not cover:

| Out of scope | Where it lives |
|-------------|----------------|
| GPU command submission (CS ioctl) | `HARDWARE-3D-ASSESSMENT.md` Tier 2 |
| GPU fence/signaling | `HARDWARE-3D-ASSESSMENT.md` Tier 2 |
| Mesa hardware Gallium driver (radeonsi/iris) | `HARDWARE-3D-ASSESSMENT.md` Tier 1 |
| AMD ring buffer programming | `local/recipes/gpu/amdgpu/` |
| Intel render ring programming | `local/recipes/gpu/redox-drm/source/src/drivers/intel/` |
| Mesa EGL platform extension for DRM | `HARDWARE-3D-ASSESSMENT.md` Tier 3 |

PRIME/DMA-BUF is a **prerequisite** for hardware-accelerated rendering, but it is not sufficient
by itself. The render pipeline (command submission + fencing + Mesa driver) is tracked separately
in `HARDWARE-3D-ASSESSMENT.md`.

## Why Not a Kernel DMA-BUF Scheme

Linux has a global `dma-buf` kernel subsystem with its own fd type. Red Bear OS does NOT need this
because:

1. **`redox-drm` IS the exporter.** In Linux, any kernel subsystem can export a dma-buf. In Redox,
   only the DRM scheme exports GPU buffers. There is no need for a generic kernel dma-buf layer.

2. **Scheme fds ARE the sharing mechanism.** In Linux, dma-buf has its own fd type with special
   mmap semantics. In Redox, scheme file descriptors already support `fmap_prep` → `FmapBorrowed`.
   The kernel maps the same physical pages. No new fd type needed.

3. **`sendfd` IS the fd passing mechanism.** In Linux, fd passing uses SCM_RIGHTS over Unix
   sockets. In Redox, `sendfd` passes `Arc<LockedFileDescription>` via scheme IPC. Same result.

If a future use case requires sharing non-DRM buffers (e.g., camera frames, video decode output),
a separate `scheme:dmabuf` could be created. But for GPU buffer sharing, the DRM scheme is
sufficient.

## Wire Protocol Design

### PRIME_HANDLE_TO_FD

Request (from libdrm client):
```c
struct DrmPrimeHandleToFdWire {
    uint32_t handle;      // GEM handle to export
    uint32_t flags;       // DRM_CLOEXEC | DRM_RDWR (hints, not critical for v1)
};
```

Response:
```c
struct DrmPrimeHandleToFdResponseWire {
    int32_t  fd;          // opaque export token (NOT a process fd or GEM handle)
    uint32_t _pad;
};
```

The scheme internally:
1. Validates handle ownership
2. Generates an opaque non-guessable export token
3. Stores `prime_exports[token] = gem_handle`
4. Returns the token as `fd`

The client then opens `/scheme/drm/card0/dmabuf/{token}` to get a real scheme fd.
The open handler validates the token, creates a DmaBuf scheme handle, and bumps
`gem_export_refs`. When that scheme fd is closed, the ref is dropped.

### PRIME_FD_TO_HANDLE

Request (from libdrm client):
```c
struct DrmPrimeFdToHandleWire {
    int32_t  fd;          // opaque export token (extracted via redox_fpath on dmabuf fd)
    uint32_t _pad;
};
```

Response:
```c
struct DrmPrimeFdToHandleResponseWire {
    uint32_t handle;      // GEM handle for the imported buffer
    uint32_t _pad;
};
```

The scheme internally:
1. Looks up the export token in `prime_exports` → gets the GEM handle
2. Validates the token exists
3. Adds the GEM handle to the caller's `owned_gems`
4. Returns the GEM handle

### open() path extension

```rust
// Existing paths:
"card0"              → NodeKind::Card
"card0Connector/{id}" → NodeKind::Connector(id)

// Export token path (validated against prime_exports):
"card0/dmabuf/{token}" → NodeKind::DmaBuf { gem_handle, export_token: token }
```

### redox_fpath() for DmaBuf

```rust
NodeKind::DmaBuf { export_token, .. } => format!("drm:card0/dmabuf/{export_token}")
```

### Token cleanup

When the last export ref for a GEM handle is dropped:
```rust
fn drop_export_ref(&mut self, gem_handle: GemHandle) {
    // ... decrement refcount ...
    if remove_entry {
        self.gem_export_refs.remove(&gem_handle);
        self.prime_exports.retain(|_, &mut h| h != gem_handle);
    }
}
```

When a GEM is destroyed via any path (GEM_CLOSE, DESTROY_DUMB, handle close, fb reap),
`prime_exports` entries are pruned:
- `maybe_close_gem()`: central helper prunes tokens on successful `driver.gem_close()`
- `GEM_CLOSE` / `DESTROY_DUMB`: explicit `prime_exports.retain()` after direct `driver.gem_close()`
- `PRIME_FD_TO_HANDLE`: `gem_size()` liveness check removes stale token on failure
- `open("card0/dmabuf/{token}")`: `gem_size()` liveness check removes stale token on failure

## Files to Modify

| File | Change | Status |
|------|--------|--------|
| `local/recipes/gpu/redox-drm/source/src/dmabuf.rs` | **DELETED** | ✅ |
| `local/recipes/gpu/redox-drm/source/src/scheme.rs` | DmaBuf nodes, opaque export tokens, PRIME handlers, refcount cleanup, stale token cleanup | ✅ |
| `local/recipes/gpu/redox-drm/source/src/gem.rs` | No changes (GEM operations unchanged) | — |
| `local/recipes/gpu/redox-drm/source/src/driver.rs` | No changes (PRIME is scheme-level) | — |
| `local/recipes/gpu/redox-drm/source/src/main.rs` | No changes (already clean) | — |
| `recipes/wip/x11/libdrm/source/xf86drm.c` | `redox_fpath()` + export token dmabuf path + `sys/redox.h` | ✅ |
| `recipes/libs/mesa/source/src/gallium/winsys/redox/drm/` | 4 scaffolding files (compile-time only) | ✅ |
| `local/recipes/tests/redox-drm-prime-test/` | Test reproducer recipe + Rust binary (incl. stale token test) | ✅ |
| `local/docs/HARDWARE-3D-ASSESSMENT.md` | PRIME status updated | ✅ |
| `local/docs/DMA-BUF-IMPROVEMENT-PLAN.md` | Implementation status updated | ✅ |

## Implementation Status (2026-04-16)

| Step | Status | Deliverable |
|------|--------|-------------|
| 1. Delete dead dmabuf.rs | ✅ Done | File removed |
| 2. PRIME export/import in scheme | ✅ Done | DmaBuf nodes, export refcounting, mmap_prep, open/close/fpath |
| 3. Reference counting for shared GEM | ✅ Done | gem_export_refs, bump/drop, gem_can_close, maybe_close_gem |
| 4. Two-process reproducer | ✅ Recipe created | `local/recipes/tests/redox-drm-prime-test/` (runtime validation pending) |
| 5. libdrm Redox dispatch | ✅ Done | __redox__ wrappers in drmPrimeHandleToFD and drmPrimeFDToHandle |
| 6a. Mesa winsys scaffolding | ✅ Done | `src/gallium/winsys/redox/drm/` (4 files, compiles but does not render) |
| 6b. Mesa runtime buffer sharing | ⏳ Blocked | Requires GPU command submission (not yet implemented) |

**Stale token cleanup**: All GEM destruction paths now prune `prime_exports`. Central cleanup
in `maybe_close_gem()`, explicit cleanup in `GEM_CLOSE`/`DESTROY_DUMB`, liveness checks in
`PRIME_FD_TO_HANDLE` and `open("dmabuf/{token}")` that remove stale tokens on failure.
Verified by Oracle across 8 rounds.

**Protocol note**: PRIME uses opaque export tokens. PRIME_HANDLE_TO_FD returns a non-guessable
token stored in `prime_exports`. The client opens `/scheme/drm/card0/dmabuf/{token}`
to get a real scheme fd. `redox_fpath()` on that fd reveals the token. PRIME_FD_TO_HANDLE
accepts the export token and resolves it via `prime_exports`. Tokens are cleaned up when the
last export ref is dropped.

## Relationship to Other Plans

- `local/docs/HARDWARE-3D-ASSESSMENT.md` — broader hardware 3D status (command submission, fencing,
  Mesa driver enablement). This document is the DMA-BUF-specific deep dive.
- `local/docs/CONSOLE-TO-KDE-DESKTOP-PLAN.md` — canonical desktop path plan. DMA-BUF is a
  prerequisite for the hardware-accelerated rendering phase.
- `local/docs/AMD-FIRST-INTEGRATION.md` — AMD-specific GPU details including GTT/VRAM programming.
- `docs/04-LINUX-DRIVER-COMPAT.md` — linux-kpi architecture reference for driver porting.