The batch BO pool in redox_drm_cs.c had three memory-safety bugs:
1. Class math: the pool is documented to cover 'sizes 4 KiB to 512
KiB' across REDOX_DRM_BATCH_POOL_SIZE_CLASSES (8) slots, but the
implementation computed the class as ceil(log2(byte_count)) from the
raw byte count without applying the 4 KiB base. A 32 KiB batch
request (which should map to class 3) ended up at class 15, which
is >= 8, so the pool returned NULL and every GPU submission
allocated a fresh BO — the pool was completely non-functional.
2. Undersized cached BOs: the BO was allocated with width0 = byte_count
(exact request size) but cached against a class index that expected
the full class capacity. A subsequent request that mapped to the
same class but with a larger byte_count could receive a BO smaller
than the write and silently overflow the heap.
3. No synchronization: the pool was shared across pipe_contexts but
accessed without any mutex — concurrent submissions from multiple
threads would race on rws->batch_pool[cls].
4. Resource leak: batch_bo_release returned silently if the entry
couldn't be cached, leaking the BO.
Fixes:
- Replace the per-call class math with a shared helper
batch_bo_size_class() that returns cls = log2_ceil(byte_count) - 12
(with clamping for sub-4-KiB requests). Oversized requests return
REDOX_DRM_BATCH_POOL_SIZE_CLASSES to signal 'fresh allocation, not
pool'.
- Add batch_bo_class_capacity() returning 4096 << cls.
- Acquire now allocates BO at the full class capacity (not the request
size), so any cached entry can serve any smaller request in the
same class on recycle.
- Acquire defensively verifies width0 >= byte_count before returning
a cached entry; undersized entries are destroyed.
- Release destroys entries that don't match the class capacity, and
destroys redundant entries when a slot is already occupied (rather
than leaking).
- Pool is guarded by a new mtx_t batch_pool_mutex field on
redox_drm_winsys (initialized in redox_drm_initialize, destroyed
in redox_drm_destroy).
The related 'format = 0 -> 0-byte GEM object' issue in
redox_resource_byte_count is a separate bug (batches need a non-NONE
format to get real storage); tracked as a follow-up.
Verified by a standalone host gcc test of the class math covering 15
boundary cases (0, 1, 4096, 4097, 8192, 32768, 524288, 524289,
1048576, etc.); all match expected class assignments.
vasilito
16f74ab87c