Anhad Singh
7d42e8d966
During the snapshot shown, interrupts would be _disabled_ on both CPUs and both
would have have the _same_ address space (i.e.
`PercpuBlock::current().current_addrsp`).
```rs
// CPU 0 CPU 1 (inside `context::switch()`)
// ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
// let guard = current_addrsp.acquire_write()
// let flusher = Flusher::with_cpu_set(&addrsp.used_by, ¤t_addrsp.tlb_ack);
// ...
// drop(flusher); // Flusher::flush()
// // send IPI to CPU 1 as it is in the `active_set`
// shootdown_tlb_ipi(Some(1));
// percpu::switch_arch_hook();
// // spin until TLB shootdown IPI has been acknowledged by CPU 1 // acquire_read() to remove the CPU from the `used_by` set in `prev_addrsp`
// while ackword < affected_cpu_count prev_addrsp.acquire_read()
// PercpuBlock::current().maybe_handle_tlb_shootdown();
// spin_loop();
// }
// drop(guard);
```
Now CPU 0 will spin _until_ CPU 1 has acknowledged the IPI. However, CPU 1 has
interrupts disabled and is in the middle of a context switch. To complete the
context switch it needs to remove itself from the `used_by` set in
`prev_addrsp` (which is the current address space atm). To do that it needs to
`acquire_read()` lock but it cannot since it is held by CPU 0 and won't release
it until CPU 1 has acknowledged the IPI; creating a deadlock.
Also note:
```rs
// src/context/memory.rs
// NOTE: Lock must be held, which must be guaranteed by the caller.
pub fn flush(&mut self) {
```
Here is some output from GDB to back this up:
```
pwndbg> info threads
Id Target Id Frame
1 Thread 1.1 (CPU#0 [running]) core::sync::atomic::atomic_compare_exchange_weak<usize> (dst=0xffffff7f801972f0, old=0, new=1, success=core::sync::atomic::Ordering::Acquire, failure=core::sync::atomic::Ordering::Relaxed)
at /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:4100
2 Thread 1.2 (CPU#1 [running]) kernel::context::memory::Flusher::flush (self=0xffff80004212f4b8) at src/context/memory.rs:2827
3 Thread 1.3 (CPU#2 [running]) spin::rwlock::RwLock<kernel::context::context::Context, spin::relax::Spin>::write<kernel::context::context::Context, spin::relax::Spin> (self=0xffffff7f801a99c0)
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spin-0.9.8/src/rwlock.rs:239
* 4 Thread 1.4 (CPU#3 [running]) 0xffffffff80059cf9 in kernel::context::memory::AddrSpaceWrapper::acquire_read (self=0xffffff7f80188110)
at /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/x86/sse2.rs:25
pwndbg> p kernel::context::arch::CONTEXT_SWITCH_LOCK
$1 = core::sync::atomic::AtomicBool {
v: core::cell::UnsafeCell<u8> {
value: 1
}
}
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[0].p.value).wants_tlb_shootdown.v.value
$2 = 0
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[1].p.value).wants_tlb_shootdown.v.value
$3 = 0
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[2].p.value).wants_tlb_shootdown.v.value
$4 = 1
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[3].p.value).wants_tlb_shootdown.v.value
$5 = 0
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[3].p.value).current_addrsp
$6 = core::cell::RefCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
borrow: core::cell::Cell<isize> {
value: core::cell::UnsafeCell<isize> {
value: 0
}
},
value: core::cell::UnsafeCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
value: core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>::Some(alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global> {
ptr: core::ptr::non_null::NonNull<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>> {
pointer: 0xffffff7f80188100
},
phantom: core::marker::PhantomData<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>>,
alloc: alloc::alloc::Global
})
}
}
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[2].p.value).current_addrsp
$7 = core::cell::RefCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
borrow: core::cell::Cell<isize> {
value: core::cell::UnsafeCell<isize> {
value: 0
}
},
value: core::cell::UnsafeCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
value: core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>::Some(alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global> {
ptr: core::ptr::non_null::NonNull<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>> {
pointer: 0xffffff7f80188100
},
phantom: core::marker::PhantomData<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>>,
alloc: alloc::alloc::Global
})
}
}
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[1].p.value).current_addrsp
$8 = core::cell::RefCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
borrow: core::cell::Cell<isize> {
value: core::cell::UnsafeCell<isize> {
value: 0
}
},
value: core::cell::UnsafeCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
value: core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>::Some(alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global> {
ptr: core::ptr::non_null::NonNull<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>> {
pointer: 0xffffff7f80188100
},
phantom: core::marker::PhantomData<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>>,
alloc: alloc::alloc::Global
})
}
}
pwndbg> p (*kernel::percpu::ALL_PERCPU_BLOCKS[0].p.value).current_addrsp
$9 = core::cell::RefCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
borrow: core::cell::Cell<isize> {
value: core::cell::UnsafeCell<isize> {
value: 0
}
},
value: core::cell::UnsafeCell<core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>> {
value: core::option::Option<alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global>>::Some(alloc::sync::Arc<kernel::context::memory::AddrSpaceWrapper, alloc::alloc::Global> {
ptr: core::ptr::non_null::NonNull<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>> {
pointer: 0xffffff7f80133ac8
},
phantom: core::marker::PhantomData<alloc::sync::ArcInner<kernel::context::memory::AddrSpaceWrapper>>,
alloc: alloc::alloc::Global
})
}
}
```
```
pwndbg> thread 4
[Switching to thread 4 (Thread 1.4)]
25 in /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/x86/sse2.rs
pwndbg> info threads
Id Target Id Frame
1 Thread 1.1 (CPU#0 [running]) core::sync::atomic::atomic_compare_exchange_weak<usize> (dst=0xffffff7f801972f0, old=0, new=1, success=core::sync::atomic::Ordering::Acquire, failure=core::sync::atomic::Ordering::Relaxed)
at /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:4100
2 Thread 1.2 (CPU#1 [running]) kernel::context::memory::Flusher::flush (self=0xffff80004212f4b8) at src/context/memory.rs:2827
3 Thread 1.3 (CPU#2 [running]) spin::rwlock::RwLock<kernel::context::context::Context, spin::relax::Spin>::write<kernel::context::context::Context, spin::relax::Spin> (self=0xffffff7f801a99c0)
at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spin-0.9.8/src/rwlock.rs:239
* 4 Thread 1.4 (CPU#3 [running]) 0xffffffff80059cf9 in kernel::context::memory::AddrSpaceWrapper::acquire_read (self=0xffffff7f80188110)
at /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/x86/sse2.rs:25
pwndbg> thread 4
[Switching to thread 4 (Thread 1.4)]
25 in /mnt/redox/prefix/x86_64-unknown-redox/sysroot/lib/rustlib/src/rust/library/core/src/../../stdarch/crates/core_arch/src/x86/sse2.rs
pwndbg> bt
```
This is *after* everything freezes.
For relibc tests, if I run `while true; do
/root/relibc-tests/pthread/once; done` it eventually does freeze up and
does so at the same location:
```
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ THREADS (4 TOTAL) ]──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
► 1 "" stopped: 0xffffffff800750a2 <kernel::context::switch::switch+1282>
4 "" stopped: 0xffffffff80074d5a <kernel::context::switch::switch+442>
3 "" stopped: 0xffffffff80059cf9 <kernel::context::context::Context::set_addr_space+217>
2 "" stopped: 0xffffffff80020839 <kernel::context::memory::Flusher::flush+361>
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
```
Related: https://gitlab.redox-os.org/redox-os/kernel/-/issues/175
Signed-off-by: Anhad Singh <andypython@protonmail.com>
Kernel
Redox OS Microkernel
Requirements
nasmneeds to be available on the PATH at build time.
Building The Documentation
Use this command:
cargo doc --open --target x86_64-unknown-none
Debugging
QEMU
Running QEMU with the -s flag will set up QEMU to listen on port 1234 for a GDB client to connect to it. To debug the redox kernel run.
make qemu gdb=yes
This will start a virtual machine with and listen on port 1234 for a GDB or LLDB client.
GDB
If you are going to use GDB, run these commands to load debug symbols and connect to your running kernel:
(gdb) symbol-file build/kernel.sym
(gdb) target remote localhost:1234
LLDB
If you are going to use LLDB, run these commands to start debugging:
(lldb) target create -s build/kernel.sym build/kernel
(lldb) gdb-remote localhost:1234
After connecting to your kernel you can set some interesting breakpoints and continue
the process. See your debuggers man page for more information on useful commands to run.
Notes
-
Always use
foo.get(n)instead offoo[n]and try to cover for the possibility ofOption::None. Doing the regular way may work fine for applications, but never in the kernel. No possible panics should ever exist in kernel space, because then the whole OS would just stop working. -
If you receive a kernel panic in QEMU, use
pkill qemu-systemto kill the frozen QEMU process.
How To Contribute
To learn how to contribute to this system component you need to read the following document:
Development
To learn how to do development with this system component inside the Redox build system you need to read the Build System and Coding and Building pages.
How To Build
To build this system component you need to download the Redox build system, you can learn how to do it on the Building Redox page.
This is necessary because they only work with cross-compilation to a Redox virtual machine, but you can do some testing from Linux.
Funding - Unix-style Signals and Process Management
This project is funded through NGI Zero Core, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.
