Option for GdbStub to own its packet buffer

[lewisfm]

I used gdbstub to create an on-device embedded debugger (in which it is working very well), but initially setting up the integration felt like I was working around the borrow-check system. The crate's packet buffer API seems to be optimized for having a function-scoped GdbStub that refers to existing data structures, whereas I need my debugger to live for the entire program with the ability to be periodically polled from any context.

Since my GdbStub instance needs to be stored globally, I need it to be 'static. However, there's no option for it to own its packet buffer; it only accepts references. Attaining such a &'static mut [u8] required me to do some unsafe shenanigans:

pub struct V5Debugger<S: Connection + ConnectionExt> {
    target: V5Target,
    gdb: Option<GdbStubStateMachine<'static, V5Target, S>>,
}

const GDB_PACKET_BUFFER_SIZE: usize = 0x2000;
static mut GDB_PACKET_BUFFER: [u8; GDB_PACKET_BUFFER_SIZE] = [0; _];
static GDB_PACKET_BUFFER_CLAIMED: AtomicBool = AtomicBool::new(false);

if GDB_PACKET_BUFFER_CLAIMED.swap(true, Ordering::Acquire) {
    panic!("Cannot create multiple debuggers");
}

// SAFETY: The mutable ownership over the buffer can only be taken once.
let gdb_buffer: &'static mut [u8] = unsafe {
    core::slice::from_raw_parts_mut(&raw mut GDB_PACKET_BUFFER[0], GDB_PACKET_BUFFER_SIZE)
};

let mut target = V5Target::new();

Self {
    gdb: {
        let stub = GdbStubBuilder::new(stream)
            .with_packet_buffer(gdb_buffer)
            .build()
            .unwrap();

        Some(stub.run_state_machine(&mut target).unwrap())
    },
    target,
}

It would be very convenient if I didn't have to worry about GDBStub's packet buffer. I'd be happy to do something like this:

const GDB_PACKET_BUFFER_SIZE: usize = 0x2000;
let mut target = V5Target::new();

Self {
    gdb: {
        let stub = StaticGdbStub::<GDB_PACKET_BUFFER_SIZE>::new(stream);
        Some(stub.run_state_machine(&mut target).unwrap())
    },
    target,
}

...or have some other API that implements the buffer as an owned array.

[daniel5151]

Technically speaking, gdbstub already has the ability to own its own packet buffer... but only when running with alloc enabled 😅

I poked around the commit history, and it seems that the GdbStubBuilder API hasn't really been touched since its original introduction way back in 2020, at which time, Rust didn't even have proper const generic support, hah.

Fortunately, I think we'll be able to squeeze in a new const-generic parameter for an internally-owned packet buffer fairly easily, and even more importantly - in a semver-compatible manner!

Roughly speaking, this would require:

• Updating GdbStubBuilder and GdbStub to include a new trailing GdbStub{Builder}<..., const OWNED_BUF_SIZE: usize = 0> const-generic param (where the default is 0, for back-compat)
• Updating the GdbStubBuilder::build logic to check if the user has explicitly set OWNED_BUF_SIZE to a non-zero value, and if so, use that internally allocated buffer as the packet buffer

One nuance to consider: at the moment, gdbstub (internally) relies on a tiny external library type (managed::ManagedSlice) in order to model the current no_std-borrowed vs. heap-alloc-owned semantics. I think we'll need to vendor this type into gdbstub itself, in order to extend it with a new OwnedStatic variant. That's totally fine though, and we can just stick the vendor'd version under src/util/managed.rs (especially since we already have src/util/managed_vec.rs)

If you'd like to send in a PR implementing this work, I'd be happy to review and iterate on it with you!

Alternatively, I can try to find time to take a stab at this myself, though, as with many other backlog items in gdbstub, it's unclear when I'll actually find the time to sit down and drive this forward myself, hah.