Replaced my homemade stack with picoquic and I have some questions

[lminiero]

Hi,

before moving to the questions, a bit of context...

In the past year and a half, I've done a deep dive in QUIC, and as a result of that I ended up writing a new very homemade QUIC stack, on top of which I then scaffolded basic WebTransport support, and support for RoQ (RTP Over QUIC) and MoQ (Media Over QUIC), as real-time media was indeed what I wanted to play with. This ended up in a library I called imquic, which has been fine as a testbed for the functional aspects of the different protocols, but has always been quite frail at the foundation, because of my homemade QUIC stack.

I knew I had to switch that part out and leverage a much more robust implementation instead, and since so many people recommended picoquic, I decided to give it a go. I was expecting a steep learning curve, mostly because the APIs are not really documented, but the guidance provided in the documentation made it pretty easy to first build some prototypes to familiarize with the APIs, and then figure out how to integrate picoquic in my library with as little disruption as possible.

Long story short, I got to a point where I could remove my old existing stack and have higher "levels" rely on picoquic instead, all while keeping my library API pretty much untouched, but there are a few things that I'm still unsure about, mostly due to features my library provided that I'm not sure I can 100% duplicate via picoquic too. I'll mention a few different things, so if opening separate discussions for each of them is more appropriate, please do let me know.

-fPIC support

As I mentioned, I'm writing a library, which means that if I want to use picoquic, it needs to be built with -fPIC. Is there any way to configure that using the existing build scripts without modifying the cmake file? At the moment, that's the way I do that, as I prepared a patch file for the purpose:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 69010393..57f63270 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -23,6 +23,8 @@ option(BUILD_PQBENCH "Build pqbench" ON)
 option(BUILD_LOGLIB "Build picoquic-log" ON)
 option(BUILD_LOGREADER "Build picolog_t the log reader" ON)
 
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+
 message(STATUS "Initial CMAKE_C_FLAGS=${CMAKE_C_FLAGS}")
 
 if(DISABLE_DEBUG_PRINTF)

but that's probably not a very robust way of doing that, since the cmake file may change in the future in ways that could break my patch.

CONNECTION_CLOSE reason

There doesn't seem to be a way to provide (or be notified about) a CONNECTION_CLOSE reason string, in picoquic. Is that intended, or am I missing some ways this can indeed be done? An inspection of the code didn't seem to unveil anything. While I understand reason strings are not strictly speaking necessary, I've found them useful so far, especially during MoQ interop sessions with other implementations where the same error code may be triggered by different root causes that a reason string may help identify.

WebTransport and protocols

I haven't done a deep dive of the WebTransport support provided by picoquic in picohttp yet. In order to keep things simple in this first integration, I've only done an integration at the QUIC level, and then for WebTransport I'm relying on the stack I had written in my own library instead, which means that when h3 is negotiated I have my own library code to deal with the SETTINGS, CONNECT, STREAM prefixes etc. I'd like in a second stage to be able to rely on picoquic's WebTransport layer too (especially considering it would give me generic HTTP/3 support as well, whereas now I only do CONNECT), but from what I see there are a couple of areas I'm not sure the existing code covers.

My stack allows negotiating raw QUIC and WebTransport at the same time both for clients and for servers, while the documentation seems to only mention this possibility for servers. Is that something that is actually possible for clients too?

Another question is related to the ability to negotiate protocols in WebTransport, which is a feature MoQ actually relies on to negotiate the specific draft version that will be used on the wire. This WebTransport protocol negotiation happens via headers exchanged in the CONNECT and its 200, a bit like you currently negotiate ALPNs via TLS. Assuming picoquic doesn't have native support for this, is there enough flexibility to inject custom headers and process custom ones so that this can be dealt with at an application level?

QLOG support

In my library, I implemented a support for QLOG that was meant to be transversal to the different layers you use in the connection. This means that, for the same connection and in the same QLOG file, you could log QUIC events, WebTransport events, and MoQ events by adding the related schemas and intertwining log items coming from different layers. Besides, QLOG generation supports both pure JSON (generating a JSON file at the end) and sequential JSON (where you write individual lines as they happen, for live inspection of the QLOG files).

Seeing how picoquic implements QLOG, it became clear I could not really do that anymore. I can still create my own QLOG files for layers I control myself (http3, roq, moq), but not for QUIC itself, since picoquic has its own optimized mechanism for collecting those. I guess my main questions about this are:

1. is sequential JSON supported as an output? (my guess would be no, due to the optimized way logging is performed in a binary format, in order to only generate a JSON at the very end)
2. is the existing QLOG support meant to be extensible, meaning I can register additional schemas and add events of my own and they'll be collected/serialized as QUIC events are?
3. is there a way to customize the target filename of the QLOG file? It looks like it may be hardcoded to using the initial connection ID as the filename, with maybe a bit of randomization if you're trying to avoid conflicts?

That's all

Apologies for the wall of text, but I wanted to try and collect all the feedback I had while it was still fresh in my mind, since just today I finished a first "working" integration where most things work as before. I plan to do more tests and studying in the next few days/weeks (especially considering I still haven't addressed how to handle, e.g., flow control or congestion from an application perspective), so in case I come up with more doubts I'll make sure to open a separate discussion for those.

[suhasHere]

@lminiero just wanted to share our implementation of using picoqcui for WT and raw QUIC here

https://github.com/Quicr/libquicr/blob/main/src/transport_picoquic.cpp

[lminiero]

Thanks @suhasHere!

[huitema]

@lminiero That's a nice list of potential improvements!

• On -fPIC: isn't it sufficient to call cmake with the option "-DCMAKE_POSITION_INDEPENDENT_CODE=ON" ? Or, do you need something more, such as changing the CMake to produce shared libraries? If the latter, we should maybe start a specific discussion on that subject.

• On Connection close: there is an internal API:

int picoquic_connection_error_ex(picoquic_cnx_t* cnx, uint64_t local_error, uint64_t frame_type, char const* local_reason);

If you really need it, maybe we should add an extended version of picoquic_close(cnx, application_reason_code) to add a char const * reason string.

• On WT: that definitely merits its own discussion. The client can send a connect proposing both H3 and the ALPN specific to the WT protocol, but it would be nice to have documentation and tests. See also issue Support WT-Availabe-Protocols #2041

• On QLOG: yes, we should extend the logging API to log application specific events, using the qlog extension processes. We should probably start by doing that for H3. This probably requires first moving to QLOG version2. In between, I have been making do with the API void picoquic_log_app_message(picoquic_cnx_t* cnx, const char* fmt, ...);, which abuses QLOG "status" messages. That's definitely worth its own specific discussion.

Note that i am more than willing to review PRs. Like for the extended close API, that's a simple one!

[lminiero]

@huitema thanks for the quick answer!

• On -fPIC: isn't it sufficient to call cmake with the option "-DCMAKE_POSITION_INDEPENDENT_CODE=ON" ?

Whoops, I was sure I had tried that already, but it turns out I didn't, as that did indeed do the trick! No, I don't need picoquic to be a shared library itself, I just need to be able to include it as part of a shared library, so just being able to compile with -fPIC support is enough.

• On Connection close: there is an internal API:

int picoquic_connection_error_ex(picoquic_cnx_t* cnx, uint64_t local_error, uint64_t frame_type, char const* local_reason);

If you really need it, maybe we should add an extended version of picoquic_close(cnx, application_reason_code) to add a char const * reason string.

Thanks, I'll have a look! I guess that, due to the asynchronous nature of the API, this will require somehow storing the string somewhere, at least temporarily (or maybe a bit longer, in case extended versions of picoquic_get_close_reasons may be needed too). Once I feel more comfortable touching the picoquic code itself I'll see what I can do.

• On WT: that definitely merits its own discussion. The client can send a connect proposing both H3 and the ALPN specific to the WT protocol, but it would be nice to have documentation and tests. See also issue Support WT-Availabe-Protocols #2041

Thanks for the clarification! As I mentioned, I haven't really done a deep dive on the picoquic WT support at the moment, but I'll make sure to open further discussions should I have feedback or contributions on that.

• On QLOG: yes, we should extend the logging API to log application specific events, using the qlog extension processes. We should probably start by doing that for H3. This probably requires first moving to QLOG version2. In between, I have been making do with the API void picoquic_log_app_message(picoquic_cnx_t* cnx, const char* fmt, ...);, which abuses QLOG "status" messages. That's definitely worth its own specific discussion.

Makes sense, thanks for the clarification!

Note that i am more than willing to review PRs. Like for the extended close API, that's a simple one!

I'll definitely be happy to start contributing as soon as I familiarize myself more with the internals. At the moment, I'm more at the stage where I'm learning all the different APIs, but as time goes by I'd love to be able to contribute what may be missing and/or helpful. The suggestion on starting from close is a good one, so I'll try and tackle that as soon as I feel I am able to.

Thanks again for the great work on the library and very useful feedback!