rtp to webrtc: How to handle audio/video synchronization?

[ldenoue]

Summary

is it possible to have both audio and video tracks?

Motivation

Currently the example https://github.com/pion/webrtc/tree/master/examples/rtp-to-webrtc seems to only send video tracks.

If I add another track, do we have to worry about synchronization between audio and video?

[Sean-Der]

You should also forward the Sender Reports if you want to synchronize.

a Sender Report allows you to map two different RTP streams together by using RTPTime + NTPTime. This lets you know at what absolute time something occured, then in your playback application you can buffer/playout to ensure they are synced.

[ldenoue]

Thank you!
So similar to https://github.com/ethand91/mediasoup3-record-demo/blob/master/server/src/gstreamer.js#L126 where they route both the RTP audio and video packets and the RTCP packets.
(Except in their case they record incoming audio/video WebRTC streams by sending the RTP/RTCP to gstreamer.

In my case of using Pion to send RTP to WebRTC, I would have to do the opposite: send the RTP like you already have done to the WebRTC tracks, and also get the RTCP from GStreamer to Pion.

But hang on: since my gstreamer pipeline runs on the same machine as the Pion app, it shouldn't have packet loss... So are RTCP even useful?

[Sean-Der]

I think it would be fine to have Pion generate the Sender Reports!

It will just use the wall clock (time.Now()) for NTPTime, you may have a slight drift.

[Diceshake]

How does Pion generate the Sender Reports? And how are they forwarded to WebRTC?

Is this all handled by RegisterDefaultInterceptors?

[Sean-Der]

@Diceshake Yea! RegisterDefaultInterceptors currently enables Sender+Receiver Reports