Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

One way audio when Kernelizing media stream #1840

Open
jmordica opened this issue Jul 26, 2024 · 7 comments
Open

One way audio when Kernelizing media stream #1840

jmordica opened this issue Jul 26, 2024 · 7 comments
Labels
bug

Comments

@jmordica
Copy link

rtpengine version the issue has been seen with

11.2.1.4+0~mr11.2.1.4 git-HEAD-1d93d9b4

Used distribution and its version

Ubuntu 22.04.3 LTS

Linux kernel version used

5.15.0-1048-gke

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

Audio continuing after entering kernel mode

Unexpected behaviour you saw

When kernelizing media stream happens (3-5 seconds after the call starts) one-way audio occurs.

Steps to reproduce the problem

Can reproduce on every call.

Additional program output to the terminal or logs illustrating the issue

[1722008079.782316] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Received command 'offer' from 10.128.15.217:44476
[1722008079.782391] NOTICE: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Creating new call
[1722008079.783197] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Replying to 'offer' from 10.128.15.217:44476 (elapsed time 0.000863 sec)
[1722008081.345332] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Received command 'answer' from 10.128.15.217:51727
[1722008081.346186] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Replying to 'answer' from 10.128.15.217:51727 (elapsed time 0.000832 sec)
[1722008081.531392] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Received command 'answer' from 10.128.15.217:60911
[1722008081.532306] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Replying to 'answer' from 10.128.15.217:60911 (elapsed time 0.000888 sec)
[1722008082.722700] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Received command 'answer' from 10.128.15.217:47254
[1722008082.723544] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Replying to 'answer' from 10.128.15.217:47254 (elapsed time 0.000825 sec)
[1722008086.013362] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20138]: [core] Confirmed peer address as 216.221.155.73:34438
[1722008086.013406] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20138]: [core] Kernelizing media stream: 216.221.155.73:34438 -> 10.206.15.219:20138 | 10.206.15.219:20158 -> 10.12.1.2:14008
[1722008086.015473] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20158]: [core] Confirmed peer address as 10.12.1.2:14008
[1722008086.015524] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20158]: [core] Kernelizing media stream: 10.12.1.2:14008 -> 10.206.15.219:20158 | 10.206.15.219:20138 -> 216.221.155.73:34438
[1722008086.369899] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20159]: [core] Confirmed peer address as 10.12.1.2:14009
[1722008086.369987] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e port 20159]: [core] Kernelizing media stream: 10.12.1.2:14009 -> 10.206.15.219:20159 | 10.206.15.219:20139 -> 216.221.155.73:34439
[1722008090.542910] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Received command 'delete' from 10.128.15.217:41376
[1722008090.543055] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Deleting call branch 'eb131e75-9998-4894-b8b7-ef3e20446bb6' (via-branch '')
[1722008090.543066] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Removing media stream from kernel: local 10.206.15.219:20158
[1722008090.543080] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Removing media stream from kernel: local 10.206.15.219:20138
[1722008090.543087] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Removing media stream from kernel: local 10.206.15.219:20159
[1722008090.543115] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Deleting call branch 'gK0485904b' (via-branch 'z9hG4bKPj9c61272d-2cfd-4a67-a3de-c0779b4071c40')
[1722008090.543132] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Deleting entire call
[1722008090.543433] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] Final packet stats:
[1722008090.543530] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --- Tag 'eb131e75-9998-4894-b8b7-ef3e20446bb6', created 0:11 ago for branch ''
[1722008090.543537] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ---     subscribed to 'gK0485904b'
[1722008090.543541] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ---     subscription for 'gK0485904b'
[1722008090.543548] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ------ Media #1 (audio over RTP/AVP) using PCMU/8000
[1722008090.543556] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --------- Port   10.206.15.219:20158 <>       10.12.1.2:14008, SSRC 67818910, in 162 p, 27864 b, 0 e, 3 ts, out 432 p, 74304 b, 0 e
[1722008090.543562] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --------- Port   10.206.15.219:20159 <>       10.12.1.2:14009 (RTCP), SSRC 67818910, in 1 p, 100 b, 0 e, 4 ts, out 0 p, 0 b, 0 e
[1722008090.543569] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --- Tag 'gK0485904b', created 0:11 ago for branch 'z9hG4bKPj9c61272d-2cfd-4a67-a3de-c0779b4071c40'
[1722008090.543572] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ---     subscribed to 'eb131e75-9998-4894-b8b7-ef3e20446bb6'
[1722008090.543575] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ---     subscription for 'eb131e75-9998-4894-b8b7-ef3e20446bb6'
[1722008090.543582] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] ------ Media #1 (audio over RTP/AVP) using PCMU/8000
[1722008090.543587] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --------- Port   10.206.15.219:20138 <>  216.221.155.73:34438, SSRC 93c7fa89, in 432 p, 74304 b, 0 e, 0 ts, out 162 p, 27864 b, 0 e
[1722008090.543594] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [core] --------- Port   10.206.15.219:20139 <>  216.221.155.73:34439 (RTCP), SSRC 0, in 0 p, 0 b, 0 e, 8 ts, out 1 p, 100 b, 0 e
[1722008090.544326] INFO: [1e8344c1-4aaa-43b9-af71-fada6348ef5e]: [control] Replying to 'delete' from 10.128.15.217:41376 (elapsed time 0.001369 sec)

Results of cat /proc/rtpengine/0/list

local inet4 10.206.15.219:20114
    expect inet4 10.12.1.2:12426
    src mismatch action: drop
    stats:                  172 bytes,                    1 packets,                    0 errors
        RTP payload type   0:                  172 bytes,                    1 packets
    SSRC in: 694544a4
    option: RTP
    option: PT filter
    option: SSRC tracking
    output #0
       src inet4 10.206.15.219:20124
       dst inet4 142.190.3.138:4012
      stats:                  172 bytes,                    1 packets,                    0 errors
local inet4 10.206.15.219:20115
    expect inet4 10.12.1.2:12427
    src mismatch action: drop
    stats:                    0 bytes,                    0 packets,                    0 errors
    SSRC in: 694544a4
    option: RTP
    option: non forwarding
    option: SSRC tracking
local inet4 10.206.15.219:20124
    expect inet4 142.190.3.138:4012
    src mismatch action: drop
    stats:                17372 bytes,                  101 packets,                    0 errors
        RTP payload type   0:                17372 bytes,                  101 packets
        RTP payload type 101:                    0 bytes,                    0 packets
    SSRC in: 728c6747
    option: RTP
    option: PT filter
    option: SSRC tracking
    output #0
       src inet4 10.206.15.219:20114
       dst inet4 10.12.1.2:12426
      stats:                17372 bytes,                  101 packets,                    0 errors
local inet4 10.206.15.219:20125
    expect inet4 142.190.3.138:4013
    src mismatch action: drop
    stats:                    0 bytes,                    0 packets,                    0 errors
    SSRC in: 728c6747
    option: RTP
    option: non forwarding
    option: SSRC tracking
local inet4 10.206.15.219:20138
    expect inet4 216.221.155.73:34438
    src mismatch action: drop
    stats:                17372 bytes,                  101 packets,                    0 errors
        RTP payload type   0:                17372 bytes,                  101 packets
    SSRC in: 93c7fa89, 282beac
    option: RTP
    option: PT filter
    option: SSRC tracking
    output #0
       src inet4 10.206.15.219:20158
       dst inet4 10.12.1.2:14008
      stats:                17372 bytes,                  101 packets,                    0 errors
local inet4 10.206.15.219:20158
    expect inet4 10.12.1.2:14008
    src mismatch action: drop
    stats:                  172 bytes,                    1 packets,                    0 errors
        RTP payload type   0:                  172 bytes,                    1 packets
    SSRC in: 67818910
    option: RTP
    option: PT filter
    option: SSRC tracking
    output #0
       src inet4 10.206.15.219:20138
       dst inet4 216.221.155.73:34438
      stats:                  172 bytes,                    1 packets,                    0 errors
local inet4 10.206.15.219:20159
    expect inet4 10.12.1.2:14009
    src mismatch action: drop
    stats:                    0 bytes,                    0 packets,                    0 errors
    SSRC in: 67818910
    option: RTP
    option: non forwarding
    option: SSRC tracking


Other leg:

[1722008079.724665] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Received command 'offer' from 10.128.15.217:50951
[1722008079.724762] NOTICE: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Creating new call
[1722008079.725665] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Replying to 'offer' from 10.128.15.217:50951 (elapsed time 0.000979 sec)
[1722008082.747001] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Received command 'answer' from 10.128.15.217:41671
[1722008082.747769] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Replying to 'answer' from 10.128.15.217:41671 (elapsed time 0.000748 sec)
[1722008086.009228] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20114]: [core] Confirmed peer address as 10.12.1.2:12426
[1722008086.009307] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20114]: [core] Kernelizing media stream: 10.12.1.2:12426 -> 10.206.15.219:20114 | 10.206.15.219:20124 -> 142.190.3.138:4012
[1722008086.020324] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20124]: [core] Confirmed peer address as 142.190.3.138:4012
[1722008086.020376] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20124]: [core] Kernelizing media stream: 142.190.3.138:4012 -> 10.206.15.219:20124 | 10.206.15.219:20114 -> 10.12.1.2:12426
[1722008087.577916] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20125]: [core] Confirmed peer address as 142.190.3.138:4013
[1722008087.577987] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20125]: [core] Kernelizing media stream: 142.190.3.138:4013 -> 10.206.15.219:20125 | 10.206.15.219:20115 -> 10.12.1.2:12427
[1722008087.749392] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20115]: [core] Confirmed peer address as 10.12.1.2:12427
[1722008087.749466] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx port 20115]: [core] Kernelizing media stream: 10.12.1.2:12427 -> 10.206.15.219:20115 | 10.206.15.219:20125 -> 142.190.3.138:4013
[1722008090.522533] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Received command 'delete' from 10.128.15.217:34650
[1722008090.522659] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Deleting call branch 'egxMm-CpY-xfSYsxYnkpndcE2V50T0.l' (via-branch '')
[1722008090.522676] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Removing media stream from kernel: local 10.206.15.219:20124
[1722008090.522688] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Removing media stream from kernel: local 10.206.15.219:20114
[1722008090.522697] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Removing media stream from kernel: local 10.206.15.219:20125
[1722008090.522702] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Removing media stream from kernel: local 10.206.15.219:20115
[1722008090.522728] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Deleting call branch 'e283b39a-1c6e-4d2e-babb-fe4097a56ad7' (via-branch 'z9hG4bKPjgKVhXvSe3ZIa4.eraGnEQzx70swzh.W60')
[1722008090.522740] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Deleting entire call
[1722008090.523101] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] Final packet stats:
[1722008090.523113] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --- Tag 'egxMm-CpY-xfSYsxYnkpndcE2V50T0.l', created 0:11 ago for branch ''
[1722008090.523116] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ---     subscribed to 'e283b39a-1c6e-4d2e-babb-fe4097a56ad7'
[1722008090.523118] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ---     subscription for 'e283b39a-1c6e-4d2e-babb-fe4097a56ad7'
[1722008090.523124] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ------ Media #1 (audio over RTP/AVP) using PCMU/8000
[1722008090.523138] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --------- Port   10.206.15.219:20124 <>   142.190.3.138:4012 , SSRC 728c6747, in 361 p, 62092 b, 0 e, 0 ts, out 165 p, 28380 b, 0 e
[1722008090.523144] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --------- Port   10.206.15.219:20125 <>   142.190.3.138:4013  (RTCP), SSRC 728c6747, in 4 p, 280 b, 0 e, 3 ts, out 1 p, 100 b, 0 e
[1722008090.523149] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --- Tag 'e283b39a-1c6e-4d2e-babb-fe4097a56ad7', created 0:11 ago for branch 'z9hG4bKPjgKVhXvSe3ZIa4.eraGnEQzx70swzh.W60'
[1722008090.523152] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ---     subscribed to 'egxMm-CpY-xfSYsxYnkpndcE2V50T0.l'
[1722008090.523155] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ---     subscription for 'egxMm-CpY-xfSYsxYnkpndcE2V50T0.l'
[1722008090.523161] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] ------ Media #1 (audio over RTP/AVP) using PCMU/8000
[1722008090.523170] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --------- Port   10.206.15.219:20114 <>       10.12.1.2:12426, SSRC 694544a4, in 165 p, 28380 b, 0 e, 3 ts, out 361 p, 62092 b, 0 e
[1722008090.523175] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [core] --------- Port   10.206.15.219:20115 <>       10.12.1.2:12427 (RTCP), SSRC 694544a4, in 1 p, 100 b, 0 e, 3 ts, out 4 p, 280 b, 0 e
[1722008090.523803] INFO: [qp1McBxQmj.TueQaVEpY0z28-pihSvFx]: [control] Replying to 'delete' from 10.128.15.217:34650 (elapsed time 0.001228 sec)

Anything else?

rtpengine.conf:

[rtpengine]
interface=internal/10.206.15.219;external/10.206.15.219!34.174.151.15
listen-ng=10.206.15.219:22222
foreground=true
log-stderr=true
port-min=10001
port-max=59999
recording-dir=/tmp
recording-method=pcap
recording-format=eth
log-level=5
delete-delay=0
offer-timeout=1800
final-timeout=21600
listen-http=10.206.15.219:22223
redis=10.206.15.219:6379/1
redis-connect-timeout=3000

This call is controlled by kamailio.
@jmordica jmordica added the bug label Jul 26, 2024
@rfuchs
Copy link
Member

rfuchs commented Jul 26, 2024

Try with a supported version, either 11.5 LTS or current 12.4 or master.

@jmordica
Copy link
Author

ok will try with 12.4 and report back.

@jmordica
Copy link
Author

Tried with 12.4.1.7+0~mr12.4.1.7 git-HEAD-07244a2f
Same results.

It's very strange that this all of a sudden started happening with no change to the vm. Non-kernel mode works fine.

@jmordica
Copy link
Author

Confirmed that the kernel module is running but this is what it shows when doing cat /proc/rtpengine/0/list

local inet4 10.128.15.216:13178
    expect inet4 216.221.155.74:14552
    src mismatch action: drop
    stats:               104060 bytes,                  605 packets,                    0 errors
        RTP payload type   0:               104060 bytes,                  605 packets
    last packet: 1722091146    SSRC in: 38686aaa [seq 605/0], e8d69f04 [seq 238/0]
    options: RTP PT-filter SSRC-tracking forward-RTCP
    output #0
       src inet4 10.128.15.216:14726
       dst inet4 10.12.4.2:17272
      stats:               104060 bytes,                  605 packets,                    0 errors
 SSRC out: 0 [seq 605+0/0], 0 [seq 238+0/0]
local inet4 10.128.15.216:14726
    expect inet4 10.12.4.2:17272
    src mismatch action: drop
    stats:                29412 bytes,                  171 packets,                    0 errors
        RTP payload type   0:                29412 bytes,                  171 packets
    last packet: 0    SSRC in: 7f91786 [seq 19193/0]
    options: RTP PT-filter SSRC-tracking forward-RTCP
    output #0
       src inet4 10.128.15.216:13178
       dst inet4 216.221.155.74:14552
      stats:                29412 bytes,                  171 packets,                    0 errors
 SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:14727
    expect inet4 10.12.4.2:17273
    src mismatch action: drop
    stats:                  180 bytes,                    2 packets,                    0 errors
    last packet: 0    SSRC in: 7f91786 [seq 19193/0]
    options: RTP RTCP SSRC-tracking forward-RTCP
    output #0 (RTCP)
       src inet4 10.128.15.216:13179
       dst inet4 216.221.155.74:14553
      stats:                  180 bytes,                    2 packets,                    0 errors
 SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:16752
    expect inet4 104.3.145.160:4006
    src mismatch action: drop
    stats:                60888 bytes,                  354 packets,                    0 errors
        RTP payload type   0:                60888 bytes,                  354 packets
        RTP payload type 101:                    0 bytes,                    0 packets
    last packet: 1722091146    SSRC in: 48df170e [seq 29525/0]
    options: RTP PT-filter SSRC-tracking forward-RTCP
    output #0
       src inet4 10.128.15.216:29332
       dst inet4 10.12.4.2:10730
      stats:                60888 bytes,                  354 packets,                    0 errors
 SSRC out: 0 [seq 29525+0/0]
local inet4 10.128.15.216:16753
    expect inet4 104.3.145.160:4007
    src mismatch action: drop
    stats:                  280 bytes,                    4 packets,                    0 errors
    last packet: 0    SSRC in: 48df170e [seq 29525/0]
    options: RTP RTCP SSRC-tracking forward-RTCP
    output #0 (RTCP)
       src inet4 10.128.15.216:29333
       dst inet4 10.12.4.2:10731
      stats:                  280 bytes,                    4 packets,                    0 errors
 SSRC out: 0 [seq 29525+0/0]
local inet4 10.128.15.216:29332
    expect inet4 10.12.4.2:10730
    src mismatch action: drop
    stats:                29756 bytes,                  173 packets,                    0 errors
        RTP payload type   0:                29756 bytes,                  173 packets
    last packet: 0    SSRC in: 7c8a8fa2 [seq 10923/0]
    options: RTP PT-filter SSRC-tracking forward-RTCP
    output #0
       src inet4 10.128.15.216:16752
       dst inet4 104.3.145.160:4006
      stats:                29756 bytes,                  173 packets,                    0 errors
 SSRC out: 0 [seq 0+0/0]
local inet4 10.128.15.216:29333
    expect inet4 10.12.4.2:10731
    src mismatch action: drop
    stats:                  100 bytes,                    1 packets,                    0 errors
    last packet: 0    SSRC in: 7c8a8fa2 [seq 10923/0]
    options: RTP RTCP SSRC-tracking forward-RTCP
    output #0 (RTCP)
       src inet4 10.128.15.216:16753
       dst inet4 104.3.145.160:4007
      stats:                  100 bytes,                    1 packets,                    0 errors
 SSRC out: 0 [seq 0+0/0]

Also confirmed nft list ruleset:

# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
	chain DOCKER {
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		XT match comment not found
 counter packets 507 bytes 34625 jump CILIUM_POST_nat
		XT match comment not found
 counter packets 734 bytes 49186 jump KUBE-POSTROUTING
		XT match comment not found
 XT match addrtype not found
 counter packets 657 bytes 44777 jump IP-MASQ
	}

	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
		XT match comment not found
 counter packets 17 bytes 1007 jump CILIUM_PRE_nat
		iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 XT match tcp not found
 XT match comment not found
 counter packets 0 bytes 0 XT target DNAT not found

		iifname != "eth0" meta l4proto tcp ip daddr 169.254.169.254 XT match tcp not found
 XT match comment not found
 counter packets 0 bytes 0 XT target DNAT not found

	}

	chain OUTPUT {
		type nat hook output priority -100; policy accept;
		XT match comment not found
 counter packets 507 bytes 34625 jump CILIUM_OUTPUT_nat
	}

	chain IP-MASQ {
		ip daddr 169.254.0.0/16 XT match comment not found
 counter packets 463 bytes 33137 return
		ip daddr 10.0.0.0/8 XT match comment not found
 counter packets 34 bytes 2040 return
		ip daddr 172.16.0.0/12 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 192.168.0.0/16 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 240.0.0.0/4 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 192.0.2.0/24 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 198.51.100.0/24 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 203.0.113.0/24 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 100.64.0.0/10 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 198.18.0.0/15 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 192.0.0.0/24 XT match comment not found
 counter packets 0 bytes 0 return
		ip daddr 192.88.99.0/24 XT match comment not found
 counter packets 0 bytes 0 return
		XT match comment not found
 counter packets 160 bytes 9600 XT target MASQUERADE not found

	}

	chain KUBE-MARK-DROP {
		counter packets 0 bytes 0 XT target MARK not found

	}

	chain KUBE-MARK-MASQ {
		counter packets 0 bytes 0 XT target MARK not found

	}

	chain KUBE-POSTROUTING {
		XT match mark not found
 counter packets 731 bytes 48858 return
		counter packets 0 bytes 0 XT target MARK not found

		XT match comment not found
 counter packets 0 bytes 0 XT target MASQUERADE not found

	}

	chain KUBE-KUBELET-CANARY {
	}

	chain CILIUM_POST_nat {
	}

	chain CILIUM_OUTPUT_nat {
	}

	chain CILIUM_PRE_nat {
	}
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
	chain DOCKER {
	}

	chain DOCKER-ISOLATION-STAGE-1 {
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
		counter packets 0 bytes 0 return
	}

	chain DOCKER-ISOLATION-STAGE-2 {
		oifname "docker0" counter packets 0 bytes 0 drop
		counter packets 0 bytes 0 return
	}

	chain FORWARD {
		type filter hook forward priority filter; policy drop;
		XT match comment not found
 counter packets 0 bytes 0 jump CILIUM_FORWARD
		counter packets 0 bytes 0 jump DOCKER-USER
		counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
		oifname "docker0" XT match conntrack not found
 counter packets 0 bytes 0 accept
		oifname "docker0" counter packets 0 bytes 0 jump DOCKER
		iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
		iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
		meta l4proto tcp counter packets 0 bytes 0 accept
		meta l4proto udp counter packets 0 bytes 0 accept
		meta l4proto icmp counter packets 0 bytes 0 accept
		meta l4proto sctp counter packets 0 bytes 0 accept
	}

	chain DOCKER-USER {
		counter packets 0 bytes 0 return
	}

	chain KUBE-FIREWALL {
		ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8 XT match comment not found
 XT match conntrack not found
 counter packets 0 bytes 0 drop
		XT match comment not found
 XT match mark not found
 counter packets 0 bytes 0 drop
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		XT match comment not found
 counter packets 38137 bytes 7488379 jump CILIUM_OUTPUT
		counter packets 49755 bytes 8309024 jump KUBE-FIREWALL
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
		ip protocol udp counter packets 14 bytes 2318 jump rtpengine
		XT match comment not found
 counter packets 90180 bytes 929670982 jump CILIUM_INPUT
		counter packets 125289 bytes 1267916661 jump KUBE-FIREWALL
	}

	chain KUBE-KUBELET-CANARY {
	}

	chain CILIUM_INPUT {
		meta mark & 0x00000f00 == 0x00000200 XT match comment not found
 counter packets 0 bytes 0 accept
	}

	chain CILIUM_OUTPUT {
		meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
 counter packets 0 bytes 0 accept
		meta mark & 0x00000e00 == 0x00000800 XT match comment not found
 counter packets 0 bytes 0 accept
		meta mark & 0x00000f00 != 0x00000e00 meta mark & 0x00000f00 != 0x00000d00 meta mark & 0x00000e00 != 0x00000a00 meta mark & 0x00000e00 != 0x00000800 meta mark & 0x00000f00 != 0x00000f00 XT match comment not found
 counter packets 38137 bytes 7488379 XT target MARK not found

	}

	chain CILIUM_FORWARD {
		oifname "cilium_host" XT match comment not found
 counter packets 0 bytes 0 accept
		iifname "cilium_host" XT match comment not found
 counter packets 0 bytes 0 accept
		iifname "lxc*" XT match comment not found
 counter packets 0 bytes 0 accept
		iifname "cilium_net" XT match comment not found
 counter packets 0 bytes 0 accept
		oifname "lxc*" XT match comment not found
 counter packets 0 bytes 0 accept
		iifname "lxc*" XT match comment not found
 counter packets 0 bytes 0 accept
	}

	chain rtpengine {
		XT target RTPENGINE not found
 counter packets 14 bytes 2318
	}
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
	chain OUTPUT {
		type route hook output priority mangle; policy accept;
		meta l4proto tcp ip saddr 169.254.169.254 XT match tcp not found
 counter packets 0 bytes 0 accept
		meta l4proto udp ip saddr 169.254.169.254 XT match udp not found
 counter packets 0 bytes 0 accept
		ip saddr 169.254.169.254 counter packets 0 bytes 0 drop
	}

	chain KUBE-IPTABLES-HINT {
	}

	chain KUBE-KUBELET-CANARY {
	}

	chain CILIUM_POST_mangle {
	}

	chain CILIUM_PRE_mangle {
		XT match socket not found
 XT match comment not found
 counter packets 0 bytes 0 XT target MARK not found

		meta l4proto tcp meta mark 0x07940200 XT match comment not found
 counter packets 0 bytes 0 XT target TPROXY not found

		meta l4proto udp meta mark 0x07940200 XT match comment not found
 counter packets 0 bytes 0 XT target TPROXY not found

	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
		XT match comment not found
 counter packets 38137 bytes 7488379 jump CILIUM_POST_mangle
	}

	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
		XT match comment not found
 counter packets 90180 bytes 929670982 jump CILIUM_PRE_mangle
	}
}
table ip6 mangle {
	chain KUBE-IPTABLES-HINT {
	}

	chain KUBE-KUBELET-CANARY {
	}
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
	chain KUBE-MARK-DROP {
		counter packets 0 bytes 0 XT target MARK not found

	}

	chain KUBE-MARK-MASQ {
		counter packets 0 bytes 0 XT target MARK not found

	}

	chain KUBE-POSTROUTING {
		XT match mark not found
 counter packets 3 bytes 240 return
		counter packets 0 bytes 0 XT target MARK not found

		XT match comment not found
 counter packets 0 bytes 0 XT target MASQUERADE not found

	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		XT match comment not found
 counter packets 3 bytes 240 jump KUBE-POSTROUTING
	}

	chain KUBE-KUBELET-CANARY {
	}
}
# Warning: table ip6 filter is managed by iptables-nft, do not touch!
table ip6 filter {
	chain KUBE-FIREWALL {
		XT match comment not found
 XT match mark not found
 counter packets 0 bytes 0 drop
	}

	chain KUBE-KUBELET-CANARY {
	}

	chain rtpengine {
		XT target RTPENGINE not found
 counter packets 0 bytes 0
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
		ip6 nexthdr udp counter packets 0 bytes 0 jump rtpengine
	}
}
# Warning: table ip raw is managed by iptables-nft, do not touch!
table ip raw {
	chain CILIUM_OUTPUT_raw {
		oifname "lxc*" meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
 counter packets 0 bytes 0 XT target CT not found

		oifname "cilium_host" meta mark & 0xfffffeff == 0x00000a00 XT match comment not found
 counter packets 0 bytes 0 XT target CT not found

		oifname "lxc*" meta mark & 0x00000e00 == 0x00000800 XT match comment not found
 counter packets 0 bytes 0 XT target CT not found

		oifname "cilium_host" meta mark & 0x00000e00 == 0x00000800 XT match comment not found
 counter packets 0 bytes 0 XT target CT not found

	}

	chain CILIUM_PRE_raw {
		meta mark & 0x00000f00 == 0x00000200 XT match comment not found
 counter packets 0 bytes 0 XT target CT not found

	}

	chain OUTPUT {
		type filter hook output priority raw; policy accept;
		XT match comment not found
 counter packets 38137 bytes 7488379 jump CILIUM_OUTPUT_raw
	}

	chain PREROUTING {
		type filter hook prerouting priority raw; policy accept;
		XT match comment not found
 counter packets 90180 bytes 929670982 jump CILIUM_PRE_raw
	}
}

Still no dice.

@jmordica
Copy link
Author

One notable observation:
The environment is GKE and has been running for years. When the kubernetes master node got upgraded from 1.27 to 1.28 the rtpengine worker node immediately stopped working in kernel mode. The worker node itself didn't update to v1.28. It was still on the previous version with no restarts when the issue occurred.

The worker node has been moved to 1.28 and the issue still happens when the media is sent to the kernel. Have tried both 11.5 and 12.4.

@rfuchs
Copy link
Member

rfuchs commented Jul 29, 2024

Might be something specific to your VM setup then, and/or some interaction with other nft rules.

@jmordica
Copy link
Author

Right. Not seeing any conflicts in the ruleset above?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jmordica @rfuchs