mavsdk_server errors when running in container

[random82]

Hi,

I have problems with running mavsdk_server from within a Docker container. My stack includes PX4 Autopilot 1.20 running on px4io/px4-dev-simulation-focal:latest image and MAVSDK 0.40.0 running on ubuntu:20.04

I broadcast MAVLINK commands to other containers with this command.

pxh>mavlink start -p -o 14560 -u 14560

The mavsdk_server connects but runs with errors. I can connect to gRPC endpoint but the stream refreshes every minute or so.
What do I miss?

mavsdk image:

FROM ubuntu:20.04 

RUN apt-get update && \
    apt-get install -y python3 \
                       python3-pip && \
    apt-get -y autoremove && \
    apt-get clean autoclean && \
    rm -rf /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*
RUN pip3 install mavsdk
WORKDIR /usr/local/lib/python3.8/dist-packages/mavsdk/bin/mavsdk_server
EXPOSE 50051

mavsdk_server log:

root@3cd878262cc4:~# mavsdk_server udp://0.0.0.0:14560
[12:38:12|Info ] MAVSDK version: v0.40.0 (mavsdk_impl.cpp:21)
[12:38:12|Debug] New: System ID: 0 Comp ID: 0 (mavsdk_impl.cpp:538)
[12:38:12|Error] Sending message failed (mavsdk_impl.cpp:261)
[12:38:12|Info ] Server started (grpc_server.cpp:48)
[12:38:12|Info ] Server set to listen on 0.0.0.0:50051 (grpc_server.cpp:49)
[12:38:12|Info ] Waiting to discover system on udp://0.0.0.0:14560... (connection_initiator.h:21)
[12:39:13|Info ] New system on: 172.17.0.2:14560 (with sysid: 1) (udp_connection.cpp:191)
[12:39:13|Debug] Component Autopilot (1) added. (system_impl.cpp:354)
[12:39:14|Debug] Setting UUID to: 5283920058631409231 (system_impl.cpp:222)
[12:39:14|Debug] Discovered 1 component(s) (UUID: 5283920058631409231) (system_impl.cpp:511)
[12:39:14|Info ] System discovered (connection_initiator.h:57)
[12:39:15|Warn ] sending again after 0.501607 s, retries to do: 3  (511). (mavlink_commands.cpp:239)
[12:39:15|Warn ] sending again, retries to do: 3  (SYS_FAILURE_EN). (mavlink_parameters.cpp:565)
[12:39:15|Debug] Falling back to Gimbal Version 1 (gimbal_impl.cpp:58)
[12:39:15|Debug] Falling back to gimbal protocol v1 (mission_impl.cpp:129)
[12:39:15|Warn ] sending again, retries to do: 2  (SYS_FAILURE_EN). (mavlink_parameters.cpp:565)
[12:39:15|Warn ] sending again after 1.00642 s, retries to do: 2  (511). (mavlink_commands.cpp:239)
[12:39:16|Warn ] sending again after 1.51094 s, retries to do: 1  (511). (mavlink_commands.cpp:239)
[12:39:16|Warn ] sending again, retries to do: 1  (SYS_FAILURE_EN). (mavlink_parameters.cpp:565)
[12:39:16|Error] Error: Retrying failed get param busy timeout: SYS_FAILURE_EN (mavlink_parameters.cpp:581)
[12:39:16|Error] Retrying failed (511) (mavlink_commands.cpp:260)
[12:39:17|Warn ] sending again after 0.501853 s, retries to do: 3  (512). (mavlink_commands.cpp:239)
[12:39:17|Warn ] sending again, retries to do: 3  (NAV_MIN_FT_HT). (mavlink_parameters.cpp:565)
[12:39:17|Info ] heartbeats timed out (system_impl.cpp:266)
[12:39:17|Debug] Lost 5283920058631409231 (mavsdk_impl.cpp:561)
[12:39:17|Info ] System timed out (connection_initiator.h:61)
[12:39:17|Warn ] sending again, retries to do: 2  (NAV_MIN_FT_HT). (mavlink_parameters.cpp:565)
[12:39:17|Warn ] sending again after 1.00698 s, retries to do: 2  (512). (mavlink_commands.cpp:239)
[12:39:18|Warn ] sending again after 1.51208 s, retries to do: 1  (512). (mavlink_commands.cpp:239)
[12:39:18|Warn ] sending again, retries to do: 1  (NAV_MIN_FT_HT). (mavlink_parameters.cpp:565)
[12:39:18|Debug] Discovered 1 component(s) (UUID: 5283920058631409231) (system_impl.cpp:511)
[12:39:18|Error] Error: Retrying failed get param busy timeout: NAV_MIN_FT_HT (mavlink_parameters.cpp:581)
[12:39:18|Error] Retrying failed (512) (mavlink_commands.cpp:260)
[12:39:19|Warn ] sending again after 0.500783 s, retries to do: 3  (520). (mavlink_commands.cpp:239)
[12:39:19|Warn ] sending again, retries to do: 3  (NAV_FT_DST). (mavlink_parameters.cpp:565)
[12:39:19|Debug] Falling back to gimbal protocol v1 (mission_impl.cpp:129)
[12:39:19|Debug] Falling back to Gimbal Version 1 (gimbal_impl.cpp:58)
[12:39:19|Warn ] sending again after 1.00654 s, retries to do: 2  (520). (mavlink_commands.cpp:239)
[12:39:19|Warn ] sending again, retries to do: 2  (NAV_FT_DST). (mavlink_parameters.cpp:565)
[12:39:20|Warn ] sending again, retries to do: 1  (NAV_FT_DST). (mavlink_parameters.cpp:565)
[12:39:20|Warn ] sending again after 1.5123 s, retries to do: 1  (520). (mavlink_commands.cpp:239)
[12:39:20|Error] Retrying failed (520) (mavlink_commands.cpp:260)
[12:39:20|Error] Error: Retrying failed get param busy timeout: NAV_FT_DST (mavlink_parameters.cpp:581)
[12:39:21|Warn ] sending again after 0.503187 s, retries to do: 3  (528). (mavlink_commands.cpp:239)
[12:39:21|Warn ] sending again, retries to do: 3  (NAV_FT_FS). (mavlink_parameters.cpp:565)
[12:39:21|Info ] heartbeats timed out (system_impl.cpp:266)
[12:39:21|Debug] Lost 5283920058631409231 (mavsdk_impl.cpp:561)

[julianoes]

When you run all this, is the computer's CPU maxed out? I'm wondering if PX4 is running slower than realtime and hence MAVSDK sometimes "loses the connection", so sees heartbeats timing out.

[random82]

That was my suspicion, so I added more cores to WSL 2, but it didn't go over 30% CPU in the PX4 container, and MAVSDK CPU utilisation was negligible.

I eventually got it working by disabling broadcast, assigning static IPs via Docker Compose and modifying the PX4 mavlink bootstrap script to

# API/Offboard link
mavlink start -t [MAVSDK_CONTAINER_IP] -x -u $udp_offboard_port_local -r 4000000 -f -m onboard -o $udp_offboard_port_remote