Getting CONNECTION_LOST on device

[AleksandarGT]

Describe the bug

Sometimes a device loses connection to iot core and from the fleet indexing status I can see CONNECTION_LOST. I'm not sure what may cause this as there aren't any logs produced by the iot client to show that internet connection has stopped.

While IoT core shows the device as offline due to the connection lost, if I connect to the device via ssh through cloudflare I see that it's online and working as expected. The problem with connection lost goes away if I restart the iot device client.

From the logs you can see that on 2025-08-20T10:02:47.248Z I manually restarted the service for iot device client.

Logs

2025-08-20T09:17:52.586Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/home/sens/.aws-iot-device-client/aws-iot-device-client.conf' is not set to recommended value... {Permissions: {desired: 640, actual: 644}}
2025-08-20T09:17:52.588Z [INFO]  {Config.cpp}: Successfully fetched JSON config file from /home/sens/.aws-iot-device-client/aws-iot-device-client.conf : {...}
2025-08-20T09:17:52.589Z [INFO]  {Config.cpp}: Not creating directory ~/.aws-iot-device-client/sample-shadow/ since sample shadow is disabled
2025-08-20T09:17:52.589Z [INFO]  {Config.cpp}: Successfully fetched JSON config file from /home/sens/.aws-iot-device-client/aws-iot-device-client-runtime.conf : {
"runtime-config": {
    "completed-fp": true,
    "cert": "/home/sens/.aws-iot-device-client/keys/active-certificate.pem.crt",
    "key": "/home/sens/.aws-iot-device-client/keys/active-private.pem.key",
    "thing-name": "hvam-xx",
    "device-config": {
        
        }
    }
}
2025-08-20T09:17:52.589Z [INFO]  {Config.cpp}: Successfully fetched Runtime config file '~/.aws-iot-device-client/aws-iot-device-client-runtime.conf' and validated its content.
2025-08-20T09:17:52.589Z [DEBUG] {Config.cpp}: Did not find a http proxy config file /home/sens/.aws-iot-device-client/http-proxy.conf, assuming HTTP proxy is disabled on this device
2025-08-20T10:02:47.248Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: `aws-c-io: AWS_IO_SOCKET_CLOSED, socket is closed.`. Device Client will retry connection until it is successfully connected to the core. 
2025-08-20T10:03:07.450Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/home/sens/.aws-iot-device-client/aws-iot-device-client.conf' is not set to recommended value... {Permissions: {desired: 640, actual: 644}}
2025-08-20T10:03:07.450Z [INFO]  {Config.cpp}: Successfully fetched JSON config file from /home/sens/.aws-iot-device-client/aws-iot-device-client.conf : {...}

Environment (please complete the following information):

• OS: Debian
• Raspberry PI 4
• Architecture: ARM
• Device Client commit: 7f9547b

Additional context

Add any other context about the problem here.

[tony-josi-aws]

@AleksandarGT Is it possible that the internet connection to your device was interrupted for a long duration? How oftern do you experience this issue?

[AleksandarGT]

@tony-josi-aws Internet may drop for a short-duration, never more than 1-2 minutes. And we get this at least 1-2 times a day for every device we currently have in the office. I believe it also may be because the connection silently dies and the iot device client doesn't pick it up.

When I run this command on devices that are in CONNECTION_LOST state I get the following response

echo "=== 4) TCP socket state to 8883"
ss -tnp state established | awk '\''$4 ~ /:8883|:8883/ || $5 ~ /:8883|:8883/'\'' || true

PID=$(systemctl show -p MainPID --value aws-iot-device-client 2>/dev/null || echo "")
if [ -n "$PID" ] && [ "$PID" -ne 0 ]; then
  echo "--- Sockets held by IDC (pid $PID)"
  ss -tnp | grep "$PID" || true
fi

=== 4) TCP socket state to 8883
--- Sockets held by IDC (pid 7512)

But when I run it on devices that are connected I get

=== 4) TCP socket state to 8883
--- Sockets held by IDC (pid 7512)
ESTAB 0      0       192.168.0.13:46232 52.58.188.232:443   users:(("aws-iot-device-",pid=7512,fd=7)) 

So now I ran this with the idea that if the connection has been lost, it would be detected faster and killed without lingering for the default 2 hours.

# 1) Fast OS keepalives: close dead TCP in ~90s so IDC immediately reconnects
sudo tee /etc/sysctl.d/99-iot-keepalive.conf >/dev/null <<'EOF'
net.ipv4.tcp_keepalive_time=45
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.tcp_keepalive_probes=3
# also make unacked data time out in 90s (covers blackholed paths)
net.ipv4.tcp_user_timeout=90000
EOF
sudo sysctl --system

I am not sure yet if this will fix the issue.

[tony-josi-aws]

Can you share the full logs of the device for an entire day or more?

If your device stays disconnected from the internet for more than the Persistent session expiry period, which is by default 3600 Seconds (it can be updated upon request) this could happen, as the device client won't be resubscribing to the topics again when it connects back and needs to be restarted for subscribing again. [refer - AWS IoT Core message broker and protocol limits and quotas]

Just curious, how do you determine how long the device stays disconnected from the internet/AWS?

[AleksandarGT]

@tony-josi-aws I don't think it's an issue with the persistent session expiry period as the issue occurs also when we actively use the device and it's not just sitting idle without internet.

For some context: We have a 'shadow monitor' python script running on the device which performs actions depending on the incoming desired shadow value. Some of the actions require restarting the iot device client service in order to apply new configuration changes. Maybe, the often restarting of the service may cause some issues with connection but we are not sure as well.

For how do we determine how long the device stays disconnected from internet/aws - we've noticed the internet occassionally drops for 1-2 minutes in the office on our personal devices but we haven't monitored the exact duration of this.

We've applied the keep alive settings for the internet connection and we will let you know if this fixes the issue. If you can keep this issue open for upcoming 2-3 weeks so I can update you on the result.

[jasonpcarroll]

@AleksandarGT any updates?

[AleksandarGT]

@jasonpcarroll Hello, I was out of work for 2 weeks. I will get back to you next week with an update from our team if the issue still persists.

[AleksandarGT]

@jasonpcarroll I've asked the team and I can see that we still have this issue. Today I also monitored that after a set of time in the connection lost state, the iot library will throw this error

2025-10-01T11:45:50.631Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: aws-c-io: AWS_IO_SOCKET_CLOSED, socket is closed.. Device Client will retry connection until it is successfully connected to the core.

After that I saw on the dashboard that the device was again in connected state.

Is it possible that if the connection gets dropped for any reason, the iot library will not attempt to reconnect until the keep alive time expires? I believe the keep alive for the library is 20 minutes?

[AleksandarGT]

@jasonpcarroll @tony-josi-aws After observing this for prolonged time I can verify that this happens exactly because of this reason. One of the devices went offline shortly after we sent a shadow update, the device itself was online, however the connection to iot core was silently dropped. The library takes exactly 20 minutes to realize the connection is lost and tries to reconnect. Is it possible to shorten the time for keeping the connection alive to force the library to reestablish communication?

You can see from the logs that the last update received and the connection retry happened exactly 20 minutes from each other

2025-10-07T07:32:46.348Z [DEBUG] {SecureTunnelingFeature.cpp}: Subscribed to tunnel notification topic
2025-10-07T07:32:46.397Z [INFO]  {SampleShadowFeature.cpp}: Stored the latest DeviceMetadata shadow document to local successfully
2025-10-07T07:52:46.319Z [ERROR] {SharedCrtResourceManager.cpp}: MQTT Connection interrupted with error: `aws-c-io: AWS_IO_SOCKET_CLOSED, socket is closed.`. Device Client will retry connection until it is successfully connected to the core. 
2025-10-07T07:53:01.627Z [INFO]  {SharedCrtResourceManager.cpp}: MQTT connection resumed with return code: 0
2025-10-07T07:53:01.785Z [INFO]  {SampleShadowFeature.cpp}: Stored the latest DeviceMetadata shadow document to local successfully

[jasonpcarroll]

Hi @AleksandarGT,

Sorry for the later reply. I have been diving into the code/how this is setup and have determined that there is currently no way to set keep alive time from the current state of this repo. There was a PR opened for this last year that is a start of an implementation to solve this - though has been inactive for some time (#468).

This line is where keep alive time can be placed by updating the Connect call to include the Keep Alive time in seconds

aws-iot-device-client/source/SharedCrtResourceManager.cpp

Line 468 in 37b1815

if (!connection->Connect(config.thingName->c_str(), false))

.
E.G.

if (!connection->Connect(config.thingName->c_str(), false, <keep-alive-timeout-in-secs>))

Currently it is not set by the Connect call so the default is 0 seconds - on AWS IoT broker side this translates to a keep alive time of 1200 seconds according to https://docs.aws.amazon.com/general/latest/gr/iot-core.html#limits_iot, which would match what you are saying about it taking 20 minutes. Note that if you use a Keep Alive time of less than 30 it should instead default to 30.

[jasonpcarroll]

I would try setting keep-alive-timeout-in-secs to 30 and rebuilding the device client. Then let me know if that fixes the issue. I may just open a small PR that sets this to 30 and call it a day if that works for you.

[AleksandarGT]

@jasonpcarroll OK, I've applied the change to a couple of our devices. I will let you know if this fixes the problem.

[AleksandarGT]

@jasonpcarroll By the way, how would you recommend that we update the aws iot device client build on multiple devices? Should we ship an OTA sh file to patch the code and rebuild the library? Or is there a more straight forward way?