Native SDK crashes on Java_com_twilio_voice_Voice_nativeHandleMessage for incoming calls

[jamesbluecrow]

Description

We see many crashes in our app when receiving incoming calls.

The trace is not very useful so we don't really know how to approach this. Everything is done in the Main thread as the push notification arrives.

Steps to Reproduce

No clear repro steps, we haven't been able to reproduce it but we see +2k crashes weekly from this. It looks like it mostly happens for incoming calls

Expected Behavior

No crash happening.

Actual Behavior

Sometimes it the app crashes and because it's in the native code we can't really catch the exception.

Reproduces How Often

7% of our users are affected by this.

Twilio Call SID(s)

CAa37e338e2a412480cd171bdd92b1fc9a

Logs

Crashed: Thread: SIGABRT  0x0000000000000000
#00 pc 0x66b38 libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)
#01 pc 0x66b08 libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)
#02 pc 0x688bc libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)
#03 pc 0x90b10 libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)
#04 pc 0x479c80 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#05 pc 0x4e02a0 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#06 pc 0x4e241c libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#07 pc 0x55a720 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#08 pc 0x55e310 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#09 pc 0x4dee54 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#10 pc 0x4df598 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#11 pc 0x4df67c libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#12 pc 0x52d128 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#13 pc 0x4c745c libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#14 pc 0x55a464 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#15 pc 0x4701d0 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 62e0d7afc022295681860f9c7a2da6256bf21058)
#16 pc 0xcb888 libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)
#17 pc 0x68370 libc.so (BuildId: 97f8621d2a4b9ffed8f5196d93a8a78e)

Voice Android SDK

com.twilio:voice-android:6.7.0
com.twilio:audioswitch:1.2.0

OS Version

It happens in multiple Android versions: Android 11, 12, 13, 14, 15

Device Model

It happens in multiple device models. Samsung, Google Pixel, etc.

[afalls-twilio]

@jamesbluecrow Thanks for the information. It looks like a null pointer of some kind but we have not been able to reproduce it on our end. Have you been able to reproduce it at all with the quickstart?

[jamesbluecrow]

@jamesbluecrow Thanks for the information. It looks like a null pointer of some kind but we have not been able to reproduce it on our end. Have you been able to reproduce it at all with the quickstart?

No unfortunately I haven't been able to reproduce it in the quickstart, but neither I have been able to reproduce it on our own app.

In perspective we see weekly around +3k crashes affecting +2k users. The ratio is of just 1.5 crash per user and with the volume of calls we have is small so really hard to reproduce.

But the numbers are too high to ignore.

Sorry I am not being able to help giving you clear repro steps.

[jamesbluecrow]

Our code to handle the notifications though is almost a copy paste of what the quickstart project is:

@AndroidEntryPoint
class FirebaseMessagingService : FirebaseMessagingService() {

    @Inject
    lateinit var twilioNotificationHandler: TwilioNotificationHandler
    
    override fun onMessageReceived(remoteMessage: RemoteMessage) {
        val isHandledByTwilio = twilioNotificationHandler.handle(remoteMessage)
        ...
    }
}

class TwilioNotificationHandler(private val context: Context) {
    fun handle(remoteMessage: RemoteMessage): Boolean {
        val isHandledByTwilio = Voice.handleMessage(
            context,
            remoteMessage.data,
            object : MessageListener {
                override fun onCallInvite(twilioCallInvite: com.twilio.voice.CallInvite) {
                    ...
                }

                override fun onCancelledCallInvite(
                    cancelledCallInvite: com.twilio.voice.CancelledCallInvite,
                    callException: CallException?,
                ) {
                    ...
                }
            },
        )

        return isHandledByTwilio
    }
}

[afalls-twilio]

@jamesbluecrow hmm... what happens when you run your app, background it and kill it, then call it... or maybe wait 5 min before calling it. I'm wondering if there is something strange regarding lifetimes... the null pointer crash in native is very unusual.

[jamesbluecrow]

I've tried that but no luck, I don't see any problem and calls come normally.

Looking at our firebase again I see there is a different variant that has different trace, but this other variant crash has happened only 6 times in the last 7 days. I'll post it here in case they are related and it helps:

          Crashed: Thread: SIGABRT  0x0000000000000000
#00 pc 0x63332 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#01 pc 0x6465d libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#02 pc 0x814c3 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#03 pc 0x1f590 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#04 pc 0x24f079 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#05 pc 0x24ef93 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#06 pc 0x293fad libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#07 pc 0x5ce6ff libtwilio_voice_android_so.so (std::__ndk1::locale::locale(std::__ndk1::locale const&)) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#08 pc 0x5f38f6 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#09 pc 0x624dc2 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#10 pc 0x2116e7 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#11 pc 0x29589f libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#12 pc 0x61850e libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#13 pc 0x6185d6 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#14 pc 0x62f88e libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#15 pc 0x61842a libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#16 pc 0x62f88e libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#17 pc 0x2e9bf3 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#18 pc 0x663a1 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#19 pc 0x380f7 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#20 pc 0x2ec7fb libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#21 pc 0x314c9 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#22 pc 0x75ba libnetd_client.so (BuildId: e8aea903333b1c0b2eb136caedf863b8)
#23 pc 0x2ec7fb libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#24 pc 0x2ec84d libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#25 pc 0x2ec7fb libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#26 pc 0x293107 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#27 pc 0x5fa262 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#28 pc 0x5e7865 libtwilio_voice_android_so.so (__dynamic_cast) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#29 pc 0x5e7849 libtwilio_voice_android_so.so (__dynamic_cast) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#30 pc 0x5e78af libtwilio_voice_android_so.so (__dynamic_cast) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#31 pc 0x25ffe libaaudio_internal.so (BuildId: cec5d2ec9a10fa21b6c5401111827576)
#32 pc 0x32db3 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#33 pc 0x24fbd9 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#34 pc 0x28bbb9 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#35 pc 0x5f64ae libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#36 pc 0x2936b5 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#37 pc 0x624dc2 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#38 pc 0x5f38f6 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#39 pc 0xfd1cf libtwilio_voice_android_so.so (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#40 pc 0x293773 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#41 pc 0xfd1cf libtwilio_voice_android_so.so (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#42 pc 0x2caab1 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#43 pc 0x2d0993 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#44 pc 0x24f933 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#45 pc 0x2b1c45 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#46 pc 0x24fbd9 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#47 pc 0x2b22a1 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#48 pc 0x5f64ae libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#49 pc 0x24fbd9 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#50 pc 0x240695 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#51 pc 0x5f64ae libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#52 pc 0x2837d3 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#53 pc 0x283819 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#54 pc 0x2e99f7 libtwilio_voice_android_so.so (Java_com_twilio_voice_Voice_nativeHandleMessage) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#55 pc 0x61c1f6 libtwilio_voice_android_so.so (__emutls_get_address) (BuildId: 162da90a073752b4b9ebf0e35dbf3e6469c405cd)
#56 pc 0xb573a libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)
#57 pc 0x5cc17 libc.so (BuildId: 82f892fd1b7c9384a320c9a49d4c95fc)

[afalls-twilio]

@jamesbluecrow that call stack looks truly odd... looking at the sourceJava_com_twilio_voice_Voice_nativeHandleMessage doesn't recursively call itself but according to the call stack reported here, it does... also, there is no thread local storage but __emutls_get_address is being called...

and there has been no luck in reproducing the issue at all, any usage patterns? we could not reproduce it with the QuickStart on our end either

Actually, Can you check something for me? what is in onCallInvite and onCancelledCallInvite? is it making any subsequent SDK calls from those callback methods?

[jamesbluecrow]

Doing some quick maths we see this crash happening for around 0.5 of the calls handled by our app which makes it extremely difficult to reproduce it.

To me it does look like there is likely some kind of race condition in the SDK (I'm surprised that other apps that have integrated the SDK are not experiencing this to the same level we see).

There is no other calls to the SDK aside of twilioCallInvite.accept() if the user clicks accept/answer

[jamesbluecrow]

@afalls-twilio does Java_com_twilio_voice_Voice_nativeHandleMessage get called at all for outgoing calls?

I'm looking more into our logs and I see something that suggests that this crash not only happens for incoming alls but also for outgoing calls.

[afalls-twilio]

@jamesbluecrow The two issues you have reported, no one has reported before. Its very unlikely that a race condition is occurring if all the SDK calls are being made from the same thread. Is it possible that the thread your application is making the SDK calls from, is not a thread with a Looper? If the thread does not have a looper, then all the callbacks from the SDK get posted to the UI thread (which has a looper), which can cause multiple SDK calls coming from different threads if subsequent sdk calls are made from those callbacks.

Also, Java_com_twilio_voice_Voice_nativeHandleMessage is only called when Voice.handleMessage is called from the java layer and is used to process incoming firebase messages and it entirely synchronous, ie, it does not interact with any other threads and onCallInvite/onCancelledCallInvite callback invocation come directly out of that function.

https://twilio.github.io/twilio-voice-android/docs/6.7.1/com/twilio/voice/Voice.html#handleMessage(android.content.Context,android.os.Bundle,com.twilio.voice.MessageListener

[jamesbluecrow]

@afalls-twilio as you can see in the code I posted we don't really have threads playing a part here #641 (comment) . This code is copied from our project (I only removed the parts that are not relevant).

I don't have data to prove my suspicion, but looking at one specific crash for one of our users it does suggest that we got registered the crash in firebase right after the user tried an outgoing call. Anyway all the other crashes breadcrumbs I see seem to be always when incoming calls so maybe I'm just misreading the logs.