Examples stopped working after update to 2.1.0

[razvandragomirescu]

The new examples compile fine but fail to start the modem. The output on the serial monitor is

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...

It appears to freeze there. I got no errors when compiling or uploading. Any way to debug this?

[razvandragomirescu]

The old examples still work fine (on the same hardware) with version 1.2.3, so it's not a hardware issue.

[PaulZC]

Hi @razvandragomirescu ,
Do you see the same behavior on all of the Iridium examples? Do the simple examples like Example7 work?
Best wishes,
Paul

[PaulZC]

Also, please make sure you have "RedBoard Artemis ATP" selected as the board type. ("Artemis Module" will not work.)

[PaulZC]

To enable debug messages for the Iridium serial communication, change the false in this line to true:

Artemis_Global_Tracker/Software/examples/Example7_GetIMEI/Example7_GetIMEI.ino

Line 63 in 8094e14

#define DIAGNOSTICS false // Change this to true to see IridiumSBD diagnostics

[PaulZC]

Hi @razvandragomirescu ,

If you have time, please do try uploading one of the new binaries onto your board. Details are here. That way we can rule out the way the code is being compiled.

If you run the Example7 binary, you should see:

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
IMEI is 3004-------4090.
Putting the 9603N to sleep.
Disabling 9603N power...
Disabling the supercapacitor charger...
Done!

Running Example7 with DIAGNOSTICS set to true should produce:

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
Calling internalBegin
Powering on modem...
>> AT

Waiting for response OK

<< AT

OK
>> ATE1

Waiting for response OK

<< ATE1

OK
>> AT&D0

Waiting for response OK

<< AT&D0

OK
>> AT&K0

Waiting for response OK

<< AT&K0

OK
Ring alerts are enabled.
>> AT+SBDMTA=1

Waiting for response OK

<< AT+SBDMTA=1

OK
>> AT+CGMR

Waiting for response OK

<< AT+CGMR

Call Processor Version: TA16005

Modem DSP Version: 1.7 svn: 2358

DBB Version: 0x0001 (ASIC)

RFA Version: 0x0007 (SRFA2)

NVM Version: KVS

Hardware Version: BOOT07d2/9603NrevDE/04/RAW0c

BOOT Version: TA16005 (rev exported)

OK
Firmware version is TA16005
MSSTM workaround is NOT enforced.
InternalBegin: success!
>> AT+CGSN

Waiting for response OK

<< AT+CGSN

3004-------4090

OK
IMEI is 3004-------4090.
Putting the 9603N to sleep.
Powering off modem...
Disabling 9603N power...
Disabling the supercapacitor charger...
Done!

Please also check that the red 9603N POWER LED turns on when you see the "Enabling 9603N power" appear.

If you want to debug further, you can try probing on the TX1 test point on the rear of the PCB. You should see the serial AT commands being sent to the modem at 19200 Baud (3.3V). The replies - if any - will appear on RX1.

Best wishes,
Paul

[razvandragomirescu]

I have tried both on a new computer (Windows this time to rule out an issue with my Mac) and with the binaries (uploaded from Windows using the binary uploader). Same result, it freezes at "Starting modem...". The orange "charging" LED lights up briefly while the console says "Waiting for supercapacitors to charge", then the RED "9603N Power" LED lights up but the modem doesn't appear to respond.

Once again, this works fine with the older 1.2.3 firmware.

[PaulZC]

Hi @razvandragomirescu ,

Thank you for investigating this.
Just to be clear, have you tried the Example7 binary? And does that freeze at the same place?
(I'm trying to rule out an issue with re-starting Serial1 after the Artemis has been in deep sleep. Example7 does not use deep sleep.)

Thanks again,
Paul

[razvandragomirescu]

Correct, it's Example 7 that I've tried, as per your suggestion. The behaviour is the same on both Mac and Windows, both when compiling from source and when loading the binary. Maybe my AGT board is faulty somehow (although that would not explain how/why it works with 1.2.3 but not with 2.1.0)? I only have one, so there's not much I can do about it.

[razvandragomirescu]

BTW, with DIAGNOSTICS=true, it stops at

Calling internalBegin
Powering on modem...

I never get that "AT" command sent to the modem, it never displays that, it just freezes at "Powering on modem...".

[PaulZC]

Thanks again for the diagnostics.

Wow, OK. This is a head-scratcher...

Best guess, some badness is happening when the new code tries to enable the modem via the 'sleep' pin. Why? I did have to change the gnssON and gnssOFF functions in v2.0 because the power glitch issue I fixed months ago on the GNSS enable pin came back again. I wonder if this issue only appear on some pins on some chips. I will try and send you a new binary to try in the next day or two. (It means hacking the IridiumSBDi2c library as the control of the 'sleepPin' is embedded in there.)

Thank you for your patience.

[razvandragomirescu]

I've just installed and run the binary for the iridiumSleepTest, result is below:

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
Calling internalBegin
custom configureSleepPin: sleepPin configured
Powering on modem...

Still appears to freeze on "Powering on modem". Would it help if I mailed you the AGT unit I'm testing on (if you can't reproduce it on your devices)?

[PaulZC]

Hi @razvandragomirescu ,

(Wow - that was fast! I was just typing a message to you but you beat me to it!)

OK. That's very interesting. The code is executing the custom configureSleepPin because we see the custom configureSleepPin: sleepPin configured message.

https://github.com/sparkfun/SparkFun_IridiumSBD_I2C_Arduino_Library/blob/4652dc95da1e2f7425a4005bbe88dd29f3889591/src/IridiumSBD.cpp#L1106

Then we see the Powering on modem... message:

https://github.com/sparkfun/SparkFun_IridiumSBD_I2C_Arduino_Library/blob/4652dc95da1e2f7425a4005bbe88dd29f3889591/src/IridiumSBD.cpp#L1114

But it must be failing in the custom setSleepPin function - or just after it - because we do not see the custom setSleepPin: sleepPin set message.

I'm based in the UK. If you return your board to SF in Boulder, it will get to me but it will take a while. If I'm not trying your patience too much, can we try a couple more custom binaries first? I'd like to understand exactly where the code is failing. I'll upload another binary in a few minutes.

Best wishes,
Paul

[PaulZC]

Updated files are there now... Thanks!

[razvandragomirescu]

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Waiting for supercapacitors to charge...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
Calling internalBegin
custom configureSleepPin: sleepPin configured
Powering on modem...
custom setSleepPin: sleepPin tristate_disable
custom setSleepPin: sleepPin set HIGH

With the latest version.

[PaulZC]

Hi @razvandragomirescu ,

Thank you for trying the new code. It is clear that the test code is crashing when the tri-state pin is set to an output:

Artemis_Global_Tracker/Software/examples/AGT_iridiumSleep_Test/AGT_iridiumSleep_Test.ino

Line 107 in df54d12

am_hal_gpio_state_write(PinName(this->sleepPin), AM_HAL_GPIO_OUTPUT_TRISTATE_ENABLE);

The crash is actually happening during the 10ms delay after that line.

I have a board here which does the same thing on the gnssEN pin. But on my board the Iridium enable (sleepPin) works fine. On my board, I first saw the issue using v1.2 of the core. I changed the way I was driving the pin to correct it. Then with v2.1 it came back again and I had to add more changes to correct it (by making the output open-drain instead of push-pull). It does not seem to be any kind of short circuit or brown-out as I do not see a spike in the current draw when it happens. Nathan (@nseidle) thinks it might be an un-trapped interrupt - the pin could be generating an interrupt for which there is no service routine and the code crashes as a result. I will try and capture exactly what happens today using a hardware debug tool.

One more thing you could try is to make sure the Artemis bootloader is up to date. On the Uploader tool there is a second button labelled "Update Bootloader". Clicking that will update the bootloader to the latest version. Then you will need to upload the binary code again in the usual way. That might help.

We need to arrange to get a replacement board to you. How did you buy your AGT? Did you buy it directly from SparkFun?

Thanks again for all the time you've put into this - we really appreciate it.

Best wishes,
Paul

[razvandragomirescu]

I've just tried to update the bootloader and reuploaded the test, it behaves exactly the same way as before. I've bought my AGT from Distrelec in Europe. Can you email me on the address in my Github profile please if you need details (or give me an address to send them to)? Thank you!

[a-schneider-fmi]

Since the update to version 2, I experience another issue which may or may not be related: the tracker seems not to wake from deep sleep after 2 messages have been sent. From the configuration tool, I see the following messages:

[...]
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Powering down the GNSS...
Going into deep sleep until next WAKEINT (60 seconds).

And then it hangs. This happens with several TXINT settings (I tried 2 minutes, 5 minutes, 10 minutes).

I have tested example 12 to debug. It seems to work, at least it runs for more than half an hour. Do you have any suggestion how to investigate further?

[PaulZC]

Hi Andreas (@a-schneider-fmi ),

I don't think this is the same issue... Let's discuss it separately in #38.

Best wishes,
Paul

[PaulZC]

Hi Andreas (@a-schneider-fmi ) & Razvan (@razvandragomirescu ),

I now suspect you are both seeing the same issue.

I confirmed this morning that holding the Serial1 RX pin low can cause the Artemis to hang - but only with v2.1 of the Apollo3 core. Details are here: sparkfun/Arduino_Apollo3#423

When the Iridium modem is powered off - or possibly when it is first powered on - there is a good chance it holds the RX pin low causing the code hang.

I may be able to work around this by changing the code to make sure that Serial1 is only started once the modem is fully powered on. I will of course keep you both informed.

Sincere apologies for the inconvenience,
Paul

[razvandragomirescu]

@PaulZC should I still return the unit to Sparkfun or is this solvable in software? I haven't shipped mine back yet, I was just about to call FEDEX.

[PaulZC]

Hi Razvan,

It may be possible to solve this with the AGT software. I'm not certain yet if I can correct this in the AGT code or whether it needs to be corrected within the Apollo3 core. I will try and let you know if a solution in the AGT code is possible as soon as possible. Perhaps you could delay calling FedEx until then?

Very best wishes,
Paul

[jerabaul29]

Following all these discussions with much interest, many thanks for your efforts making this whole environment better :) .

A small note to @razvandragomirescu : if you do not really NEED the core v2, have you considered using the core v1 for production for now? I am still sticking to the core v1 for my use, the only thing I miss is the bluetooth, all other stuff is working nicely. The core v1 has got really quite robust. I have deployed AGTs in the Arctic in February on drift ice based on the core v1, and my instruments are still working today, 7 months later. I had never had such success with deploying instrumentation in the Arctic before. So IMHO the core v1 is definitely production ready :) .

Still looking forward to a core v2 production ready though, many thanks @PaulZC for all your efforts :) .

[PaulZC]

Hi @razvandragomirescu / @a-schneider-fmi ,

Good news! I'm 99% confident I have a solution for this. I was having difficulty understanding why Example7 was failing for Razvan, but I do now understand why. I should be able to release updated examples late today / early tomorrow.

Thank you for your patience.

Very best wishes,
Paul

[PaulZC]

Hi Razvan (@razvandragomirescu ),

I have updated the examples to include a fix for the Apollo3 Serial1 RX pin issue. Please give them a try. As usual, there are pre-compiled binaries you can upload. Or, you can compile the code yourself: please use v2.1.0 of the Apollo3 core; please update the IridiumSBDi2c library to v3.0.4.

Please let me know if this resolves your issue.

Very best wishes,
Paul

[razvandragomirescu]

I'm afraid it doesn't fix it @PaulZC :

Artemis Global Tracker
Example: Get IMEI

Please check that the Serial Monitor is set to 115200 Baud
and that the line ending is set to Newline.
Then click Send to start the example.

Enabling the supercapacitor charger...
Supercapacitors charged!
Enabling 9603N power...
Starting modem...
Calling internalBegin
configureSleepPin: sleepPin configured
Powering on modem...
setSleepPin: sleepPin set HIGH
custom IridiumSBD::beginSerialPort

Freezes at this point (I've only tried the Example 7, I can try the others if you want).

Razvan

[PaulZC]

Hi Razvan (@razvandragomirescu ),

Thank you for your persistence!

One last try... Please try Example7 again using v3.0.5 of the IridiumSBDi2c library. I have made a small change to the timing of beginSerialPort which I hope will help. I have updated the binaries to include the change - the examples themselves have not changed.

If this does not work then it really is time to call FedEx.

Very best wishes,
Paul

[razvandragomirescu]

I'm afraid it's the same @PaulZC , I've just tried examples 7, 19 and 21, they all fail (freeze) at the same point. I'll ship the AGT back to Sparkfun on Monday next week, so if you have anything else you'd like me to try before then I'm happy to do it. Thank you!!

[PaulZC]

Thank you Razvan. I'm out of ideas... I need to get my hands on your board so I can figure out what is going wrong. Those examples all run OK on the board I have here. I do suspect it is the Serial1-hanging-when-RX-is-pulled-low issue but the changes I made should have fixed that. Thank you for your patience and understanding.

Let's leave this issue open until I have investigated your board.
Very best wishes,
Paul

[razvandragomirescu]

Unit shipped back to Sparkfun earlier today, I hope it helps @PaulZC .

[PaulZC]

Thank you Razvan. It will take a few days to reach me. I will update you on what I discover as soon as I have the board.
Best wishes,
Paul

[PaulZC]

Hi Razvan (@razvandragomirescu ),
As we discussed yesterday, commit c6f3dbc does resolve this issue.
Thank you again for all your help.
Closing...!
Very best wishes,
Paul