ReceiveSMS sketch hangs after SMS received

[joshbober]

In the Arduino example sketch ReceiveSMS.ino, when a message is received by the device, the response gets cut off and the sketch hangs. This was discussed partially in #50, but is a separate issue from what has been discussed there. #80 hypothesized that it was an issue with the '/' character, but the issue was closed without clear resolution (using another library may have solved this). With Serial debugging active, I see that the response returned over the serial monitor is cut off abruptly, like this:

14:32:05.683 -> AT+CMGL="REC UNREAD"
14:32:05.683 -> +CMGL: 6,"REC UNREAD","+1##########",,"21

After this, the code gets stuck in a while() loop, waiting for MODEM.ready() to return something other than 0. This is where the problem becomes more clear. Incoming uart data is stored in the RX ring buffer, which has a size of 64 ints. The first 64 characters of the response are being stored in this buffer, and the rest seems to be dropped. I verified this by adding a Serial.print statement with the size of the RX buffer after each character is read. This means that the response code at the end is not stored or read in the modem's poll() function, causing the return value for "ready" to always remain 0.

I have verified that this is NOT an issue with processing the '/' character; if you change the AT command in the library from "AT+CMGL="REC UNREAD"" to just "AT+CGML" (identical commands, see AT commands reference), the serial monitor will print more of the date/time, like this:

15:45:05.700 -> AT+CMGL
15:45:05.746 -> +CMGL: 11,"REC UNREAD","+1##########",,"21/09/08,12:45

I see that buffering is handled differently for the NBClient functions (the NBSocketBuffer files), where the buffer is extended to 512 ints. Does a similar solution need to be applied here to handle SMS message reading? Is there something that I am missing in my understanding?

Steps to replicate:

1. Upload ReceiveSMS.ino to an NB1500 with an activated SIM card that has SMS service. Set debugging to true in the NB constructor- nbAccess(true)
2. Wait for the device to initialize and be in the "Waiting for messages" state
3. Send an SMS message to the device

Link to gist of Serial Monitor output following the 3 steps above (with phone number redacted)
https://gist.github.com/joshbober/9a9a01e35c540c0f4d4771276bfee0be

Other relevant information:

• My device firmware is updated to the latest release 05.12
• Using the SerialSARAPassthrough sketch and u-blox tool m-center, the AT+CGML command returns the full expected response

[CptHolzschnauz]

Just a short reply. Sorry, i was unclear with "other library". I meant i compiled my sketch with another computer with a pristine MKRNB lib (without any changes made by me ;=) and then it worked. Thats what i meant with "another" lib.

You are right, the / char was coincidence.

I just replicate the error, but with another board (and SIM) with the same sketch it works perfektly.

There is a bug and it's going to be a tough one...

PS: I loaded the serialPasstrough (the modem keeping powered on), with
AT+CMGL="REC UNREAD" there was nothing
after AT+CMGL="ALL" it listed 13 sms, i deleted all by hand, after that AT+CMGL="REC UNREAD" worked.
Loaded back the sketch, the sms lib failed with the same behaviour.

Joshbober, you wrote
After this, the code gets stuck in a while() loop, waiting for MODEM.ready() to return something other than 0.
Where do you mean? Is the buffer handling not within NB_SMS.cpp, 350 ?
Let's track down that bug!

[CptHolzschnauz]

PPS: I reproduced the bug with
07:57:59.400 -> AT+CMGL="REC UNREAD"
07:57:59.400 -> +CMGL: 1,"REC UNREAD","+41xxxxxxxxx",,"21

Manually, it works perfectly:
08:08:19.720 -> AT+CMGL="REC UNREAD"
08:08:19.720 -> +CMGL: 2,"REC UNREAD","+41xxxxxxxxx",,"21/09/10,08:08:16+08"
08:08:19.756 -> Test
08:08:19.756 ->
08:08:19.756 -> OK

Where do you count 64 characters?
And again, stoped at the "/" char, despite this is more then coincidence? Different results with the same AT command AT+CMGL="REC UNREAD" send by serialPASS and by the lib, it must be some issue with the buffer handler of the lib!?

This issue comes out of a sudden and vanish the same way...strange flaky thing.
Or is it just happend to Josh and me?

[joshbober]

Thanks for looking into this, Cpt.

First off, I have also seen that manually sending the AT+CMGL command using the passthrough sketch works perfectly.

Where do you mean? Is the buffer handling not within NB_SMS.cpp, 350?
I think the issue actually lies in the handling of AT responses in Modem.cpp, poll() function (starting at line 274). I did some digging into the "_uart->available()" code, which took me to the Arduino files Uart.cpp, Uart.h, and RingBuffer.h. The Uart class uses a "ring buffer" to store incoming data, and the available() command returns the number of elements stored in the buffer. This check is done in Modem.cpp, 276, where it tests to see if there is data in the ring buffer that can be read. If there is data, the next char is read from the uart buffer, printed to Serial monitor (if debug enabled), then added to the Modem's "_buffer" variable (lines 277-283).

Where do you count 64 characters?
In the file RingBuffer.h (C:\Users\[user]\AppData\Local\Arduino15\packages\arduino\hardware\samd\1.8.11\cores\arduino\api\RingBuffer.h), line 33, the defined SERIAL_BUFFER_SIZE is 64. This could be modified elsewhere, or not even used, but I'm not sure, it just confirmed my assumption.
Using the information above, I added a Serial.print command to return the number of elements in the Uart buffer as each character gets read and printed to Serial monitor. Here's a link to the Gist of that code, and the output (it is a little hard to read since it prints the number after each character).

You can see from here that the "AT+CGML="REC UNREAD"" command is also stored in this buffer, which explains why the total characters printed is 64 and not only 40. A way to test this is to remove "="REC UNREAD"" from "NB_SMS.cpp", line 226; the "AT+CGML" command by default will return the unread messages (see section 9.12.4 of SARA-R4 series AT commands manual). 13 more characters of the response get printed since the part of command that is removed is 13 characters long.

I think the response for the AT+CGML command is longer than expected for most AT responses, and thus the Modem.cpp code is not equipped to process it. Only the first 64 characters of the total command+response get stored in the buffer, so the result code (OK, ERROR, NO CARRIER, or CME ERROR) is not found. The value of _ready remains 0, and the code remains forever in the while loop found in NB_SMS.cpp, line 283 (this can be verified by putting a Serial.print statement within the while loop and watching on the Serial monitor).

Sorry for the wall of text, hopefully this gets us closer to an understanding of the issue.

[CptHolzschnauz]

No worries! I'm glad i found a mate with this issue!
I got a feeling the cause is much simpler (just a feeling i know ;=)
Main question: Why was it working perfectly the other day?

[joshbober]

I fully support the theory that it's a simple fix that I'm just not understanding, hopefully someone who understands or contributed to this library can chime in.

As for your question- I have no clue. I have not gotten it to work even once yet. I tried using a fresh library install on the same computer, with a fresh compile (temp folder emptied), to no avail.

I noticed you said in another comment that it worked for you on a different computer; I will attempt that as well and report my results.

[joshbober]

UPDATE:

Tested on different computer, but I have found the real culprit: the Arduino SAMD boards library.

Versions 1.8.10 and 1.8.11 exhibit this buffer/hanging issue.
Downgrading to version 1.8.9 or below makes it work (except one of the characters gets eaten in the sketch Serial monitor output on a different line). Link to SM output

@CptHolzschnauz please test with the downgraded SAMD library and let me know if you see the same result.

If this is the case, and the issue is not with the MKRNB library, but with the SAMD, should there be a disclaimer or something to not use v1.8.10+ of the SAMD core until they fix it? I will pursue submitting this issue over there if it hasn't already been brought up.

[CptHolzschnauz]

Oh man, you are right! I can confirm.
I was the discoverer of the big SPI bug in 1.8.10, so i never updated. That's why it works compiled on my old computer.
With my new one, the 1.8.11 was automatically installed on so compiled on that machine it didn't work.
My gut feeling was right.. ;=)

[joshbober]

Thank you for confirming! I'm glad that we got to the bottom of this, and that your gut feeling was right.

I just created an issue over on the SAMD repo, hopefully they can provide some insight. @per1234, not sure if this should be closed here, with the stipulation that this is an issue with dependencies rather than the MKRNB lib itself.

[CptHolzschnauz]

Thank you for finding! My gut was half right, i think you were on the right path to the bug deep down in the core...

[CptHolzschnauz]

Update: I had also a strange behaviour with the same symptons with the nbscanner class, call getSignalStrength() with core 1.8.9. I downgraded the core lib to 1.6.12, the last version of the mighty Sandeep Mistry. The age of the mkrnb and the core are now more equal. I will let u know.

[rhowell209]

I have been following this thread (and all the precursors to this thread on Arduino forums including the board soldering to get the R410M firmware upgraded on the MKRNB) and am very interested in the progress on this. In my particular experience, I have been trying to get SMS messages to send and receive, but have achieved neither of these even once. I seem to be able to run all of the other examples (SATA pass through, NTP time pull from a server, Network type configuration (LTE-M, NB, then the ones with failover order)) but the SMS side of it has never worked. My AT&T account swears SMS is enabled for this SIM, although I never took it out and tried it on a cell phone because I wasn't sure if the SIM had to be tied to a device IMEI to work and feared that may alias the truth.

I'm having a bit of an embarassing moment here, but I can't seem to find this library in my Arduino Library Manager. What can I search for specifically to find it and ensure it is downgraded? I feel like I've seen it before but it's just not showing up.

Thanks,
Richard

EDIT: I realized the board manager core firmwares are in separate managers than the code libraries. All good here, will post any unique results that improve further guidance. When I get a moment, I'll keep downgrading them until I can possibly avoid the character loss issue described above.

EDIT 2: Still can not send or receive SMS using either MKR NB 1500 included examples. m-center shows "success" when sent manually over AT commands, but it's never received on my device, nor vice versa.

[CptHolzschnauz]

The issue is between the modem and the mobile network, nothing with the lib or your sketch.
You have to update the firmware of the modem first of all. The last release solves some sms problems and some issues with AT&T as far as i know.
https://forum.arduino.cc/t/firmware-upgrade-for-ublox-sara-r410m-02b-on-the-mkr-nb-1500-2/699292/12
It looks a bit scary for the first time, but you can do this. With some experience it's a 5 minute task.
You need a small solder iron tip. First bring some fresh solder to the mkr, take a good pretinned cable, start the EasyFlash as Admin, try several times when it does not suceed at first time. Good luck!

[rhowell209]

The issue is between the modem and the mobile network, nothing with the lib or your sketch.
You have to update the firmware of the modem first of all. The last release solves some sms problems and some issues with AT&T as far as i know.
https://forum.arduino.cc/t/firmware-upgrade-for-ublox-sara-r410m-02b-on-the-mkr-nb-1500-2/699292/12
It looks a bit scary for the first time, but you can do this. With some experience it's a 5 minute task.
You need a small solder iron tip. First bring some fresh solder to the mkr, take a good pretinned cable, start the EasyFlash as Admin, try several times when it does not suceed at first time. Good luck!

What I meant was that I have done this. I do appreciate all that great information. It really was quite easy.
Current response:

ATI9
L0.0.00.00.05.12,A.02.19
OK

Unfortunately I still cannot send nor receive SMS.

[CptHolzschnauz]

Firmware is fine. sms don't work with MKRNB lib, nor manually, nor u-blox center?
Can you try another network? Maybe the actual dont' allow sms over NBIoT / LTE M no matter what they sweare?
Try the SIM / sms in your mobile phone, that's no problem IMHO.

[rhowell209]

Nope. M-Center is my primary test tool when I boot it up and always nothing. It typically doesn't ever populate the "Radio access technology" field under the Network tab either. I have seen it do that maybe once.

I was going to try that but worried with SIM being 3-in-1 if I popped it down from micro/3ff to nano/4ff for my phone, I might have trouble getting it back into the micro slot without a special adapter. Not sure if just pushing it back into the micro plastic template will work after it is detached from plastic, but I'll try.

[rhowell209]

One interesting thing is that the AT+URAT= commands don’t seem to be accepted in M-Center after updating the firmware for the uBlox, and it says command not supported. I have attached a few images to show you what works and what doesn’t.

Do you ever hear of people saying this doesn’t work with standard AT&T LTE-M network? Or maybe SMS doesn’t work over that network? I have ordered some Hologram sim cards since I see a lot of people using those and was wondering if you might recommend a US network that has been demonstrated/proven to work with this board.

Attached some images to help you get an idea. Strangely the radio access technology field in the network tab almost never populates (I've seen it once) with either version of ublox firmware, old or new.

[CptHolzschnauz]

AT+URAT= is not workin anymore, probably this is automatic now.
I have no idea about AT&T capabilities, i'm in switzerland. It works here, somehow..
Try AT+CFUN=15,1 to reset also the SIM card.
Tick the box in "Automatic reading of RX SMS"
When you send a SMS to the device whats happen with AT+CMGL="ALL" ?

[rhowell209]

It will OK it just like everything else, but not actually show/read anything. From: CptHolzschnauz ***@***.***> Sent: Monday, September 20, 2021 9:20 AM To: arduino-libraries/MKRNB ***@***.***> Cc: Richard Howell ***@***.***>; Comment ***@***.***> Subject: Re: [arduino-libraries/MKRNB] ReceiveSMS sketch hangs after SMS received (#86) AT+URAT= is not workin anymore, probably this is automatic now. I have no idea about AT&T capabilities, i'm in switzerland. It works here, somehow.. Try AT+CFUN=15,1 to reset also the SIM card. Tick the box in "Automatic reading of RX SMS" When you send a SMS to the device whats happen with AT+CMGL="ALL" ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#86 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKCNE3RFOEBN2ZBOPJRVB7LUC47JNANCNFSM5DVR56YQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[CptHolzschnauz]

Just OK means there are no SMS in the memory of the modem.
IMHO it's a network problem. Try another SIM/Network before you turn crazy over this s...

[rhowell209]

My friend I hope you are right. I turned crazy days ago already. I am working on trying something with Verizon or some other network with my Hologram sim that should be coming.

Do you have any knowledge of why this link for the NB is still active but the more general link is still dead? Noticed they appended a "-B" to the product ID. May be just a US thing.

[CptHolzschnauz]

I am, thx. I'm fighting with this modem problem since months, you are not alone in the loony bin ;=)

[rhowell209]

Regarding your issue above, while I was reading page after page of uBlox AT manuals -- which is almost as exciting as a moto/freescale MC9SO8 manual -- somewhere it mentioned 3GPP release 13 power saving modes / deep sleep modes shutting off the actual UART channels until it could realize edge triggering on tx/rx lines if not using the dedicated wake pin to wake unit up. Might explain the missing character at start on some of these responses. This may have been the SARA-N3 series though.

[CptHolzschnauz]

You can turn off power saving with AT+UPSV=0
What answer you get with AT+UMNOPROF=?

[rhowell209]

AT+UMNOPROF=?

AT+UMNOPROF=?

+UMNOPROF: 
0: SW default
1: SIM ICCID select

2: ATT

198: AT&T Profile 2-4-12
31: DT
19: Vodafone

100: Standard Europe
21: TELUS
101: Standard Europe No-ePCO
4: Telstra
8: Sprint
43: ROGERS
46: Orange France
33: VIVO
45: TIM Brasil
44: Claro Brasil

206: FN ATT
3:n

[rhowell209]

Digi International posts of u-blox Memory Housekeeping Issue on XBee3 Cat M Cellular

[CptHolzschnauz]

Sorry, i meant AT+UMNOPROF?
So try also AT+UMNOPROF=198

[rhowell209]

AT+UMNOPROF? returns "0"
AT+UMNOPROF=198 is accepted but it doesn't retain. I tried the following as well:

AT+CFUN=4
AT+UMNOPROF=198
AT+CFUN=1

and..

AT+CFUN=0
AT+UMNOPROF=198
AT+CFUN=1

Still returns 0 when checking AT+UMNOPROF?

[CptHolzschnauz]

Sorry, i forgot, after AT+UMNOPROF=198 you have to AT+CFUN=15

[rhowell209]

Oh dang, thanks, forgot that. After doing this, it doesn't appear to register with the network.

OK

AT+COPS?

AT+COPS?

+COPS: 0

OK

AT+CEREG=2

AT+CEREG=2

OK

AT+CEREG?

AT+CEREG?

+CEREG: 2,0

OK

AT+CEREG=0

AT+CEREG=0

OK


AT+CSQ

AT+CSQ

+CSQ: 99,99

OK

[CptHolzschnauz]

So then, back to start... Have you also tried AT+UMNOPROF=1 ?

[rhowell209]

So you were right, it did end up being something on the carrier side. They finally fessed up and told me they didn't "have data activated" on their end. After that got resolved, I started seeing data. The texts from the device to the phones never miss a beat -- very good. Sometimes the receiption from my phone to the device, however, can lag quite a bit. But it is looking more responsive tonight, the difference being that I have improved my antenna situation. Perhaps that's what's causing it, we will see. Thanks for the tips!

[CptHolzschnauz]

Cheers!

[joshbober]

Resolved with version 1.8.12 of the SAMD core. I tested the ReceiveSMS sketch with the new core and a MKR NB1500. Though the first character of each message is still dropped when writing the message to the SM, the device no longer hangs, and multiple messages can be received without resetting the modem.

Thanks for your help chasing this down, and special thanks to @sandeepmistry for finding the issue with the SAMD core.