Fix for hanging response body download #57

nikiromagnoli · 2019-02-20T18:24:38Z

Hi everyone,

Scenario

I was using ArduinoHttpClient library to handle HTTPS over GSM on a Arduino MKR 1400 and MKRGSM library to simple custom AWS endpoint, whose body was "hello". Soon I noticed that retrieving that body (after a header section of about 7 entries) took a very very long time (apparently hanging forever).

Exact download time was: 4 minutes and 32 seconds.

Debugging

I debugged the library and found why it was taking so long: when the chars scan is matching pivot strings (so for example "Content-Length:") it goes fast; then when no pivot matches and a simple chars collection takes place (that is the "default" case for the switch in readHeader()), for some reason chars are not available at every loop of skipResponseHeaders()'s while, so the if(available())returnsfalse, moving the flow in the elsesection where lies adelay` that gives breath before searching for other chars.

I don't know exactly why chars are not available at every loop (didn't go so deep), but the actual result was that the delay takes place at every downloaded char.
A delay of 1 second per char is very much for an HTTP response (headers + body).

Fixing

So I fetched the constant with the amount of milliseconds to wait in that delay and reduced the wait time from 1 second to 30 milliseconds (fairly above 20 milliseconds that is the manual suggested minimum wait time between requests to the uBlox chip).

Now download time to the same request is: 12 seconds.

Don't know if this involves side-effects that I'm not aware of, but I rely on your wisdom for this.

Greetings

By the way, this is my first ever pull-request, gotta learn how this is done properly... so go easy on me, okay? ;)
Cheers

Sidenoting

Searching through the issues I found that few could be related to this, links follow:

sandeepmistry · 2019-03-08T21:41:52Z

Hi @TechNyquist, thanks for opening this pull request!

Do you have an example sketch to reproduce the issue? Also, what version of the MKRGSM library are you using?

nikiromagnoli · 2019-03-12T10:21:24Z

Hi @sandeepmistry, thanks to you for bringing up all this stuff.

So, my MKRGSM library version is 1.3.3 and I did not provide any example because I did nothing special but the normal usecase of GSM/GPRS connection, then a simple GET to some HTTPS client. By the way, actually I did something: just added GSMSecurity support to GPRS in order to enable GSMSSLClient for connecting to HTTPS host. Anyway below is the script used for testing.
Note: I bold this point, I'm connecting to AWS simple Python lambda that is replying with status, 6 headers and "hello".

#include <MKRGSM.h>
#include <ArduinoHttpClient.h>
#include "arduino_secret.h"

// in "arduino_secret" separate file are PIN number, GSM/GPRS credentials,
// SSL root certificate and HTTP server and path

// HTTP_SERVER is something like "xxxxxxxxxx.execute-api.yy-yyyy-y.amazonaws.com"
// HTTP_PATH something like "/zzzz"

// initialize the library instance
GSMSSLClient client;
GPRS gprs;
GSM gsmAccess;
GSMSecurity profile;

int port = 443;

void setup()
{
  Serial.begin(9600);
  while (!Serial);

  Serial.println("Starting Arduino web client.");
  bool connected = false;

  while (!connected) {
    if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
        (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
      connected = true;
    } else {
      Serial.println("Not connected");
      delay(1000);
    }
  }

  Serial.println("Importing certificates...");

  profile.setRootCertificate(SECRET_ROOT_CERT);
  profile.setValidation(SSL_VALIDATION_ROOT_CERT);
  profile.setVersion(SSL_VERSION_TLS_1_2);
  profile.setCipher(SSL_CIPHER_AUTO);
  client.setSecurityProfile(profile);
  
  // after GPRS connection try HTTPS request
  Serial.println("Sending HTTPS request...");
  HttpClient http (client, HTTP_SERVER, port);
  http.get(HTTP_PATH);

  // output
  Serial.println("Receiving response...");

  Serial.print("Status code: ");
  int statusCode = http.responseStatusCode();
  Serial.println(statusCode);

  Serial.print("Response: ");
  String response = http.responseBody();    // here hangs 4 minutes. Down to 12 seconds with the patch
  Serial.println(response);
}

void loop()
{
  // nothing to do here
}

Aside: also note that it's not the only problem: I noticed that on SSL transport, sometimes servers drop connection at the end of body while responding, and response reading routine maybe does not expect this (content-length?) and continues to wait for data, hanging till timeout. Anyway this is not related to this PR, just a bonus tip.

sandeepmistry · 2019-03-25T19:55:14Z

Hi @TechNyquist,

I'm having trouble reproducing this, here's my example sketch:


#include <ArduinoHttpClient.h>
#include <MKRGSM.h>

#include "arduino_secrets.h"
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
// Please enter your sensitive data in the Secret tab or arduino_secrets.h
// PIN Number
const char PINNUMBER[]     = SECRET_PINNUMBER;
// APN data
const char GPRS_APN[]      = SECRET_GPRS_APN;
const char GPRS_LOGIN[]    = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;

const char serverAddress[] = "example.org";  // server address
int port = 80;

// initialize the library instance
GSMClient gsmClient;
GPRS gprs;
GSM gsmAccess;

HttpClient client = HttpClient(gsmClient, serverAddress, port);

void setup() {
  Serial.begin(9600);
  while (!Serial);
  // connection state
  bool connected = false;

  // After starting the modem with GSM.begin()
  // attach the shield to the GPRS network with the APN, login and password
  while (!connected) {
    if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
        (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
      connected = true;
    } else {
      Serial.println("Not connected");
      delay(1000);
    }
  }
}

void loop() {
  // assemble the path for the GET message:
  String path = "/";

  // send the GET request
  Serial.println("making GET request");
  client.get(path);

  // read the status code and body of the response
  int statusCode = client.responseStatusCode();
  String response = client.responseBody();
  Serial.print("Status code: ");
  Serial.println(statusCode);
  Serial.print("Response: ");
  Serial.println(response);


  Serial.println("Wait ten seconds\n");
  delay(10000);
}

Would you be able to share the response headers and body of the server you are using?

Also,

  profile.setRootCertificate(SECRET_ROOT_CERT);
  profile.setValidation(SSL_VALIDATION_ROOT_CERT);
  profile.setVersion(SSL_VERSION_TLS_1_2);
  profile.setCipher(SSL_CIPHER_AUTO);
  client.setSecurityProfile(profile);

These API's are part of the 1.3.3 of the library ...

sandeepmistry · 2019-03-25T20:07:12Z

I've tried another test sketch, this time using HTTPS:


#include <ArduinoHttpClient.h>
#include <MKRGSM.h>

#include "arduino_secrets.h"
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
// Please enter your sensitive data in the Secret tab or arduino_secrets.h
// PIN Number
const char PINNUMBER[]     = SECRET_PINNUMBER;
// APN data
const char GPRS_APN[]      = SECRET_GPRS_APN;
const char GPRS_LOGIN[]    = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;

const char serverAddress[] = "www.microsoft.com";  // server address
int port = 443;

// initialize the library instance
GSMSSLClient gsmClient;
GPRS gprs;
GSM gsmAccess;

HttpClient client = HttpClient(gsmClient, serverAddress, port);

void setup() {
  Serial.begin(9600);
  while (!Serial);
  // connection state
  bool connected = false;

  // After starting the modem with GSM.begin()
  // attach the shield to the GPRS network with the APN, login and password
  while (!connected) {
    if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
        (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
      connected = true;
    } else {
      Serial.println("Not connected");
      delay(1000);
    }
  }
}

void loop() {
  // assemble the path for the GET message:
  String path = "/robots.txt";

  // send the GET request
  Serial.println("making GET request");
  client.get(path);

  // read the status code and body of the response
  int statusCode = client.responseStatusCode();
  String response = client.responseBody();
  Serial.print("Status code: ");
  Serial.println(statusCode);
  Serial.print("Response: ");
  Serial.println(response);


  Serial.println("Wait ten seconds\n");
  delay(10000);
}

All seems ok, here's the output of the serial monitor:

making GET request
Status code: 200
Response: # Robots.txt file for www.microsoft.com

User-agent: *
Disallow: /en-us/windows/si/matrix.html
Disallow: /en-us/windows/si/matrix.html
Disallow: /*/security/search-results.aspx?
Disallow: /*/music/*/search/
Disallow: /*/search/
Disallow: /*/music/*/Search/
Disallow: /*/Search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Disallow: /*/store/d/groove-music-pass/cfq7ttc0k5dq/0001
Allow: /*/store/*/search/
Allow: /*/store/*/layout/
Allow: /*/store/music/groove-music-pass/*
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: /music/*/wal/
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: */mediaapi/
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/
Disallow: /windows/compatibility/windows-7/*
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: https://www.microsoft.com/en-us/explore/msft_sitemap_index.xml
Sitemap: https://www.microsoft.com/learning/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/licensing/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/legal/sitemap.xml
Sitemap: https://www.microsoft.com/filedata/sitemaps/RW5xN8
Sitemap: https://www.microsoft.com/store/collections.xml
Sitemap: https://www.microsoft.com/store/productdetailpages.index.xml
Sitemap: https://www.microsoft.com/store/sitemaps/custom-index.xml
Sitemap: https://www.microsoft.com/en-us/store/locations/store-locations-sitemap.xml

Wait ten seconds

making GET request
Status code: 200
Response: # Robots.txt file for www.microsoft.com

User-agent: *
Disallow: /en-us/windows/si/matrix.html
Disallow: /en-us/windows/si/matrix.html
Disallow: /*/security/search-results.aspx?
Disallow: /*/music/*/search/
Disallow: /*/search/
Disallow: /*/music/*/Search/
Disallow: /*/Search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Disallow: /*/store/d/groove-music-pass/cfq7ttc0k5dq/0001
Allow: /*/store/*/search/
Allow: /*/store/*/layout/
Allow: /*/store/music/groove-music-pass/*
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: /music/*/wal/
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: */mediaapi/
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/
Disallow: /windows/compatibility/windows-7/*
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: https://www.microsoft.com/en-us/explore/msft_sitemap_index.xml
Sitemap: https://www.microsoft.com/learning/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/licensing/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/legal/sitemap.xml
Sitemap: https://www.microsoft.com/filedata/sitemaps/RW5xN8
Sitemap: https://www.microsoft.com/store/collections.xml
Sitemap: https://www.microsoft.com/store/productdetailpages.index.xml
Sitemap: https://www.microsoft.com/store/sitemaps/custom-index.xml
Sitemap: https://www.microsoft.com/en-us/store/locations/store-locations-sitemap.xml

Wait ten seconds

making GET request
Status code: 200
Response: # Robots.txt file for www.microsoft.com

User-agent: *
Disallow: /en-us/windows/si/matrix.html
Disallow: /en-us/windows/si/matrix.html
Disallow: /*/security/search-results.aspx?
Disallow: /*/music/*/search/
Disallow: /*/search/
Disallow: /*/music/*/Search/
Disallow: /*/Search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Disallow: /*/store/d/groove-music-pass/cfq7ttc0k5dq/0001
Allow: /*/store/*/search/
Allow: /*/store/*/layout/
Allow: /*/store/music/groove-music-pass/*
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: /music/*/wal/
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: */mediaapi/
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/
Disallow: /windows/compatibility/windows-7/*
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: https://www.microsoft.com/en-us/explore/msft_sitemap_index.xml
Sitemap: https://www.microsoft.com/learning/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/licensing/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/legal/sitemap.xml
Sitemap: https://www.microsoft.com/filedata/sitemaps/RW5xN8
Sitemap: https://www.microsoft.com/store/collections.xml
Sitemap: https://www.microsoft.com/store/productdetailpages.index.xml
Sitemap: https://www.microsoft.com/store/sitemaps/custom-index.xml
Sitemap: https://www.microsoft.com/en-us/store/locations/store-locations-sitemap.xml

Wait ten seconds

making GET request
Status code: 200
Response: # Robots.txt file for www.microsoft.com

User-agent: *
Disallow: /en-us/windows/si/matrix.html
Disallow: /en-us/windows/si/matrix.html
Disallow: /*/security/search-results.aspx?
Disallow: /*/music/*/search/
Disallow: /*/search/
Disallow: /*/music/*/Search/
Disallow: /*/Search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Disallow: /*/store/d/groove-music-pass/cfq7ttc0k5dq/0001
Allow: /*/store/*/search/
Allow: /*/store/*/layout/
Allow: /*/store/music/groove-music-pass/*
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: /music/*/wal/
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: */mediaapi/
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/
Disallow: /windows/compatibility/windows-7/*
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: https://www.microsoft.com/en-us/explore/msft_sitemap_index.xml
Sitemap: https://www.microsoft.com/learning/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/licensing/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/legal/sitemap.xml
Sitemap: https://www.microsoft.com/filedata/sitemaps/RW5xN8
Sitemap: https://www.microsoft.com/store/collections.xml
Sitemap: https://www.microsoft.com/store/productdetailpages.index.xml
Sitemap: https://www.microsoft.com/store/sitemaps/custom-index.xml
Sitemap: https://www.microsoft.com/en-us/store/locations/store-locations-sitemap.xml

Wait ten seconds

making GET request
Status code: 200
Response: # Robots.txt file for www.microsoft.com

User-agent: *
Disallow: /en-us/windows/si/matrix.html
Disallow: /en-us/windows/si/matrix.html
Disallow: /*/security/search-results.aspx?
Disallow: /*/music/*/search/
Disallow: /*/search/
Disallow: /*/music/*/Search/
Disallow: /*/Search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Disallow: /*/store/d/groove-music-pass/cfq7ttc0k5dq/0001
Allow: /*/store/*/search/
Allow: /*/store/*/layout/
Allow: /*/store/music/groove-music-pass/*
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: /music/*/wal/
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: */mediaapi/
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/
Disallow: /windows/compatibility/windows-7/*
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: https://www.microsoft.com/en-us/explore/msft_sitemap_index.xml
Sitemap: https://www.microsoft.com/learning/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/licensing/sitemap.xml
Sitemap: https://www.microsoft.com/en-us/legal/sitemap.xml
Sitemap: https://www.microsoft.com/filedata/sitemaps/RW5xN8
Sitemap: https://www.microsoft.com/store/collections.xml
Sitemap: https://www.microsoft.com/store/productdetailpages.index.xml
Sitemap: https://www.microsoft.com/store/sitemaps/custom-index.xml
Sitemap: https://www.microsoft.com/en-us/store/locations/store-locations-sitemap.xml

Wait ten seconds

making GET request

nikiromagnoli · 2019-06-11T10:04:53Z

So long... but I found the time to test and I'm here again!

I took your ino code @sandeepmistry and tested on it in a brand new project, so I'm here to share the results with you all.

I replaced the endpoint with my one from AWS (just for more succinct reply, but tested microsoft robots file too and the behaviour is the same as well, despite the different durations). The final result is that the issue showed. It is successful 4 times, then from the fifth on it fails. Follows the original print out from your code:

Port open
making GET request
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Status code: -3
Response:
Wait ten seconds

making GET request
Status code: -3
Response:
Wait ten seconds

So I tried to go deeper and printed out all the chars the client gets back from server (char followed by its ASCII code in square brackets), and the print out is the following:

Port open
Wait 5 seconds before start...
making GET request

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49] [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13] 
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request

------- STATUS

------- BODY
�[-1]
------- OUTCOME
Status code: -3
Response:
Wait ten seconds

making GET request

------- STATUS

------- BODY
�[-1]
------- OUTCOME
Status code: -3
Response:
Wait ten seconds

Then I had the suspect that connection to server was failing after a preset amount of previous sessions, so I added a outcome test on connection to host, and the following is the print out:

Port open
making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS

------- BODY
�[-1]
------- OUTCOME
Status code: -3
Response:
Wait ten seconds

making GET request
Connection failed: -2
Wait ten seconds

So I tought that the problem was GSM session drop, so I placed a re-registration on GSM when connection fails, and the final print out is the following:

Opening port
Port open
Wait 5 seconds before start...
Trying to register to GSM...
Registered on GSM network
making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49] [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34] 
------- OUTCOME
	Status code: 200
	Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34] 
------- OUTCOME
	Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS
H[72] T[84] T[84] P[80] /[47] 1[49] .[46] 1[49]  [32] 2[50] 0[48] 0[48]  [32] O[79] K[75] [13]
[10]
------- BODY
"[34] h[104] e[101] l[108] l[108] o[111] "[34]
------- OUTCOME
Status code: 200
Response: "hello"
Wait ten seconds

making GET request
Connected to host

------- STATUS

------- BODY
�[-1]
------- OUTCOME
Status code: -3
Response:
Wait ten seconds

making GET request
Connection failed: -2
Wait ten seconds

Trying to register to GSM...
Registered on GSM network
making GET request
Connection failed: -2
Wait ten seconds

Trying to register to GSM...
Registered on GSM network
making GET request
Connection failed: -2
Wait ten seconds

Trying to register to GSM...
Registered on GSM network
making GET request
Connection failed: -2
Wait ten seconds

So re-registration actually does not solve the problem.

As conclusion I'm not so sure about what's going on here. The problem seems to show after 4 retries for me and seems different from the first assumptions.
I bold the environment in use:

board: Arduino MKRGSM 1400
ArduinoHttpClient: 0.3.2
MKRGSM: 1.3.3
development IDE: Visual Studio w/ Visual Micro extension

I attach the final code file of the last test, both "ino" file and modified "HttpClient.cpp" for chars print out.
HttpClient_PR_testing.zip
PS: I grayed out the AWS endpoint and path.

I hope to have shed some more light on the issue.

shortened retry delay as fix of long lasting body download

f7393c5

Rocketct requested review from Rocketct and sandeepmistry February 21, 2019 09:38

anctsys mentioned this pull request Jun 25, 2019

Excessive Memory consumption using ArduinoHttpClient & MKRGSM arduino-libraries/MKRGSM#97

Closed

per1234 added type: imperfection Perceived defect in any part of project topic: code Related to content of the project itself labels Apr 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for hanging response body download #57

Fix for hanging response body download #57

nikiromagnoli commented Feb 20, 2019 •

edited by per1234

Loading

sandeepmistry commented Mar 8, 2019

nikiromagnoli commented Mar 12, 2019

sandeepmistry commented Mar 25, 2019 •

edited

Loading

sandeepmistry commented Mar 25, 2019

nikiromagnoli commented Jun 11, 2019 •

edited

Loading

Fix for hanging response body download #57

Are you sure you want to change the base?

Fix for hanging response body download #57

Conversation

nikiromagnoli commented Feb 20, 2019 • edited by per1234 Loading

Scenario

Debugging

Fixing

Greetings

Sidenoting

sandeepmistry commented Mar 8, 2019

nikiromagnoli commented Mar 12, 2019

sandeepmistry commented Mar 25, 2019 • edited Loading

sandeepmistry commented Mar 25, 2019

nikiromagnoli commented Jun 11, 2019 • edited Loading

nikiromagnoli commented Feb 20, 2019 •

edited by per1234

Loading

sandeepmistry commented Mar 25, 2019 •

edited

Loading

nikiromagnoli commented Jun 11, 2019 •

edited

Loading