Excessive Memory consumption using ArduinoHttpClient & MKRGSM

[anctsys]

Hi
I have an excessive memory consumption when i used methods responseStatusCode() & responseBody() more than 50%.I using the code you post for resolve arduino-libraries/ArduinoHttpClient#57
I also add a code to check memory using on the arduino.

Please can you help me

MyConfig is
Arduino MKR1400
ArduinoHttpClient : 0.4.0
MKRGSM : 1.4.2

Memory Consumtion Displaying on the terminal

memory at startup of setup : 28347
memory at end of setup : 28279
making GET request
memory before responseStatusCode : 27871
memory after responseStatusCode: 24027
memory before responseBody : 24027
memory after responseBody : 19079
Status code: 200
Response: Wait ten seconds

making GET request
memory before responseStatusCode : 19079
memory after responseStatusCode: 19079
memory before responseBody : 19079
memory after responseBody : 14131
Status code: 200
Response: Wait ten seconds

making GET request
memory before responseStatusCode : 14131
memory after responseStatusCode: 14131
memory before responseBody : 14131
memory after responseBody : 14131
Status code: 200
Response: Wait ten seconds

The sketch is below

#include <MKRGSM.h>

    #ifdef __arm__
    // should use uinstd.h to define sbrk but Due causes a conflict
    extern "C" char* sbrk(int incr);
    #else  // __ARM__
    extern char *__brkval;
    #endif  // __arm__

    

#include "arduino_secrets.h"
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
// Please enter your sensitive data in the Secret tab or arduino_secrets.h
// PIN Number
const char PINNUMBER[]     = SECRET_PINNUMBER;
// APN data
const char GPRS_APN[]      = SECRET_GPRS_APN;
const char GPRS_LOGIN[]    = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;

const char serverAddress[] = "www.microsoft.com";  // server address
int port = 443;

// initialize the library instance
GSMSSLClient gsmClient;
GPRS gprs;
GSM gsmAccess;

HttpClient client = HttpClient(gsmClient, serverAddress, port);

String response="";
int statusCode=0;

void setup() {
  Serial.begin(9600);
  while (!Serial);
  Serial.print("memory at startup of setup : ");
  print_FreeMemory();
  // connection state
  bool connected = false;

  // After starting the modem with GSM.begin()
  // attach the shield to the GPRS network with the APN, login and password
  while (!connected) {
    if ((gsmAccess.begin(PINNUMBER) == GSM_READY) &&
        (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY)) {
      connected = true;
    } else {
      Serial.println("Not connected");
      delay(1000);
    }
  }
  Serial.print("memory at end of setup : ");
  print_FreeMemory();
}

void loop() {
  // assemble the path for the GET message:
  String path = "/robots.txt";

  // send the GET request
  Serial.println("making GET request");
  client.get(path);

  // read the status code and body of the response
  Serial.print("memory before responseStatusCode : ");
  print_FreeMemory();
  statusCode = client.responseStatusCode();
  Serial.print("memory after responseStatusCode: ");
  print_FreeMemory();
  Serial.print("memory before responseBody : ");
  print_FreeMemory();
  response = client.responseBody();
  Serial.print("memory after responseBody : ");
  print_FreeMemory();
  Serial.print("Status code: ");
  Serial.println(statusCode);
  Serial.print("Response: ");
  //Serial.println(response);
  response="";


  Serial.println("Wait ten seconds\n");
  delay(10000);
}

        int print_FreeMemory() {
            char top;
            #ifdef __arm__
              Serial.println(&top - reinterpret_cast<char*>(sbrk(0)));
              return &top - reinterpret_cast<char*>(sbrk(0));
            #elif defined(CORE_TEENSY) || (ARDUINO > 103 && ARDUINO != 151)
                Serial.print("Mémoire dispo  : ");
                Serial.println(&top - __brkval);
              return &top - __brkval;
            #else  // __arm__
              //this->set_freememory(__brkval ? &top - __brkval : &top - __malloc_heap_start);
              return __brkval ? &top - __brkval : &top - __malloc_heap_start;
            #endif  // __arm__
        }```

----------------------------------------------------------------------------------------

[Rocketct]

Hi @anctsys i have made some test, really there is not a problem, the point is how the memory is managed.
What is happen is that each time that you receive a new response the memory manager reserve a portion of memory for the members of the object MKRGSM and ArduinoHttpClient moving the first free memory slot pointer,until when the response receive not exceed this area portion the memory allocated is ever the same, but if for some reason the received one is higher the allocated, the previous one is free and a new big section of memory is allocated, moving the first free memory slot pointer, this cause a memory fragmentation.
a consequence of this fragmentation is that &top - reinterpret_cast<char*>(sbrk(0)); will follow to show = 14131 because not take in account of the memory allocated before the heap pointer.

[anctsys]

Hi @Rocketct Rocketct

Thank you for your answer
If I understand the problem comes from the script that calculates the memory and you confirm that the memory is well released.
Also, do you have an alogrithme to check the free memory taking into account the fragmentation.

Bruno

[pnndra]

Hi @anctsys there are two overlaping issues here. the first is that static string objects within the httpclient object are "grow only", so this means that whenever you reveive a message that does not fit in the string object the realloc function is called causing old memory buffer to be released and a new one to be allocated. what will happen next is that even if you assign the string object with a smaller string it will keep the bigger buffer. this is wanted behaviour and means that you will always have an allocated buffer that is as big as the biggest message you received.
the second issue is that whenever you deallocate memory,you allocate a bigger buffer and memory just after the released buffer is already taken, then the new buffer must be allocated from heap and the buffer just released will be added to the free memory list. this is really free memory but it's not possible to use it unless you request portions that are equal or smaller than the sizes of each released buffer. this means that yes, this is free memory but for example if you try to allocate a buffer as big as all the "free memory" you won't be able to achieve it as it is fragmented.
what the script does is just subtract the top and the bottom pointer of the heap, which is correct if you consider only the unfragmented RAM. considering also the fragmented one, that would be achievable by summing the free memory list, may be misleading because worst case you may have a lot of free memory but fragmented in 1 byte blocks which would make it completely useless unless you want to allocate a byte at once...

[anctsys]

Hi @pnndra and @Rocketct
For the first case. Is there a way to clear this cache? Because if you only receive a large amount of data once, and receive a lot of small messages after, it will reduce the memory using by httpclient for the next smaller messages.
For the second problem thank you for allowing me to understand the memory fragmentation in the Arduino
Can you give me the numbers of the two problems to track them.

Thank You So Much

[pnndra]

Hi @anctsys, if you look at the header file for the http client library (https://github.com/arduino-libraries/ArduinoHttpClient/blob/7f36561e0bced5458516066ae09636e119cae0ed/src/HttpClient.h#L287) you'll see that there are several string variables. C++ string objects can only grow so even if you set them to "" they will stay same size. One thing you can do is to set them to empty string and then call the reserve method which will reallocate the buffer with the new size. Note that this will call realloc which will likely reuse the same pointer if you shrink the buffer and that will further fragment the memory. Anyway if you want you can add a new API that does this with all the string objects ( there are a few) in this class...

[anctsys]

Thank you
I will close this issue for MKRGSM