Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Repeated calls to client.get() fail. #2

Open
damellis opened this issue Feb 20, 2016 · 9 comments
Open

Repeated calls to client.get() fail. #2

damellis opened this issue Feb 20, 2016 · 9 comments
Assignees
@damellis
Labels
topic: code Related to content of the project itself type: imperfection Perceived defect in any part of project

Comments

@damellis
Copy link

I've been trying the SimpleGet example and get the below result. This is connecting to www.google.com, port 80, and calling: client.get("/");. It looks like the first call succeeds but gets only part of the page. Subsequent calls fail. Maybe we're running out of RAM?

I think we may need to support a stream version of this API, where the RestClient class parses the headers but doesn't read the full body of the response. Instead, the user would call client.read() and other stream methods to process the response as needed.

SSID: CalVisitor
IP Address: 10.105.168.227
Status code: 0
making GET request
Response: <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search the world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp" name="robots"><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'yszHVuywJ9GKjwPi95SACA',kEXPI:'18167,3700246,3700388,4003510,4028875,4029815,4031109,4032678,4033307,4036509,4036527,4038012,4039268,4042784,4042792,4043492,4044606,4045841,4046304,4048042,4048882,4049501,4049549,4049557,4050161,4050912,4051242,4051558,4051596,4052304,4053233,4054284,4054513,4055202,4055381,4056038,4056590,4057170,4057586,4057836,4057920,4058004,4058117,4058228,4058277,4058316,4058329,4058330,4058337,4058381,4058384,4058624,4058636,4059024,4059318,4059328,4059359,4059438,4059789,4059861,4059977,4060342,4060657,4060681,4060683,4061097,8300096,8300272,8300310,8500572,8502095,8502315,8502349,8502452,8502690,8502986,8503012,8503038,8503135,8503216,8503303,8503306,8503404,8503585,8503611,10200083',authuser:0,kscs:'c9c918f0_24'};google.kHL='en';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.wl=function(a,b){try{google.ml(Error(a),!1,b)}catch(d){}};google.time=function(){return(new Date).getTime()};google.log=function(a,b,d,e,g){a=google.logUrl(a,b,d,e,g);if(""!=a){b=new Image;var c=google.lc,f=google.li;c[f]=b;b.onerror=b.onload=b.onabort=function(){delete c[f]};window.google&&window.google.vel&&window.google.vel.lu&&window.google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,d,e,g){var c="",f=google.ls||"";if(!d&&-1==b.search("&ei=")){var h=google.getEI(e),cHTTP/1.1 200 OK
Date: Sat, 20)&&((e=google.getLEI(e))?c+="&lei="+e:h!=google.kEI&&(c+="&lei="+google.kEI))}a=d||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+c+f+"&zx="+google.time();/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};google.y={};google.x=function(a,b){google.y[a.id]=[a,b];return!1};google.load=function(a,b,d){google.x({id:a+k++},function(){google.load(a,b,d)})};var k=0;})();var _gjwl=location;function _gjuc(){var a=_gjwl.href.indexOf("#");if(0<=a&&(a=_gjwl.href.substring(a),0<a.indexOf("&q=")||0<=a.indexOf("#q="))&&(a=a.substring(1),-1==a.indexOf("#"))){for(var d=0;d<a.length;){var b=d;"&"==a.charAt(b)&&++b;var c=a.indexOf("&",b);-1==c&&(c=a.length);b=a.substring(b,c);if(0==b.indexOf("fp="))a=a.substring(0,d)+a.substring(c,a.length),c=d;else if("cad=h"==b)return 0;d=c}_gjwl.href="/search?"+a+"&cad=h";return 1}return 0}
function _gjh(){!_gjuc()&&window.google&&google.x&&google.x({id:"GJH"},function(){google.nav&&google.nav.gjh&&google.nav.gjh()})};window._gjh&&_gjh();</script><style>#gb{font:13px/27px Arial,sans-serif;height:30px}#gbz,#gbg{position:absolute;white-space:nowrap;top:0;height:30px;z-index:1000}#gbz{left:0;padding-left:4px}#gbg{right:0;padding-right:5px}#gbs{background:transparent;position:absolute;top:-999px;visibility:hidden;z-index:998;right:0}.gbto #gbs{background:#fff}#gbx3,#gbx4{background-color:#2d2d2d;background-image:none;_backgro9557,4050908,4051241,4051596,4,4054284,4054513,4055202,4055381,4056038,4056590,4057170,4057586,4057836,4057920,4058004,4058117,4058228,4058277,4058316,4058329,4058330,4058337,4058381,4058384,4058624,4058636,4059024,4059318,4059328,4059359,4059438,4059789,4059861,4059977,4060342,4060657,4060681,4060683,4061097,8300096,8300272,8300310,8500572,8502095,8502315,8502349,8502452,8502690,8502986,8503012,8503038,8503135,8503216,8503303,8503306,8503404,8503585,8503611,10200083',authuser:0,kscs:'c9c918f0_24'};google.kHL='en';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.wl=function(a,b){try{google.ml(Error(a),!1,b)}catch(d){}};google.time=function(){return(new Date).getTime()};google.log=function(a,b,d,e,g){a=google.logUrl(a,b,d,e,g);if(""!=a){b=new Image;var c=google.lc,f=google.li;c[f]=b;b.onerror=b.onload=b.onabort=function(){delete c[f]};window.google&&window.google.vel&&window.google.vel.lu&&window.google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,d,e,g){var c="",f=google.ls||"";if(!d&&-1==b.search("&ei=")){var h=google.getEI(e),c�gle.ls||"";if(!d&&-1==b.search("&ei=")){var h=google.getEI(e),c������������������������������052304,4054117,4054284,4054552tion:0 -138px;background-repeat:repeat-x;border-bottom:1px solid #000;font-size:24px;height:29px;_height:30px;opacity:1;filter:alpha(opacity=100);position:absolute;top:0;width:100%;z-index:990}#gbx3{left:0}#gbx4{right:0}#gbb{position:relative}#gbbw{left:0;position:absolute;top:30px;width:100%}.gbtcb{position:absolute;visibility:hidden}#gbz .gbtcb{right:0}#gbg .gbtcb{left:0}.gbxx{display:none !important}.gbxo{opacity:0 !important;filter:alpha(opacity=0) !important}.gbm{position:absolute;z-index:999;top:-999px;visibility:hidden;text-align:left;border:1px solid #bebebe;background:#fff;-moz-box-shadow:-1px 1px 1px rgba(0,0,0,.2);-webkit-box-shadow:0 2px 4px rgba(0,0,0,.2);box-shadow:0 2px 4px rgba(0,0,0,.2)}.gbrtl .gbm{-moz-box-shadow:1px 1px 1px rgba(0,0,0,.2)}.gbto .gbm,.gbto #gbs{top:29px;visibility:visible}#gbz .gbm{left:0}#gbg .gbm{right:0}.gbxms{background-color:#ccc;display:block;position:absolute;z-index:1;top:-1px;left:-2px;right:-2px;bottom:-2px;opacity:.4;-moz-border-radius:3px;filter:progid:DXImageTransform.Microsoft.Blur(pixelradius=5);*opacity:1;*top:-2px;*left:-5px;*right:5px;*bottom:4px;-ms-filter:"progid:DXImageTransform
Wait five seconds
Status code: 200
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
Status code: 0
making GET request
Response: 
Wait five seconds
@tigoe
Copy link
Contributor

tigoe commented Feb 21, 2016

I agree with a Stream implementation on this, but I am not the person who can do it. Please assign someone else with more familiarity with the guts of Stream.

@tigoe tigoe assigned damellis and unassigned tigoe Feb 21, 2016
@sandeepmistry
Copy link

There's a HTTPSClient utility class in the AzureIoT library that we might be able to use. It currently is designed for keep alive connections and to bridge the HTTP layer needed by the Azure IoT SDK.

As for implementing Stream maybe we could extend Stream and forward Stream API's to the Client passed in:

int RestClient::available() {
  return client.available();
}

int RestClient::read() {
  return client.read();
}

int RestClient::peek() {
  return client.peek();
}

void RestClient::flush() {
  client.flush();
}

It would be nice to stream the response header values, but some users might not care about those.

@tigoe
Copy link
Contributor

tigoe commented Feb 22, 2016

Perhaps we could add a flag to ignore headers. Or just add a function to parse them out if you want them, and making ignoring them the default.

@sandeepmistry
Copy link

We could make the library parse and ignore the headers if readResponse() is called before readHeader().

For example (this is loosely based on the HTTPSClient I link to earlier):

  1. Ignore headers:
statusCode = client.get("/");

// read body one byte at a time
while (client.available()) {
  Serial.println(client.read());
}
  1. Process headers
statusCode = client.get("/");

// process headers
String headerName;
String headerValue;

while (client.readHeader(headerName, headerValue) != -1) {
  // process header
}

// read body one byte at a time
while (client.available()) {
  Serial.println(client.read());
}

@tigoe
Copy link
Contributor

tigoe commented Feb 22, 2016

Kinda works. Could also maybe do something like readUntil(BODY), which would be readUntil("\n\n") and variants thereon.

@damellis
Copy link
Author

I like Sandeep's suggestion, although we might want to do headerValue = client.readHeader(headerName), followed by a check for a valid String: if (!headerValue). I think that's more in line with the existing Stream.readString().

@tigoe
Copy link
Contributor

tigoe commented Feb 22, 2016

I like that, Dave.

@tigoe

This comment has been minimized.

Sign in to view
@sandeepmistry
Copy link

@damellis's suggestion sounds good to me!

@per1234 per1234 added type: imperfection Perceived defect in any part of project topic: code Related to content of the project itself labels Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@damellis damellis

Labels
topic: code Related to content of the project itself type: imperfection Perceived defect in any part of project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tigoe @damellis @sandeepmistry @per1234