pylightning: Correctly return the remainder of a message back

We read a JSON message from the buffer, after converting it from raw bytes to
UTF-8, and returning the remainder of the byte array back to the
caller. However the return value of `raw_decode` refers to symbols in the
UTF-8 decoded string, not the raw bytes underlying byte-array, which means
that if we have multi-byte encoded UTF-8 symbols in the byte-array we end up
with a misaligned offset and will return part of the message as
remainder. This would then end up being interpreted as the result of the next
call.

This could not be exploited currently since we use a socket only for a single
JSON-RPC call and will close the connection afterwards, but since we want to
eventually recycle connections for multiple calls, this could have been very
dangerous.

Signed-off-by: Christian Decker <[email protected]>
Reported-by: Corné Plooy <@bitonic-cjp>

Author: cdecker

contrib/pylightning/lightning/lightning.py

@@ -46,8 +46,10 @@ def _readobj_compat(self, sock, buff=b''):
                     continue
                 # Convert late to UTF-8 so glyphs split across recvs do not
                 # impact us
-                objs, len_used = self.decoder.raw_decode(buff.decode("UTF-8"))
-                return objs, buff[len_used:].lstrip()
+                buff = buff.decode("UTF-8")
+                objs, len_used = self.decoder.raw_decode(buff)
+                buff = buff[len_used:].lstrip().encode("UTF-8")
+                return objs, buff
             except ValueError:
                 # Probably didn't read enough
                 pass