Snakebite doesn't work with HDFS RPC encryption

[elukey]

This is a tracking task to list all the work needed to solve one outstanding issue with snakebite. When RPC encryption is enabled for HDFS, the following happens:

• snakebite contacts the HDFS namenode via Hadoop RPC, negotiating the encryption settings using GSS-API via SASL. It needs to retrieve the list of blocks to read/write and the related datanodes to talk to. This part works fine.
• snakebite then has to contact every HDFS datanode, using a specific RPC protocol that is not Hadoop RPC. The authentication is done via DIGEST-MD5 via SASL, that also allows to set the encryption level if needed (to then allow the negotiation of AES encryption). This bit currently doesn't work because the code that would be needed relies on functionalities of SASL that are not implemented in pure-sasl (namely DIGEST-MD5).

I opened an issue to pure-sasl (thobbs/pure-sasl#32) but some work would be needed to add the missing features.

The alternative would be to use sasl (https://github.com/cloudera/python-sasl) but unfortunately the library is not maintained since 2016. There is a fork that we could consider that should support DIGEST-MD5 + GSS-API: cloudera/python-sasl#15 (comment)

[elukey]

Updating this issue with some info about testing done. The sasl library was forked to sasl3 (updating bindings etc..) but I wasn't still able to make any RPC to the Namenode to work. The bug that I was getting reported the following error in the Namenode's logs:

INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 10.64.5.32 threw exception [javax.security.sasl.SaslException: Problems unwrapping SASL buffer [Caused by GSSException: Defective token detected (Mechanism level: Kerberos GSS-API Mechanism Token:Defective Token ID!)]]
javax.security.sasl.SaslException: Problems unwrapping SASL buffer [Caused by GSSException: Defective token detected (Mechanism level: Kerberos GSS-API Mechanism Token:Defective Token ID!)]

After a lot of tests and debugging comparing the code running pure-sasl (working) vs the one usingsasl3 (ending up in the above failure) I noticed that the encoded token sent to the Namenode (to wrap the RPC) was always 4 bytes longer, and the diff was at the front (so the first four bytes were different for some reason, dumping binary was a joy). Google then helped and I found:

https://lists.andrew.cmu.edu/pipermail/cyrus-sasl/2017-March/003002.html

So the extra 4 bytes in front are the length of the payload, added by GSS-API. Cyrus Sasl follows a RFC, Java another one. The trick to make everything to work is to:

• remove the first 4 bytes after encoding with sasl.encode(), and then send to the Namenode.
• add 4 bytes with the len of the token received from the Namenode (something like struct.pack('!I', len(response.token)) + response.token is sufficient).

All the above was to make cyrus sasl to work with RPCs between Client and Namenode, that use GSS-API/kerberos in my case. This makes snakebite to work with calls like ls, but it doesn't when we want to get the content of a file for example, because the code to encrypt the RPC between client and Datanode is still missing. The RPC in fact doesn't use GSS-API, but Digest-MD5, and it follows a different protocol (Datanode protocol).

So next step is to implement the missing code and see if it works. If so, we'll be able to switch snakebite back to cyrus sasl and finally use it in Hadoop clusters with Kerberos and RPC encryption.

[elukey]

As reference https://issues.apache.org/jira/browse/HDFS-6606

[elukey]

I tried very hard to make everything working, but for the moment I am declaring defeated after many unsuccessful tries. The main issue is that the encrypt/decrypt part of python-sasl (and hence cyrus sasl) seems not working for the DIGEST-MD5 encryption. This is an example of what snakebite receives after a successful sasl handshake:

payload: "rspauth=d6141c413171a5eb119aedbbeb8db387"
cipherOption {
  suite: AES_CTR_NOPADDING
  inKey: "\343;p\214\224V\326\026L\367\371d\177\222x\271tf\241N\345\236\206\321F~\000\001\000\000\000\000"
  inIv: "\370l&\317\246\304x.\177\307\336\236C\336\332\225"
  outKey: "\307\214\314\'\355\276A\'\302\360N\261uw\177\347\323T\303\322\342-As\372$\000\001\000\000\000\001"
  outIv: "#\326N\260\237\337s\232N\027N\255L\3265F"
}

The various Iv and Key fields are encrypted, by the Datanode, using the DIGEST-MD5 session keys negotiated via SASL just before (when using auth-conf). I planned to use PyCryptodome for AES encryption but I wasn't able to get to test it.

The first issue was the error described in cyrusimap/cyrus-sasl#614. I found a workaround, namely patching python-sasl, but then I just ended up in a more generic error with more specific hints about what to change/patch.