privacy

[martindurant]

@bdrosen96 , would any of your work allow RCP "privacy" mode ("hadoop.rpc.protection": "privacy"}) to be enabled?

[bdrosen96]

yes definitely.

[martindurant]

As you can see from the commit log in this repo, I have already folded in some of your work. Do you know what it would take to get "privacy" to work? Even better, would you be willing to contribute :) ?

[bdrosen96]

Here are some changes that I think are still needed:

https://github.com/bdrosen96/libhdfs3/pull/1/files
https://github.com/bdrosen96/libhdfs3/pull/3/files
https://github.com/bdrosen96/libhdfs3/pull/6/files
https://github.com/bdrosen96/libhdfs3/pull/9/files
https://github.com/bdrosen96/libhdfs3/pull/10/files

Secure Data Transfer:

bdrosen96/libhdfs3#11
bdrosen96/libhdfs3#12
bdrosen96/libhdfs3#13
bdrosen96/libhdfs3#14

KMS:

bdrosen96/libhdfs3#15
bdrosen96/libhdfs3#16
bdrosen96/libhdfs3#17

[bdrosen96]

There are three more new PRs from today:

bdrosen96/libhdfs3#18
bdrosen96/libhdfs3#19
bdrosen96/libhdfs3#20

They really should have just been one PR.

[martindurant]

What do these ones do, @bdrosen96 ?
Currently #3 is up to just before KMS on your list, but I suspect that those changes already exist in HAWQ so might be redundant here. Is there any way you can test #3? I can confirm that it builds and hdfs3 seems to work with it without changes, but I don't have a secured cluster to test against right now.

[bdrosen96]

This fixes a bug with SASL negotiation when protection setting is privacy or integrity for kerberos/GSSAPI. The odd thing is that this only seems to reproduce on some clusters so it was not noticed before. In addition, I also added two new session config variables to allow a user to manually specify RPC protection and data transfer protection rather than just getting it via SASL in case that logic has further issues. I will have one more follow up PR, likely tomorrow (#21) that with fix the new config variable to use the string names used by hadoop instead of the SASL QOPS number equivalent.

As for testing 3, I can try to do so, assuming I can easily build it using same process as I do now in our internal tests, but I'm not sure I will definitely be able to get to it tomorrow.

Which changes do you think already exist in HAWQ?

[martindurant]

Well I'm glad that at least you have this in production, to pick up those bugs for as many configurations as possible.
I would appreciate description of the config variable changes - or are these things that are picked up by LIBHDFS3_CONF?

I think HAWQ implemented the KMS stuff, or at least these is some code touching that, and I haven't yet looked at your PRs to see if they amount to the same or not.

Side question: do you know if the SASL implementation can be called to wrap/unwrap general bytes for encrypting yarn RPC streams? I would like to be able to command yarn from python directly... Or, alternatively, there are various SASL implementations in python (or wrappers), but I have no idea whether they implement the more complicated security layers. Any insight would be very useful.

[bdrosen96]

The two new config vars are:

hadoop.rpc.protection
dfs.data.transfer.protection

and with last PR that went in this morning, they mean same thing as the XML hadoop vars of the same name and can have same values (privacy, authentication or integrity)

I'm not sure how the SASL implementation will work with yarn RPC since I have never tried to do so. I would not be surprised if it would work with it, assuming YARN uses similiar RPC as name node. Not sure that yarn has as many options for configuring RPC though.

[martindurant]

Excellent stuff, looks good. Do you intend to work further on this? I wonder, is https://github.com/martindurant/libhdfs3-downstream now a good place for you to work against? It might be convenient for you to be nearer the upstream.

As far as I know, YARN RPC uses exactly the same mechanisms as HDFS, even being defined by the same 'hadoop.rpc.protection' parameter. cerastes attempts to do this, but using a wrapper around cyrus-sasl.

[bdrosen96]

I don't have any current plans to make more changes unless/until a new issue may come up.

Once all the needed changes are in the downstream repo and it has been confirmed to work with all the various cases my current repo does, then I could possibly switch.

That probably explains why hadoop.rpc.protection is in core-site.xml and not hdfs-site.xml.

[martindurant]

@bdrosen96 : as expected, I am having trouble merging more of your code into the current HAWQ-based version, because it implements KMS, but in a different way to your code. For a trivial change, at https://github.com/ContinuumIO/libhdfs3-downstream/blob/master/libhdfs3/src/client/OutputStreamImpl.cpp#L255 it uses fileStatus.getFileEncryption() where you have getEncryptionInfo() - I am assuming these do the same, but the interface is different.

Do you have any recommendations for moving forward? It is possible to pull only the "private"-related material out of your code and trust that HAWQ's KMS stuff is right? I care much more about the former than the latter. Any other idea for how to consolidate would be appreciated. Unfortunately, I am no star c++ coder; just applying only your changes post-KMS is proving no simpler, since the base code is now different in places...

[bdrosen96]

From what I can see, the version of KMS supported in HAWQ may have some functionality gaps.

1 It does not support kerberos at all, where my version does.

2 It does not support specifying KMS auth type.

3 It does not support tokens.

4 I don't think it supports the Base64 variant:

std::replace(encoded.begin(), encoded.end(), '+', '-');
std::replace(encoded.begin(), encoded.end(), '/', '_');

on encrypt

or decrypt:

int rem = data.length() % 4;
if (rem) {
rem = 4 - rem;
while (rem != 0 ) {
data = data + "=";
rem -= 1;
}
}
std::replace(data.begin(), data.end(), '-', '+');
std::replace(data.begin(), data.end(), '_', '/');

I have not yet looked at how the kms itself is integrated into the encrypt/decrypt portions.

[martindurant]

Would you, then, advocate working from your branch's tip and integrating whatever is needed on the HAWQ end, rather than the other way around as I've been trying?

[bdrosen96]

For point 4, looks like this is actually partially handled, just in CryptoCodec.cpp.

As for encoding, I see that in one place, but I don't yet see decode and I think there is at least one other place that might need encoding.

I would suggest first intergrating existing PR (without KMS) plus the new PRs from last week as one unit into the code base. Then start with new PR to handle the KMS/HAWQ integration. Most likely way to try to handle this would be to first to try to add token and kerberos SPNEGO support for existing HAWQ KMS as one PR.

[bdrosen96]

Interesting thing is it looks like the HAWQ kms stuff may have copied mine to some extent (or we both copied from same reference). If you look at HttpClient.cpp you see we have same macros for CURL.