binary builder should check authenticity and integrity of downloaded packages

[gberche-orange]

Thanks for sharing this great work for building buildpack binaries!

As discussed into cloudfoundry/java-buildpack-dependency-builder#6 the buildpacks binary builder should strive to verify authenticiy and integrity of downloaded artefacts

For example in https://github.com/cloudfoundry/binary-builder/blob/master/templates/httpd_blueprint.sh.erb#L8-L9 the builder is currently vulnerable to man in the midddle attacks that could return compromised packages.

In the case of apache, apache source packages are digitally signed with apache public keys, and should be checked by the binary builder.

Other cases might check against checksum downloaded through HTTPS on the main distrib (and not mirror).
http://apache.mirrors.tds.net/apr/#sig

[mhoran]

Hey @gberche-orange. Thanks for bringing this to our attention. I think we should definitely download sources via HTTPS at all times, and figure out a non-intrusive way to verify signatures.

We had started this project off by downloading sources via git, in particular GitHub, and relied on the intrinsic "security" provided by the git protocol. However, we should probably spend some time improving this mechanism, as one of the major reasons behind this effort is to ensure we're confident in the integrity of our packages.

I'm wondering if the preferred mechanism would be to provide a hash of the download file to the binary builder at run time -- e.g. ./bin/binary-builder httpd <version> <hash>, or if we should bake in the hash verification to the binary builder itself.

Fetching the hashes from the distribution sites seems like it still leaves open a man-in-the-middle attack, though if we move to all HTTPS URLs that would be more difficult -- but as history has shown, not impossible.

I'd love to hear your thoughts, and will discuss with the team tomorrow.

[gberche-orange]

I understand that fetching the source distrib checksum from the distribution through https is the standard way to check packages as they come released without including manual operation each time. I'd rather then have the binary builder do the download and verification itself.

As for signatures, since they typically don't change much and in particular not with each new package version it seems better to embed the public keys (e.g. apache public keys) into the binary builder repo I would think, and then check the signature of the downloaded sources against the public key.

(Sorry for late response, was out if office for a while)

[flavorjones]

Thanks for pointing this out. I'm going to schedule work on making sure we're specifying the expected checksum for any downloads.

[gberche-orange]

Thanks @flavorjones

Do you mean "making sure we're verifying the expected checksum for any downloads, against checksum securely downloaded from source repos" and verifying signature for digitally signed source packages ?

[flavorjones]

I believe we've completed the work you described above -- all the builder pipelines now require a checksum or a GPG key, see https://github.com/cloudfoundry/binary-builder/blob/master/bin/binary-builder#L34-L40.

[flavorjones]

And, to your point above, all the pipelines then use that checksum or GPG key to verify the source tarball.

If you see anything we're not doing, please do let me know. I'm putting together a track of work related to this and to your request at #5 to provide more structured metadata around a "chain of custody" so that end users of a buildpack can verify all the binaries being delivered.

[gberche-orange]

Thanks @flavorjones !

I see the source files are indeed downloaded and verified against checksum or signature upfront but then the some template steps don't seem to use the downloaded and extracted tar (e.g. httpd-apr jruby ) and are therefore still vulnerable to man in the middle attacks.

Is that something you plan on fixing ?

Also, the docker file is vulnerable to man in the middle attacks in fetching
~/.vim/install that is sourced

[flavorjones]

Heya @gberche-orange,

Thanks for pointing this out. In the jruby case we do verify a checksum:

binary-builder/templates/jruby_blueprint.sh.erb

Line 12 in 1430d38

if [[ $(md5sum apache-maven-<%= maven_version %>-bin.tar.gz) != *"<%= maven_md5 %>"* ]]; then

but in the httpd case we're not. I've filed a bug report:

https://www.pivotaltracker.com/story/show/103622774

Also worth noting that we're not yet verifying checksums when we build some dependencies in the php-buildpack, which I've filed another bug for here:

https://www.pivotaltracker.com/story/show/103623654

[gberche-orange]

Thanks @flavorjones for sharing the additional stories added.

Thanks for pointing this out. In the jruby case we do verify a checksum:

Sorry, I had missed that, thanks for correcting me on this.

[flavorjones]

Just an update that the current code on master verifies checksums for all the components of httpd.

The PHP story is in motion, and nearly done.

[flavorjones]

@gberche-orange The code for verifying checksums is here: https://github.com/cloudfoundry/binary-builder/blob/master/recipe/httpd_meal.rb#L152-L171

[gberche-orange]

@flavorjones great thanks for the update, looks good to me.