[C++][Python] Potential improvements around supply chain security

[pitrou]

Describe the enhancement requested

For now this is more of a wishlist/discussion issue, but could grow into a more precise meta-task if we want to move forward.

There have been growing concerns over the years over the fragility of software supply chains, particularly when open source software is concerned. Some standards and practices have been proposed to help prevent such attacks:

• SLSA (apparently pronounced "salsa") is, AFAIU, a specification that help projects evaluate and improve their build and test practices
• Software bills of materials (SBOM) are a type of artifact that precisely describe the provenance of code shipped within a package (related link: announcement of a "SBOM for Python packages" project; also: Accelerating SBOM success with the help of SLSA)
• OpenSSF scorecards provide a standard vocabulary to evaluate a software projects' security practices
• Reproducible builds help ensure that binary artifacts have not been compromised, by allowing independent verification of build outputs

Arrow C++ in particular has a non-trivial set of dependencies that are incorporated in the build process in various ways. For example, for Python wheels we use vcpkg on a specific changeset, potentially with home-grown patches. This of course applies to other bindings of Arrow C++ where we may produce binary packages (such as R).

We should evaluate whether any of these could help us improve our intrinsic quality, or would merely amount to additional bureaucracy (related link: concerns by a prominent member of the Python packaging community).

Note: if desirable, this could, and should, typically be funded by interested companies.

Component(s)

C++, Python

[pitrou]

cc @raulcd @assignUser

[assignUser]

+1
Reproducible builds could also improve security but will likely be a big effort (except for java where there are afaik plugins that handle it).

[pitrou]

Good point, I think we should strive to achieve reproduce builds.

[raulcd]

Reproducible builds would help on releases too. It could potentially make patch and minor releases easier to achieve.

[pitrou]

Apparently someone has been producing automated OpenSSF scorecards, e.g. for PyArrow: https://deps.dev/pypi/pyarrow

[assignUser]

🧐
pyarrow 18.0.0 has no dependencies.