Move storage_adapters(S3, HDFS, GCS, ABFS) to a separate repository

[majetideepak]

There are a bunch of improvements to storage_adapters pending merge.
The storage_adapters also require more features with respect to credentials and authorization.
Meta does not use these storage_adapters but has to maintain them internally as storage_adapters live in Velox.
The code merge process is also delayed because of the current import to Meta's mono-repo.

The idea is to move storage_adapters to a separate repo and include them as a submodule in Velox.
All the tests for the storage_adapters will live in this new repo.

There will be no CI in the new repo. All testing will continue in the Velox CI.

The storage_adapters development will happen in the new repo.
Build scripts will be set up to get the latest Velox.
Once the unit tests pass locally, the developer can create a Velox branch, change the submodule to the development branch, and open PR against it. Once CI passes, the development branch will be merged in the new repo.
The developer then has to advance the submodule to the storage_adapters and merge that PR.
The development would look similar to Prestissimo <-> Velox development except for the CI part.

Having a single CI also simplifies making breaking FileSystem API changes in both the repos.

[majetideepak]

CC: @pedroerp @kgpai @assignUser

[kgpai]

@majetideepak On the whole I am aligned with moving storage adapters.

I have some questions though .

There will be no CI in the new repo. All testing will continue in the Velox CI.

How does this work say if you need to add or upgrade a dependency ? Does it mean that the only way to know if a change builds/tests/works is to open a PR against velox ?

Once the unit tests pass locally, the developer can create a Velox branch, change the submodule to the development branch, and open PR against it.

This unit test is on the dev machine I presume ? If so its not reliable. We should have some CI for the adapters repo too.

We should also think about the developer flow and how this impacts Prestissimo.

[majetideepak]

Yes. Developers will test all changes locally, open a PR against the new repo, pull the PR as part of a Velox PR, and test CI to be able to merge in the new repo.
We will need an adapters CI in Velox to track breaking FileSystem API changes until we have Velox versioning. Making breaking FileSystem API changes will not be possible if we have a CI job in the new repo. Having a single CI will allow us to make these changes in sort of an atomic manner in the Velox repo.

The developer flow would look very similar to Prestissimo <-> Velox development. Git submodules and make scripts will handle this.

Also, see my reply for Pedro below.

[pedroerp]

There will be no CI in the new repo. All testing will continue in the Velox CI.

Why is that? Since most of the adapter's code would be in the new repo, shouldn't we have most of the CI that tests these things in the new repo, and then just have sort of integration test CI jobs in Velox?

[majetideepak]

We must keep one adapters CI in Velox to track breaking FileSystem API changes until we have Velox versioning. Most Velox clients consume the latest Velox today. Making a breaking FileSystem API change in Velox will not be possible if we have a CI job in the new repo.
We have a cycle between the repos. Base FileSystem class (Velox repo) -> storage_adapters (new repo) -> Velox consumers (Velox repo). To be able to break this cycle, we will need to have a single CI in Velox repo.
Users can test unit tests/integration tests locally with their changes. The final merge will happen after testing in the Velox repo against the adapters CI.
Also, see my comment above for Krishna.

[pedroerp]

I see. Could you present the proposal at the open source sync tomorrow? May be a bit easier to discuss over VC.

[majetideepak]

During the OSS sync, we felt that the Storage Adapters are a key component of Velox and probably should continue to live in Velox. Meta will address the velocity issues. We decided to table this for now.