refactor: ABFS implementation #11419

majetideepak · 2024-11-04T13:54:57Z

Combine AbfsAccount and AbfsConfig in a separate file.
Clean up API naming and clarify semantics.
Add a new constructor for AbfsWriteFile to specify a client. This is used for testing.

netlify · 2024-11-04T13:55:14Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`5475558`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/673625f6696151000840c72a

FelixYBW · 2024-11-04T23:36:08Z

@zhli1142015

zhli1142015 · 2024-11-05T00:12:19Z

velox/connectors/hive/storage_adapters/abfs/AdlsFileClient.h

+ * To facilitate unit testing of file write scenarios, we define the
+ * AzureDatalakeFileClient here, which can be mocked during testing.
+ */
+class AdlsFileClient {


Please use AzureDatalakeFileClient, AdlsFileClient indicates a different thing in our context.
Thanks.

The Azure client name for writing is DataLakeFileClient. I renamed the implementations accordingly and tried to keep the name short.

zhli1142015 · 2024-11-05T00:13:52Z

velox/connectors/hive/storage_adapters/abfs/tests/MockAdlsFileClient.h

+ * https://github.com/Azure/Azurite/wiki/ADLS-Gen2-Implementation-Guidance
+ *
+ * To facilitate unit testing of file write scenarios, we define the
+ * IBlobStorageFileClient here, which can be mocked during testing.


Please update the name here also.

zhli1142015 · 2024-11-05T00:35:01Z

velox/connectors/hive/storage_adapters/abfs/AbfsConfig.h

+
+namespace facebook::velox::filesystems {
+
+static std::string kAzureBlobEndpoint{"fs.azure.blob-endpoint"};


What does this property do? Is this for testing only, if yes please add comments.
Thanks.

majetideepak · 2024-11-05T21:07:55Z

@zhli1142015 thanks for your review! I addressed your comments. Can you take another look?

majetideepak · 2024-11-05T21:23:28Z

@zhli1142015 I noticed that the usage of DataLakeFileClient::Flush is not optimal. We flush when the file is closed
and this might consume a lot of memory if the file written is big. We need to flush similarly to S3 periodically (10Mib chunks). What do you think?

czentgr

nice refactor. Looks much more clean now.

czentgr · 2024-11-05T22:56:45Z

velox/connectors/hive/storage_adapters/abfs/AbfsConfig.cpp

+  std::string_view file;
+  bool isHttps = true;
+  if (path.find(kAbfssScheme) == 0) {
+    file = path.substr(8);


Lets use

file = path.substr(kAbfssScheme.length());

czentgr · 2024-11-05T22:56:56Z

velox/connectors/hive/storage_adapters/abfs/AbfsConfig.cpp

+  if (path.find(kAbfssScheme) == 0) {
+    file = path.substr(8);
+  } else if (path.find(kAbfsScheme) == 0) {
+    file = path.substr(7);


Lets use

file = path.substr(kAbfsScheme.length());

velox/connectors/hive/storage_adapters/abfs/AbfsReadFile.h

czentgr · 2024-11-06T04:22:01Z

velox/connectors/hive/storage_adapters/abfs/AbfsWriteFile.cpp

+    const config::ConfigBase& config) {
+  auto abfsAccount = AbfsConfig(path, config);
+  std::shared_ptr<AzureDataLakeFileClient> client =
+      std::make_shared<DataLakeFileClientWrapper>(


Just curious why the DataLakeFileClientWrapper is shared_ptr and not unique_ptr? Would something else access this if it is part of the client?

shared_ptr made it easier to write tests. But I changed this to a unique_ptr as I think it should be as well.

czentgr · 2024-11-06T04:27:04Z

velox/connectors/hive/storage_adapters/abfs/tests/AbfsCommonTest.cpp

+      abfssConfig.connectionString(),
+      "DefaultEndpointsProtocol=https;AccountName=foobar;AccountKey=456;EndpointSuffix=core.windows.net;");
+
+  // test with special characters


Nit. Commit not complete sentence.

czentgr · 2024-11-06T04:36:53Z

velox/connectors/hive/storage_adapters/abfs/tests/AzuriteServer.cpp

+      {{"fs.azure.account.key.test.dfs.core.windows.net", key_},
+       {kAzureBlobEndpoint, endpoint}});
+
+  // Update the default config map with the supplied configOverride map


Nit. Missing .. Or do we even need this comment?

I updated the function documentation instead.

czentgr · 2024-11-06T04:40:40Z

velox/connectors/hive/storage_adapters/abfs/tests/AzuriteServer.h


  virtual ~AzuriteServer();

 private:
  int64_t port_;
+  std::string account_{"test"};


Maybe all of these 4 new members are const?

zhli1142015 · 2024-11-06T06:11:42Z

ient::Flush is not optimal. We flush when the file is closed
and this might consume a lot of memory if the file written is big. We need to flush similarly to S3 periodically (10Mib chunks). What do you think?

I thought we don't cache data and send by chunk. The append API sends data to remote directly.
The behavior you mentioned in S3, may be better, we can take a look later.
Thanks.

majetideepak · 2024-11-06T14:11:23Z

@czentgr thanks for the review. I addressed your comments.

majetideepak · 2024-11-06T19:29:32Z

Filed #11456 for the write improvements.

czentgr

Looks good. Just one nit.

czentgr · 2024-11-07T22:48:36Z

velox/connectors/hive/storage_adapters/abfs/tests/MockDataLakeFileClient.h

    filePath_ = tempFile->getPath();
  }

+  MockDataLakeFileClient(std::string_view filePath) : filePath_(filePath) {}
+
+  std::string_view path() {


Nit, this could be std::string_view path() const.

kevinwilfong · 2024-11-11T17:56:40Z

This needs a maintainer to approve, cc: @Yuhta @xiaoxmeng

https://velox-lib.io/docs/community/components-and-maintainers/

majetideepak · 2024-11-13T12:00:43Z

@kevinwilfong I am the maintainer for the storage_adapters and I approve this :).

kevinwilfong · 2024-11-13T23:32:18Z

I just double checked with the PLC and it sounds like that's not sufficient.

majetideepak · 2024-11-14T16:32:24Z

@zhli1142015 Can you please take a look and approve this PR?

Yuhta

Stamp since @zhli1142015 has the most knowledge about this and he is good with the change.

facebook-github-bot · 2024-11-15T18:47:53Z

@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

kevinwilfong · 2024-11-16T00:56:42Z

It looks like if we merge this it will break Presto, could you make a VELOX_ENABLE_FORWARD_COMPATIBILITY change in Presto to handle changing the namespace

https://github.com/prestodb/presto/blob/ecfda83fe2987a191b1cd8f722e9fe2e9d4c8b0e/presto-native-execution/presto_cpp/main/PrestoServer.cpp#L1304

majetideepak · 2024-11-18T09:46:43Z

@kevinwilfong Added Presto change here prestodb/presto#24063

kevinwilfong · 2024-11-18T17:58:05Z

Thanks! I'll try merging this again.

facebook-github-bot · 2024-11-18T23:23:43Z

@kevinwilfong has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-11-19T05:41:22Z

@kevinwilfong merged this pull request in 7d0b84e.

conbench-facebook · 2024-11-19T06:10:42Z

Conbench analyzed the 1 benchmark run on commit 7d0b84e8.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Summary: Combine AbfsAccount and AbfsConfig in a separate file. Clean up API naming and clarify semantics. Add a new constructor for AbfsWriteFile to specify a client. This is used for testing. Pull Request resolved: facebookincubator#11419 Reviewed By: Yuhta Differential Revision: D66015694 Pulled By: kevinwilfong fbshipit-source-id: 7224aaa1e3cda99c1596546e8050676c635396a5

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 4, 2024

majetideepak force-pushed the abfs-gcs-multifs branch 3 times, most recently from 58c4444 to be9b010 Compare November 4, 2024 16:12

zhli1142015 reviewed Nov 5, 2024

View reviewed changes

majetideepak marked this pull request as ready for review November 5, 2024 20:59

majetideepak force-pushed the abfs-gcs-multifs branch from 890d571 to cd4a59f Compare November 5, 2024 21:06

majetideepak force-pushed the abfs-gcs-multifs branch from cd4a59f to 9c88379 Compare November 5, 2024 21:12

czentgr reviewed Nov 6, 2024

View reviewed changes

czentgr approved these changes Nov 7, 2024

View reviewed changes

majetideepak force-pushed the abfs-gcs-multifs branch from 3502540 to 48ea8d7 Compare November 8, 2024 22:58

majetideepak added the ready-to-merge PR that have been reviewed and are ready for merging. PRs with this tag notify the Velox Meta oncall label Nov 8, 2024

majetideepak added 2 commits November 14, 2024 11:30

refactor: ABFS implementation

dad9603

fix rebase conflict

eee3afa

majetideepak force-pushed the abfs-gcs-multifs branch from 48ea8d7 to 5475558 Compare November 14, 2024 16:31

majetideepak added 4 commits November 14, 2024 11:31

fix test

a986331

address review comments

f016c1a

address review comments

a84bab5

address review comments

5475558

zhli1142015 approved these changes Nov 15, 2024

View reviewed changes

Yuhta approved these changes Nov 15, 2024

View reviewed changes

majetideepak mentioned this pull request Nov 17, 2024

[native] Allow incoming Velox abfs namespace change prestodb/presto#24063

Merged

facebook-github-bot closed this in 7d0b84e Nov 19, 2024

facebook-github-bot added the Merged label Nov 19, 2024

zuyu mentioned this pull request Nov 19, 2024

[native] Advance Velox and remove abfs namespace prestodb/presto#24085

Merged

zhouyuan mentioned this pull request Nov 20, 2024

[GLUTEN-6887][VL] Daily Update Velox Version (2024_11_20) apache/incubator-gluten#7997

Merged

majetideepak deleted the abfs-gcs-multifs branch June 19, 2025 20:18


		namespace facebook::velox::filesystems {

		static std::string kAzureBlobEndpoint{"fs.azure.blob-endpoint"};

refactor: ABFS implementation #11419

refactor: ABFS implementation #11419

Uh oh!

Conversation

majetideepak commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

FelixYBW commented Nov 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majetideepak Nov 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

majetideepak commented Nov 5, 2024

Uh oh!

majetideepak commented Nov 5, 2024

Uh oh!

czentgr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhli1142015 commented Nov 6, 2024

Uh oh!

majetideepak commented Nov 6, 2024

Uh oh!

majetideepak commented Nov 6, 2024

Uh oh!

czentgr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinwilfong commented Nov 11, 2024

Uh oh!

majetideepak commented Nov 13, 2024

Uh oh!

kevinwilfong commented Nov 13, 2024

Uh oh!

majetideepak commented Nov 14, 2024

Uh oh!

Yuhta left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Nov 15, 2024

Uh oh!

kevinwilfong commented Nov 16, 2024

Uh oh!

majetideepak commented Nov 18, 2024

Uh oh!

kevinwilfong commented Nov 18, 2024

Uh oh!

majetideepak commented Nov 4, 2024 •

edited

Loading

netlify bot commented Nov 4, 2024 •

edited

Loading

majetideepak Nov 5, 2024 •

edited

Loading

czentgr left a comment •

edited

Loading