feat(services/azdls): list start from #5242

alexwilcoxson-rel · 2024-10-24T20:51:48Z

Which issue does this PR close?

Closes #.

Rationale for this change

This changes proposes implementation of the start_after workaround used by Hadoop to improve listing in ADLS from a particular point.

This is important for table formats like Delta Lake that make use of list_with_offset in object store to get the latest version of a table.

In my testing listing directories with > 10k records similar to what Delta Lake would do improves times from seconds to milliseconds (~3seconds to 60ms).

Also this adds SAS token support which could be split into its own PR but needed it for testing against my company's ADLS accounts.

What changes are included in this PR?

Changes to azdls module to include list with start_after functionality. This includes a module crc64 which computes crc for use in the continuation token generated for start after

Are there any user-facing changes?

Yes, start_after will function for azdls, and SAS token support.

@Xuanwo I'm opening this as a draft to get initial feedback. This is not directly supported by Microsoft, or I should say documented. In pursuit of them adding this capability to the azblob endpoints, they were the ones that directed me to Hadoop example, so I have confidence in it but could see if its something they would officially support/document.

One caveat as well is I believe (per the hadoop code) the change is will only work if hierarchical namespace (xns) is enabled which isn't something I believe we can tell when the list request is being made. It could be the user has to opt in and inform if xns is enabled. Hadoop code does another request to figure that out dynamically which I'd personally want to avoid.

Xuanwo

Hi, thank you @alexwilcoxson-rel for working on this. I've been considering porting logic from Hadoop to OpenDAL for a while but haven't taken action yet.

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

core/src/services/azdls/backend.rs

core/src/services/azdls/crc64.rs

core/src/services/azdls/core.rs

core/src/services/azdls/crc64.rs

alexwilcoxson-rel · 2024-10-28T20:13:51Z

hey @Xuanwo as I was refactoring the code I wrote a test against one of our actual storage accounts. I then executed the list call with recursive = true as the object_store integration does. However when I do that list result from azure contains everything.

I'm trying to figure out how recursive could be impacting the azdls list but I don't see it used by azdls service.

EDIT: ah I see the CompleteLayer, looking into this.

…t list complete layer

alexwilcoxson-rel · 2024-10-29T01:31:41Z

Ok so I found the FlatLister does not retain the OpList args.

Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls
start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

Xuanwo · 2024-10-29T02:29:43Z

Thank you @alexwilcoxson-rel for catching this.

Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.

It should be fine as long as we handle the capability correctly.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls

start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

core/src/raw/oio/list/flat_list.rs

core/src/services/azdls/backend.rs

alexwilcoxson-rel · 2024-10-30T00:55:32Z

The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls

start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

Yes I will create a follow up issue to switch to use recursive parameter available on azdls list. Mentioning though the seeming incompatibility with this start_after workaround

alexwilcoxson-rel · 2024-10-30T00:56:50Z

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

gate behind feature

alexwilcoxson-rel · 2024-11-04T19:22:09Z

Hey @Xuanwo just letting you know I probably won't get back to this until later in the week. I will share though that I am using this in our fork of version ~0.48 I think. We're using it along size the base object_store azure store (since those have all the put support for delta), and it has gone well!

Looking forward to getting this in along with the put if-not-match changes that are in progress as well!

Xuanwo · 2025-04-07T11:44:07Z

Hi, @alexwilcoxson-rel, do we have a status change here? I'm open to get this feature merged under a new config called enable_list_with_start_after.

alexwilcoxson-rel · 2025-05-05T17:49:06Z

Hey @Xuanwo sorry for the delay on this, I have some cycles to try to get it across the line

alexwilcoxson-rel · 2025-05-05T18:11:53Z

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

done

alexwilcoxson-rel · 2025-05-05T19:01:27Z

In testing this with deltalake table loads, i see the object_store integration is doing a stat per list entry

is there a way around this if the list response contains the needed metadata? i notice the metadata key stuff was removed (for the better 😄 )

Xuanwo · 2025-05-21T09:53:50Z

sas_token support has been added in #6205

core/src/services/azdls/backend.rs

alexwilcoxson-rel · 2025-05-27T15:40:10Z

In testing this with deltalake table loads, i see the object_store integration is doing a stat per list entry

is there a way around this if the list response contains the needed metadata? i notice the metadata key stuff was removed (for the better 😄 )

@Xuanwo can you provide input here please?

Xuanwo · 2025-05-28T03:40:59Z

In testing this with deltalake table loads, i see the object_store integration is doing a stat per list entry

is there a way around this if the list response contains the needed metadata? i notice the metadata key stuff was removed (for the better 😄 )

I think it's fine for us to simply remove the stat from the list, since object_store doesn't do that anyway. We always generate the same metadata as object_store does.

Xuanwo

Thank you for working on this!

core/Cargo.toml

Xuanwo · 2025-05-28T03:51:49Z

core/src/layers/complete.rs

@@ -162,11 +162,11 @@ impl<A: Access> CompleteAccessor<A> {
            (true, false) => {
                // Forward path that ends with /
                if path.ends_with('/') {
-                    let p = FlatLister::new(self.inner.clone(), path);
+                    let p = FlatLister::new(self.inner.clone(), path, args);


We don't need to touch here since start_after is handled by azdls directly.

I thought this was related to this comment: #5242 (comment)

I thought this was related to this comment: #5242 (comment)

Oh, I see. Sorry for overlooking that.

How about we split them into two PRs to handle them separately? I can also find some time to benchmark azdls’s recursive list.

We can add start_after support in this PR and consider removing recursive support for azdls if we confirm that its performance is really poor.

I wish we can merge this PR quickly so you can use the mainline opendal ASAP 💌

so should i should remove all changes in flat_list and complete and then there will be a test with recursive and azdls?

so should i should remove all changes in flat_list and complete and

Yes, let's remove all changes in flat_list and complete.

then there will be a test with recursive and azdls?

This has been covered in our behavior tests, so as long as our CI passes, everything should be fine.

core/src/services/azdls/backend.rs

core/src/services/azdls/core.rs

core/src/services/azdls/lister.rs

core/src/services/azdls/core.rs

alexwilcoxson-rel · 2025-05-28T20:18:05Z

@Xuanwo if your okay with latest changes, I can create follow up issues for

recursive investigation on azdls
object_store stat removal during listing

Xuanwo · 2025-05-29T04:47:43Z

if your okay with latest changes, I can create follow up issues for

Perfect! Most changes looks good to me. We only need to remove the FlatLister changes.

feat(services/azdls): list start from and sas token support

ddfd3ae

github-actions bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Oct 24, 2024

alexwilcoxson-rel mentioned this pull request Oct 24, 2024

Use reconstructed ListBlobs marker to provide list offset support in MicrosoftAzure store apache/arrow-rs#6174

Closed

chore(services/azdls): clippy

384c71e

Xuanwo reviewed Oct 25, 2024

View reviewed changes

Xuanwo mentioned this pull request Oct 25, 2024

Oct 21, 2024: This week in OpenDAL #5220

Closed

alexwilcoxson-rel added 2 commits October 28, 2024 20:29

feat(services-azdls): refactored crc64 and pass list args through fla…

aba400e

…t list complete layer

feat(services-azdls): fmt

c370249

Merge branch 'main' into azdls-list-start-from

df6cf90

Xuanwo reviewed Oct 29, 2024

View reviewed changes

core/src/raw/oio/list/flat_list.rs Outdated Show resolved Hide resolved

core/src/services/azdls/backend.rs Outdated Show resolved Hide resolved

Xuanwo mentioned this pull request Oct 29, 2024

Oct 28, 2024: This week in OpenDAL #5251

Closed

Merge branch 'main' into azdls-list-start-from

7be7756

alexwilcoxson-rel added 3 commits May 5, 2025 12:41

Merge branch 'main' into azdls-list-start-from

3777f54

chore: merge fix

7d1285f

chore: fmt

53b5769

chore: pr feedback

6313a28

feat: add azdls-enable-list-with-start-after

413aa30

alexwilcoxson-rel marked this pull request as ready for review May 5, 2025 18:14

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label May 5, 2025

alexwilcoxson-rel added 2 commits May 5, 2025 13:16

chore: lint

fcfb897

chore: lint

03301db

Merge branch 'main' into azdls-list-start-from

2185b87

alexwilcoxson-rel commented May 27, 2025

View reviewed changes

core/src/services/azdls/backend.rs Outdated Show resolved Hide resolved

chore: post merge fix

d2f0b30

Xuanwo reviewed May 28, 2025

View reviewed changes

core/src/services/azdls/core.rs Outdated Show resolved Hide resolved

alexwilcoxson-rel changed the title ~~feat(services/azdls): list start from and sas token support~~ feat(services/azdls): list start from May 28, 2025

alexwilcoxson-rel added 3 commits May 28, 2025 13:54

refactor(azdls): move crc and continuation token to core for

b830f6a

refactor(azdls): config argument for start_after

2d15502

chore: spelling

52a7805

feat(services/azdls): list start from #5242

Are you sure you want to change the base?

feat(services/azdls): list start from #5242

Uh oh!

Conversation

alexwilcoxson-rel commented Oct 24, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexwilcoxson-rel commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexwilcoxson-rel commented Oct 29, 2024

Uh oh!

Xuanwo commented Oct 29, 2024

Uh oh!

Uh oh!

Uh oh!

alexwilcoxson-rel commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexwilcoxson-rel commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexwilcoxson-rel commented Nov 4, 2024

Uh oh!

Xuanwo commented Apr 7, 2025

Uh oh!

alexwilcoxson-rel commented May 5, 2025

Uh oh!

alexwilcoxson-rel commented May 5, 2025

Uh oh!

alexwilcoxson-rel commented May 5, 2025

Uh oh!

Xuanwo commented May 21, 2025

Uh oh!

Uh oh!

alexwilcoxson-rel commented May 27, 2025

Uh oh!

Xuanwo commented May 28, 2025

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Xuanwo May 28, 2025

Choose a reason for hiding this comment

Uh oh!

alexwilcoxson-rel May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexwilcoxson-rel May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexwilcoxson-rel commented May 28, 2025

Uh oh!

Xuanwo commented May 29, 2025

Uh oh!

alexwilcoxson-rel commented Oct 28, 2024 •

edited

Loading

alexwilcoxson-rel commented Oct 30, 2024 •

edited

Loading

alexwilcoxson-rel commented Oct 30, 2024 •

edited

Loading

Xuanwo May 28, 2025 •

edited

Loading