Oct 21, 2024: This week in OpenDAL

[Xuanwo]

Introduction

Goal of this ticket is a weekly summary if interesting things happening in OpenDAL over the last week. Note this is not a complete list. Please feel free to comment on this ticket about things that I may have missed or you think should get wider attention by the community.

Strongly inspired by This week in DataFusion

Highlights

• [ANNOUNCE] New OpenDAL PMC Member: Morris Tai #5245, congrats to @morristai.

Xuanwo's TLDR

We are preparing for the 0.50.1 release and I am personally pretty excited about:

• feat(core): add version(bool) for List operation to include version d… #5106 from @meteorgan
• feat(ofs): introduce ofs macos support #5136 from @oowl
• fix(bindings/c): use ManuallyDrop instead of forget to make sure pointer is valid #5166 from @ethe
• feat: Add if_none_match for write #5129 from @ForestLH
• feat: Add OpenDAL Compat #5185 from @Xuanwo
• feat(core): abstract HttpFetch trait for raw http client #5184 from @everpcpc
• feat: Support NebulaGraph #5116 from @GG2002

Upcoming Releases

• Tracking issues of OpenDAL v0.50.1 Release #5203
• Release Plan of Apache OpenDAL v0.50.2 #5241

Help Wanted Issues

• Tracking Issues with Conditional Write #5218
• Tracking issues of user metadata support #4842

Projects Working On

• D binding: feat(bindings/d): add D bindings support #5181 by @kassane
• kv scan API: feat(adapter/kv): support async iterating on scan results #5208 by @PragmaTwice
• List start_after for azdls: feat(services/azdls): list start from and sas token support #5242 by @@alexwilcoxson-rel

Interesting discussions underway

• The Way to OpenDAL Rust Core v1.0 #5189
• ideas: Split opendal rust core into multiple crates #5206
• [DISCUSS] Move "Who is using OpenDAL" to discussions "Show and tell" #5221
• Add `if_not_exists` in `OpWrite` to make behavior more clear #5223
• How close are we from runtime agnostic? #5225
• OpenDAL Grafana Dashboard Template #5232

Community

• Discussion
• Discord

[Xuanwo]

Thank you, @alexwilcoxson-rel, for initiating a draft with #5242. This utilizes a preview API of azdls to enhance listing performance:

This is important for table formats like Delta Lake that make use of list_with_offset in object store to get the latest version of a table.

In my testing listing directories with > 10k records similar to what Delta Lake would do improves times from seconds to milliseconds (~3seconds to 60ms).

[Xuanwo]

• Next week: Oct 28, 2024: This week in OpenDAL #5251