feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API (#8845)

- **Bump `opendal` / `object_store_opendal` to 0.57.0** (in the new
`vortex-object-store-opendal` crate); this version enables COS
versioning by default.
- **Use the upstream `pyo3-object_store` 0.11.0** from crates.io (no
longer vendored via `[patch.crates-io]`). The built-in store classes
(S3/Azure/GCS/HTTP/Local/Memory) keep their `PyObjectStore` interop, and
obstore users can still pass external stores.
- **Introduce `vortex-object-store-opendal`**, a new workspace crate
that builds an `object_store::ObjectStore` from an OpenDAL
`services::Cos` `OpendalStore` (`make_opendal_store(url, properties)`),
reused by both the URL registry and the Python store class.
- **Add a `CosStore` pyclass (opendal feature)** that wraps an
OpenDAL-backed Tencent COS `OpendalStore` and holds it as `Arc<dyn
object_store::ObjectStore>`. It is constructed with explicit config
(`bucket`, `endpoint`, `secret_id`, `secret_key`, `root`,
`disable_config_load`) and can be handed to
`vortex.io.read_url(store=...)` / `vortex.io.write(..., store=...)`.
- **Introduce `AnyVortexStore` in `io.rs`** so `read_url` / `write`
accept both the built-in `pyo3-object_store` classes and the new
OpenDAL-backed `CosStore` via a single `store=` argument.
- **`cos://` and `oss://` URL resolution** now go through the
OpenDAL-backed store, configured via OpenDAL environment variables
(`TENCENTCLOUD_SECRET_ID` / `TENCENTCLOUD_SECRET_KEY` / `COS_ENDPOINT`,
and `ALIBABACLOUD_ACCESS_KEY_ID` / `ALIBABACLOUD_ACCESS_KEY_SECRET` /
`OSS_ENDPOINT`).
- **Document the standalone-store usage and add unit tests**
(`cos_store_builds_object_store`).

**Rationale for this change**

Closes: #8844

Vortex's Python API already exposes concrete, standalone store objects
for S3/Azure/GCS/HTTP/Local/Memory, but Tencent COS and Alibaba OSS were
only reachable through URL-based resolution that relied on environment
variables. This adds first-class OpenDAL-backed stores so COS/OSS can be
used exactly like the other stores (construct once, pass via `store=`),
with better ergonomics and type checking.

**What changes are included in this PR?**

- new `vortex-object-store-opendal` crate (OpenDAL `services-cos`,
`opendal`/`object_store_opendal` 0.57.0)
- `pyo3-object_store` pinned to the upstream 0.11.0 (object_store
0.13.x, matching the workspace) — vendored fork removed
- new `CosStore` pyclass (gated on the `opendal` feature) wrapping an
OpenDAL COS `OpendalStore`, holding `Arc<dyn object_store::ObjectStore>`
- `AnyVortexStore` extractor in `io.rs` so `read_url` / `write` accept
the new store
- `cos://` / `oss://` URL resolution backed by OpenDAL
- docs + unit tests for the standalone-store path

**What APIs are changed? Are there any user-facing changes?**

New public API (Python, `opendal` feature):

- `vortex._lib.CosStore(bucket, endpoint, *, secret_id=None,
secret_key=None, root=None, disable_config_load=False)` — a standalone
`ObjectStore` you can pass to `vortex.io.read_url(url, store=...)` /
`vortex.io.write(arrays, path, store=...)`.
- `vortex.store.CosStore` / `vortex.store.from_url` now document
`cos://` and `oss://` OpenDAL-backed support.

---------

Signed-off-by: forwardxu <forwardxu@apache.org>

Author: XuQianJin-Stars

Cargo.lock

@@ -12,6 +12,7 @@ members = [
     "vortex-flatbuffers",
     "vortex-metrics",
     "vortex-io",
+    "vortex-object-store-opendal",
     "vortex-proto",
     "vortex-array",
     "vortex-arrow",

Cargo.toml

@@ -13,6 +13,7 @@ Vortex arrays support reading and writing to many object storage systems:
    store/http
    store/local
    store/memory
+   store/opendal
    store/config
 
 .. autofunction:: vortex.store.from_url

docs/api/python/store.rst

@@ -0,0 +1,87 @@
+=============
+OpenDAL (COS)
+=============
+
+Vortex can read from and write to Tencent Cloud COS through
+`OpenDAL <https://opendal.apache.org/>`_, which provides native service support.
+
+This store is available only when Vortex is built with the ``opendal`` feature
+(e.g. ``maturin develop --features opendal`` or ``cargo build -p vortex-jni --features opendal``).
+
+:class:`vortex.store.CosStore`
+==============================
+
+.. py:class:: vortex.store.CosStore(bucket, endpoint, *, secret_id=None, secret_key=None, root=None, disable_config_load=False)
+
+   A Tencent Cloud COS object store, backed by OpenDAL. Construct it with explicit
+   configuration and pass it to
+   :func:`vortex.io.read_url` / :func:`vortex.io.write` via the ``store=`` argument,
+   exactly like the built-in S3/Azure/GCS stores.
+
+   The class is only available when Vortex is built with the ``opendal`` feature; on
+   a default build, instantiating it raises :class:`ImportError`.
+
+   :param bucket: COS bucket name (e.g. ``"my-bucket"``).
+   :param endpoint: COS endpoint (e.g. ``"https://cos.ap-guangzhou.myqcloud.com"``).
+   :param secret_id: Optional Tencent Cloud secret id. Maps to the ``TENCENTCLOUD_SECRET_ID``
+       environment variable when unset.
+   :param secret_key: Optional Tencent Cloud secret key. Maps to the
+       ``TENCENTCLOUD_SECRET_KEY`` environment variable when unset.
+   :param root: Optional key prefix applied to every operation.
+   :param disable_config_load: When ``True``, disable OpenDAL's automatic config loading
+       and rely only on the explicit configuration. Defaults to ``False``.
+
+Reading from COS
+================
+
+Pass a ``cos://`` URL directly. Credentials and the endpoint are picked up from the environment
+variables OpenDAL's COS builder reads (``TENCENTCLOUD_SECRET_ID``, ``TENCENTCLOUD_SECRET_KEY`` and
+``COS_ENDPOINT``):
+
+.. code-block:: python
+
+   import vortex as vx
+
+   a = vx.io.read_url("cos://my-bucket/path/to/dataset.vortex")
+
+Or configure explicitly with :class:`~vortex.store.CosStore` and pass it to
+:func:`vortex.io.read_url` via ``store=``:
+
+.. code-block:: python
+
+   from vortex.io import read_url
+   from vortex.store import CosStore
+
+   store = CosStore(
+       bucket="my-bucket",
+       endpoint="https://cos.ap-guangzhou.myqcloud.com",
+       secret_id="AKID...",
+       secret_key="...",
+   )
+
+   # When `store=` is supplied, the path is a key within the store, so the scheme and
+   # bucket are not part of the path passed to read_url.
+   a = read_url("path/to/dataset.vortex", store=store)
+
+Passing a store object directly
+===============================
+
+``CosStore`` is a concrete store object. You can build one once and pass it
+directly to :func:`vortex.io.read_url` / :func:`vortex.io.write` via the ``store=`` argument,
+exactly like the built-in S3/Azure/GCS stores:
+
+.. code-block:: python
+
+   from vortex.io import read_url
+   from vortex.store import CosStore
+
+   store = CosStore(
+       bucket="my-bucket",
+       endpoint="https://cos.ap-guangzhou.myqcloud.com",
+       secret_id="AKID...",
+       secret_key="...",
+   )
+
+   # When `store=` is supplied, the path is resolved as a key within the store, so the scheme
+   # and bucket are not part of the path passed to read_url.
+   a = read_url("path/to/dataset.vortex", store=store)

docs/api/python/store/opendal.rst

@@ -54,7 +54,12 @@
 git_root = Path(__file__).parent.parent
 
 nitpicky = True  # ensures all :class:, :obj:, etc. links are valid
-nitpick_ignore = []
+nitpick_ignore = [
+    # `vortex.store.CosStore` is re-exported through the private `vortex.store._cos` module,
+    # and the `ObjectStore` type alias resolves to the private path. The public class is
+    # fully documented in `opendal.rst`; the private path is intentionally not.
+    ("py:class", "vortex.store._cos.CosStore"),
+]
 
 doctest_global_setup = "import pyarrow; import vortex; import vortex as vx; import random; random.seed(a=0)"
 doctest_default_flags = (

docs/conf.py

@@ -34,11 +34,17 @@ url = { workspace = true }
 vortex = { workspace = true, features = ["object_store", "files"] }
 vortex-arrow = { workspace = true }
 vortex-geo = { workspace = true }
+vortex-object-store-opendal = { path = "../vortex-object-store-opendal", optional = true }
 vortex-parquet-variant = { workspace = true }
 
 [dev-dependencies]
 jni = { workspace = true, features = ["invocation"] }
 
+[features]
+# Enable OpenDAL-backed object stores (Tencent COS) for `cos://` URLs. This pulls in the
+# `opendal` dependency, so it is opt-in.
+opendal = ["dep:vortex-object-store-opendal"]
+
 [lib]
 crate-type = ["cdylib"]