Skip to content

Parquet connector supports partition and read path with start and length #16

@jinchengchenghh

Description

@jinchengchenghh

Description

This is the usage of Gluten hive connector split, but now cudf parquet connector only supports specify the file path.

split = std::make_shared<velox::connector::hive::HiveConnectorSplit>(
            kHiveConnectorId,
            paths[idx],
            format,
            starts[idx],
            lengths[idx],
            partitionKeys,
            std::nullopt /*tableBucketName*/,
            std::unordered_map<std::string, std::string>(),
            nullptr,
            std::unordered_map<std::string, std::string>(),
            std::unordered_map<std::string, std::string>(),
            0,
            true,
            metadataColumn,
            properties[idx]);

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions