Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] GenericBenchmark docs need update when access HDFS file #7887

Open
Yohahaha opened this issue Nov 11, 2024 · 2 comments
Open

[VL] GenericBenchmark docs need update when access HDFS file #7887

Yohahaha opened this issue Nov 11, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@Yohahaha
Copy link
Contributor

Description

Seems we need set several environment variables to access HDFS file in GenericBenchmark,

/root/oap-velox/velox/external/hdfs/ArrowHdfsInternal.cpp:366: /opt/HADOOP_COMMON/lib/native/libhdfs.so
Environment variable CLASSPATH not set!
getJNIEnv: getGlobalJNIEnv failed
E20241111 10:31:28.080675 33605 Exceptions.h:66] Line: /root/oap-velox/velox/exec/Driver.cpp:598, Function:operator(), Expression:  Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: string pointer is null, Source: RUNTIME, ErrorCode: INVALID_STATE
I20241111 10:31:28.080758 33605 Task.cpp:1972] Terminating task Gluten_Stage_0_TID_0_VTID_0 with state Failed after running for 2ms
I20241111 10:31:28.080828 33605 Task.cpp:1207] All drivers (1) finished for task Gluten_Stage_0_TID_0_VTID_0 after running for 2ms
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Operator::getOutput failed for [operator: TableScan, plan node ID: 0]: string pointer is null
Retriable: False
Function: operator()

@JkSelf @marin-ma would you help clarify the usage and update docs? thank you!

@Yohahaha Yohahaha added the enhancement New feature or request label Nov 11, 2024
@JkSelf
Copy link
Contributor

JkSelf commented Nov 12, 2024

@Yohahaha Yes. We need set this command
export CLASSPATH=$HADOOP_HOME/bin/hdfs classpath --glob
before running generic_benchmark. I will update the docs later. Thanks.

@FelixYBW
Copy link
Contributor

@JkSelf how can we use libhdfs3.so in generic benchmark?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants