[VL] VeloxBackend should know it run in executor or driver

[leoluan2009]

Description

VeloxBackend show know where it run, executor or driver? for example if if run driver ,it should not init velox cache. There are two methods to this enhancement:

1. pass a config spark.gluten.isDriver when creating VeloxBackend instance, this is simple.
2. Add a member variables in VeloxBackend class, need to change jni code.

[leoluan2009]

@zhztheplayer @zhouyuan can you give some thoughts? thanks!

[zhztheplayer]

Driver and executor do have different plugin entrypoints, https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxListenerApi.scala, are you suggesting a new approach?

[leoluan2009]

Driver and executor do have different plugin entrypoints, https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxListenerApi.scala, are you suggesting a new approach?

But in VeloxBackend.cc, we can not know where it run. The info do not pass from java code to cpp code

[zhztheplayer]

Driver and executor do have different plugin entrypoints, https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxListenerApi.scala, are you suggesting a new approach?

But in VeloxBackend.cc, we can not know where it run. The info do not pass from java code to cpp code

I see. Do you know which part of C++ code requires for this information?

[leoluan2009]

Driver and executor do have different plugin entrypoints, https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxListenerApi.scala, are you suggesting a new approach?

But in VeloxBackend.cc, we can not know where it run. The info do not pass from java code to cpp code

I see. Do you know which part of C++ code requires for this information?

if it run in driver, it should not init velox cache. https://github.com/apache/incubator-gluten/blob/main/cpp/velox/compute/VeloxBackend.cc#L197

[zhztheplayer]

I am curious why it matters to initialize the cache in driver or not. Do you already see some issues or errors in your circumstance?

BTW I'll prefer changing the JNI API to have different paths for driver / executor native initializations if we have to do it.

[leoluan2009]

I am curious why it matters to initialize the cache in driver or not. Do you already see some issues or errors in your circumstance?

BTW I'll prefer changing the JNI API to have different patches for driver / executor native initializations if we have to do it.

Yes, when initialize the cache, it will create cache dir and check remaining disk capacity while spark driver node may has smaller disk than executor.

[FelixYBW]

Do we start any Velox pipeline on driver today? Where the cache is initialized?

Looks only the BHJ's hash build may be run on driver which we haven't implemented yet.

[LoseYSelf]

Do we start any Velox pipeline on driver today? Where the cache is initialized?

Looks only the BHJ's hash build may be run on driver which we haven't implemented yet.

this line will check ssd space.

incubator-gluten/cpp/velox/compute/VeloxBackend.cc

Line 217 in c653337

const std::filesystem::space_info si = std::filesystem::space(ssdCachePathPrefix, ec);

[FelixYBW]

Thank you. Then we should initialize velox on driver and worker differently.