[Feature Request] Enable Apache Arrow classes to be shared among plugins

[rishabhmaurya]

Is your feature request related to a problem? Please describe

Currently Arrow Flight Stream based SPIs introduced in #16691 make use of generics. Since plugins have a separate classloaders, so its not possible to share BufferAllocators and VectorSchemaRoots from arrow-memory to be shared between plugins thus making SPIs not usable.

Describe the solution you'd like

@reta has an idea to make use of extension plugin classloader for such purposes #17580. Still evaluating this one.

Related component

Search:Performance

Describe alternatives you've considered

One way is to shade all the dependencies and include them in server classpath. That's the solution I tried to implement here but its not a clean solution and can make classloader bulky and dirty -

arrow-memory shaded deps - https://github.com/rishabhmaurya/OpenSearch/blob/flight-producer/libs/arrow-memory-shaded/build.gradle
flight-core shaded deps - https://github.com/rishabhmaurya/OpenSearch/blob/flight-producer/libs/flight-core-shaded/build.gradle

Additional context

No response

[jngz-es]

Thanks @rishabhmaurya for creating this issue. A new feature remote model streaming inference in ml plugin is depending on it.

[rishabhmaurya]

@jngz-es good to see the progress you've made. I will try to share some solution with you by end of this week from my feature branch to unblock your prototype.
Also, once we have client support opensearch-project/opensearch-java#1502 and easy way to port any TransportRequest response in arrow format #17695 we can get your complete use case working end to end.