Bring Apache Arrow classes into arrow-spi library so those could be shared between plugins #17580
Conversation
|
❌ Gradle check result for fa8f0d8: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for e81963a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #17580 +/- ##
============================================
+ Coverage 72.27% 72.37% +0.09%
- Complexity 65611 65697 +86
============================================
Files 5311 5311
Lines 304942 304946 +4
Branches 44225 44226 +1
============================================
+ Hits 220407 220690 +283
+ Misses 66448 66197 -251
+ Partials 18087 18059 -28 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…hared between plugins Signed-off-by: Andriy Redko <[email protected]>
|
❌ Gradle check result for 107264e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 107264e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for 107264e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❕ Gradle check result for 107264e: UNSTABLE
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
reta
requested review from
anasalkouz,
andrross,
ashking94,
bugmakerrrrrr,
Bukhtawar,
CEHENKLE,
cwperks,
dblock,
dbwiddis,
gbbafna,
jainankitk,
kotwanikunal and
linuxpi
as code owners
reta
requested review from
mch2,
msfroh,
nknize,
owaiskazi19,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
|
@reta I'm guessing this logic will not work for any code in server module assuming the invariant that server cannot depend on netty ( |
@rishabhmaurya AFAIK, there is no direct or transitive dependencies on
It all depends how the classes will be accessed. The |
You're right! If we choose to use netty based buffer allocator, which is what we are planning to be the default, then Currently, we are not creating VectorSchemaRoot as part of the integ test (https://github.com/opensearch-project/OpenSearch/blob/main/plugins/arrow-flight-rpc/src/internalClusterTest/java/org/opensearch/arrow/flight/ArrowFlightServerIT.java#L51), but with #17446, where we have added integration tests creating buffers and vectorSchemaRoot in the plugin, my fear is these might fail because of above reasoning |
|
Allocation manager is set here |
I believe as far as the exposed types are backed by arrow-format / arrow-memory / arrow-vector, and referenced as such, we should be fine |
This is plugin specific setting, should work believe (all other plugins should only see |
|
@reta please set following system properties in Ideally, it should have been caught as part of internal cluster test, but classpath and dependencies of all tests including This is the error i'm getting - |
Thanks @rishabhmaurya , this is a rabbit hole indeed, closing this one since we need a different solution. |
|
@reta I agree! I know you don't have much time these days to take up tasks like these so thanks for working on it. Please let me know if you have any ideas which we can experiment? One was to shade all the dependencies and include them in server classpath. That's the solution I tried to implement here -
Maybe we can revive this discussion here or I can open another issue for it |
Thanks @rishabhmaurya , I think that the way we approached the problem is not working with Apache Arrow in principle. What I have in mind is to use OpenSearch's plugin |
|
@reta I agree that shaded dependencies will pollute the classpath and comes with maintenance overhead so should be avoided. PS: I see what you're talking about - https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/plugins/PluginsService.java#L759 |
Exactly, thanks @rishabhmaurya |
Description
Bring Apache Arrow classes into
arrow-spilibrary so those could be shared between pluginsRelated Issues
N/A
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.