[Feature Request] Tracking recent stable build of Lucene 10.x for main branch

[jainankitk]

Is your feature request related to a problem? Please describe

We should consider tracking recent stable build of Lucene 10.x for OpenSearch main branch. That will allow OpenSearch to leverage recent Lucene optimizations and actively build on top of that. Recently introduced BulkCollection APIs (#19324) can be leveraged well for significantly improving the aggregation performance in OpenSearch. https://jenkins.thetaphi.de/ gives good idea regarding the stability of most recent Lucene builds.

Describe the solution you'd like

We don't need to upgrade Lucene for every build/nightly, but whenever something significant is added in Lucene that we want to build on top of. Although, I am wondering if this has any implications on the release schedule for OpenSearch?

Related component

Build

Describe alternatives you've considered

No response

Additional context

No response

[harshavamsi]

@jainankitk do you mean tracking lucene snapshot?

[jainankitk]

@jainankitk do you mean tracking lucene snapshot?

Yes! I meant the snapshot repo from here - https://repository.apache.org/content/groups/snapshots/org/apache/lucene/lucene-core/10.4.0-SNAPSHOT/

[msfroh]

IIRC, @reta used to sometimes update us to the latest 9.x Lucene snapshot on the OpenSearch main branch (when it was the "future major version" branch), before we moved main to Lucene 10.x ahead of the 3.0 release.

I like the idea of tracking Lucene snapshots, since e.g. we might have caught the performance regression in 10.3 that @expani had fixed in time for 10.2. (That should have shown up in some of the first 10.3 snapshots.) It's also been discussed in the past, in #5608 and #9315.

The big challenge now is that we don't have a separate "bleeding edge" main branch distinct from the "releasable" 3.x branch. If we do start running Lucene snapshots on main, we might have to "move backwards" to the latest stable release before we cut an OpenSearch release branch.

[mch2]

Yeah we still have the workflow to build Lucene snapshots and publish on demand to I think ci.opensearch.org, though it looks broken at the moment...

On a somewhat unrelated note...we've been discussing introducing some form of ongoing longevity testing that is triggered daily and runs for a few weeks. It would be great to perform these with Lucene snapshots as well.

For the branching issue maybe we cut a 3.x-lucene for now and periodically merge from main?

[jainankitk]

The big challenge now is that we don't have a separate "bleeding edge" main branch distinct from the "releasable" 3.x branch. If we do start running Lucene snapshots on main, we might have to "move backwards" to the latest stable release before we cut an OpenSearch release branch.

Yeah, I was also concerned about how this impacts release schedule for OpenSearch. I can think of mainly 2 ways of getting around this issue:

• Revert to old way of having main and 3.x branch
• Create new branch that track lucene snapshot and strategic changes relying on unreleased lucene features can be included in that. We can occasionally merge from main to keep the diff minimal and update the lucene snapshot as and when required.

My preference is the latter as we don't need to backport every PR on main. Would like to hear from others.

[jainankitk]

For the branching issue maybe we cut a 3.x-lucene for now and periodically merge from main?

3.x-lucene as the branch name seems quite appropriate to me! Went ahead with the creation of 3.x-lucene branch

@mch2 - I haven't had much experience with the github workflows in general. Will you be able to help get that going again?

[reta]

I was thinking, if we do branching of 3.x, than:

• main could use Lucene 11 snapshots (+ whatever is coming in 4.x), it is expected to be released very soon [1]
• 3.x could use fresh 10.x snapshots during the development and switch to release version before the next release

[1] https://lists.apache.org/thread/c8sjqg0t79zzo06j6wj3pvhdlg5wph34

[jainankitk]

main could use Lucene 11 snapshots (+ whatever is coming in 4.x), it is expected to be released very soon [1]

As per my understanding, the release schedule for Lucene 11 is not decided yet, and very likely not this year

3.x could use fresh 10.x snapshots during the development and switch to release version before the next release

I am not sure if that will work. For example, if we track 10.x snapshot in 3.x and add some features depending on those snapshots directly into 3.x, we cannot release until Lucene releases, right?

Hence, I prefer 3.x-lucene branch instead (much more explicit and clear) to strategically add changes depending on unreleased lucene features which can be merged into main/3.x post Lucene release.