Adopt a convention for upgrading Spark to a minor release (3rd release number)

[PHILO-HE]

Currently, maven profile is used to customize Gluten's build with selective code modules and dependencies for different Spark releases. And only one Spark minor release is picked as the officially support version in each release line. E.g., currently, Spark-3.4.3 is officially supported in Spark-3.4 release line. Previously, we can always upgrade Spark to its latest minor release, which possibly requires some Gluten code to be changed to adapt to that. But with Iceberg/Hudi introduced into Gluten, the situation is different. One release of Iceberg/Hudi only officially supports some certain Spark releases (this is quite like Gluten). The direct upgradation for Spark can break the use of Iceberg or Hudi.

So when upgrading Spark, we should make sure the currently bound Iceberg/Hudi supports the new Spark release (this may be possible even though Iceberg/Hudi doesn't officially claim this), or we may have to upgrade Iceberg/Hudi meanwhile to guarantee that. For the latter choice, if only their next release meets our requirement, we will be blocked until new Iceberg/Hudi release comes. Then, this breaks our previous convention in the community, i.e., move quickly to always support latest Spark minor release.

Any suggestion is welcome!

[PHILO-HE]

cc @zhztheplayer, @zhouyuan, @weiting-chen, @FelixYBW, @zzcclp

[FelixYBW]

So it's the question that which one is more important, Iceberg/Hudi support or latest spark minor release support. I vote that Iceberg/Hudi is more important.

We may investigate but I'd believe most of Gluten users doesn't use the exact minor release version as Gluten support. The rebase work between minor release isn't much.

Maybe even iceberg and Hudi will have different minor spark release support as well, in this way, we need to prioritize iceberg and Hudi one day.

[PHILO-HE]

@FelixYBW, yes, now that Iceberg/Hudi are integrated with Gluten, we should guarantee the basic functionality is good at least.

[zhztheplayer]

+1 to keeping support iceberg / delta / hudi in releases