[FLINK-35966][runtime] Introduce the TASKS for TaskManagerLoadBalanceMode enum

[RocMarshal]

What is the purpose of the change

• [FLINK-35966][runtime] Introduce the TASKS for TaskManagerLoadBalanceMode enum
• [FLINK-36691][runtime] Support configuration to enable tasks balancing at slot level for Default Scheduler

Brief change log

• [FLINK-35966][runtime] Introduce the TASKS for TaskManagerLoadBalanceMode enum
• [FLINK-36691][runtime] Support configuration to enable tasks balancing at slot level for Default Scheduler

Verifying this change

This change is already covered by existing tests,

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• 17a1b97 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[RocMarshal]

hi, @zhuzhurk @1996fanrui Could you help take a look for the PR if you had the free time ?
Many thanks !

[1996fanrui]

As I understand, task balancing needs 2 level:

• Task to Slot
• Slot to TM

This PR will Introduce the TASKS for TaskManagerLoadBalanceMode enum, and only support task balancing at slot level. And Slot to TM will be supported later.

I think it's acceptable for Flink 2.0, and I hope hear the suggestion from @zhuzhurk .

[zhuzhurk]

As I understand, task balancing needs 2 level:

• Task to Slot
• Slot to TM

This PR will Introduce the TASKS for TaskManagerLoadBalanceMode enum, and only support task balancing at slot level. And Slot to TM will be supported later.

I think it's acceptable for Flink 2.0, and I hope hear the suggestion from @zhuzhurk .

Does you mean that the balance of Slot to TM[1] is not planned to be supported in Flink 2.0?

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-370%3A+Support+Balanced+Tasks+Scheduling#FLIP370:SupportBalancedTasksScheduling-3.2.3ImplementslotstoTMsbalancingforDefaultScheduler:

[RocMarshal]

Thanks @zhuzhurk @1996fanrui for the comments.

Please let me to synchronize the background information and introduce the reasons for this change.

The feature freeze for version 2.0 will take place at the end of November 2024.

Does you mean that the balance of Slot to TM[1] is not planned to be supported in Flink 2.0?

We expect to release this complete feature in 2.0 ideally
but I estimate that the two core PRs at the TaskManager level may face some risks based on the current progress,

• [FLINK-33391][runtime] Support tasks balancing at TM level for Adaptive Scheduler. #25517 TM level balancing for AdaptiveSheduler)
• [FLINK-33387][runtime] Introduce the abstraction and the interface about loading #25174 TM level balancing for DefaultSheduler)

Meanwhile, for FLIP-370,

• enabling the switch to support slot-level balancing could address a significant portion of the tasks skew issues.
• The solutions for these two PRs are already determined and are currently awaiting review.
  • [FLINK-35966][runtime] Introduce the TASKS for TaskManagerLoadBalanceMode enum #25647
  • [FLINK-33389][runtime] Support tasks balancing at slot level for Adaptive Scheduler #25504

Therefore, I'd like to attempt to first support the slot-level balancing for the two schedulers.

Please let me know what's your opinion~ Thx!

[zhuzhurk]

Thanks @zhuzhurk @1996fanrui for the comments.

Please let me to synchronize the background information and introduce the reasons for this change.

The feature freeze for version 2.0 will take place at the end of November 2024.

Does you mean that the balance of Slot to TM[1] is not planned to be supported in Flink 2.0?

We expect to release this complete feature in 2.0 ideally but I estimate that the two core PRs at the TaskManager level may face some risks based on the current progress,

• [FLINK-33391][runtime] Support tasks balancing at TM level for Adaptive Scheduler. #25517 TM level balancing for AdaptiveSheduler)
• [FLINK-33387][runtime] Introduce the abstraction and the interface about loading #25174 TM level balancing for DefaultSheduler)

Meanwhile, for FLIP-370,

• enabling the switch to support slot-level balancing could address a significant portion of the tasks skew issues.

• The solutions for these two PRs are already determined and are currently awaiting review.

  • [FLINK-35966][runtime] Introduce the TASKS for TaskManagerLoadBalanceMode enum #25647
  • [FLINK-33389][runtime] Support tasks balancing at slot level for Adaptive Scheduler #25504

Therefore, I'd like to attempt to first support the slot-level balancing for the two schedulers.

Please let me know what's your opinion~ Thx!

The feature freeze for version 2.0 will take place at the end of November 2024

The feature freeze date has been postponed to 2025-01-13[1]. Will it give enough time to complete the FLIP?

[1] "[DISCUSS] Feature Freeze for Flink 2.0 - Jan 13" in dev ML