[Improvement-88][SERVER] Kubernetes Connection Pool Implementation

[Cat-Drink]

Purpose of the pull request

This pull request introduces support for the Kubernetes client connection pool, optimizing the interaction performance and resource management between Apache DolphinScheduler and Kubernetes clusters. By maintaining a reuse mechanism for client connections, this feature significantly reduces the overhead caused by frequent creation and destruction of Kubernetes clients, thereby improving the overall efficiency and stability of the system.

Brief change log

• Core Backend Implementation: Added the KubernetesClientPool class, which implements a singleton-pattern connection pool manager to maintain an independent connection pool for each Kubernetes cluster.
• Connection Pool Configuration: Implemented the PoolConfig inner class, which supports configuration items such as minimum idle connections, maximum idle connections, maximum connections, maximum waiting time, and idle timeout. These configurations can be customized through properties in TaskConstants.
• Cluster Connection Pool: Implemented the ClusterClientPool inner class, which is responsible for managing client connections of specific clusters, including the creation, acquisition, return, and closure of connections.
• Idle Connection Cleanup: Added a periodic cleanup thread that executes the cleanupIdleClients method every 30 seconds to clear timed-out idle connections, effectively saving resources.
• Cluster Identification: Implemented a cluster identification generation mechanism based on kubeconfig, using the SHA-256 algorithm to generate a unique identifier for each cluster.
• Client Validation: Added the isClientValid method to ensure that the returned client connection remains valid.
• Integration into Utility Classes: Updated utility classes such as K8sUtils and KubernetesApplicationManager to enable them to acquire and return clients using the connection pool.

Verify this pull request

This change includes comprehensive test cases and can be verified through the following methods:

Automated Testing

• Basic Functionality Test: KubernetesClientPoolTest.testKubernetesClientPoolBasicFunction verifies the basic operations of the connection pool (acquiring cluster ID, obtaining client, returning client, and closing the connection pool).
• Configuration Test: KubernetesClientPoolTest.testKubernetesClientPoolConfig verifies the correctness of connection pool configurations.
• Cluster ID Generation Test: KubernetesClientPoolTest.testClusterIdGeneration verifies the function of generating a unique cluster identifier based on kubeconfig.
• Singleton Pattern Test: KubernetesClientPoolTest.testSingletonPattern verifies the singleton pattern implementation of the connection pool.
• Concurrent Access Test: Verifies the thread safety and stability of the connection pool in a multi-threaded environment.
• Timeout Handling Test: KubernetesClientPoolTest.testClusterClientPoolBorrowObjectTimeout verifies the handling mechanism for client acquisition timeouts.
• Idle Connection Cleanup Test: KubernetesClientPoolTest.testCleanupIdleClients and KubernetesClientPoolTest.testClusterClientPoolCleanupIdle verify the periodic cleanup function of idle connections.

Manual Verification Steps

1. Ensure that appropriate connection pool parameters (such as k8s.client.pool.max.size) are set in the configuration file.
2. Submit multiple Kubernetes tasks to different clusters.
3. Check the logs to verify whether the connection pool correctly creates and manages client connections.
4. Monitor system resource usage to verify whether the connection pool effectively reduces the overhead caused by connection creation and destruction.

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

[SbloodyS]

This pr's title and DSIP-88 are no related. Please modify the correct title and with full design detail and description. @Cat-Drink

[sonarqubecloud]

Quality Gate passed

Issues
74 New issues
0 Accepted issues

Measures
0 Security Hotspots
78.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud