Description
Git Servers may throttle operations when a given threshold (operations/time) is met.
If the controller continues to attempt cloning whilst blocked, it will impair the controller's ability to self-heal as some servers will keep extending the ban window.
The ban window length varies per service provider, in public GitLab for example this is set to 60 minutes.
Considering that the current default --max-retry-delay is set to 15min, in case of error caused by being banned, the controller will continue to try cloning at most 4 times per hour, and the server would reject all those attempts, and keep extending the ban window indefinitely. To avoid this problem users have to increase the --max-retry-delay to be higher than the ban window used by the Git servers being used.
In the long term, the controller could observe the header Retry-After RFC-6585 header, which indicates the amount of seconds remaining on the current ban. This can be used in conjunction with Kubernetes' RateLimiter to define a per item retry policy.