Feature request: Enterprise rolling restarts should terminate more than 1 worker at a time #2651
Description
As previously discussed we operate a large pool of workers in our static Passenger application pool. We're interested in getting our new code into service as soon as possible and are considering the mechanism by which Passenger rotates processes yet again.
Have you considered allowing the removal of >1 process from the application pool for replacement? This might look like a setting likepassenger_min_rolling_restart_processes (default: Pool Size - 1) that specifies we will accept going down to some amount of workers rather than always have it set to Pool Size - 1.
This would allow Passenger to remove up to Pool Size - Min Rolling Restart Processes workers at a time from request serving and in parallel await their current requests to finish and then kill them and fork/spawn new ones. I think this would speed up the pace at which new workers come into service for users with large application pools and some measure of idle/low use capacity
An example: we have 96 processes. If we set it to 76 workers then during a rolling restart Passenger would:
- Boot the preloader
- Select the initial 20 processes for termination and take them out of request serving
- The first 20 processes are essentially idle, they will terminate immediately and fork 20 new ones. The new ones go into service for requests immediately since they are a newer generation.
- As each one of the first 20 dies and is replaced, another one is selected to enter the the termination pool.
For our use-case we've around 50 active Passenger processes in our pool and the rest handles surges in traffic. With the above model around 40-45 workers will be replaced in very likely < 30 seconds after initiating a rolling restart. Those processes will serve around ~80-90% of the inbound request volume, so our new code is in serve a lot faster. This also handles situations where Passenger cannot kill/replace any other worker if the one it has chosen to replace is handling a slow request or is deadlocked; by processing multiple workers for replacement at a time the slow one will take up 1 of N replacement slots, not the only replacement slot.