Restart job when pre-empted

[richardrl]

My SLURM cluster has a pre-emptible queue.
I want to be able to:

• Detect when a job has been pre-empted, or otherwise exits
• When it exists, I have a server (your application?) that detects when the job is pre-empted and re-sends it to the pre-emption queue to try again

Can you explain how to do this with Slurm-web?

[rezib]

Hello @richardrl, I just converted your issue #473 into this discussion as this is not a bug nor a feature request.

If we put aside GANG scheduling, there are two 2 modes of preemption in Slurm: cancel and requeue.

In cancel mode, Slurm basically cancel the preempted jobs. In this case, Slurm-web will report the job as being cancelled (as well as sacct and squeue commands). AFAIK, there is no way to requeue a job that is cancelled in Slurm, it is definitely over and there not much Slurm-web can do about it.

In requeue mode, Slurm puts the jobs back in the queue to start over. In this case, Slurm-web will report these jobs as pending (as well as sacct and squeue) until they start for another run.

As far as I understand, the requeue mode is what you need. What do expect from Slurm-web in addition to this preemption mode?