[Slurm] Add retry_delays when waiting for workers

[David96]

Fixes JuliaParallel/SlurmClusterManager.jl#55

Inspired by the LSF code. It's not very well tested though, especially about the error path I'm not sure. From reading the code I think just throwing an exception is fine?

[kescobo]

LGTM - I'll wait to merge for a couple of days in case anyone has bright ideas for how to test. This package needs a lot more testing in general, but as it is, I think it's worth merging

[David96]

Just something maybe worth noting: for people using SlurmManager directly (not addprocs_slurm) this is actually a breaking change as the retry_delays parameter is required.

[kescobo]

Oh, thank you for noting that, it's definitely important to know about that for SemVer purposes. Can you set a default value? I think you should be able to just define a constructor like SlurmManager(n::Int) = SlurmManager(n, 3) or something.

[David96]

Added the default constructor, I used the exponential backoff as used in addprocs_slurm (which I got from LSF), too, though. If you want to minimize changes in behavior I can change both of them to a repeated iterator.

[kescobo]

I don't think minimizing changes is the goal, I think we want the behavior that makes most sense as a default. I'd consider this not technically breaking if the same code works to launch jobs, but I'm happy to defer to others. Incrementing a version isn't the worst thing in the world.

[kescobo]

Not sure if you want to use #154 to try to add tests, but AFAIC this is ready to merge. Let me know

[David96]

I'm not exactly sure how a test on this would look like so from my side, it's ready to merge, too.