Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic batch-system discovery #157

Open
meliache opened this issue Nov 17, 2021 · 0 comments
Open

Automatic batch-system discovery #157

meliache opened this issue Nov 17, 2021 · 0 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@meliache
Copy link
Collaborator

When re-reading the b2luigi documentation, I noticed that @nils-braun had written:

In the future, there will even be an automatic discovery of the batch system to use.

The current state is LSF as a default batch system. A very simple approach would be to define a priority order as an iterable, and then if no batch system is given via setting, go through the default batch systems in the list and choose the first one that is available. This check could simply be if the submission executable exists or maybe also if some status command shows that the system is alive.

For a start I would use ("lsf", "htcondor") as defaults, maybe fall back to "local". The only thing that I'm worried with "local" in the default listis a user accidentally sending a job with 800 workers to the login node when the remote batch is down if `"local"``is down and issues like that, so maybe better leave it out. In any case it should be made very obvious to the user which batch system is chosen.

@meliache meliache added enhancement New feature or request help wanted Extra attention is needed labels Nov 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant