Add SLURM support for multi-node tests #65
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
roclark
added
enhancement
Member
Author
|
This is currently a draft based on the ongoing discussion in #1. At this point, the NCCL tests should be fully functional using the Python wheel. As I see it, the following items are still required:
|
Draft
roclark
force-pushed
the
slurm-support
branch
2 times, most recently
from
b41f382 to
4866c75
Compare
fredvx
reviewed
Apr 5, 2021
roclark
force-pushed
the
slurm-support
branch
10 times, most recently
from
e1cfc68 to
e1125ff
Compare
To make it easier to run on large clusters, Bobber should be able to run on SLURM clusters with Pyxis and Enroot installed. This would replace the need for mpirun and SSH keys/daemons inside the containers, making it easier to run tests without copying images between nodes or synchronizing SSH keys. Signed-Off-By: Robert Clark <[email protected]>
roclark
force-pushed
the
slurm-support
branch
from
e1125ff to
5b8711e
Compare
While using Slurm, it is entirely possible to still use Bobber but not have Docker installed on the head node where the jobs will be launched. In this case, Docker should be ignored unless one of the commands directly needs the Docker runtime. Signed-Off-By: Robert Clark <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To make it easier to run on large clusters, Bobber should be able to run on SLURM clusters with Pyxis and Enroot installed. This would replace the need for mpirun and SSH keys/daemons inside the containers, making it easier to run tests without copying images between nodes or synchronizing SSH keys.
Closes #1
Signed-Off-By: Robert Clark [email protected]