[ft] Skip extra quorum when using semi-sync training #1221
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
H-Huang
force-pushed
the
diloco
branch
2 times, most recently
from
93f426d to
6fab72e
Compare
H-Huang
changed the title
fegin
reviewed
May 27, 2025
torchtitan/components/optimizer.py
Outdated
| @@ -192,7 +192,8 @@ def __init__( | |||
| } | |||
| self.cache_state_dict: dict[str, Any] = {} | |||
| self._ft_optimizer = ft.Optimizer(ft_manager, self) | |||
| self._call_from_ft: bool = False | |||
| # Originally this is False, True means we just call the step() as normally | |||
| self._call_from_ft: bool = True | |||
There was a problem hiding this comment.
Why changes this to True? step() will manually update _call_from_ft() to ensure that the call path is correctly routed through ft.Optimizer.step() then OptimizersContainer.step(). If we set to False, it will only go though OptimizersContainer.step() not ft.Optimizer.step().
There was a problem hiding this comment.
Yeah I had hardcoded this originally to get it working for the semi-sync training path. I updated this to be set by an argument in the constructor.
We only want to go through OptimizersContainer.step() not ft.Optimizer.step() when doing localsgd/diloco
H-Huang
force-pushed
the
diloco
branch
2 times, most recently
from
d8acd57 to
abdadd9
Compare
fegin
approved these changes
May 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Missed that we were using the ftOptimizer when doing fault tolerant communication for HSDP, we should skip this when we have semi-sync training enabled and only do quorum when the replica groups sync.