Log config only on rank zero at torchtune/recipes/full_finetune_distributed.py
#2711
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2711
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 1 New FailureAs of commit dbab003 with merge base 3310946 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
recipes/full_finetune_distributed.py
Outdated
| @@ -1056,7 +1057,9 @@ def recipe_main(cfg: DictConfig) -> None: | |||
| - Parameters specified in config (see available configs through ``tune ls``) | |||
| - Overwritten by arguments from the command-line | |||
| """ | |||
| config.log_config(recipe_name="FullFinetuneRecipeDistributed", cfg=cfg) | |||
| if torch.distributed.get_rank() == 0: | |||
There was a problem hiding this comment.
Could we move this inside the recipe's setup and use if self._is_rank_zero?
There was a problem hiding this comment.
Thanks for the PR @kabir2505! In addition to @SalmanMohammadi's comment, if you want to go a bit further here we can also make the same changes to our other distributed recipes: full_dpo_distributed.py, lora_dpo_distributed.py, knowledge_distillation_distributed.py, qat_distributed.py, qat_lora_finetune_distributed.py, early_exit_finetune_distributed.py, generate_v2_distributed.py, grpo_full_finetune_distributed.py, lora_finetune_distributed_multi_dataset.py
|
Hey @SalmanMohammadi @ebsmothers — just to confirm, should |
Correct! |
| @@ -13,6 +13,7 @@ | |||
| from warnings import warn | |||
|
|
|||
| import torch | |||
| import torch.distributed | |||
| @@ -287,9 +287,9 @@ def setup(self, cfg: DictConfig) -> None: | |||
| """ | |||
| if self._is_rank_zero: | |||
| self._metric_logger = config.instantiate(cfg.metric_logger) | |||
|
|
|||
| # log config with parameter override | |||
There was a problem hiding this comment.
I think this still needs to log config here. The naming is bad, but one logs it to the metric logger (meaning it's saved into the WandB or Tensorboard run), and the other just prints it to disk
Context
What is the purpose of this PR? Is it to
This PR ensures that log_config is only called on rank zero in distributed runs to avoid redundant logging across devices.
Address: #2700
Please link to any issues this PR addresses.
Changelog
What are the changes made in this PR?
- Added a rank check to only log config on rank zero
Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example