Cm3 integration #727
Cm3 integration #727
Conversation
- make cm3 compliant with main branch
- revert inference mode
- revert inference mode
urielsinger
requested review from
ArmenAg,
Xirider,
adampolyak,
andrewPoulton,
bashnick,
davides,
igormolybogFB,
klshuster,
lilisierrayu,
moyapchen,
ngoyal2707,
punitkoura,
suchenzang,
tangbinh and
zycalice
as code owners
|
This has some specific logic to CM3leon project (i.e., img token conversions), are we sure we want to land this in main? Do we want to possible pick a branch and merge everything into there and periodically update to main? |
| process_group=distributed_utils.get_data_parallel_group(), | ||
| ) | ||
| model = task.build_model(cfg.model) | ||
| if not isinstance(model, FullyShardedDataParallel): |
There was a problem hiding this comment.
Just to confirm, this is for loading up consolidated model for training?
There was a problem hiding this comment.
Yes.
I added support to change the MP size during job lunch, and for that I need to wrap it in FullyShardedDataParallel inside the build_model.
As I don't want to double wrap it, I needed to add this if..
| # Copyright (c) Meta Platforms, Inc. and affiliates. All Rights Reserved. | ||
| # | ||
| # This source code is licensed under the MIT license found in the | ||
| # LICENSE file in the root directory of this source tree. |
There was a problem hiding this comment.
These are changes to the cm3 objectives that i landed in scaling_racm3 correct?
|
Code LGTM. @suchenzang to give guidance on whether or not to land here. |
| def _create_cm3_special_tokens(self): | ||
| self.cm3_sentinel_end = "<eoss>" | ||
| self.cm3_break = "<racm3:break>" | ||
| self.dictionary.add_symbol(self.cm3_break) |
There was a problem hiding this comment.
It's all looking great.
We want to make a change here to recycle unused embedding index of cm3_break and sentinel for the next version.
Should i just add a commit on top of this PR? Or do I file a different PR?
| if boundaries[0] != 0: | ||
| boundaries = [0] + boundaries | ||
| if boundaries[-1] != item.size(0) - 1: | ||
| boundaries = boundaries + [item.size(0)] |
There was a problem hiding this comment.
is get_document_boundaries() robust to the case that there is no break tokens
No description provided.