Add OLMo2 models (1B, 7B, 13B) support to fairseq2 #1410
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do? Please describe:
Add OLMo2 model architecture (1B, 7B, 13B) support in fairseq2
The key architecture changes include:
OLMO2 is similar to the LLaMA architecture with the following differences:
An integration test is added to ensure the output is consistent with HF Transformers. The integration test has passed for the 1B model.
Note:
OLMO2MultiheadAttention inherits from StandardMultiheadAttention (marked
@final)because the only difference is the order of normalization in
_project_q()and_project_kv().Reimplementing the entire class would duplicate ~150 lines of boilerplate code. Right now, the type checker warning is suppressed.
Fixes #1402
Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.
Check list: