Skip to content

[WIP] Support export of Llama with DynamicCache and transformers>=4.51 #24379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 25 commits into from
Closed

[WIP] Support export of Llama with DynamicCache and transformers>=4.51 #24379

wants to merge 25 commits into from

Conversation

xadupre
Copy link
Member

@xadupre xadupre commented Apr 10, 2025

Description

Description

Replaces #24291.

transformers>=4.51 makes DynamicCache exportable.
The modification were tested with a tiny LLM:

python -m onnxruntime.transformers.models.llama.convert_to_onnx -m arnir0/Tiny-LLM --output Tiny-LLM --precision fp16 --execution_provider cuda --small_gp --use_dynamo_export

xadupre added 24 commits March 28, 2025 15:01
@xadupre
first draft to migrate to newer version of transformers
5453405
@xadupre
add patches
31e82a9
@xadupre
Merge branch 'main' of https://github.com/microsoft/onnxruntime into …
299f116 
…llama2
@xadupre
fix import
cdec2d0
@xadupre
fix build and import
827d3bd
@xadupre
build
18b649e
@xadupre
fix lint
0e77ed4
@xadupre
lint
4633a3e
@xadupre
lint
b12287a
@xadupre
rename
1b926cb
@xadupre
lint
6646e61
@xadupre
lint
a14b8b3
@xadupre
remove args.dynamo
9f3a816
@xadupre
fix issues
0c88e42
@xadupre
copy inputs
8b60535
@xadupre
fix shape
741285b
@xadupre
fix validation
f8490a5
@xadupre
Merge branch 'main' of https://github.com/microsoft/onnxruntime into …
dbe202c 
…llama2
@xadupre
add use_dynamo_export
ca43041
@xadupre
lint
19d4dfb
@xadupre
remove unncessary changes
cee2181
@xadupre
Merge branch 'main' of https://github.com/microsoft/onnxruntime into …
ebc7b77 
…llama2
@xadupre
simplification
1a9847f
@xadupre
Merge branch 'main' of https://github.com/microsoft/onnxruntime into …
c7d80a3 
…llama2
github-advanced-security[bot]
github-advanced-security bot found potential problems Apr 10, 2025
View reviewed changes
onnxruntime/python/tools/transformers/models/llama/llama_inputs.py
@@ -7,6 +7,7 @@

import numpy as np
import torch
import transformers

Check notice

Code scanning / CodeQL

Module is imported with 'import' and 'import from' Note

Module 'transformers' is imported with both 'import' and 'import from'.
Module 'onnxruntime.test.python.transformers' is imported with both 'import' and 'import from'.
onnxruntime/python/tools/transformers/models/llama/llama_parity.py
import torch
import transformers

Check notice

Code scanning / CodeQL

Module is imported with 'import' and 'import from' Note

Module 'transformers' is imported with both 'import' and 'import from'.
Module 'onnxruntime.test.python.transformers' is imported with both 'import' and 'import from'.
github-advanced-security[bot]
github-advanced-security bot found potential problems Apr 10, 2025
View reviewed changes
onnxruntime/python/tools/transformers/models/torch_export_patches/__init__.py
return {torch_deepcopy(v) for v in value}
if isinstance(value, dict):
return {k: torch_deepcopy(v) for k, v in value.items()}
if isinstance(value, np.ndarray):

Check failure

Code scanning / lintrunner

RUFF/F821 Error

Undefined name np.
See https://docs.astral.sh/ruff/rules/undefined-name
@xadupre xadupre closed this Apr 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@xadupre