Closed
Description
System Info
transformersversion: 4.45.0- Platform: Linux-5.14.0-284.73.1.el9_2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.15
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:
- Using GPU in script?:
- GPU type: NVIDIA A100-SXM4-80GB
Who can help?
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
Torch.Compiling the Generate method using the following:
model.generate = torch.compile(model.generate)
This leads to a ValueError as follows:
Traceback (most recent call last):
File "/workspace/infer.py", line 28, in <module>
main()
File "/workspace/infer.py", line 22, in main
generate_ids = model.generate(input_ids, max_new_tokens=length)
File "/opt/conda/envs/cg-test/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 465, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/cg-test/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 40, in inner
return fn(*args, **kwargs)
File "/opt/conda/envs/cg-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/cg-test/lib/python3.10/site-packages/transformers/generation/utils.py", line 1805, in generate
self._validate_model_kwargs(model_kwargs.copy())
File "/opt/conda/envs/cg-test/lib/python3.10/site-packages/transformers/generation/utils.py", line 1205, in _validate_model_kwargs
raise ValueError(
ValueError: The following `model_kwargs` are not used by the model: ['max_new_tokens'] (note: typos in the generate arguments will also show up in this list)
Expected behavior
No errors, identical behaviour as the eager version.
Activity