qint4 failed for diffusers: QBitsTensor cannot be changed

[liyihao1230]

When I used qfloat8 to quantize the unet model of Kolors-diffusers, it works well. But failed with qint4.

use qint4/(qfloat8)

class KolorsUNet2DConditionModel(QuantizedDiffusersModel):
base_class = UNet2DConditionModel
model = UNet2DConditionModel.from_pretrained("./Kolors-diffusers", variant="fp16", subfolder="unet")
qmodel = KolorsUNet2DConditionModel.quantize(model, weights=qint4) # qfloat8
qmodel.save_pretrained("./kolors-qint4") # kolors-qfloat8

work well in qfloat8 but failed with qint4

unet = KolorsUNet2DConditionModel.from_pretrained('./model_zoo/kolors-int4')
#unet = KolorsUNet2DConditionModel.from_pretrained('./model_zoo/kolors-qfloat8')
pipe = KolorsPipeline.from_pretrained(
KOLORS_MODEL_PATH,
unet=unet._wrapped.to(dtype=torch.float16),
text_encoder=text_encoder,
torch_dtype=torch.float16, # torch.float16
variant="fp16",# use_safetensors=True
).to('cuda')

error log

Traceback (most recent call last):
File "/mnt/data/liyihao/workspace/text2img/text2img_kolors_v5_light.py", line 101, in
unet=unet._wrapped.to(dtype=torch.float16),
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1174, in to
return self._apply(convert)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 780, in _apply
module._apply(fn)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 805, in _apply
param_applied = fn(param)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1160, in convert
return t.to(
File "/usr/local/lib/python3.10/dist-packages/optimum/quanto/tensor/qtensor.py", line 93, in torch_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/quanto/tensor/qbits/qbits.py", line 258, in torch_dispatch
return qdispatch(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/quanto/tensor/qbits/qbits_ops.py", line 54, in _to_copy
raise ValueError("The dtype of a QBitsTensor cannot be changed")
ValueError: The dtype of a QBitsTensor cannot be changed

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.