ValueError: Input X contains NaN.

[tianyumyum]

在运行量化（PTQ）任务时，当进行平滑量化（Smooth Quantization）并调用 KMeans 聚类时，输入数据中出现了 NaN 值，导致程序崩溃。该问题发生在模型的 linear_217 层。

Layer shift_smooth_help_layer_37 Piece piece_1, loss: 0.0002034902572631836, alpha : 4.5
Find Better K-Piece 2
Search 3 Piece
search for piece piece_0; centroids value is 1.232672095298767
Layer shift_smooth_help_layer_37 Piece piece_0, loss: 0.00023603439331054688, alpha : 1.0
search for piece piece_1; centroids value is 3.4432828426361084
Layer shift_smooth_help_layer_37 Piece piece_1, loss: 0.0002543926239013672, alpha : 1.0
search for piece piece_2; centroids value is 7.919163227081299
Layer shift_smooth_help_layer_37 Piece piece_2, loss: 0.00019800662994384766, alpha : 5.5
Find Better K-Piece 3
[smooth search] search input of linear_133
Search 1 Piece
search for piece piece_0; centroids value is 1.7744877338409424
Layer linear_133 Piece piece_0, loss: 0.0002846717834472656, alpha : 0.79
Find Better K-Piece 1
Search 2 Piece
search for piece piece_0; centroids value is 1.7283637523651123
Layer linear_133 Piece piece_0, loss: 0.00033354759216308594, alpha : 1.0
search for piece piece_1; centroids value is 9.28531265258789
Layer linear_133 Piece piece_1, loss: 0.00025582313537597656, alpha : 2.0
Find Better K-Piece 2
LAUNCH INFO 2025-02-28 06:31:39,766 Pod failed
LAUNCH ERROR 2025-02-28 06:31:39,766 Container failed !!!
Container rank 0 status failed cmd ['python', 'run_quantization_peft.py', './config/repllama_02/ptq_argument_fp.json'] code 1 log log/workerlog.0
LAUNCH INFO 2025-02-28 06:31:39,766 ------------------------- ERROR LOG DETAIL -------------------------
ce_2, loss: 0.0151214599609375, alpha : 0.44
Find Better K-Piece 3
[smooth search] search input of linear_217
Search 1 Piece
Traceback (most recent call last):
File "/mnt/ceph_home/[MASKED_NAME]/PaddleNLP/llm/run_quantization_peft.py", line 722, in
main()
File "/mnt/ceph_home/[MASKED_NAME]/PaddleNLP/llm/run_quantization_peft.py", line 643, in main
apply_smooth(quant_args, trainer, ptq_dataloader, ptq_model_config)
File "/mnt/ceph_home/[MASKED_NAME]/PaddleNLP/llm/utils/quant.py", line 182, in apply_smooth
smooth.update_weight()
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/paddleslim-0.0.0.dev0-py3.10.egg/paddleslim/quant/advanced/smooth.py", line 196, in update_weight
s = self.search_function.search(
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/paddleslim-0.0.0.dev0-py3.10.egg/paddleslim/quant/advanced/piecewise_search.py", line 87, in search
centroids, labels = k_means(act_abs_max, k_piece)
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/paddleslim-0.0.0.dev0-py3.10.egg/paddleslim/quant/advanced/utils.py", line 33, in k_means
k_means.fit(weight)
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/base.py", line 1389, in wrapper
return fit_method(estimator, *args, **kwargs)
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/cluster/_kmeans.py", line 1454, in fit
X = validate_data(
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/utils/validation.py", line 2944, in validate_data
out = check_array(X, input_name="X", **check_params)
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/utils/validation.py", line 1107, in check_array
_assert_all_finite(
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/utils/validation.py", line 120, in _assert_all_finite
_assert_all_finite_element_wise(
File "/mnt/ceph_home/[MASKED_NAME]/miniconda3/envs/paddle/lib/python3.10/site-packages/sklearn/utils/validation.py", line 169, in _assert_all_finite_element_wise
raise ValueError(msg_err)
ValueError: Input X contains NaN.
KMeans does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
LAUNCH INFO 2025-02-28 06:31:39,766 Exit code 1

   python -m paddle.distributed.launch --gpus 0,1 \
     run_quantization_peft.py \
     ./config/repllama/ptq_argument.json

模型为merge后的castorini/repllama-v1-7b-lora-passage，可以跑通awq_argument.json，但是ptq_argument.json的时候会报错