" Exception: Jacobian mismatch for output 0 with respect to input 0" in test/xpu/test_decomp_xpu.py

[libohao1201]

Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_matmul_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_addr_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_inplace_forward_mode_AD_addmm_decomposed_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_inplace_forward_mode_AD_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_inplace_forward_mode_AD_addmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD___rmatmul___xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad___rmatmul___xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_mv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_matmul_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_fn_fwgrad_bwgrad_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_inplace_forward_mode_AD_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_addbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_inplace_forward_mode_AD_addbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_mv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_fwd_gradients_xpu.TestFwdGradientsXPU,test_forward_mode_AD_addmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_gradgrad_addr_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_gradgrad_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_addr_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad___rmatmul___xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_matmul_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_grad_addmm_decomposed_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_grad_addbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_addbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_grad_addmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_addmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_grad_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_mv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad___rmatmul___xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_matmul_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_inplace_grad_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_grad_nn_functional_multi_head_attention_forward_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_mv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_ops_gradients_xpu.TestBwdGradientsXPU,test_fn_gradgrad_addmv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_autograd_xpu.TestAutogradDeviceTypeXPU,test_mv_grad_stride_0_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_modules_xpu.TestModuleXPU,test_grad_nn_MultiheadAttention_eval_mode_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_modules_xpu.TestModuleXPU,test_grad_nn_MultiheadAttention_train_mode_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_modules_xpu.TestModuleXPU,test_gradgrad_nn_MultiheadAttention_eval_mode_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_modules_xpu.TestModuleXPU,test_gradgrad_nn_MultiheadAttention_train_mode_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_decomp_xpu.TestDecompXPU,test_quick_core_backward_mv_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_decomp_xpu.TestDecompXPU,test_comprehensive_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_decomp_xpu.TestDecompXPU,test_quick_baddbmm_xpu_float64
op_ut,third_party.torch-xpu-ops.test.xpu.test_decomp_xpu.TestDecompXPU,test_quick_addmv_xpu_float64

[libohao1201]

🐛 Describe the bug

third_party.torch-xpu-ops.test.xpu.test_decomp_xpu.TestDecompXPU: test_quick_core_backward_mv_xpu_float64 got error " Exception: Jacobian mismatch for output 0 with respect to input 0"

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
____________ TestDecompXPU.test_quick_core_backward_mv_xpu_float64 _____________
Traceback (most recent call last):
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 1151, in test_wrapper
    return test(*args, **kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 1473, in only_fn
    return fn(self, *args, **kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1622, in wrapper
    fn(*args, **kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 2435, in wrapper
    fn(*args, **kwargs)
  File "/home/gta/libohao/pt210/pytorch/third_party/torch-xpu-ops/test/xpu/test_decomp_xpu.py", line 617, in test_quick_core_backward
    torch.autograd.gradcheck(func, args)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/autograd/gradcheck.py", line 2086, in gradcheck
    return _gradcheck_helper(**args)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/autograd/gradcheck.py", line 2115, in _gradcheck_helper
    _gradcheck_real_imag(
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/autograd/gradcheck.py", line 1518, in _gradcheck_real_imag
    gradcheck_fn(
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/autograd/gradcheck.py", line 1659, in _slow_gradcheck
    raise GradcheckError(
torch.autograd.gradcheck.GradcheckError: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ -7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -1.9073,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -3.8147,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -3.8147,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000, -11.4441,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -7.6294,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   7.6294,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -3.8147,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -3.8147,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -3.8147,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -3.8147,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,  -7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,  -7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294]], device='xpu:0',
       dtype=torch.float64)
analytical:tensor([[-8.4417, -0.0000, -0.0000, -0.0000, -0.0000],
        [-7.6030, -0.0000, -0.0000, -0.0000, -0.0000],
        [-1.2849, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 6.5677,  0.0000,  0.0000,  0.0000,  0.0000],
        [-3.1797, -0.0000, -0.0000, -0.0000, -0.0000],
        [-1.0773, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 1.0570,  0.0000,  0.0000,  0.0000,  0.0000],
        [-3.3636, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 6.6357,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 7.5713,  0.0000,  0.0000,  0.0000,  0.0000],
        [-0.0000, -8.4417, -0.0000, -0.0000, -0.0000],
        [-0.0000, -7.6030, -0.0000, -0.0000, -0.0000],
        [-0.0000, -1.2849, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  6.5677,  0.0000,  0.0000,  0.0000],
        [-0.0000, -3.1797, -0.0000, -0.0000, -0.0000],
        [-0.0000, -1.0773, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  1.0570,  0.0000,  0.0000,  0.0000],
        [-0.0000, -3.3636, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  6.6357,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  7.5713,  0.0000,  0.0000,  0.0000],
        [-0.0000, -0.0000, -8.4417, -0.0000, -0.0000],
        [-0.0000, -0.0000, -7.6030, -0.0000, -0.0000],
        [-0.0000, -0.0000, -1.2849, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  6.5677,  0.0000,  0.0000],
        [-0.0000, -0.0000, -3.1797, -0.0000, -0.0000],
        [-0.0000, -0.0000, -1.0773, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  1.0570,  0.0000,  0.0000],
        [-0.0000, -0.0000, -3.3636, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  6.6357,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  7.5713,  0.0000,  0.0000],
        [-0.0000, -0.0000, -0.0000, -8.4417, -0.0000],
        [-0.0000, -0.0000, -0.0000, -7.6030, -0.0000],
        [-0.0000, -0.0000, -0.0000, -1.2849, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  6.5677,  0.0000],
        [-0.0000, -0.0000, -0.0000, -3.1797, -0.0000],
        [-0.0000, -0.0000, -0.0000, -1.0773, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  1.0570,  0.0000],
        [-0.0000, -0.0000, -0.0000, -3.3636, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  6.6357,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  7.5713,  0.0000],
        [-0.0000, -0.0000, -0.0000, -0.0000, -8.4417],
        [-0.0000, -0.0000, -0.0000, -0.0000, -7.6030],
        [-0.0000, -0.0000, -0.0000, -0.0000, -1.2849],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  6.5677],
        [-0.0000, -0.0000, -0.0000, -0.0000, -3.1797],
        [-0.0000, -0.0000, -0.0000, -0.0000, -1.0773],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  1.0570],
        [-0.0000, -0.0000, -0.0000, -0.0000, -3.3636],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  6.6357],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  7.5713]], device='xpu:0',
       dtype=torch.float64)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3300, in wrapper
    method(*args, **kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 428, in instantiated_test
    result = test(self, **param_kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1707, in wrapper
    fn(*args, **kwargs)
  File "/home/gta/.conda/envs/pt210/lib/python3.10/site-packages/torch/testing/_internal/common_device_type.py", line 1163, in test_wrapper
    raise e_tracked from e
Exception: Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[ -7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -1.9073,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -3.8147,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [ -3.8147,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [  7.6294,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000, -11.4441,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -7.6294,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,  -3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   3.8147,   0.0000,   0.0000,   0.0000],
        [  0.0000,   7.6294,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -3.8147,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,  -3.8147,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   7.6294,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -3.8147,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,  -3.8147,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   7.6294,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,  -7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,  -7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   0.0000],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294],
        [  0.0000,   0.0000,   0.0000,   0.0000,   7.6294]], device='xpu:0',
       dtype=torch.float64)
analytical:tensor([[-8.4417, -0.0000, -0.0000, -0.0000, -0.0000],
        [-7.6030, -0.0000, -0.0000, -0.0000, -0.0000],
        [-1.2849, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 6.5677,  0.0000,  0.0000,  0.0000,  0.0000],
        [-3.1797, -0.0000, -0.0000, -0.0000, -0.0000],
        [-1.0773, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 1.0570,  0.0000,  0.0000,  0.0000,  0.0000],
        [-3.3636, -0.0000, -0.0000, -0.0000, -0.0000],
        [ 6.6357,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 7.5713,  0.0000,  0.0000,  0.0000,  0.0000],
        [-0.0000, -8.4417, -0.0000, -0.0000, -0.0000],
        [-0.0000, -7.6030, -0.0000, -0.0000, -0.0000],
        [-0.0000, -1.2849, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  6.5677,  0.0000,  0.0000,  0.0000],
        [-0.0000, -3.1797, -0.0000, -0.0000, -0.0000],
        [-0.0000, -1.0773, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  1.0570,  0.0000,  0.0000,  0.0000],
        [-0.0000, -3.3636, -0.0000, -0.0000, -0.0000],
        [ 0.0000,  6.6357,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  7.5713,  0.0000,  0.0000,  0.0000],
        [-0.0000, -0.0000, -8.4417, -0.0000, -0.0000],
        [-0.0000, -0.0000, -7.6030, -0.0000, -0.0000],
        [-0.0000, -0.0000, -1.2849, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  6.5677,  0.0000,  0.0000],
        [-0.0000, -0.0000, -3.1797, -0.0000, -0.0000],
        [-0.0000, -0.0000, -1.0773, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  1.0570,  0.0000,  0.0000],
        [-0.0000, -0.0000, -3.3636, -0.0000, -0.0000],
        [ 0.0000,  0.0000,  6.6357,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  7.5713,  0.0000,  0.0000],
        [-0.0000, -0.0000, -0.0000, -8.4417, -0.0000],
        [-0.0000, -0.0000, -0.0000, -7.6030, -0.0000],
        [-0.0000, -0.0000, -0.0000, -1.2849, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  6.5677,  0.0000],
        [-0.0000, -0.0000, -0.0000, -3.1797, -0.0000],
        [-0.0000, -0.0000, -0.0000, -1.0773, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  1.0570,  0.0000],
        [-0.0000, -0.0000, -0.0000, -3.3636, -0.0000],
        [ 0.0000,  0.0000,  0.0000,  6.6357,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  7.5713,  0.0000],
        [-0.0000, -0.0000, -0.0000, -0.0000, -8.4417],
        [-0.0000, -0.0000, -0.0000, -0.0000, -7.6030],
        [-0.0000, -0.0000, -0.0000, -0.0000, -1.2849],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  6.5677],
        [-0.0000, -0.0000, -0.0000, -0.0000, -3.1797],
        [-0.0000, -0.0000, -0.0000, -0.0000, -1.0773],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  1.0570],
        [-0.0000, -0.0000, -0.0000, -0.0000, -3.3636],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  6.6357],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  7.5713]], device='xpu:0',
       dtype=torch.float64)


Caused by sample input at index 0: SampleInput(input=Tensor[size=(5, 10), device="xpu:0", dtype=torch.float64], args=TensorList[Tensor[size=(10,), device="xpu:0", dtype=torch.float64]], kwargs={}, broadcasts_input=False, name='')

To execute this test, run the following from the base repo dir:
    PYTORCH_OPINFO_SAMPLE_INPUT_INDEX=0 PYTORCH_TEST_WITH_SLOW=1 python test/xpu/test_decomp_xpu.py TestDecompXPU.test_quick_core_backward_mv_xpu_float64

Versions

#2262

[daisyden]

Duplicated with #2360

[BBBela]

I did a little research on this issue and found some potential solution.

Root cause

All those test cases are failing due to precision loss in addmm_out and baddbmm_out operators.
The problem is that both those operators don't handle float64 properly and that leads to accuracy issues and tests' fails.
Going deeper -> both addmm_out and baddbmm_out are using under the hood the onednn::matmul with post operations for matrix-scalar multiplication and matrix addition. The problem is that those post operations don't support flaot64 and the precision is rounded to float32. Although the matrix multiplication itself works well. Problem is just with post operations.
That was deduced numerically and by tests, but the documentation of oneDNN confirms it. It says (oneDNN data types) that fp64 is supported just for several kernels, including matmul. Tests confirm that. But post operations are matrix addition and matrix-scalar multiplications, where the precision is lost.

Solution proposal

I tried to resolve this issue by splitting the single onednn::matmul call (which was supposed to do all the needed operations at once) into separate matrix-matrix multiplication, matrix-scalar multiplication and matrix addition. All those separate ops handle float64 properly, so the whole chain also works properly and tests are passing.

For now I created PR from my fork to my fork to consult the solution before potentially creating PR directly to pytorch.
Here are the proposed changes: PR_WITH_CHANGES

Considerations

I prepared some tests to check the performance of this solution.
More details and data are in the PR, but in general float64 computations are much slower than float32. For small matrices there is no big difference, but for bigger ones there is noticeable performance drop. The float64 execution is about 12 to 15 times slower.

As far as I know it may be expected. I tested it on BMG and from what I know it is much more optimized for float32 computations than for high precision float64 and there is also twice the memory usage and data transfers. Although I think it would be good to confirm with someone that such performance drop for float64 may be expected.

Additional info

Proposed solution also fixes #2356 and #2687.

[LuFinch]

Hi @BBBela. Great thanks your detailed analysis and proposed solutions.

Before reviewing your change, I open a JIRA for OneDNN to discuss whether they could support float64 post operations MFDNN-14599.