[Issue]: add_rmsnorm_quant_kernel precision

[Jacob0226]

Problem Description

This is a follow-up tracking issue for `add_rmsnorm_quant_kernel. Two related PRs are listed below:

1. Aiter: fix residual_out accuracy of hip rmsnorm fused add #2011
   This PR contains the latest changes to add_rmsnorm_quant_kernel, including updates related to kernel accuracy.
2. SGLang: [AMD] Skip the flaky test for lora ci test. sgl-project/sglang#20175
   SGLang CI started failing after add_rmsnorm_quant_kernel was merged into AITER. Please refer to this PR for detailed analysis at both the kernel level and model level precision/accuracy test.
   Our experiments show that the AITER kernel (SGLANG_USE_AITER=1) add_rmsnorm_quant_kernel has a larger numerical difference compared to the vLLM kernels (SGLANG_USE_AITER=0) fused_add_rms_norm and rms_norm compared to the ground truth LlamaRMSNorm from the lib transformers.

Operating System

22.04.5 LTS (Jammy Jellyfish)

CPU

AMD EPYC 9655 96-Core Processor

GPU

AMD Instinct MI325X

ROCm Version

ROCm 7.0.0

ROCm Component

No response

Steps to Reproduce

• Kernel level
  Docker: rocm/sgl-dev:v0.5.9-rocm700-mi30x-20260308
  python test_rmsnorm_3way_consistency.py
  Check rmsnorm_seed_sweep_4panel.png
  test_rmsnorm_3way_consistency.py

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[junhaha666]

#2241
This PR should be able to solve the problem of this sglang ci test.
The v_fmac_f32 instruction in add_rmsnorm_quant_kernel performs a Fused Multiply-Add (FMA), which computes sum = sum + x * x with only one rounding at the end — the intermediate product x * x is kept at full precision internally.
In contrast, the CK kernel (and PyTorch/HF reference) uses separate multiply and add: tmp = x * x (round once), then sum = sum + tmp (round again) — two roundings total.
This single-vs-double rounding difference produces a tiny discrepancy (~1 ULP) in the square_sum, which propagates through inv_rms = rsqrt(square_sum / n + eps) and the final output = x * inv_rms * weight, accumulating across 32+ transformer layers during autoregressive generation. Eventually, this accumulated drift can flip the top-1 token in softmax sampling, causing visible output divergence from the HF reference.
The fix is to replace v_fmac_f32 with explicit mul + add (with #pragma clang fp contract(off) to prevent the compiler from re-fusing them back into FMA), aligning the rounding behavior with CK/HF.