Skip to content

[INTERPRETER] Compute bf16 ops in a value dtype#10945

Open
Hughshine wants to merge 1 commit into
triton-lang:mainfrom
Hughshine:fix/interp-int1-bf16-binop
Open

[INTERPRETER] Compute bf16 ops in a value dtype#10945
Hughshine wants to merge 1 commit into
triton-lang:mainfrom
Hughshine:fix/interp-int1-bf16-binop

Conversation

@Hughshine

@Hughshine Hughshine commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Refs #10919 (bf16 half; int1 half fixed in #10923).

The interpreter stores bf16 as a uint16 bit pattern, but binary_op, unary_op, create_fma and create_dot ran the numpy op directly on that storage, so bf16 arithmetic and comparisons computed on the bit pattern instead of the value, diverging from the GPU. 2.5 + 2.0 gave -2.9e-39 (from 16416 + 16384 as uint16) instead of 4.5; -5.0 < -2.0 gave 0 instead of 1 (bf16 is sign-magnitude, so uint16 order != value order).

This converts operands to float32 before the op (as the GPU does) and rounds the result back to bf16 (RTNE) after, covering binary_op, unary_op and create_fma; create_dot now converts any sub-32-bit float operand, not just 8-bit fp8. Adds interpreter tests for bf16 binary_op, comparison, neg and fma.

@Hughshine
Hughshine requested a review from ptillet as a code owner July 18, 2026 09:41
@Jokeren

Jokeren commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

These are two different problems. int1 is handled in #10923

bfloat16 I'm not sure why we need an upcast. How does it match the GPU side?

The interpreter stores bf16 as a uint16 bit pattern, but binary_op, unary_op,
fma and dot ran the numpy op directly on that storage, so bf16 arithmetic and
comparisons computed on the bit pattern instead of the value, diverging from
the GPU. Convert to float32 before the op (as the GPU does) and round back to
bf16 after. Adds interpreter tests for bf16 binary_op, comparison, neg and fma.

(int1, the other half of triton-lang#10919, was fixed separately in triton-lang#10923.)
@Hughshine
Hughshine force-pushed the fix/interp-int1-bf16-binop branch from 545ed87 to 9faa170 Compare July 18, 2026 20:38
@Hughshine Hughshine changed the title [INTERPRETER] Compute int1 and bf16 ops in a value dtype [INTERPRETER] Compute bf16 ops in a value dtype Jul 18, 2026
@Hughshine

Hughshine commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @Jokeren. Dropped the int1 changes (handled by #10923) and rebased; this is now bf16-only.

The bug: bf16 is stored as a uint16 bit pattern (_get_np_dtype, interpreter.py#L162-L163). That storage is fine, but binary_op runs the numpy op straight on the stored array without first converting it to the value dtype, so it computes on the uint16 storage instead of the bf16 value. 2.5 + 2.0 computes 16416 + 16384 = 32800, read back as bf16 = -2.9e-39 instead of 4.5. Comparison is wrong too (bf16 is sign-magnitude, so uint16 order != value order — -5.0 < -2.0 gives 0). neg and fma have the same bug; create_dot converts 8-bit fp8 but not bf16.

On the upcast to fp32: numpy has no native bf16 (smallest float is fp16), so the op has to run in some numpy float type. fp32 holds bf16 losslessly, and a single RTNE round back to bf16 is bit-identical to a native bf16 op (verified for add/sub/mul), so this changes representation only, not numerics. Same as the int1 branch in #10923 (compute in a wider domain, narrow back). Tests fail-before/pass-after on + - * < neg fma.

Same-type fp8 is out of scope here. It is also uint8-stored and semantic.py#L106 keeps the dtype rather than promoting, so the interpreter has the same bug (2.5 + 2.0 -> 128.0), but the GPU rejects same-type fp8 arithmetic (PassManager::run failed), so the interpreter should diverge/reject there rather than compute it. Left for a separate change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants