Skip to content

[Autodiff] Support backward in nn.Modules containing Helion kernels - #3473

Open
hinriksnaer wants to merge 1 commit into
mainfrom
hinriksnaer/stack/24
Open

[Autodiff] Support backward in nn.Modules containing Helion kernels#3473
hinriksnaer wants to merge 1 commit into
mainfrom
hinriksnaer/stack/24

Conversation

@hinriksnaer

@hinriksnaer hinriksnaer commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

When HELION_EXPERIMENTAL_DIFFERENTIABLE=1 is set, Helion kernels
automatically participate in the autograd graph. No manual
torch.autograd.Function boilerplate needed:

import helion
import helion.language as hl

@helion.kernel
def square_plus(x: Tensor) -> Tensor:
    out = torch.empty_like(x)
    for tile in hl.tile(x.size()):
        v = x[tile]
        out[tile] = v * v + v
    return out

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(128, 256)
        self.fc2 = nn.Linear(256, 32)

    def forward(self, x):
        x = self.fc1(x)
        x = square_plus(x)
        return self.fc2(x)

model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
x = torch.randn(16, 128, device="cuda")
target = torch.randn(16, 32, device="cuda")

for step in range(100):
    optimizer.zero_grad()
    loss = F.mse_loss(model(x), target)
    loss.backward()
    optimizer.step()
HELION_EXPERIMENTAL_DIFFERENTIABLE=1 python train.py

Kernel.__call__ detects tensors with requires_grad=True and routes
through a thin torch.autograd.Function wrapper. The backward pass
calls
helion.experimental.backward(), which auto-generates a backward Helion
kernel from the forward kernel's FX graph. Both forward and backward are
real Helion kernels compiled through the normal pipeline.

hinriksnaer added a commit that referenced this pull request Aug 27, 2026
When `HELION_EXPERIMENTAL_DIFFERENTIABLE=1` is set, Helion kernels
automatically participate in the autograd graph. No manual
`torch.autograd.Function` boilerplate needed:

```python
import helion
import helion.language as hl

@helion.kernel
def square_plus(x: Tensor) -> Tensor:
    out = torch.empty_like(x)
    for tile in hl.tile(x.size()):
        v = x[tile]
        out[tile] = v * v + v
    return out

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(128, 256)
        self.fc2 = nn.Linear(256, 32)

    def forward(self, x):
        x = self.fc1(x)
        x = square_plus(x)
        return self.fc2(x)

model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
x = torch.randn(16, 128, device="cuda")
target = torch.randn(16, 32, device="cuda")

for step in range(100):
    optimizer.zero_grad()
    loss = F.mse_loss(model(x), target)
    loss.backward()
    optimizer.step()
```

```bash
HELION_EXPERIMENTAL_DIFFERENTIABLE=1 python train.py
```

`Kernel.__call__` detects tensors with `requires_grad=True` and routes
through a thin `torch.autograd.Function` wrapper. The backward pass
calls
`helion.experimental.backward()`, which auto-generates a backward Helion
kernel from the forward kernel's FX graph. Both forward and backward are
real Helion kernels compiled through the normal pipeline.

stack-info: PR: #3473, branch: hinriksnaer/stack/24
@hinriksnaer
hinriksnaer force-pushed the hinriksnaer/stack/24 branch from 26a8b9e to facfddd Compare August 27, 2026 16:17
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 27, 2026
When `HELION_EXPERIMENTAL_DIFFERENTIABLE=1` is set, Helion kernels
automatically participate in the autograd graph. No manual
`torch.autograd.Function` boilerplate needed:

```python
import helion
import helion.language as hl

@helion.kernel
def square_plus(x: Tensor) -> Tensor:
    out = torch.empty_like(x)
    for tile in hl.tile(x.size()):
        v = x[tile]
        out[tile] = v * v + v
    return out

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(128, 256)
        self.fc2 = nn.Linear(256, 32)

    def forward(self, x):
        x = self.fc1(x)
        x = square_plus(x)
        return self.fc2(x)

model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
x = torch.randn(16, 128, device="cuda")
target = torch.randn(16, 32, device="cuda")

for step in range(100):
    optimizer.zero_grad()
    loss = F.mse_loss(model(x), target)
    loss.backward()
    optimizer.step()
```

```bash
HELION_EXPERIMENTAL_DIFFERENTIABLE=1 python train.py
```

`Kernel.__call__` detects tensors with `requires_grad=True` and routes
through a thin `torch.autograd.Function` wrapper. The backward pass
calls
`helion.experimental.backward()`, which auto-generates a backward Helion
kernel from the forward kernel's FX graph. Both forward and backward are
real Helion kernels compiled through the normal pipeline.

stack-info: PR: #3473, branch: hinriksnaer/stack/24
@hinriksnaer
hinriksnaer marked this pull request as draft August 27, 2026 16:21
@hinriksnaer
hinriksnaer force-pushed the hinriksnaer/stack/24 branch from facfddd to 5ed77f5 Compare August 27, 2026 16:21
@hinriksnaer
hinriksnaer marked this pull request as ready for review August 27, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant