add FusedApplyRotaryEmbGradKernel

[cccddd77]

No description provided.

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

[github-actions]

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10517/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.8ms (= 4380.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 5762.3ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.32 (= 57.6ms / 43.8ms)

OneFlow resnet50 time: 26.2ms (= 2616.0ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 38.5ms (= 3849.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.47 (= 38.5ms / 26.2ms)

OneFlow resnet50 time: 18.7ms (= 3740.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 37.1ms (= 7410.0ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.98 (= 37.1ms / 18.7ms)

OneFlow resnet50 time: 17.4ms (= 3481.2ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 30.3ms (= 6059.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.74 (= 30.3ms / 17.4ms)

OneFlow resnet50 time: 17.4ms (= 3488.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.5ms (= 5906.2ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.69 (= 29.5ms / 17.4ms)

OneFlow swin dataloader time: 0.199s (= 39.765s / 200, num_workers=1)
PyTorch swin dataloader time: 0.127s (= 25.482s / 200, num_workers=1)
Relative speed: 0.641 (= 0.127s / 0.199s)

OneFlow swin dataloader time: 0.057s (= 11.455s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.555s / 200, num_workers=4)
Relative speed: 0.572 (= 0.033s / 0.057s)

OneFlow swin dataloader time: 0.030s (= 5.997s / 200, num_workers=8)
PyTorch swin dataloader time: 0.016s (= 3.273s / 200, num_workers=8)
Relative speed: 0.546 (= 0.016s / 0.030s)

❌ OneFlow resnet50 time: 49.3ms (= 4925.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.3ms (= 6627.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 66.3ms / 49.3ms)

OneFlow resnet50 time: 37.3ms (= 3733.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.6ms (= 4664.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 46.6ms / 37.3ms)

OneFlow resnet50 time: 27.9ms (= 5575.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 40.2ms (= 8045.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 40.2ms / 27.9ms)

OneFlow resnet50 time: 25.5ms (= 5104.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.5ms (= 7893.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 39.5ms / 25.5ms)

OneFlow resnet50 time: 24.5ms (= 4908.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.2ms (= 7242.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.48 (= 36.2ms / 24.5ms)

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10517/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.3ms (= 4332.8ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.5ms (= 5750.4ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.33 (= 57.5ms / 43.3ms)

OneFlow resnet50 time: 26.6ms (= 2659.2ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.9ms (= 3790.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.43 (= 37.9ms / 26.6ms)

OneFlow resnet50 time: 18.8ms (= 3753.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 37.3ms (= 7462.4ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.99 (= 37.3ms / 18.8ms)

OneFlow resnet50 time: 16.5ms (= 3296.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 31.0ms (= 6198.3ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.88 (= 31.0ms / 16.5ms)

OneFlow resnet50 time: 17.3ms (= 3453.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.0ms (= 5801.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.68 (= 29.0ms / 17.3ms)

OneFlow swin dataloader time: 0.200s (= 40.008s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.651s / 200, num_workers=1)
Relative speed: 0.641 (= 0.128s / 0.200s)

OneFlow swin dataloader time: 0.057s (= 11.313s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.482s / 200, num_workers=4)
Relative speed: 0.573 (= 0.032s / 0.057s)

OneFlow swin dataloader time: 0.030s (= 5.980s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.339s / 200, num_workers=8)
Relative speed: 0.558 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 49.2ms (= 4924.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.2ms (= 6616.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 66.2ms / 49.2ms)

OneFlow resnet50 time: 37.2ms (= 3717.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 47.2ms (= 4721.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 47.2ms / 37.2ms)

OneFlow resnet50 time: 27.6ms (= 5525.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.3ms (= 7869.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.42 (= 39.3ms / 27.6ms)

OneFlow resnet50 time: 25.1ms (= 5026.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.8ms (= 7754.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.54 (= 38.8ms / 25.1ms)

OneFlow resnet50 time: 25.0ms (= 4992.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.0ms (= 7200.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 36.0ms / 25.0ms)