Performance of PyTorch in EB compared to upstream wheels

[lexming]

I've run multiple inference and training benchmarks from https://github.com/pytorch/benchmark to compare the performance of PyTorch build from source in EB with a simple installation using the official wheels from pytorch.org.

Test configurations

The following tables show the results for PyTorch 2.6.0 in 2024a. I made 3 different installations:

1. Standard build in EB from source:

   • PyTorch-2.6.0-foss-2024a.eb
   • PyTorch-2.6.0-foss-2024a-CUDA-12.6.0.eb

2. Official wheels from pytorch.org on top of Python/SciPy-bundle (2024a) and CUDA 12.6.0 on EB:

   • Similar to the following PR, but for PyTorch 2.6.0 and CUDA 12.6.0
     {ai}[foss/2025a] PyTorch v2.9.1, Triton v3.5.1 w/ CUDA 12.8.0 from wheels easybuild-easyconfigs#25267

3. Official wheels for PyTorch and CUDA on top of a minimal Python/SciPy-bundle (2024a) environment on EB

Hardware/Job

All benchmarks run on the same system:

• 16 AMD Zen5 CPU cores
• 1 NVIDIA L40S GPU (48GB VRAM)
• 96GB of RAM
• each benchmark ran at least 5 times

Results

All values are relative Operations per Second (OPS) with respect the configuration (3) with PyTorch and CUDA wheels.

Training on GPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--TOTAL--	100.37%	99.88%	100.00%
BERT_pytorch	100.02%	99.61%	100.00%
alexnet	99.63%	99.92%	100.00%
detectron2_fasterrcnn_r_101_c4	x	97.90%	100.00%
detectron2_fasterrcnn_r_101_dc5	x	99.80%	100.00%
detectron2_fasterrcnn_r_101_fpn	x	102.76%	100.00%
LearningToPaint	101.59%	101.26%	100.00%
dlrm	99.88%	99.91%	100.00%
hf_Bert_large	101.30%	98.47%	100.00%
mobilenet_v3_large	99.98%	99.45%	100.00%
timm_regnet	100.39%	99.98%	100.00%
yolov3	100.14%	99.61%	100.00%

Note: the runs with an x failed with errors.

Inference on GPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--TOTAL--	99.80%	99.56%	100.00%
BERT_pytorch	100.29%	97.69%	100.00%
alexnet	100.41%	98.62%	100.00%
detectron2_fasterrcnn_r_101_c4	x	99.80%	100.00%
detectron2_fasterrcnn_r_101_dc5	x	101.15%	100.00%
detectron2_fasterrcnn_r_101_fpn	x	98.73%	100.00%
LearningToPaint	98.01%	100.25%	100.00%
dlrm	99.67%	99.83%	100.00%
hf_Bert_large	99.14%	101.19%	100.00%
hf_T5_large	98.42%	96.71%	100.00%
mobilenet_v3_large	100.55%	100.39%	100.00%
timm_regnet	100.53%	100.30%	100.00%
yolov3	101.19%	100.01%	100.00%

Note: the runs with an x failed with errors.

Training on CPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--TOTAL--	31.94%	99.91%	100.00%
BERT_pytorch	7.16%	101.44%	100.00%
alexnet	28.28%	101.51%	100.00%
detectron2_fasterrcnn_r_101_c4	71.91%	100.92%	100.00%
LearningToPaint	29.47%	101.39%	100.00%
dlrm	22.87%	100.61%	100.00%
hf_Bert_large	7.29%	101.23%	100.00%
mobilenet_v3_large	9.93%	93.36%	100.00%
timm_vision_transformer_large	77.08%	100.63%	100.00%
timm_regnet	33.44%	98.12%	100.00%

Inference on CPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--TOTAL--	35.54%	99.63%	100.00%
BERT_pytorch	2.08%	100.66%	100.00%
alexnet	28.77%	98.30%	100.00%
detectron2_fasterrcnn_r_101_c4	68.24%	99.39%	100.00%
LearningToPaint	32.40%	100.97%	100.00%
dlrm	8.27%	96.77%	100.00%
hf_Bert_large	25.18%	94.50%	100.00%
hf_T5_large	5.22%	101.43%	100.00%
mobilenet_v3_large	21.25%	113.94%	100.00%
timm_vision_transformer_large	67.20%	99.87%	100.00%
timm_regnet	61.48%	98.80%	100.00%
yolov3	70.84%	91.34%	100.00%

Observations

• Performance of inference/training on GPU is practically the same between builds from source and the official wheels
• Performance of inference/training on CPU is unreasonably low on the builds from source on EB compared to official wheels

[boegel]

Could this be due to overloading available cores by using too many threads or something?

What happens if you us BLIS as backend for FlexiBLAS rather than OpenBLAS?

[lexming]

Results for PyTorch 2.9.1 in 2025a

Test configurations

The following tables show the results for the following 3 different installations:

1. Standard build in EB from source (backported from 2025b):

   • PyTorch-2.9.1-foss-2025a-CUDA-12.8.0.eb

2. Official wheels from pytorch.org on top of Python/SciPy-bundle (2025a) and CUDA 12.8.0 on EB:

   • PyTorch-2.9.1-foss-2025a-CUDA-12.8.0-whl.eb

3. Official wheels for PyTorch and CUDA on top of a minimal Python/SciPy-bundle (2025a) environment on EB. Virtual environment with the following:

   pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
   

Hardware/Job

All benchmarks run on the same system:

• 16 AMD Zen5 CPU cores
• 1 NVIDIA L40S GPU (48GB VRAM)
• 96GB of RAM
• average of at least 5 benchmark runs

Benchmarks

Official suite of benchmarks from PyTorch:
https://github.com/pytorch/benchmark

Independently installed for each of the 3 configurations on its own virtual environment.

Results

All values are relative Operations per Second (OPS) with respect the configuration (3) with PyTorch and CUDA wheels.

Training on GPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--AVERAGE--	100.42%	100.41%	100.00%
BERT_pytorch	100.04%	100.10%	100.00%
alexnet	99.54%	99.68%	100.00%
detectron2_fasterrcnn_r_50_c4	104.72%	105.04%	100.00%
detectron2_fasterrcnn_r_50_fpn	99.54%	99.73%	100.00%
LearningToPaint	100.68%	99.88%	100.00%
dlrm	100.03%	99.79%	100.00%
hf_Bert_large	100.62%	100.38%	100.00%
hf_Longformer	100.12%	100.23%	100.00%
resnet50	100.38%	100.35%	100.00%
mobilenet_v3_large	99.86%	99.78%	100.00%
timm_regnet	99.96%	99.88%	100.00%
vision_maskrcnn	100.02%	100.05%	100.00%
yolov3	99.96%	100.39%	100.00%

Inference on GPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
--AVERAGE--	100.02%	99.96%	100.00%
BERT_pytorch	100.19%	100.17%	100.00%
alexnet	100.12%	100.06%	100.00%
detectron2_fasterrcnn_r_50_c4	99.78%	99.43%	100.00%
detectron2_fasterrcnn_r_50_fpn	99.53%	99.57%	100.00%
LearningToPaint	98.04%	99.76%	100.00%
dlrm	100.25%	100.11%	100.00%
hf_Bert_large	100.37%	99.38%	100.00%
hf_Longformer	99.91%	99.95%	100.00%
resnet50	99.66%	100.00%	100.00%
mobilenet_v3_large	99.05%	99.02%	100.00%
timm_regnet	99.41%	99.59%	100.00%
vision_maskrcnn	100.17%	100.05%	100.00%
yolov3	100.03%	100.00%	100.00%

Training on CPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
*--AVERAGE--	42.81%	100.22%	100.00%
BERT_pytorch	45.81%	99.45%	100.00%
alexnet	33.88%	99.99%	100.00%
LearningToPaint	34.50%	101.33%	100.00%
dlrm	56.27%	99.90%	100.00%
hf_Bert_large	34.49%	100.43%	100.00%
hf_Longformer	28.59%	101.88%	100.00%
mobilenet_v3_large	9.27%	99.22%	100.00%
resnet50	30.31%	100.79%	100.00%
vision_maskrcnn	41.12%	100.36%	100.00%
timm_vision_transformer_large	122.34%	100.21%	100.00%
timm_regnet	34.29%	98.91%	100.00%

Inference on CPU

	(1) Build source	(2) PyTorch wheels	(3) PyTorch/CUDA wheels
*--AVERAGE--	42.69%	100.51%	100.00%
BERT_pytorch	2.52%	100.69%	100.00%
alexnet	28.66%	98.75%	100.00%
LearningToPaint	34.81%	100.36%	100.00%
dlrm	55.94%	96.51%	100.00%
hf_Bert_large	22.72%	100.47%	100.00%
hf_Longformer	19.46%	107.67%	100.00%
mobilenet_v3_large	18.60%	99.42%	100.00%
resnet50	48.89%	97.42%	100.00%
vision_maskrcnn	74.93%	100.73%	100.00%
timm_vision_transformer_large	123.90%	99.60%	100.00%
timm_regnet	59.03%	96.10%	100.00%

Observations

Results are very similar to PyTorch 2.6.0:

• Performance on GPU (inference/training) is practically the same between builds from source and the official wheels, which is expected as CUDA is doing the heavy lifting here.
• Performance on CPU (inference/training) is unreasonably low on the builds from source from EB, which just achieve 42% of the performance of the official wheels.

[Flamefire]

Any chance you can get a profile of BERT_pytorch inference on CPU? That looks like the highest offender so I'd expect to see a single issue very visibly there. Really feels like we have a wrong configuration option in our builds. Many can be seen with python -c 'import torch; print(torch.__config__.show())'

[lexming]

Sure, I can try to profile BERT_pytorch on CPU. No big promises though as I'm a bit busy.

In the meantime, these are the configs:

• PyTorch-2.9.1-foss-2025a-CUDA-12.8.0.eb (from source)

  PyTorch built with:
  - GCC 14.2
  - C++ Version: 201703
  - Intel(R) MKL-DNN v3.7.1 (Git Hash N/A)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.8
  - NVCC architecture flags: -gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90,code=sm_90
  - CuDNN 91.0.1  (built against CUDA 12.9)
  - Magma 2.9.0
  - Build settings: BLAS_INFO=flexi, BUILD_TYPE=Release, COMMIT_SHA=Unknown, CUDA_VERSION=12.8, CUDNN_VERSION=9.10.1, CXX_COMPILER=/tmp/eb-befbb0j1/tmp5s6iaq_s/rpath_wrappers/gxx_wrapper/g++, CXX_FLAGS=-O2 -ftree-vectorize -march=native -fno-math-errno -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=flexi, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.1, USE_CUDA=1, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=0, USE_GLOG=0, USE_GLOO=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=0, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,
  

• PyTorch-2.9.1-foss-2025a-CUDA-12.8.0-whl.eb (from wheels)

  PyTorch built with:
  - GCC 13.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.8
  - NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120
  - CuDNN 91.0.1  (built against CUDA 12.9)
    - Built with CuDNN 90.8
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=5811a8d7da873dd699ff6687092c225caffcf1bb, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF,USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,
  

Main differences I see:

• wheels use their own bundled MKL from Intel, while our builds use an external math library (OpenBLAS through FlexiBLAS by default)
• our build has -ftree-vectorize

So I suspect that it is either the math library or the vectorization.

update: I'm benchmarking the build from source with different math libs through FlexiBLAS. Which will also reply the question from @boegel

[lorisercole]

The build from source uses USE_MPI=ON, whereas the wheel uses USE_MPI=OFF. Does it make any difference?

[lexming]

Math libraries on CPU with PyTorch 2.9.1 in 2025a

Performance of PyTorch 2.9.1 in 2025a built from source with EB on top of FlexiBLAS. Math libraries are switched at runtime by loading the corresponding module and setting $FLEXIBLAS.

The reference value are the wheels of PyTorch 2.9.1 from pytorch.org.

Inference

	OpenBLAS	BLIS	AOCL_MT	IMKL	Wheel
AVERAGE	27.89%	20.77%	38.73%	46.03%	100.00%
BERT_pytorch	0.77%	7.64%	19.31%	12.43%	100.00%
alexnet	11.26%	24.89%	82.28%	71.60%	100.00%
LearningToPaint	15.43%	25.62%	50.96%	50.83%	100.00%
dlrm	42.82%	8.66%	2.37%	0.99%	100.00%
hf_Bert_large	11.35%	7.79%	81.58%	64.46%	100.00%
hf_Longformer	7.34%	9.35%	35.66%	25.44%	100.00%
mobilenet_v3_large	10.78%	21.00%	28.57%	26.63%	100.00%
resnet50	30.06%	41.57%	52.13%	54.58%	100.00%
timm_vision_transformer_large	91.29%	5.37%	7.99%	87.65%	100.00%
timm_regnet	42.50%	43.58%	56.03%	56.77%	100.00%
vision_maskrcnn	43.18%	33.02%	9.11%	54.94%	100.00%

Training

	OpenBLAS	BLIS	AOCL_MT	IMKL	Wheel
AVERAGE	30.58%	23.24%	24.68%	42.35%	100.00%
BERT_pytorch	33.42%	19.80%	3.57%	27.34%	100.00%
alexnet	12.19%	18.58%	55.17%	50.88%	100.00%
LearningToPaint	17.28%	23.96%	47.11%	45.30%	100.00%
dlrm	38.70%	21.07%	3.31%	2.76%	100.00%
hf_Bert_large	12.53%	14.93%	4.03%	48.66%	100.00%
hf_Longformer	7.45%	18.21%	4.67%	47.58%	100.00%
mobilenet_v3_large	29.41%	35.44%	34.95%	38.70%	100.00%
resnet50	16.51%	16.68%	20.73%	20.57%	100.00%
timm_vision_transformer_large	79.17%	5.56%	4.86%	83.10%	100.00%
timm_regnet	45.87%	41.84%	53.15%	50.79%	100.00%
vision_maskrcnn	43.87%	39.53%	39.92%	50.12%	100.00%

Resources used (inference + training)

	CPU Cores	Max Mem (GB)	Wheel Mem Use	Walltime (s)	CPU Time (s)	CPU Use (s)	CPU Efficiency
Wheel	32	38.00	100.00%	1027	32864	21898	66.63%
OpenBLAS	32	38.90	102.35%	3082	98624	63796	64.69%
BLIS	32	39.18	103.09%	8082	258624	18780	7.26%
AOCL_MT	32	39.78	104.67%	10217	326944	301832	92.32%
IMKL	32	37.77	99.38%	1507	48224	22463	46.58%

Observations

• Intel MKL does indeed perform better than OpenBLAS, but it does not achieve 50% the performance on average of the upstream wheels.
• There is an issue with the CPU usage of BLIS, it is probably over-threading.
• Unsurprisingly, memory usage is basically the same across the board.

[boegel]

@lexming Maybe the culprit is not so much oneMKL, but the oneDNN library, is that used by the wheels?

[lexming]

@boegel see comment above: #931 (comment)

[Flamefire]

Doing a comparison of PIP, Source and Source+MKL performances using test_eval[BERT_pytorch-cpu]:

MKL: 96,67%
Source: 0,02%

USE_MPI has no influence. Raw data of 5 runs each:

========================= mkl ============================
Name (time in ms)                              Min     Max                  Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
test_eval[BERT_pytorch-cpu]     141.9732  144.3895  143.0586  0.7562  143.2505  0.9384       2;0  6.9901       8           1
test_eval[BERT_pytorch-cpu]     132.5835  139.3600  135.0440  2.2714  134.5504  2.8692       2;0  7.4050       8           1
test_eval[BERT_pytorch-cpu]     142.6957  163.0360  146.2602  6.8097  144.0426  1.0649       1;1  6.8371       8           1
test_eval[BERT_pytorch-cpu]     136.7400  143.0240  139.9472  2.2250  140.3815  3.7012       3;0  7.1456       8           1
test_eval[BERT_pytorch-cpu]     144.0075  169.3238  149.3803  8.8536  146.4246  1.2019       1;2  6.6943       7           1
========================= source ============================
Name (time in s)                               Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
test_eval[BERT_pytorch-cpu]     4.8005  6.2588  5.3578  0.5478  5.2056  0.5663       2;0  0.1866       5           1
test_eval[BERT_pytorch-cpu]     5.8991  7.4897  6.7907  0.5879  6.9446  0.6870       2;0  0.1473       5           1
test_eval[BERT_pytorch-cpu]     5.3113  6.6919  6.0879  0.5139  6.0678  0.6249       2;0  0.1643       5           1
test_eval[BERT_pytorch-cpu]     5.7495  6.4186  6.1221  0.2616  6.1351  0.3918       2;0  0.1633       5           1
test_eval[BERT_pytorch-cpu]     4.0841  5.7676  4.7597  0.6250  4.5754  0.6463       2;0  0.2101       5           1

  OPS: Operations Per Second, computed as 1 / Mean

[boegel]

related:

• MKL missing for PyTorch easybuild-easyconfigs#10940

[boegel]

@boegel see comment above: #931 (comment)

That mentions MKL-DNN, which I guess is actually oneDNN (https://github.com/uxlfoundation/oneDNN)