[REQUEST]Does the current version support distributed fine-tuning on mac devices (M2-M4)?

[hsoftxl]

Does the current version support distributed fine-tuning on mac devices (M2-M4)?

Thanks

[tjruwase]

@hsoftxl, a user previously created initial support for Apple accelerators. However, I have not seen much activity since. This is a feature of great interest to us, but we lack both bandwidth and hardware to maintain on-par with other supported accelerators. Would you be interested in working on this?

[hsoftxl]

@tjruwase i can test this feature.

I don't know if I can make any relevant code modifications because I haven't done this before But I am willing to do this

[tjruwase]

@hsoftxl, thanks for offering to help with this. You can start by installing deepspeed and running the unit tests to see how much succeeds in your environment.

[hsoftxl]

pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.11.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/anaconda3/envs/deepspeed/bin/python
[2025-04-02 16:52:02,224] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to mps (auto detect)
cachedir: .pytest_cache
Using --randomly-seed=4228339676
rootdir: /Users/mac1/projects/source/DeepSpeed
plugins: randomly-3.16.0, forked-1.6.0, xdist-3.6.1
collected 1969 items / 20 errors / 42 skipped

================================================ ERRORS ================================================
____________________________ ERROR collecting deepspeed/nvme/test_ds_aio.py ____________________________
import file mismatch:
imported module 'deepspeed.nvme.test_ds_aio' has this file attribute:
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/nvme/test_ds_aio.py
which is not the same as the test file we want to collect:
/Users/mac1/projects/source/DeepSpeed/deepspeed/nvme/test_ds_aio.py
HINT: remove pycache / .pyc files and/or use a unique basename for your test file modules
_________________________ ERROR collecting deepspeed/nvme/test_ds_aio_utils.py _________________________
import file mismatch:
imported module 'deepspeed.nvme.test_ds_aio_utils' has this file attribute:
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/nvme/test_ds_aio_utils.py
which is not the same as the test file we want to collect:
/Users/mac1/projects/source/DeepSpeed/deepspeed/nvme/test_ds_aio_utils.py
HINT: remove pycache / .pyc files and/or use a unique basename for your test file modules
______________________ ERROR collecting tests/hybrid_engine/hybrid_engine_test.py ______________________
tests/hybrid_engine/hybrid_engine_test.py:13: in
deepspeed.runtime.utils.see_memory_usage('pre test', force=True)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/utils.py:784: in see_memory_usage
CA {round(torch_memory_reserved() / (1024 * 1024 * 1024),2)} GB
E TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,742] [INFO] [utils.py:781:see_memory_usage] pre test
___________________________ ERROR collecting tests/lightning/test_simple.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/lightning/test_simple.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/lightning/test_simple.py:7: in
from pytorch_lightning import LightningModule, Trainer
E ModuleNotFoundError: No module named 'pytorch_lightning'
_____________________ ERROR collecting tests/model/BingBertSquad/test_e2e_squad.py _____________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/model/BingBertSquad/test_e2e_squad.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/model/BingBertSquad/test_e2e_squad.py:14: in
import evaluate as eval
E ModuleNotFoundError: No module named 'evaluate'
_____________________ ERROR collecting tests/model/Megatron_GPT2/run_perf_test.py ______________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/model/Megatron_GPT2/run_perf_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/model/Megatron_GPT2/run_perf_test.py:11: in
from test_common import BaseTestCase
E ModuleNotFoundError: No module named 'test_common'
_______________________ ERROR collecting tests/onebit/test_compressed_backend.py _______________________
tests/onebit/test_compressed_backend.py:20: in
deepspeed.init_distributed(dist_backend=get_accelerator().communication_backend_name())
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,766] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:52:06,767] [INFO] [comm.py:673:init_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment...
[2025-04-02 16:52:06,862] [INFO] [comm.py:728:mpi_discovery] Discovered MPI settings of world_rank=0, local_rank=0, world_size=1, master_addr=192.168.50.92, master_port=29500
[2025-04-02 16:52:06,862] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
------------------------------------------- Captured stderr --------------------------------------------
hostname: illegal option -- I
usage: hostname [-f] [-s | -d] [name-of-host]
________________________ ERROR collecting tests/onebit/test_compressed_perf.py _________________________
tests/onebit/test_compressed_perf.py:24: in
deepspeed.init_distributed(dist_backend=get_accelerator().communication_backend_name())
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,910] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:52:06,911] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
__________________________ ERROR collecting tests/onebit/test_mpi_backend.py ___________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_mpi_backend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_mpi_backend.py:12: in
from deepspeed.runtime.comm.mpi import MpiBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/mpi.py:7: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
____________________________ ERROR collecting tests/onebit/test_mpi_perf.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_mpi_perf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_mpi_perf.py:10: in
from deepspeed.runtime.comm.mpi import MpiBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/mpi.py:7: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
__________________________ ERROR collecting tests/onebit/test_nccl_backend.py __________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_nccl_backend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_nccl_backend.py:13: in
from deepspeed.runtime.comm.nccl import NcclBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/nccl.py:8: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
___________________________ ERROR collecting tests/onebit/test_nccl_perf.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_nccl_perf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_nccl_perf.py:13: in
from deepspeed.runtime.comm.nccl import NcclBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/nccl.py:8: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
_____________________________ ERROR collecting tests/perf/adagrad_test.py ______________________________
tests/perf/adagrad_test.py:37: in
_main()
tests/perf/adagrad_test.py:33: in _main
ds_time = _test_perf(param, DeepSpeedCPUAdagrad)
tests/perf/adagrad_test.py:14: in _test_perf
optimizer = optimizer_func(param)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adagrad/cpu_adagrad.py:22: in init
self.ds_opt_adagrad = CPUAdagradBuilder().load()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/op_builder/cpu/no_impl.py:21: in load
raise ValueError("This op had not been implemented on CPU backend.")
E ValueError: This op had not been implemented on CPU backend.
_______________________________ ERROR collecting tests/perf/adam_test.py _______________________________
tests/perf/adam_test.py:37: in
_main()
tests/perf/adam_test.py:33: in _main
ds_time = _test_perf(param, DeepSpeedCPUAdam)
tests/perf/adam_test.py:14: in _test_perf
optimizer = optimizer_func(param)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py:94: in init
self.ds_opt_adam = CPUAdamBuilder().load()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/op_builder/cpu/no_impl.py:21: in load
raise ValueError("This op had not been implemented on CPU backend.")
E ValueError: This op had not been implemented on CPU backend.
_________________ ERROR collecting tests/small_model_debugging/partial_offload_test.py _________________
tests/small_model_debugging/partial_offload_test.py:106: in
model, _, _, _ = deepspeed.initialize(args=args,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/init.py:144: in initialize
dist.init_distributed(dist_backend=dist_backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
config_dict["zero_optimization"] {'stage': 0, 'sub_group_size': 8, 'reduce_bucket_size': 20, 'offload_optimizer': {'device': 'cpu', 'pin_memory': True, 'ratio': 0.3}}
[2025-04-02 16:58:19,841] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.6+3054b934, git-hash=3054b934, git-branch=master
[2025-04-02 16:58:19,841] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:19,842] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
------------------------------------------- Captured stderr --------------------------------------------
Exception ignored in: <function DeepSpeedCPUAdam.del at 0x10a88b920>
Traceback (most recent call last):
File "/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in del
self.ds_opt_adam.destroy_adam(self.opt_id)
^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdagrad.del at 0x10bdd2980>
Traceback (most recent call last):
File "/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adagrad/cpu_adagrad.py", line 29, in del
self.ds_opt_adagrad.destroy_adagrad(self.opt_id)
^^^^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdagrad' object has no attribute 'ds_opt_adagrad'
_____________________ ERROR collecting tests/small_model_debugging/stage3_test.py ______________________
tests/small_model_debugging/stage3_test.py:90: in
test_driver()
tests/small_model_debugging/stage3_test.py:56: in test_driver
with deepspeed.zero.Init():
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py:955: in init
init_distributed()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------

BUILDING MODEL
[2025-04-02 16:58:20,103] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,103] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
___________________ ERROR collecting tests/small_model_debugging/test_mics_config.py ___________________
tests/small_model_debugging/test_mics_config.py:108: in
with deepspeed.zero.MiCS_Init(config_dict_or_path=config_dict):
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/zero/mics.py:151: in init
dist.init_distributed()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
[2025-04-02 16:58:20,153] [INFO] [config.py:734:init] Config mesh_device None world_size = 1
[2025-04-02 16:58:20,154] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,154] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
______________________ ERROR collecting tests/small_model_debugging/test_model.py ______________________
tests/small_model_debugging/test_model.py:103: in
model, _, _, _ = deepspeed.initialize(args=args,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/init.py:144: in initialize
dist.init_distributed(dist_backend=dist_backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
config_dict["zero_optimization"] {'stage': 0, 'reduce_bucket_size': 20, 'zero_hpz_partition_size': 1, 'reduce_scatter': True, 'zero_quantized_weights': False, 'zero_quantized_gradients': False}
[2025-04-02 16:58:20,294] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.6+3054b934, git-hash=3054b934, git-branch=master
[2025-04-02 16:58:20,294] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,294] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
_________________________ ERROR collecting tests/torch_compile/test_compile.py _________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/torch_compile/test_compile.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/torch_compile/test_compile.py:12: in
import intel_extension_for_pytorch # noqa: F401 # type: ignore
E ModuleNotFoundError: No module named 'intel_extension_for_pytorch'
________________________ ERROR collecting tests/unit/ops/lion/test_cpu_lion.py _________________________
tests/unit/ops/lion/test_cpu_lion.py:17: in
pytest.cpu_vendor = get_cpu_info()["vendor_id_raw"].lower()
E KeyError: 'vendor_id_raw'
=========================================== warnings summary ===========================================
:8
:8: PytestDeprecationWarning: A private pytest class or function was used.

tests/unit/checkpoint/test_universal_checkpoint.py:217
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:217: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/checkpoint/test_universal_checkpoint.py:221
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:221: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/checkpoint/test_universal_checkpoint.py:225
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:225: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/inference/test_human_eval.py:12
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_human_eval.py:12: PytestUnknownMarkWarning: Unknown pytest.mark.evaluation - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.evaluation

tests/unit/inference/test_inference.py:351
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:351: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_inference.py:437
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:437: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:483
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:483: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_inference.py:515
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:515: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:556
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:556: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:611
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:611: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(3)

tests/unit/inference/test_inference.py:652
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:652: PytestUnknownMarkWarning: Unknown pytest.mark.nightly - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.nightly

tests/unit/inference/test_inference_config.py:13
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference_config.py:13: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_stable_diffusion.py:16
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_stable_diffusion.py:16: PytestUnknownMarkWarning: Unknown pytest.mark.stable_diffusion - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.stable_diffusion

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:79
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:79: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:92
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:92: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:36
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:36: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:41
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:41: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:63
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:63: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:112
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:112: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:118
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:118: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:124
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:130
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_post_ln.py:26
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_post_ln.py:26: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_pre_ln.py:28
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_pre_ln.py:28: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:73
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:73: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:23
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:23: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:107
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:107: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_atom_builder.py:16
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_atom_builder.py:16: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:129
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:129: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:140
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:140: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:154
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:154: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:167
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:167: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:177
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:177: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:189
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:189: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:14
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:14: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:47
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:47: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:84
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:84: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:128
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:128: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:172
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:172: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:218
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:30
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:30: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:52
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:52: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:76
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:76: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:81
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:81: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:104
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:104: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_scatter.py:29
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_scatter.py:29: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:106
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:106: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:120
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:120: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:130
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:148
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:148: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:162
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:162: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:90
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:90: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:101
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:101: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:110
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:110: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:148
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:148: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/model_implementations/parameters/test_contiguify.py:51
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_contiguify.py:51: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_layer_inheritance.py:31
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_layer_inheritance.py:31: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:73
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:73: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:93
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:93: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:120
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:120: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:133
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:133: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:153
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:153: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:35
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:35: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:55
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:55: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:37
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:37: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:26
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:26: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:92
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:92: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:49
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:85
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:85: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:40
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:40: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:67
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:67: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:112
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:112: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:146
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:146: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:193
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:193: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:209
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:209: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:228
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:228: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:240
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:240: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/modules/test_blas_linear_module.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blas_linear_module.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blas_linear_module.py:106
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blas_linear_module.py:106: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:127
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:127: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:138
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:138: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:152
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:152: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:165
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:165: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:175
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:175: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:187
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:187: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:198
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:198: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:74
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:74: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:86
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:86: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_custom_module.py:44
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_custom_module.py:44: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:153
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:153: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:165
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:165: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:181
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:181: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:193
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:193: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:205
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:205: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:273
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:273: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_post_ln_module.py:28
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_post_ln_module.py:28: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:74
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:74: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:86
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:86: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_quantized_linear_module.py:154
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_quantized_linear_module.py:154: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_quantized_linear_module.py:170
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_quantized_linear_module.py:170: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/ragged/test_blocked_allocator.py:15
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:15: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:22
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:22: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:50
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:50: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:58
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:58: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:90
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:90: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:99
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:99: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:117
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:117: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:13
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:13: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:19
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:25
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:25: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:31
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:31: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:37
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:37: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:43
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:43: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:49
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:55
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:55: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:19
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:32
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/model_parallelism/test_autotp_training.py:164
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_autotp_training.py:164: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/model_parallelism/test_autotp_training.py:275
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_autotp_training.py:275: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/model_parallelism/test_configurable_parallel_mp.py:61
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_mp.py:61: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_mp.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_mp.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:234
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:234: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:241
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:241: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:248
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:248: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:255
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:255: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:262
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:262: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/ops/transformer/inference/test_attention.py:22
/Users/mac1/projects/source/DeepSpeed/tests/unit/ops/transformer/inference/test_attention.py:22: PytestUnknownMarkWarning: Unknown pytest.mark.inference_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_ops

tests/unit/runtime/half_precision/onebit/test_onebit.py:1163
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/half_precision/onebit/test_onebit.py:1163: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_nvme_checkpointing.py:21
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_nvme_checkpointing.py:21: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_zeropp.py:64
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_zeropp.py:64: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_zeropp.py:190
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_zeropp.py:190: PytestUnknownMarkWarning: Unknown pytest.mark.nightly - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.nightly

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================= short test summary info ========================================
ERROR deepspeed/nvme/test_ds_aio.py
ERROR deepspeed/nvme/test_ds_aio_utils.py
ERROR tests/hybrid_engine/hybrid_engine_test.py - TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
ERROR tests/lightning/test_simple.py
ERROR tests/model/BingBertSquad/test_e2e_squad.py
ERROR tests/model/Megatron_GPT2/run_perf_test.py
ERROR tests/onebit/test_compressed_backend.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/onebit/test_compressed_perf.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/onebit/test_mpi_backend.py
ERROR tests/onebit/test_mpi_perf.py
ERROR tests/onebit/test_nccl_backend.py
ERROR tests/onebit/test_nccl_perf.py
ERROR tests/perf/adagrad_test.py - ValueError: This op had not been implemented on CPU backend.
ERROR tests/perf/adam_test.py - ValueError: This op had not been implemented on CPU backend.
ERROR tests/small_model_debugging/partial_offload_test.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/stage3_test.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/test_mics_config.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/test_model.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/torch_compile/test_compile.py
ERROR tests/unit/ops/lion/test_cpu_lion.py - KeyError: 'vendor_id_raw'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 20 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================= 42 skipped, 146 warnings, 20 errors in 388.69s (0:06:28) =======================
Abort(868846479): Fatal error in internal_Finalize: Other MPI error, error stack:
internal_Finalize(50)............: MPI_Finalize failed
MPII_Finalize(439)...............:
MPID_Finalize(803)...............:
MPIDI_OFI_mpi_finalize_hook(1065):
flush_send_queue(1024)...........:
MPIDI_OFI_handle_cq_error(788)...: OFI poll failed (default nic=utun4: Input/output error)

[hsoftxl]

run pytest , got this result

[tjruwase]

@hsoftxl, thanks for sharing this. I now have a Mac laptop, so I will try to repro.

Can you share a details of your hardware environment?

[hsoftxl]

mac studio M2, python = 3.11

[hsoftxl]

@tjruwase