Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REQUEST]Does the current version support distributed fine-tuning on mac devices (M2-M4)? #7148

Open
hsoftxl opened this issue Mar 18, 2025 · 5 comments
Open

[REQUEST]Does the current version support distributed fine-tuning on mac devices (M2-M4)? #7148

hsoftxl opened this issue Mar 18, 2025 · 5 comments
Labels
enhancement New feature or request

Comments

@hsoftxl
Copy link

hsoftxl commented Mar 18, 2025

Does the current version support distributed fine-tuning on mac devices (M2-M4)?

Thanks
@hsoftxl hsoftxl added the enhancement New feature or request label Mar 18, 2025
@tjruwase
Copy link
Contributor

@hsoftxl, a user previously created initial support for Apple accelerators. However, I have not seen much activity since. This is a feature of great interest to us, but we lack both bandwidth and hardware to maintain on-par with other supported accelerators. Would you be interested in working on this?

@hsoftxl
Copy link
Author

hsoftxl commented Mar 19, 2025

@tjruwase i can test this feature.

I don't know if I can make any relevant code modifications because I haven't done this before But I am willing to do this

@tjruwase
Copy link
Contributor

@hsoftxl, thanks for offering to help with this. You can start by installing deepspeed and running the unit tests to see how much succeeds in your environment.

@hsoftxl
Copy link
Author

hsoftxl commented Apr 3, 2025

pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.11.11, pytest-8.3.5, pluggy-1.5.0 -- /opt/anaconda3/envs/deepspeed/bin/python
[2025-04-02 16:52:02,224] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to mps (auto detect)
cachedir: .pytest_cache
Using --randomly-seed=4228339676
rootdir: /Users/mac1/projects/source/DeepSpeed
plugins: randomly-3.16.0, forked-1.6.0, xdist-3.6.1
collected 1969 items / 20 errors / 42 skipped

================================================ ERRORS ================================================
____________________________ ERROR collecting deepspeed/nvme/test_ds_aio.py ____________________________
import file mismatch:
imported module 'deepspeed.nvme.test_ds_aio' has this file attribute:
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/nvme/test_ds_aio.py
which is not the same as the test file we want to collect:
/Users/mac1/projects/source/DeepSpeed/deepspeed/nvme/test_ds_aio.py
HINT: remove pycache / .pyc files and/or use a unique basename for your test file modules
_________________________ ERROR collecting deepspeed/nvme/test_ds_aio_utils.py _________________________
import file mismatch:
imported module 'deepspeed.nvme.test_ds_aio_utils' has this file attribute:
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/nvme/test_ds_aio_utils.py
which is not the same as the test file we want to collect:
/Users/mac1/projects/source/DeepSpeed/deepspeed/nvme/test_ds_aio_utils.py
HINT: remove pycache / .pyc files and/or use a unique basename for your test file modules
______________________ ERROR collecting tests/hybrid_engine/hybrid_engine_test.py ______________________
tests/hybrid_engine/hybrid_engine_test.py:13: in
deepspeed.runtime.utils.see_memory_usage('pre test', force=True)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/utils.py:784: in see_memory_usage
CA {round(torch_memory_reserved() / (1024 * 1024 * 1024),2)} GB
E TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,742] [INFO] [utils.py:781:see_memory_usage] pre test
___________________________ ERROR collecting tests/lightning/test_simple.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/lightning/test_simple.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/lightning/test_simple.py:7: in
from pytorch_lightning import LightningModule, Trainer
E ModuleNotFoundError: No module named 'pytorch_lightning'
_____________________ ERROR collecting tests/model/BingBertSquad/test_e2e_squad.py _____________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/model/BingBertSquad/test_e2e_squad.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/model/BingBertSquad/test_e2e_squad.py:14: in
import evaluate as eval
E ModuleNotFoundError: No module named 'evaluate'
_____________________ ERROR collecting tests/model/Megatron_GPT2/run_perf_test.py ______________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/model/Megatron_GPT2/run_perf_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/model/Megatron_GPT2/run_perf_test.py:11: in
from test_common import BaseTestCase
E ModuleNotFoundError: No module named 'test_common'
_______________________ ERROR collecting tests/onebit/test_compressed_backend.py _______________________
tests/onebit/test_compressed_backend.py:20: in
deepspeed.init_distributed(dist_backend=get_accelerator().communication_backend_name())
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,766] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:52:06,767] [INFO] [comm.py:673:init_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment...
[2025-04-02 16:52:06,862] [INFO] [comm.py:728:mpi_discovery] Discovered MPI settings of world_rank=0, local_rank=0, world_size=1, master_addr=192.168.50.92, master_port=29500
[2025-04-02 16:52:06,862] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
------------------------------------------- Captured stderr --------------------------------------------
hostname: illegal option -- I
usage: hostname [-f] [-s | -d] [name-of-host]
________________________ ERROR collecting tests/onebit/test_compressed_perf.py _________________________
tests/onebit/test_compressed_perf.py:24: in
deepspeed.init_distributed(dist_backend=get_accelerator().communication_backend_name())
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
[2025-04-02 16:52:06,910] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:52:06,911] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
__________________________ ERROR collecting tests/onebit/test_mpi_backend.py ___________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_mpi_backend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_mpi_backend.py:12: in
from deepspeed.runtime.comm.mpi import MpiBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/mpi.py:7: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
____________________________ ERROR collecting tests/onebit/test_mpi_perf.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_mpi_perf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_mpi_perf.py:10: in
from deepspeed.runtime.comm.mpi import MpiBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/mpi.py:7: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
__________________________ ERROR collecting tests/onebit/test_nccl_backend.py __________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_nccl_backend.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_nccl_backend.py:13: in
from deepspeed.runtime.comm.nccl import NcclBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/nccl.py:8: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
___________________________ ERROR collecting tests/onebit/test_nccl_perf.py ____________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/onebit/test_nccl_perf.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/onebit/test_nccl_perf.py:13: in
from deepspeed.runtime.comm.nccl import NcclBackend
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/comm/nccl.py:8: in
import cupy
E ModuleNotFoundError: No module named 'cupy'
_____________________________ ERROR collecting tests/perf/adagrad_test.py ______________________________
tests/perf/adagrad_test.py:37: in
_main()
tests/perf/adagrad_test.py:33: in _main
ds_time = _test_perf(param, DeepSpeedCPUAdagrad)
tests/perf/adagrad_test.py:14: in _test_perf
optimizer = optimizer_func(param)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adagrad/cpu_adagrad.py:22: in init
self.ds_opt_adagrad = CPUAdagradBuilder().load()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/op_builder/cpu/no_impl.py:21: in load
raise ValueError("This op had not been implemented on CPU backend.")
E ValueError: This op had not been implemented on CPU backend.
_______________________________ ERROR collecting tests/perf/adam_test.py _______________________________
tests/perf/adam_test.py:37: in
_main()
tests/perf/adam_test.py:33: in _main
ds_time = _test_perf(param, DeepSpeedCPUAdam)
tests/perf/adam_test.py:14: in _test_perf
optimizer = optimizer_func(param)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py:94: in init
self.ds_opt_adam = CPUAdamBuilder().load()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/op_builder/cpu/no_impl.py:21: in load
raise ValueError("This op had not been implemented on CPU backend.")
E ValueError: This op had not been implemented on CPU backend.
_________________ ERROR collecting tests/small_model_debugging/partial_offload_test.py _________________
tests/small_model_debugging/partial_offload_test.py:106: in
model, _, _, _ = deepspeed.initialize(args=args,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/init.py:144: in initialize
dist.init_distributed(dist_backend=dist_backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
config_dict["zero_optimization"] {'stage': 0, 'sub_group_size': 8, 'reduce_bucket_size': 20, 'offload_optimizer': {'device': 'cpu', 'pin_memory': True, 'ratio': 0.3}}
[2025-04-02 16:58:19,841] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.6+3054b934, git-hash=3054b934, git-branch=master
[2025-04-02 16:58:19,841] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:19,842] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
------------------------------------------- Captured stderr --------------------------------------------
Exception ignored in: <function DeepSpeedCPUAdam.del at 0x10a88b920>
Traceback (most recent call last):
File "/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in del
self.ds_opt_adam.destroy_adam(self.opt_id)
^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdagrad.del at 0x10bdd2980>
Traceback (most recent call last):
File "/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/ops/adagrad/cpu_adagrad.py", line 29, in del
self.ds_opt_adagrad.destroy_adagrad(self.opt_id)
^^^^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdagrad' object has no attribute 'ds_opt_adagrad'
_____________________ ERROR collecting tests/small_model_debugging/stage3_test.py ______________________
tests/small_model_debugging/stage3_test.py:90: in
test_driver()
tests/small_model_debugging/stage3_test.py:56: in test_driver
with deepspeed.zero.Init():
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py:955: in init
init_distributed()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------

BUILDING MODEL
[2025-04-02 16:58:20,103] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,103] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
___________________ ERROR collecting tests/small_model_debugging/test_mics_config.py ___________________
tests/small_model_debugging/test_mics_config.py:108: in
with deepspeed.zero.MiCS_Init(config_dict_or_path=config_dict):
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/runtime/zero/mics.py:151: in init
dist.init_distributed()
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
[2025-04-02 16:58:20,153] [INFO] [config.py:734:init] Config mesh_device None world_size = 1
[2025-04-02 16:58:20,154] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,154] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
______________________ ERROR collecting tests/small_model_debugging/test_model.py ______________________
tests/small_model_debugging/test_model.py:103: in
model, _, _, _ = deepspeed.initialize(args=args,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/init.py:144: in initialize
dist.init_distributed(dist_backend=dist_backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/comm.py:691: in init_distributed
cdb = TorchBackend(dist_backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:118: in init
self.init_process_group(backend, timeout, init_method, rank, world_size)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/deepspeed/comm/torch.py:148: in init_process_group
torch.distributed.init_process_group(backend,
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:81: in wrapper
return func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:95: in wrapper
func_return = func(*args, **kwargs)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1721: in init_process_group
default_pg, _ = _new_process_group_helper(
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:1923: in _new_process_group_helper
backend_config = BackendConfig(backend)
/opt/anaconda3/envs/deepspeed/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:385: in init
raise ValueError(
E ValueError: We detected accelerator mps on your machine. But we don't know which communication backend to use for this accelerator. Please specify the backend argument in the init_process_group call.
------------------------------------------- Captured stdout --------------------------------------------
seed: 2222
config_dict["zero_optimization"] {'stage': 0, 'reduce_bucket_size': 20, 'zero_hpz_partition_size': 1, 'reduce_scatter': True, 'zero_quantized_weights': False, 'zero_quantized_gradients': False}
[2025-04-02 16:58:20,294] [INFO] [logging.py:107:log_dist] [Rank -1] DeepSpeed info: version=0.16.6+3054b934, git-hash=3054b934, git-branch=master
[2025-04-02 16:58:20,294] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-02 16:58:20,294] [INFO] [comm.py:689:init_distributed] Initializing TorchBackend in DeepSpeed with backend None
_________________________ ERROR collecting tests/torch_compile/test_compile.py _________________________
ImportError while importing test module '/Users/mac1/projects/source/DeepSpeed/tests/torch_compile/test_compile.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/anaconda3/envs/deepspeed/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/torch_compile/test_compile.py:12: in
import intel_extension_for_pytorch # noqa: F401 # type: ignore
E ModuleNotFoundError: No module named 'intel_extension_for_pytorch'
________________________ ERROR collecting tests/unit/ops/lion/test_cpu_lion.py _________________________
tests/unit/ops/lion/test_cpu_lion.py:17: in
pytest.cpu_vendor = get_cpu_info()["vendor_id_raw"].lower()
E KeyError: 'vendor_id_raw'
=========================================== warnings summary ===========================================
:8
:8: PytestDeprecationWarning: A private pytest class or function was used.

tests/unit/checkpoint/test_universal_checkpoint.py:217
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:217: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/checkpoint/test_universal_checkpoint.py:221
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:221: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/checkpoint/test_universal_checkpoint.py:225
/Users/mac1/projects/source/DeepSpeed/tests/unit/checkpoint/test_universal_checkpoint.py:225: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/inference/test_human_eval.py:12
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_human_eval.py:12: PytestUnknownMarkWarning: Unknown pytest.mark.evaluation - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.evaluation

tests/unit/inference/test_inference.py:351
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:351: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_inference.py:437
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:437: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:483
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:483: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_inference.py:515
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:515: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:556
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:556: PytestUnknownMarkWarning: Unknown pytest.mark.seq_inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.seq_inference

tests/unit/inference/test_inference.py:611
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:611: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(3)

tests/unit/inference/test_inference.py:652
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference.py:652: PytestUnknownMarkWarning: Unknown pytest.mark.nightly - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.nightly

tests/unit/inference/test_inference_config.py:13
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_inference_config.py:13: PytestUnknownMarkWarning: Unknown pytest.mark.inference - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference

tests/unit/inference/test_stable_diffusion.py:16
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/test_stable_diffusion.py:16: PytestUnknownMarkWarning: Unknown pytest.mark.stable_diffusion - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.stable_diffusion

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:79
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:79: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:92
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_bias_activation.py:92: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:36
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:36: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_blas_linear.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:41
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:41: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:63
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:63: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:112
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:112: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:118
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:118: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:124
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:124: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:130
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_gated_activation.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_post_ln.py:26
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_post_ln.py:26: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_pre_ln.py:28
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_pre_ln.py:28: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:73
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/core_ops/test_rms_norm.py:73: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:23
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:23: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:107
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/cutlass_ops/test_moe_gemm.py:107: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_atom_builder.py:16
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_atom_builder.py:16: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:129
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:129: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:140
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:140: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:154
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:154: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:167
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:167: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:177
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:177: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:189
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_flash.py:189: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:14
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:14: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:47
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:47: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_kv_copy.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:84
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:84: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:128
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:128: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:172
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:172: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:218
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_blocked_rotary_emb.py:218: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:30
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:30: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:52
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:52: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:76
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_logits_gather.py:76: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:81
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:81: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:104
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_gather.py:104: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_moe_scatter.py:29
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_moe_scatter.py:29: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:106
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:106: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:120
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:120: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:130
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:130: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:148
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:148: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:162
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_ragged_embed.py:162: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:90
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:90: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:101
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:101: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:110
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:110: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:148
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/kernels/ragged_ops/test_top_k_gating.py:148: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/model_implementations/parameters/test_contiguify.py:51
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_contiguify.py:51: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_layer_inheritance.py:31
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_layer_inheritance.py:31: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:73
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:73: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:93
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:93: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:120
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:120: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:133
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:133: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:153
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_mapping.py:153: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:35
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:35: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:55
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_multi_parameter_layer.py:55: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:37
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:37: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:98
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/parameters/test_parameter_list.py:98: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:26
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:26: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:54
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:54: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:92
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_attn_out_sharding.py:92: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:49
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:85
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_mlp_sharding.py:85: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:40
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:40: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:67
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:67: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:112
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:112: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:146
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:146: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:193
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:193: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:209
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:209: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:228
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:228: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:240
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/model_implementations/sharding/test_qkv_sharding.py:240: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/modules/test_blas_linear_module.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blas_linear_module.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blas_linear_module.py:106
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blas_linear_module.py:106: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:127
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:127: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:138
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:138: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:152
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:152: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:165
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:165: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:175
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:175: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:187
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:187: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_blocked_attn.py:198
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_blocked_attn.py:198: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:74
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:74: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:86
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cuda_pre_ln_module.py:86: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_custom_module.py:44
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_custom_module.py:44: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:153
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:153: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:165
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:165: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:181
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:181: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:193
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:193: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:205
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:205: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_cutlass_moe.py:273
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_cutlass_moe.py:273: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_post_ln_module.py:28
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_post_ln_module.py:28: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:74
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:74: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:80
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:80: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_pre_rms_module.py:86
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_pre_rms_module.py:86: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_quantized_linear_module.py:154
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_quantized_linear_module.py:154: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/modules/test_quantized_linear_module.py:170
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/modules/test_quantized_linear_module.py:170: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2_ops

tests/unit/inference/v2/ragged/test_blocked_allocator.py:15
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:15: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:22
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:22: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:50
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:50: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:58
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:58: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:90
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:90: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:99
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:99: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_blocked_allocator.py:117
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_blocked_allocator.py:117: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:13
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:13: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:19
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:25
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:25: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:31
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:31: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:37
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:37: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:43
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:43: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:49
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_manager_configs.py:55
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_manager_configs.py:55: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:19
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:32
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:32: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/inference/v2/ragged/test_ragged_wrapper.py:66
/Users/mac1/projects/source/DeepSpeed/tests/unit/inference/v2/ragged/test_ragged_wrapper.py:66: PytestUnknownMarkWarning: Unknown pytest.mark.inference_v2 - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_v2

tests/unit/model_parallelism/test_autotp_training.py:164
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_autotp_training.py:164: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/model_parallelism/test_autotp_training.py:275
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_autotp_training.py:275: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/model_parallelism/test_configurable_parallel_mp.py:61
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_mp.py:61: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_mp.py:89
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_mp.py:89: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:234
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:234: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:241
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:241: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(1)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:248
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:248: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(2)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:255
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:255: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/model_parallelism/test_configurable_parallel_pp.py:262
/Users/mac1/projects/source/DeepSpeed/tests/unit/model_parallelism/test_configurable_parallel_pp.py:262: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.world_size(4)

tests/unit/ops/transformer/inference/test_attention.py:22
/Users/mac1/projects/source/DeepSpeed/tests/unit/ops/transformer/inference/test_attention.py:22: PytestUnknownMarkWarning: Unknown pytest.mark.inference_ops - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.inference_ops

tests/unit/runtime/half_precision/onebit/test_onebit.py:1163
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/half_precision/onebit/test_onebit.py:1163: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_nvme_checkpointing.py:21
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_nvme_checkpointing.py:21: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_zeropp.py:64
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_zeropp.py:64: PytestUnknownMarkWarning: Unknown pytest.mark.sequential - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.sequential

tests/unit/runtime/zero/test_zeropp.py:190
/Users/mac1/projects/source/DeepSpeed/tests/unit/runtime/zero/test_zeropp.py:190: PytestUnknownMarkWarning: Unknown pytest.mark.nightly - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
@pytest.mark.nightly

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================= short test summary info ========================================
ERROR deepspeed/nvme/test_ds_aio.py
ERROR deepspeed/nvme/test_ds_aio_utils.py
ERROR tests/hybrid_engine/hybrid_engine_test.py - TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
ERROR tests/lightning/test_simple.py
ERROR tests/model/BingBertSquad/test_e2e_squad.py
ERROR tests/model/Megatron_GPT2/run_perf_test.py
ERROR tests/onebit/test_compressed_backend.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/onebit/test_compressed_perf.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/onebit/test_mpi_backend.py
ERROR tests/onebit/test_mpi_perf.py
ERROR tests/onebit/test_nccl_backend.py
ERROR tests/onebit/test_nccl_perf.py
ERROR tests/perf/adagrad_test.py - ValueError: This op had not been implemented on CPU backend.
ERROR tests/perf/adam_test.py - ValueError: This op had not been implemented on CPU backend.
ERROR tests/small_model_debugging/partial_offload_test.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/stage3_test.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/test_mics_config.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/small_model_debugging/test_model.py - ValueError: We detected accelerator mps on your machine. But we don't know which communication back...
ERROR tests/torch_compile/test_compile.py
ERROR tests/unit/ops/lion/test_cpu_lion.py - KeyError: 'vendor_id_raw'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 20 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================= 42 skipped, 146 warnings, 20 errors in 388.69s (0:06:28) =======================
Abort(868846479): Fatal error in internal_Finalize: Other MPI error, error stack:
internal_Finalize(50)............: MPI_Finalize failed
MPII_Finalize(439)...............:
MPID_Finalize(803)...............:
MPIDI_OFI_mpi_finalize_hook(1065):
flush_send_queue(1024)...........:
MPIDI_OFI_handle_cq_error(788)...: OFI poll failed (default nic=utun4: Input/output error)

@hsoftxl
Copy link
Author

hsoftxl commented Apr 3, 2025

run pytest , got this result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tjruwase @hsoftxl