ValueError: Input and output must have the same number of spatial dimensions,

[deepakkupanda]

deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/work_dirs/pspnet_r101-d8_512x512_160k_p3m10k
2022-05-28 12:13:16,781 - mmseg - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) PolyLrUpdaterHook
(NORMAL ) CheckpointHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook

before_train_epoch:
(VERY_HIGH ) PolyLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook

before_train_iter:
(VERY_HIGH ) PolyLrUpdaterHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook

after_train_iter:
(ABOVE_NORMAL) OptimizerHook
(NORMAL ) CheckpointHook
(LOW ) IterTimerHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook

after_train_epoch:
(NORMAL ) CheckpointHook
(LOW ) DistEvalHook
(VERY_LOW ) TextLoggerHook

before_val_epoch:
(LOW ) IterTimerHook
(VERY_LOW ) TextLoggerHook

before_val_iter:
(LOW ) IterTimerHook

after_val_iter:
(LOW ) IterTimerHook

after_val_epoch:
(VERY_LOW ) TextLoggerHook

after_run:
(VERY_LOW ) TextLoggerHook

2022-05-28 12:13:16,781 - mmseg - INFO - workflow: [('train', 1)], max: 160000 iters
2022-05-28 12:13:16,781 - mmseg - INFO - Checkpoints will be saved to /mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/work_dirs/pspnet_r101-d8_512x512_160k_p3m10k by HardDiskBackend.
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 241, in
main()
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 229, in main
train_segmentor(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/apis/train.py", line 191, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 61, in train
outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/parallel/distributed.py", line 59, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/base.py", line 138, in train_step
losses = self(**data_batch)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/fp16_utils.py", line 110, in new_func
return old_func(*args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/base.py", line 108, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 143, in forward_train
loss_decode = self._decode_head_forward_train(x, img_metas,
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 86, in _decode_head_forward_train
loss_decode = self.decode_head.forward_train(x, img_metas,
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/decode_heads/decode_head.py", line 204, in forward_train
losses = self.losses(seg_logits, gt_semantic_seg)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/fp16_utils.py", line 198, in new_func
return old_func(*args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/decode_heads/decode_head.py", line 235, in losses
seg_logit = resize(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda7/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/ops/wrappers.py", line 27, in resize
return F.interpolate(input, size, scale_factor, mode, align_corners)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/torch/nn/functional.py", line 3835, in interpolate
raise ValueError(
ValueError: Input and output must have the same number of spatial dimensions, but got input with with spatial dimensions of [64, 64] and output size of torch.Size([512, 512, 3]). Please provide input tensor in (N, C, d1, d2, ...,dK) format and output size in (o1, o2, ...,oK) format.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 18487) of binary: /anaconda/envs/open-mmlab/bin/python
tools/dist_train.sh: line 19: 18484 Segmentation fault (core dumped) python -m torch.distributed.launch --nnodes=$NNODES --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --nproc_per_node=$GPUS --master_port=$PORT $(dirname "$0")/train.py $CONFIG --seed 0 --launcher pytorch ${@:3}

[deepakkupanda]

I ran the code using
tools/dist_train.sh configs/pspnet/pspnet_r101-d8_512x512_160k_p3m10k.py 1 --work-dir work_dirs/pspnet_r101-d8_512x512_160k_p3m10k/ --deterministic

[deepakkupanda]

2022-05-28 18:27:51,182 - mmseg - INFO - workflow: [('train', 1)], max: 160000 iters
2022-05-28 18:27:51,183 - mmseg - INFO - Checkpoints will be saved to /mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/work_dirs/upernet_beit-base_640x640_160k_p3m5_ms by HardDiskBackend.
Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 241, in
main()
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 229, in main
train_segmentor(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/apis/train.py", line 191, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 61, in train
outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/parallel/distributed.py", line 59, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/base.py", line 138, in train_step
losses = self(**data_batch)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/fp16_utils.py", line 110, in new_func
return old_func(*args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/base.py", line 108, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 143, in forward_train
loss_decode = self._decode_head_forward_train(x, img_metas,
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py", line 86, in _decode_head_forward_train
loss_decode = self.decode_head.forward_train(x, img_metas,
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/decode_heads/decode_head.py", line 204, in forward_train
losses = self.losses(seg_logits, gt_semantic_seg)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/fp16_utils.py", line 198, in new_func
return old_func(*args, **kwargs)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/models/decode_heads/decode_head.py", line 235, in losses
seg_logit = resize(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/ops/wrappers.py", line 27, in resize
return F.interpolate(input, size, scale_factor, mode, align_corners)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/torch/nn/functional.py", line 3835, in interpolate
raise ValueError(
ValueError: Input and output must have the same number of spatial dimensions, but got input with with spatial dimensions of [160, 160] and output size of torch.Size([640, 640, 3]). Please provide input tensor in (N, C, d1, d2, ...,dK) format and output size in (o1, o2, ...,oK) format.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 19455) of binary: /anaconda/envs/open-mmlab/bin/python
tools/dist_train.sh: line 19: 19333 Segmentation fault (core dumped) python -m torch.distributed.launch --nnodes=$NNODES --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --nproc_per_node=$GPUS --master_port=$PORT $(dirname "$0")/train.py $CONFIG --seed 0 --launcher pytorch ${@:3}

[deepakkupanda]

Same error after running it with this command:
tools/dist_train.sh configs/beit/upernet_beit-base_640x640_160k_p3m5_ms.py 1 --work-dir work_dirs/upernet_beit-base_640x640_160k_p3m5_ms/ --deterministic

[MeowZheng]

From the error log, I guess you trained the model with your own datasets, and the seg map label shape is (640,640,3)

I suggest you might add a suggest break point to check the seg map label, before L235 in decode_head.py

mmsegmentation/mmseg/models/decode_heads/decode_head.py

Lines 235 to 239 in ef90522

	seg_logit = resize(
	input=seg_logit,
	size=seg_label.shape[2:],
	mode='bilinear',
	align_corners=self.align_corners)

If the seg map label is 3 channel, I suggest you convert the 3 channel color map to the 1 channel mask, please refer

mmsegmentation/tools/convert_datasets/isaid.py

Lines 14 to 45 in ef90522

	iSAID_palette = \
	{
	0: (0, 0, 0),
	1: (0, 0, 63),
	2: (0, 63, 63),
	3: (0, 63, 0),
	4: (0, 63, 127),
	5: (0, 63, 191),
	6: (0, 63, 255),
	7: (0, 127, 63),
	8: (0, 127, 127),
	9: (0, 0, 127),
	10: (0, 0, 191),
	11: (0, 0, 255),
	12: (0, 191, 127),
	13: (0, 127, 191),
	14: (0, 127, 255),
	15: (0, 100, 155)
	}
	iSAID_invert_palette = {v: k for k, v in iSAID_palette.items()}
	def iSAID_convert_from_color(arr_3d, palette=iSAID_invert_palette):
	"""RGB-color encoding to grayscale labels."""
	arr_2d = np.zeros((arr_3d.shape[0], arr_3d.shape[1]), dtype=np.uint8)
	for c, i in palette.items():
	m = np.all(arr_3d == np.array(c).reshape(1, 1, 3), axis=2)
	arr_2d[m] = i
	return arr_2d

[deepakkupanda]

I changed the segmap and make the background to [0,0,0] and foreground to [1,1,1].
Still, Im getting this error.
2022-06-01 00:12:23,560 - mmseg - INFO - Saving checkpoint at 16000 iterations
2022-06-01 00:12:27,752 - mmseg - INFO - Exp name: pspnet_r101-d8_512x512_160k_p3m10k.py
2022-06-01 00:12:27,753 - mmseg - INFO - Iter [16000/160000] lr: 9.104e-03, eta: 2 days, 6:23:58, time: 1.443, data_time: 0.010, memory: 9161, decode.loss_ce: 0.0001, decode.acc_seg: 100.0000, aux.loss_ce: 0.0001, aux.acc_seg: 100.0000, loss: 0.0002
[ ] 0/500, elapsed: 0s, ETA:Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 241, in
main()
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/tools/train.py", line 229, in main
train_segmentor(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/apis/train.py", line 191, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run
iter_runner(iter_loaders[i], **kwargs)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/iter_based_runner.py", line 67, in train
self.call_hook('after_train_iter')
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/base_runner.py", line 309, in call_hook
getattr(hook, fn_name)(self)
File "/anaconda/envs/open-mmlab/lib/python3.10/site-packages/mmcv/runner/hooks/evaluation.py", line 262, in after_train_iter
self._do_evaluate(runner)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/core/evaluation/eval_hooks.py", line 113, in _do_evaluate
results = multi_gpu_test(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/apis/test.py", line 219, in multi_gpu_test
result = dataset.pre_eval(result, indices=batch_indices)
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/datasets/custom.py", line 301, in pre_eval
intersect_and_union(
File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/deepakpanda9/code/Users/deepakpanda/segmentation/mmsegmentation/mmseg/core/evaluation/metrics.py", line 75, in intersect_and_union
pred_label = pred_label[mask]
IndexError: too many indices for tensor of dimension 2
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 12098) of binary: /anaconda/envs/open-mmlab/bin/python
tools/dist_train.sh: line 19: 12095 Segmentation fault (core dumped) python -m torch.distributed.launch --nnodes=$NNODES --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --nproc_per_node=$GPUS --master_port=$PORT $(dirname "$0")/train.py $CONFIG --seed 0 --launcher pytorch ${@:3}

[deepakkupanda]

I have defined the dataset like this. Is it right that I have defined the background as [120,120,120] and foreground as [6,230,230]. Do I need to change the values to [0,0,0] and [1,1,1].

Copyright (c) OpenMMLab. All rights reserved.

import os.path as osp

import mmcv
import numpy as np
from PIL import Image

from .builder import DATASETS
from .custom import CustomDataset

@DATASETS.register_module()
class P3M10kDataset(CustomDataset):
CLASSES = ('background', 'foreground')
PALETTE = [[120, 120, 120], [6, 230, 230]]

def __init__(self, **kwargs):
    super(P3M10kDataset, self).__init__(
        img_suffix='.jpg',
        seg_map_suffix='.png',
        reduce_zero_label=False,
        **kwargs)

[deepakkupanda]

@linfangjian01 @MeowZheng @kahkeng @nizhib Please help

[MeowZheng]

The shape of the seg map must be [H, W], so the pixel value is 0 or 1 rather than [0,0,0] or [1,1,1]

[hakpr]

Have you solved this problem? I also encountered this problem when using my own data

[deepakkupanda]

Segmap will be of HxWX3 but make sure that values are in the range of [0, num_class-1]
If you are doing binary class then segmask will only be 0 and 1.
#1641

[jinwonkim93]

i aslo encountered this problem with custom data. Is there a solution?