Add DecoupledSegNet

Author: wuyefeilin

README.md

@@ -40,6 +40,7 @@ Welcome to PaddleSeg! PaddleSeg is an end-to-end image segmentation development
 |[U<sup>2</sup>-Net](./configs/u2net)|-|-|-|-|
 |[Att U-Net](./configs/attention_unet)|-|-|-|-|
 |[U-Net++](./configs/unet_plusplus)|-|-|-|-|
+|[DecoupledSegNet](./configs/decoupled_segnet)|✔|✔|||
 
 ## Dataset
 

README_CN.md

@@ -40,6 +40,7 @@ PaddleSeg是基于飞桨[PaddlePaddle](https://www.paddlepaddle.org.cn)开发的
 |[U<sup>2</sup>-Net](./configs/u2net)|-|-|-|-|
 |[Att U-Net](./configs/attention_unet)|-|-|-|-|
 |[U-Net++](./configs/unet_plusplus)|-|-|-|-|
+|[DecoupledSegNet](./configs/decoupled_segnet)|✔|✔|||
 
 ## 数据集
 

configs/decoupled_segnet/README.md

@@ -0,0 +1,14 @@
+# Improving Semantic Segmentation via Decoupled Body and Edge Supervision
+
+## Reference
+
+> Li X, Li X, Zhang L, et al. Improving semantic segmentation via decoupled body and edge supervision[J]. arXiv preprint arXiv:2007.10035, 2020.
+
+## Performance
+
+### Cityscapes
+
+| Model | Backbone | Resolution | Training Iters | mIoU | mIoU (flip) | mIoU (ms+flip) | Links |
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+|DecoupledSegNet|ResNet50_OS8|1024x512|80000|80.86%|81.34%|81.49%|[model](https://bj.bcebos.com/paddleseg/dygraph/cityscapes/decoupledsegnet_resnet50_os8_cityscapes_1024x512_80k/model.pdparams) \| [log](https://bj.bcebos.com/paddleseg/dygraph/cityscapes/decoupledsegnet_resnet50_os8_cityscapes_1024x512_80k/train.log) \| [vdl](https://www.paddlepaddle.org.cn/paddle/visualdl/service/app/scalar?id=3c5cba5e6f89b33dc75b43c62026dc12)|
+|DecoupledSegNet|ResNet50_OS8|832x832|80000|81.26%|81.56%|81.80%|[model](https://bj.bcebos.com/paddleseg/dygraph/cityscapes/decoupledsegnet_resnet50_os8_cityscapes_832x832_80k/model.pdparams) \| [log](https://bj.bcebos.com/paddleseg/dygraph/cityscapes/decoupledsegnet_resnet50_os8_cityscapes_832x832_80k/train.log) \| [vdl](https://paddlepaddle.org.cn/paddle/visualdl/service/app?id=e3e8f9044d96a57f7337f5928f2c265f)|

configs/decoupled_segnet/decoupledsegnet_resnet50_os8_cityscapes_1024x512_80k.yml

@@ -0,0 +1,28 @@
+_base_: '../_base_/cityscapes.yml'
+
+model:
+  type: DecoupledSegNet
+  backbone:
+    type: ResNet50_vd
+    output_stride: 8
+    multi_grid: [1, 2, 4]
+    pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
+  num_classes: 19
+  backbone_indices: [0, 3]
+  aspp_ratios: [1, 12, 24, 36]
+  aspp_out_channels: 256
+  align_corners: False
+  pretrained: null
+
+loss:
+  types:
+    - type: OhemCrossEntropyLoss
+    - type: RelaxBoundaryLoss
+    - type: BCELoss
+      weight: 'dynamic'
+      edge_label: True
+    - type: OhemEdgeAttentionLoss
+  coef: [1,1,25,1]
+
+train_dataset:
+  edge: True

configs/decoupled_segnet/decoupledsegnet_resnet50_os8_cityscapes_832x832_80k.yml

@@ -0,0 +1,44 @@
+_base_: '../_base_/cityscapes.yml'
+
+model:
+  type: DecoupledSegNet
+  backbone:
+    type: ResNet50_vd
+    output_stride: 8
+    multi_grid: [1, 2, 4]
+    pretrained: https://bj.bcebos.com/paddleseg/dygraph/resnet50_vd_ssld_v2.tar.gz
+  num_classes: 19
+  backbone_indices: [0, 3]
+  aspp_ratios: [1, 12, 24, 36]
+  aspp_out_channels: 256
+  align_corners: False
+  pretrained: null
+
+loss:
+  types:
+    - type: OhemCrossEntropyLoss
+    - type: RelaxBoundaryLoss
+    - type: BCELoss
+      weight: 'dynamic'
+      edge_label: True
+    - type: OhemEdgeAttentionLoss
+  coef: [1,1,25,1]
+
+train_dataset:
+  transforms:
+    - type: ResizeStepScaling
+      min_scale_factor: 0.75
+      max_scale_factor: 2.0
+      scale_step_size: 0.25
+    - type: RandomPaddingCrop
+      crop_size: [832, 832]
+    - type: RandomHorizontalFlip
+    - type: RandomDistort
+      brightness_range: 0.4
+      contrast_range: 0.4
+      saturation_range: 0.4
+    - type: Normalize
+  edge: True
+
+optimizer:
+  weight_decay: 5.0e-4

docs/apis/models.md

@@ -18,6 +18,7 @@ The models subpackage contains the following model for image sementic segmentaio
 - [U<sup>2</sup>Net+](#U2Net-1)
 - [AttentionUNet](#AttentionUNet)
 - [UNet++](#UNet-1)
+- [DecoupledSegNet](#DecoupledSegNet)
 
 ## [DeepLabV3+](../../paddleseg/models/deeplab.py)
 > CLASS paddleseg.models.DeepLabV3P(num_classes, backbone, backbone_indices=(0, 3), aspp_ratios=(1, 6, 12, 18), aspp_out_channels=256, align_corners=False, pretrained=None)
@@ -402,3 +403,32 @@ The models subpackage contains the following model for image sementic segmentaio
             is even, e.g. 1024x512, otherwise it is True, e.g. 769x769.  Default: False.
 > > > - **pretrained** (str, optional): The path or url of pretrained model for fine tuning. Default: None.
 > > > - **is_ds** (bool): use deep supervision or not. Default: True
+
+## [DecoupledSegNet](../../paddleseg/models/decoupled_segnet.py)
+> class DecoupledSegNet(num_classes,
+                 backbone,
+                 backbone_indices=(0, 3),
+                 aspp_ratios=(1, 6, 12, 18),
+                 aspp_out_channels=256,
+                 align_corners=False,
+                 pretrained=None)
+
+    The DecoupledSegNet implementation based on PaddlePaddle.
+
+    The original article refers to
+    Xiangtai Li, et, al. "Improving Semantic Segmentation via Decoupled Body and Edge Supervision"
+    (https://arxiv.org/pdf/2007.10035.pdf)
+
+> > Args
+> > > - **num_classes** (int): The unique number of target classes.
+> > > - **backbone** (paddle.nn.Layer): Backbone network, currently support Resnet50_vd/Resnet101_vd.
+> > > - **backbone_indices** (tuple, optional): Two values in the tuple indicate the indices of output of backbone.
+           Default: (0, 3).
+> > > - **aspp_ratios** (tuple, optional): The dilation rate using in ASSP module.
+            If output_stride=16, aspp_ratios should be set as (1, 6, 12, 18).
+            If output_stride=8, aspp_ratios is (1, 12, 24, 36).
+            Default: (1, 6, 12, 18).
+> > > - **aspp_out_channels** (int, optional): The output channels of ASPP module. Default: 256.
+> > > - **align_corners** (bool, optional): An argument of F.interpolate. It should be set to False when the feature size is even,
+            e.g. 1024x512, otherwise it is True, e.g. 769x769. Default: False.
+> > > - **pretrained** (str, optional): The path or url of pretrained model. Default: None.

paddleseg/core/train.py

@@ -35,16 +35,16 @@ def check_logits_losses(logits_list, losses):
 
 def loss_computation(logits_list, labels, losses, edges=None):
     check_logits_losses(logits_list, losses)
-    loss = 0
+    loss_list = []
     for i in range(len(logits_list)):
         logits = logits_list[i]
         loss_i = losses['types'][i]
-        # Whether to use edges as labels According to loss type .
+        # Whether to use edges as labels According to loss type.
         if loss_i.__class__.__name__ in ('BCELoss', ) and loss_i.edge_label:
-            loss += losses['coef'][i] * loss_i(logits, edges)
+            loss_list.append(losses['coef'][i] * loss_i(logits, edges))
         else:
-            loss += losses['coef'][i] * loss_i(logits, labels)
-    return loss
+            loss_list.append(losses['coef'][i] * loss_i(logits, labels))
+    return loss_list
 
 
 def train(model,
@@ -115,6 +115,7 @@ def train(model,
 
     timer = Timer()
     avg_loss = 0.0
+    avg_loss_list = []
     iters_per_epoch = len(batch_sampler)
     best_mean_iou = -1.0
     best_model_iter = -1
@@ -140,11 +141,12 @@ def train(model,
                 logits_list = ddp_model(images)
             else:
                 logits_list = model(images)
-            loss = loss_computation(
+            loss_list = loss_computation(
                 logits_list=logits_list,
                 labels=labels,
                 losses=losses,
                 edges=edges)
+            loss = sum(loss_list)
             loss.backward()
 
             optimizer.step()
@@ -154,10 +156,18 @@ def train(model,
                 optimizer._learning_rate.step()
             model.clear_gradients()
             avg_loss += loss.numpy()[0]
+            if not avg_loss_list:
+                avg_loss_list = [l for l in loss_list]
+            else:
+                for i in range(len(loss_list)):
+                    avg_loss_list[i] += loss_list[i]
             train_batch_cost += timer.elapsed_time()
 
             if (iter) % log_iters == 0 and local_rank == 0:
                 avg_loss /= log_iters
+                avg_loss_list = [
+                    l.numpy()[0] / log_iters for l in avg_loss_list
+                ]
                 avg_train_reader_cost = train_reader_cost / log_iters
                 avg_train_batch_cost = train_batch_cost / log_iters
                 train_reader_cost = 0.0
@@ -171,12 +181,22 @@ def train(model,
                             avg_train_reader_cost, eta))
                 if use_vdl:
                     log_writer.add_scalar('Train/loss', avg_loss, iter)
+                    # Record all losses if there are more than 2 losses.
+                    if len(avg_loss_list) > 1:
+                        avg_loss_dict = {}
+                        for i, value in enumerate(avg_loss_list):
+                            avg_loss_dict['loss_' + str(i)] = value
+                        for key, value in avg_loss_dict.items():
+                            log_tag = 'Train/' + key
+                            log_writer.add_scalar(log_tag, value, iter)
+
                     log_writer.add_scalar('Train/lr', lr, iter)
                     log_writer.add_scalar('Train/batch_cost',
                                           avg_train_batch_cost, iter)
                     log_writer.add_scalar('Train/reader_cost',
                                           avg_train_reader_cost, iter)
                 avg_loss = 0.0
+                avg_loss_list = []
 
             if (iter % save_interval == 0
                     or iter == iters) and (val_dataset is not None):

paddleseg/models/__init__.py

@@ -30,3 +30,4 @@
 from .u2net import U2Net, U2Netp
 from .attention_unet import AttentionUNet
 from .unet_plusplus import UNetPlusPlus
+from .decoupled_segnet import DecoupledSegNet