When we start to consider DNN model partitioning, we need to come up with a data distribution scheme that can correctly divide and scatter data cross evaluators. For example, evaluators that have partitions of the same model (not replicas) need to receive a scatter of the same training example, whereas evaluators of different model replicas need to receive totally different training examples.
When we start to consider DNN model partitioning, we need to come up with a data distribution scheme that can correctly divide and scatter data cross evaluators. For example, evaluators that have partitions of the same model (not replicas) need to receive a scatter of the same training example, whereas evaluators of different model replicas need to receive totally different training examples.