implement on-the-fly dataset, generate data during training.

[r1cheu]

Hi, I'm using mmengine to build my custom toolbox. I need to randomly sample data from a text file(~3G) during the trainning. I plan to save the text path and other information in the dataset meta info and sample it using tranforms. Any advice? thanks a lot for your time.

[HAOCHENYE]

Hi, commonly dataset meta is used to save the CLASSES name, visualization palette, .etc meta information. If text path is the gt file path, maybe overriding the load_data_list and save them in data_list could be more proper

[r1cheu]

thanks for your advice. I chose another strategy to generate data and overriding the load_data_list. But now I get another problem.
The situation is I need to training a model that with multiple heads and a shared backbone on multiple datasets. Can you provide some
code to reference? thanks a lot. Here is a pesudo code:

class Model(Module):
   def __init__(self):
        ....
        self.backone = ...
        self.head1 = ...
        self.head2 = ...
        self.head3 = ...
    def forward(self, x)
        out = dict{}
        embedding = self.backbone(x)
        for i in range(num_heads):
            head_name = f'head_{i+1}'
            head = getattr(self, head_name)
            out[head_name] = head(embedding)
        return out


def train_step(batch, head):
    X, y = batch
    optim.zero_grad()
    output = model(X)[head]
    loss = loss_fn(output, y)
    loss.backward()
    optim.step()
    return loss.item()

for batch1, batch2, batch3 in zip(train_dataloader1, train_dataloader2, train_dataloader3,....):
    loss1 = train_step(batch1, head1)
    loss2 = train_step(batch2, head2)
    loss3 = train_step(batch3, head3)
    ....

for batch1, batch2, batch3 in zip(valid_dataloader1, valid_dataloader2, valid_dataloader3,....):
    metric1 = val_step(batch1, head1)
    metric2 = val_step(batch2, head2)
    metric3 = val_step(batch3, head3)
    ....

[HAOCHENYE]

You can override the default EpochBasedTrainLoop or IterbasedTrainLoop, and implement the multiple for-loop there.

@LOOPS.register_module()
class CustomLoop(BaseLoop):
    def __init__(...):
        # Built multiple dataloader here
    
    def run(...):
        # Implement your for-loop here.

MMRazor has overridden many loops there, and you can refer to it.

[r1cheu]

ah, thanks. got it.