Dynamic Generation of Tests

[diaconuccalin]

Current Status

The tests under DeeployTest, that are also used by the CI pipeline, are currently static, in the sense that they are pre-generated (using external scripts) ONNX and .npz files, containing the model definition, sample inputs, as well as expected outputs. This makes them rather opaque (the ONNX files need to be opened and analyzed separately, as well as the input and output), and thus difficult to debug.

Also, storing these files in their entirety, rather than generating them when they are needed, together with random sample inputs and weights, increases the memory footprint of the repo.

Proposed Solution

Replace the current tests with Python scripts, in which the model is defined, together with a procedure to generate random testing data for it (ONNX with weights randomly initialized, random inputs of predefined dimensions, and the expected output, computed by doing a forward pass).

Alternatives

Keep the current tests as they are, but generate future tests in the manner described in the previous point.

[Xeratec]

This is something we had prior to open-sourcing, but Moritz removed it. I think he did not like that I used onnx-graphsurgeon and numpy. We could move the old scripts here or build a pipeline to build the tests directly from PyTorch.

[Victor-Jung]

This is something we had prior to open-sourcing, but Moritz removed it. I think he did not like that I used onnx-graphsurgeon and numpy. We could move the old scripts here or build a pipeline to build the tests directly from PyTorch.

I don't see why all tests should stem from pytorch. Making some tests directly with numpy and onnx-graphsurgeon seems valid to me.

[lukamac]

One point for having inputs/outputs checked in, instead of generated, is that it is easier to reproduce. I think it would be good to have the script that generated the values, but keep using concrete values for testing.