[WIP] Add association-list-based helper functions into Rosette backend

[gussmith23]

What are the reasons/motivation for this change?

Currently, the Rosette code emitted by Yosys requires the user to ensure they correctly order module inputs when creating the "inputs" struct to be passed to the Rosette function generated for the module. This makes it difficult to use the backend in an automated way, and may lead to silent failures -- e.g., if we provide the right number of ports but in the wrong order, it won't be clear why our module isn't behaving as expected. (As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!)

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

This change adds an optional helper functions (guarded by a flag) that can construct inputs/convert outputs to/from string-keyed association lists. Users can thus explicitly specify and access each input/output port by name. This allows users to be more confident that they are constructing the input struct correctly, and allows users to more easily access outputs by name.

Explain how this is achieved.

Two new Rosette/Racket functions are (optionally) written out. One wraps over the current inputs struct constructor, taking a string-keyed association list mapping input port names to values. The other takes an outputs struct and converts it into a string-keyed association list mapping output port name strings to values.

If applicable, please suggest to reviewers how they can test the change.

I would appreciate feedback on how to test. I see that the Python tests currently run the generated Racket files. Here are two options:

1. Add the new -keyword-constructor flag in the existing tests, which will not actually cause the keyword constructors to be used, but it will cause them to be generated. This will at least test that we do not have syntax errors.
2. Add another set of nearly-identical Python tests that instantiate the input structs via the helper function, rather than the original constructor.

TODO

• Add tests (see above, need guidance from maintainers)
• access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

[gussmith23]

I am making one more change to this and then it will be ready for review. Converting to draft until then.

[gussmith23]

Ready for review.

[KrystalDelusion]

As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!

Do you have an example of this? I remember they were backwards at one point so I had to reverse them, but they should be in the right order now:

module \$fa (A, B, C, X, Y);
input [WIDTH-1:0] A, B, C;
output [WIDTH-1:0] X, Y;

(struct _$fa_Inputs (A B C) #:transparent
  ; A (bitvector 1)
  ; B (bitvector 1)
  ; C (bitvector 1)
)
(struct _$fa_Outputs (X Y) #:transparent
  ; X (bitvector 1)
  ; Y (bitvector 1)
)

If there is any rearranging of ports I would suspect it's in Yosys or the frontend rather than the Rosette backend.

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

As far as I understand Rosette/Racket, this should be possible with (e.g.) _$fa_Outputs-X, similarly to how inputs are (internally) referenced by name (let ((A (_$fa_Inputs-A inputs))). IIRC that's why they're marked #:transparent for both input and output, and also why there are comments for each of the port names/expected types, though I totally agree that being able to construct the input struct from named values would be useful (and the current state is mostly a reflection of me being unfamiliar with Racket).

access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

I think you want to iterate over auto output : ir.outputs() rather than auto name : output_struct.get_field_names(), because that'll give you the list of base names. But again, I'm not sure how necessary this is since you can already access outputs by name.

I would appreciate feedback on how to test.

I don't think adding the flag to the current tests is the right move, since that invites the possibility that the output stops being valid without the flag (though I realise this is already true for the -provides flag). I also don't think it's necessary to test the flag for the full suite of cells. I would say it's sufficient to add one test (or potentially a couple tests) specifically for checking that instantiation via the helper structs works. The current pytest framework may or may not be the best place to add that, but be aware that the functional tests are disabled by default and only run on the private runner during CI because of the additional requirements (though looking at it seems that even then the smtlib and rosette tests are skipped, which seems unideal...).

[gussmith23]

As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!

Do you have an example of this? I remember they were backwards at one point so I had to reverse them, but they should be in the right order now:

module \$fa (A, B, C, X, Y);
input [WIDTH-1:0] A, B, C;
output [WIDTH-1:0] X, Y;

(struct _$fa_Inputs (A B C) #:transparent
  ; A (bitvector 1)
  ; B (bitvector 1)
  ; C (bitvector 1)
)
(struct _$fa_Outputs (X Y) #:transparent
  ; X (bitvector 1)
  ; Y (bitvector 1)
)

If there is any rearranging of ports I would suspect it's in Yosys or the frontend rather than the Rosette backend.

Example: https://github.com/gussmith23/lakeroad/blob/50226012951d66190ea0e36a67c75035943b9d31/modules_for_importing/xilinx_ultrascale_plus/DSP48E2.v

yosys -p "read_verilog -sv DSP48E2.v; prep; clk2fflogic; write_functional_rosette -provides -assoc-list-helpers"

This should produce a file with the ports reversed; that's true on my machine at least. If it's not true on yours...something extra weird is going on!

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

As far as I understand Rosette/Racket, this should be possible with (e.g.) _$fa_Outputs-X, similarly to how inputs are (internally) referenced by name (let ((A (_$fa_Inputs-A inputs))). IIRC that's why they're marked #:transparent for both input and output, and also why there are comments for each of the port names/expected types, though I totally agree that being able to construct the input struct from named values would be useful (and the current state is mostly a reflection of me being unfamiliar with Racket).

You're right that the accessors _$fa_Outputs-X are generated. However, they're hard to use programmatically. Imagine we have a module we're trying to use from Rosette, but we don't actually know its name/its ports at the time of writing the code. It would be natural, then, to take the name and ports list as strings. If we have an output port name o as a string "o", how do we actually use the accessor mod_Outputs-o? We have to generate the symbol for the function using quasiquoting and then eval it -- see https://docs.racket-lang.org/guide/eval.html. It's possible, but it's very worth avoiding, especially when there's easy ways around it.

As an aside, I really do feel like this should be easier in Racket. I'm shocked that there's not a default way to look up a field on a struct if you know the field's name as a string. It could be that I'm totally wrong about this, though; maybe it's easy!

access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

I think you want to iterate over auto output : ir.outputs() rather than auto name : output_struct.get_field_names(), because that'll give you the list of base names. But again, I'm not sure how necessary this is since you can already access outputs by name.

Thanks! Fixed.

I would appreciate feedback on how to test.

I don't think adding the flag to the current tests is the right move, since that invites the possibility that the output stops being valid without the flag (though I realise this is already true for the -provides flag). I also don't think it's necessary to test the flag for the full suite of cells. I would say it's sufficient to add one test (or potentially a couple tests) specifically for checking that instantiation via the helper structs works. The current pytest framework may or may not be the best place to add that, but be aware that the functional tests are disabled by default and only run on the private runner during CI because of the additional requirements (though looking at it seems that even then the smtlib and rosette tests are skipped, which seems unideal...).

Any concrete suggestions on where I should put the tests? Should I not write them in pytest? Should I add them as "functional tests"? Are those the same things as the pytest tests?

[KrystalDelusion]

...
This should produce a file with the ports reversed; that's true on my machine at least. If it's not true on yours...something extra weird is going on!

Huh. It's not just backwards, it's alphabetised too, but only from the functional backends (and not just Rosette). I wonder if it's related to them being stored as a Yosys::hashlib::dict, and potentially below a certain size the order is maintained but above that they get sorted?

However, they're hard to use programmatically. Imagine we have a module we're trying to use from Rosette, but we don't actually know its name/its ports at the time of writing the code.

Ah right, that is true, particularly with the renaming/escaping that happens when it creates a unique name. In that case it does make sense to have an output converter as well.

Any concrete suggestions on where I should put the tests? Should I not write them in pytest? Should I add them as "functional tests"? Are those the same things as the pytest tests?

The tests/functional directory is gated by the make flag ENABLE_FUNCTIONAL_TESTS. Currently the tests/functional/run-test.sh script is just a single call to pytest -v -m "not smt and not rkt" "$@", which runs pytest, but skips the smt and rkt (Rosette) tests. I think what happened is that the smt and rkt tests were disabled because they have extra prerequisites that are not currently installed on the normal git runner, and would also break regular calls to make test without them. The ENABLE_FUNCTIONAL_TESTS was then added later, to limit running them to just the private runner which we also use to run the verific tests, meaning breaking make test is no longer a concern so the -m "not smt and not rkt" should be safe to remove.

Where you write the tests depends on if you prefer bash or python for scripting; if you prefer bash then you should put them in tests/functional/run-test.sh, if instead you prefer python then you're welcome to use the pytest infrastructure that already exists in that directory. In either case, any extra .v or .il files you need can go in the same directory.

[KrystalDelusion]

hierarchy; write_functional_rosette gives A B C.
prep; write_functional_rosette gives C B A... Looks like it's opt_clean that sorts them, but I'm unclear on why this affects the functional backends but not the others.

EDIT:
Ah, write_verilog uses module->ports which maintains order, the functional IR generator uses module->wires() (and then tests wire->port_input and wire->port_output), which gets sorted for faster look ups.

I can get it to not alphabetise if I use module->ports in functional.cc, but Yosys::hashlib::dict iterates in reverse??

iterator operator++() { index--; return *this; }

Relatively straightforward to solve in c++20 with for (auto input : ir.inputs() | std::views::reverse)... Less so in c++17

[gussmith23]

@KrystalDelusion I'm working on tests now, I will re-request you when it's ready! Feel free to unsubscribe for now.