Oct 20, 2024, 4:12 AM

@adrian Huh, looking at the dataset_spec-to-rtl directory, it looks like the evaluation set -- and their reference models -- are at least mostly synthesizable, though some examples are definitely not how I would write things.

That said, indeed, that benchmark seems to test in simulation, rather than, for instance, demanding that `yosys` make gates out of it, or something. I guess the real tell here, though, is that not only is the LLM garbage at writing synthesizable RTL, but it also seems to be garbage at writing Verilog at all (their August 2024 paper citing a <60% pass rate from GPT-4 Turbo!)...

Replies