Login
You're viewing the discuss.systems public feed.

Replies

  • Oct 16, 2024, 8:35 PM

    Mainstream HDLs are a foundation of sand, with no consistent toolchain-independent semantics. The only reasonable way to understand Verilog is to use a tiny, well-behaved subset.

    “Generates Verilog that actually means something” appears to be a non-goal for current LLM research. (As a rough software analogy: it’s as if the generative AI people were exclusively focused on generating C, but didn’t care at all about UB.) If it were a goal, the rational thing to do would be to generate some representation much closer to a netlist.

    💬 1🔄 1⭐ 11
  • Oct 16, 2024, 5:20 PM

    @adrian

    > The computer architecture community has a rich history of designing chips with methodologies and tools that have evolved over time.

    Kind of curious what examples they would give of this. Compared to software the field has been incredibly stagnant for the past 30 years. HLS is a big change I guess but I'm not sure it lives up to the hype.

    I guess there is also a bit of change on the verification side 🤔

    💬 1🔄 0⭐ 1
  • Oct 16, 2024, 5:50 PM

    @thezoq2 @adrian I guess synthesizing Verilog vs. just using it from simulation and then more recently there is HLS

    💬 0🔄 0⭐ 2
  • Oct 19, 2024, 6:23 AM
    @adrian I have to say, I was similarly skeptical of HLS, for a very similar reason -- HLS originally came in with the promise of 'some day, you will be able to type some SystemC, and I'll just make the hardware for you that does it'. The last I looked at HLS was a few years ago, but certainly that was not what the state of the art accomplished then! Xilinx's tool, at the time, needed a lot of handholding -- to the same extent as you need to write only the synthesis subset of Verilog in order to effectively specify digital logic in Verilog, it seemed like you needed to write some ill-defined synthesis subset of SystemC to make it generate the logic that you wanted, and to guide the tool towards generating an architecture that would meet your PPA goals.

    On the other hand, the higher end commercial HLS tools were actually relatively usable, or so said my RTL colleagues at nvidia -- it might have been the case that that Catapult actually had a meaningfully-specified subset of what you could type that would result in a usable architecture.

    So, that said, if you think about "Verilog is a language in which you write things that look like netlists in" rather than "Verilog is a language that you can use the entire grammar of", is *that* particular thing outside of what's possible to do with an LLM? Having spent no time actually playing with the darned things, I had kind of assumed that's what you would do LLM fine-tuning for (teach the LLM to please stop generating non-synthesizable Verilog, and instead be more careful about writing things that explicitly look like DFFs and RAMs and muxes). *That* part seems ok with me [1].

    The real hard part about designing hardware, for me, is that per gate that I produce, or per line of RTL that I write, there's a lot more architectural exploration and macro-level planning, as compared to when I write software. That's the kind of thing that trips up novice hardware designers, but more than that, it is the kind of thing that I expect an LLM to be particularly bad at. I spend a *lot* of time going back up and thinking "hm, no, I just realized that is going to be a long path if I don't precompute this in a previous pipeline stage", or some such, and then rewriting a lot of architecture.

    Are there mitigations for *that*? People seem to be doing research on making LLMs plan ahead. I am skeptical that that is going to happen in any meaningful capacity, and that seems like a prerequisite for an LLM being a tolerable hardware architect. My priors are still that the things are mostly a waste of perfectly good ocean boiling, and that my hopes are that the music will stop soon so that we can all go back to doing useful things with our lives. But, all this is to say, I was surprised by HLS actually being useful on the high end, and I guess I could at least believe that there would be a path to surprising me again.

    [1] Sort of. I mean, the other big problem is that the training set for good Verilog basically does not exist. Big companies write Verilog in their own in-house higher-level wrappers, and anyway, you can't get their stuff. Open source stuff mostly just isn't industrial-strength, for many reasons. And the *vast* majority of *any* content that you find written in Verilog is undergrads pasting truly unfixably atrocious code into forums asking why the tool dumped core!
    💬 1🔄 0⭐ 1
  • Oct 19, 2024, 9:08 PM

    @joshua I agree with basically all of this! e.g., LLM-generated "netlisty verilog" seems like a worthy goal; I don't think it's at all close to what the current wave is doing—as evidenced by VerilogEval, the main benchmark in the space github.com/NVlabs/verilog-eval

    💬 1🔄 0⭐ 0
  • Oct 20, 2024, 4:12 AM
    @adrian Huh, looking at the dataset_spec-to-rtl directory, it looks like the evaluation set -- and their reference models -- are at least mostly synthesizable, though some examples are definitely not how I would write things.

    That said, indeed, that benchmark seems to test in simulation, rather than, for instance, demanding that `yosys` make gates out of it, or something. I guess the real tell here, though, is that not only is the LLM garbage at writing synthesizable RTL, but it also seems to be garbage at writing Verilog at all (their August 2024 paper citing a <60% pass rate from GPT-4 Turbo!)...
    💬 0🔄 0⭐ 0