@dain @hweimer There's a paper which argues that an inevitable model collapse will happen because future models will be trained on more and more LLM generated data. From a physical point of view, I think this makes perfect sense. https://www.nature.com/articles/s41586-024-07566-y
Oct 24, 2026, 7:09 PM@hannorein @hweimer that might be true in the recursive case, but empirically, especially small models benefit from synthetic output from larger models (if you want I can find the paper). Also, while post-training has been yielding most of the gains in the last couple of years, what also happened in pre-training is more and more 1) aggressive filtering of training data so slop, SEO fodder, repetition, etc are removed using previous model generations and 2) synthetic data is generated to fill in gaps.
So I think collapse is far from inevitable even with just filtering, but also there are massive gaps in data that are still being filled in. My favourite recent (open model) example is Kolibri that got much more German training data: https://aleph-alpha.com/en/blog/kolibri-has-landed-a-sovereign-open-weight-model/