This write-up is very intriguing: https://nathan.rs/posts/gzip-lm/

Actually, this isn't the first time that I see comparing LLM models with compression as a valid abstraction, and it isn't gonna be the last for sure.

IMO, there is indeed a blurry line among both aspects of theory of information. Creating good corpus/models, where methods/approaches of compression and LLMs complement each other, is going to be an interesting field to have fun in coming months/years.