Jul 3, 2026, 6:22 PMI think .tar.gz will technically do this to some degree because gzip doesn't know what files are, but this approach still limited by the size of its sliding window (eg it can't deduplicate two files whose size is over 258 bytes or distance apart in the tar is larger than 32K)
@mcc does the ZFS stream format count? Also the brotli sliding window is much larger (up 16M)
@evert @mcc Yeah, if the question is whether *any* such format exists, you could make a ZFS filesystem with dedup and compression enabled, write the data, then ‘zfs send’ it to a file. The file could then be ‘zfs receive’d or mounted. Since dedup only matters on write, you just don’t use dedup on the receiving end, and you don’t pay the RAM penalty.
That’s more or less how I make base images for zones.
@mcc You can use xz instead of gz and get larger sliding windows.
Jul 3, 2026, 6:42 PM@mcc Besides .tar.gz's limited deduplication, there's also git's packs, though unsure how well those work in practice. Iirc they should deduplicate file contents.
@mcc .tar with a more algorithm than .gz. .tar.xz or .tar.zst. There are probably dedicated file-deduplicating solutions too.
Jul 3, 2026, 6:48 PM@mcc I guess squashfs's deduplication could accomplish that, but I'm sure most people would be confused by squashfs as an archive.
@mcc anything that's operating in "solid archive" mode should perform similar to a tarball, I think?
I've seen game roms where every region of a game is in one zip, and those compress as if they are de-duped.
@mcc Some people have already said "tar + something with a large compression window", but in particular tar + lrzip may be best if you go that way. It should be packaged on most distros.
It's intended to have a compression window up to the size of your RAM (or 2 GB on a 32-bit OS), or optionally even larger than RAM, though that's slower.
The "LR" stands for Long Range, even!
It's also multithreaded, which not everything is.
It can also be combined with other compression algorithms if you are a hardcore compression algorithm enthusiast.
(The only thing that bothers me about lrzip is that it doesn't preserve the file's modification time.)
https://wiki.archlinux.org/title/Lrzip
https://en.wikipedia.org/wiki/Rzip