RE: https://fosstodon.org/@Blosc2/117359796684529053

The Blosc team is committed to help you survive the data deluge ☔️

This write-up is very intriguing: https://nathan.rs/posts/gzip-lm/

Actually, this isn't the first time that I see comparing LLM models with compression as a valid abstraction, and it isn't gonna be the last for sure.

IMO, there is indeed a blurry line among both aspects of theory of information. Creating good corpus/models, where methods/approaches of compression and LLMs complement each other, is going to be an interesting field to have fun in coming months/years.

We are making a lot of progress on accessing remote PyTables tables via Blosc2 and fsspec. Stay tuned!

But the currently fashionable practice of pointing a powerful AI tool at the task of answering a problem X, unguided by any human expert in the field X resides in, has created an unprecedented divergence between the production of answers, and the production of insight, to the point where the two questions have become _negatively correlated_: further optimization in the direction of generating answers for X can now _decrease_ the amount of insight one could have gained from the study of X, thanks to factors discussed in previous posts, such as contamination of reasoning with the solution, degradation of the signal-to-noise ratio, or in reduction of incentives to analyze the problem further.

A priori, there is no reason that AI use _necessarily_ has to perform this tradeoff. For instance, the use of AI for literature search is already well established as a basic way to gain initial insight into a problem, or to connect any newly discovered technique to the literature. But the current incentive to race for priority for solutions above all other concerns works against such more deliberate use cases. (2/3)

RE: https://fosstodon.org/@Blosc2/117274732437947791

New Blosc2 is all about *interoperability*.

RE: https://fosstodon.org/@Blosc2/117252592714449579

Our data capsules for remote data are premiering. Check them out!

Je lisais la section "talk" de la page "human penis" sur wikipédia et je vois cette phrase : "Humans are, on average, very interested in the human penis, whether they are fans, detractors, or have a more complex opinion."

Du coup je fais un sondage.

VOTRE OPINION SUR LE PÉNIS HUMAIN

RE: https://mastodon.social/@ironArray/117201347845857107

More intelligent data handling in the server is coming. Check it out!

📢 Python-Blosc2 4.12.0 is out!

Remote arrays now work over S3, GCS and HTTP via fsspec. With lazy=True, Blosc2 fetches only the compressed blocks a slice needs, overlaps requests and reuses a validated cache — 5–17x faster for slices in our S3 benchmarks.

Also: concurrent Caterva2 chunk writers and lean UTF-8 indexes (739 → 5.5 MiB peak memory).

📖 Release notes: https://github.com/Blosc/python-blosc2/releases
🚀 pip install "blosc2[fsspec]" --upgrade

#Python #HPC #Compression

RE: https://fosstodon.org/@Blosc2/117155144727655042

Dicts are nice data structures, and super-useful for storing repeated data.

RE: https://mastodon.social/@ryancheley/117116553954346301

This post about "languishing" really touched a nerve with me and my current situation. I never felt burned out nor did I ever feel like there was nothing more to learn about programming or teaching. To the contrary, I was always trying to learn new things.

However, the current climate of LLMs has just made me think what's the fucking point of programming? Or even trying to teach programmers for that matter? Neither seems like a meaningful path for growth at this point.

RE: https://mastodon.ie/@psneeze/117117444546244052

Also while I was frantically running between telescopes and students and photographers, the physics department head showed me on his phone that the cover article for Scientific American this month is "The Battle for the Night Sky" without knowing that the very first sentence in the article is about me!

Another great article about the situation in orbit by @astrojonny.bsky.social!

https://archive.ph/QcYZ8

📣 Python-Blosc2 4.11.0 continues its alignment with Arrow conventions.

This release rebuilds CTable nullability on a per-column validity mask instead of a sentinel value ✨

Also, wheels are now a single abi3 build per platform covering CPython 3.11+, including 3.15 (tested), with free-threaded 3.14 and 3.15 available as well 🚅

Try it out!

More info: https://github.com/Blosc/python-blosc2/releases

Compress Better, Compute Bigger 🚀

#Python #DataScience #Compression #HPC #OpenSource #Blosc2

RE: https://fosstodon.org/@Blosc2/117078008603047358

The JIT in Blosc2 is quite capable, but requires some understanding to squeeze the last drop of performance out of it.

💡 Python‑Blosc2 tip #10: store unbounded text as `utf8()`, not fixed‑width `string()`.

`string(max_length=200)` rides on NumPy's `<U` dtype, which is UCS‑4: 800 bytes per row in memory on *every* read, three characters or two hundred.

Instead of:
title: str = blosc2.field(blosc2.string(max_length=200))

Do:
title: str = blosc2.field(blosc2.utf8())

Massive memory savings. More info:

https://blosc.org/python-blosc2/guides/optimization_tips.html#store-unbounded-text-as-utf8-not-fixed-width-string

Compress Better, Compute Bigger 🚀

#Python #DataScience #Blosc2

RE: https://fosstodon.org/@Blosc2/117036908162947297

On-disk broadcasting can save lots of memory.

📢 In case you missed it: "Not an island: bringing compression to the tabular ecosystem".

Most compression libraries ask you to move in — learn their API, convert your data, leave your tools behind. We think that's backwards: compression should be a fast, compact layer *underneath* the tools you already use.

Blosc2 4.9.1: DuckDB, Polars, PyArrow and pandas 3 read a CTable directly via Arrow's PyCapsule protocol.

📖 https://blosc.org/posts/not-an-island-tabular-ecosystem/

#Arrow #DuckDB #Pandas #Compression #OpenSource

💡 Python‑Blosc2 tip #9: memory-map read-only opens — skip the syscalls.

`mmap_mode="r"` maps the file once, reads pages directly — no per‑chunk syscalls.

Instead of:
arr = blosc2.open(path)

Do:
arr = blosc2.open(path, mmap_mode="r")

~15% faster for scattered reads, same memory — and it is free. The payoff multiplies with concurrent readers.

https://blosc.org/python-blosc2/guides/optimization_tips.html#memory-map-read-only-opens

Compress Better, Compute Bigger 🚀

#Python #DataScience #Blosc2

💡 Python‑Blosc2 tip #8: push filters into `where=`.

Column reductions accept `where=` — one chunk scan, no intermediates.

Instead of:
t["temperature"][:][t["region"][:] == 3].sum()

Do:
t["temperature"].sum(where=t.region == 3)

~1.9× faster, ~7× less memory.

https://blosc.org/python-blosc2/guides/optimization_tips.html#filtered-reductions-push-the-predicate-down-with-where

Compress Better, Compute Bigger 🚀

#Python #DataScience #Blosc2

💡 Python-Blosc2 tip #7: reduce columns directly — don't slice them first.

t["col"][:] materialises the whole column as one big NumPy array. Column reductions — sum(), mean(), min(), max() — work chunk by chunk over the compressed data instead, while still handling null values and deleted rows correctly.

On a 50M-row column: 1.7× faster and ~12× less peak memory.

📖 https://blosc.org/python-blosc2/guides/optimization_tips.html#reduce-columns-directly-don-t-slice-them-first

#Python #HPC #OpenSource #Compression