Transcription
Meta just open-sourced something that might just change how we handle data compression. It's called Open ZL. And unlike traditional compressors that just look at bytes, this one actually understands what's inside your data. This is called format-aware compression. A system that adapts to the structure of your data, not just the bits.
Today we're going to look at what makes Open ZL different, how it works under the hood, and why this is a big deal for developers building data pipelines, analytics tools, or even databases. It's going to be a lot of fun. So, let's dive into it.
So, let's start with the problem. Tools like Zstandard or GZIP are great general-purpose compressors, but they treat everything as a flat stream of bytes. Whether it's a CSV, a protobuff, or a nested JSON, they see the same thing, just bytes to pack tighter. That means they often miss opportunities. For example, if a column contains repeated enums or integer ranges, a human could design a better encoding, but general compressors can't see that structure.
On the other hand, some companies do build bespoke compressors for specific formats, which squeeze data more efficiently. But this approach scales badly. You end up maintaining dozens of custom decoders, all which need auditing, testing, and security reviews. So Meta asks the question, can we get the best of both worlds? And Open ZL is the solution to this problem. It separates the plan from the decompressor. First, you train or design a compression plan for each data format. But every file, regardless of the format, can still be decompressed by a single universal decoder. And that is the main difference.
So here's how it works. First, you describe the structure. You start with a schema using something called SDDDL, the simple data description language. It tells Open ZL how bytes map to actual fields and types. Then you generate a plan. An offline trainer searches through different transform sequences. These transforms are basically reversible operations that expose patterns like delta encoding, dictionary building or rearranging nested records. And then you encode that data. When you compress it, the chosen plan is embedded right into the file. And finally, you can decode it universally. The Open ZL decoder just reads the recipe from the file and runs the transforms in reverse. It has the same binary no matter the format. So instead of having one decompressor per data type, you get one decompressor that can handle any format as long as it knows how to interpret the embedded plan.
And now here's the fun part, performance. On structured data sets, Open ZL beats general compressors by a wide margin. In one of the Meta tests, Open ZL reduced data from 5.5 megabytes down to 3.5 while compressing at 340 megabytes per second, faster than Zstandard's 220 and decompressing at over 1.2 GB per second. The reason why it's so fast is because the transforms expose structure before entropy coding, so the backend compressor works more efficiently.
But this is not a magic bullet. If the data is unstructured like plain text, Open ZL can't infer structure and falls back to ZSTD internally. Also, parsing formats like CSV introduces overhead which can slow down compression even if ratios improve. But still for anything structured logs, telemetry, analytics data, the gains can be huge.
The bigger idea behind Open ZL is modular, evolvable compression. And this is a big change. You can update or retain compression plans without changing the decompressor at all. That means no breaking backward compatibility, easier security auditing, and faster innovation on the compression side. And since Open ZL is open-source, anyone can add new transforms or parsers for specialized data, time series, nested objects, scientific data, you name it.
So that is Open ZL in a nutshell. It brings a format-aware approach to compression. If you work with data pipelines or large-scale storage, this is definitely worth keeping an eye on. So, what do you think about this new framework? Have you tried it? Will you use it? Let us know in the comments down below. And folks, if you like these types of technical breakdowns, give this video a like and don't forget to subscribe to our channel. This has been Andress from Better Stack, and I will see you in the next videos. [Music]