So there I was, browsing crates.io at an hour that most reasonable people would describe as "the middle of the night", doing what all emotionally stable developers do at that hour: auditing dead Rust crates for signs of life.
Not crc32fast. Not crc. Not crc32c. The original. The stubby, ancient, crc32 crate. Last published in 2015. No changelog. No CI. A Cargo.toml that predates most of the stable Rust syntax I use daily. And sitting there, abandoned and unloved.
People were using this thing. Were probably shipping software with it in their dependency tree. They just didn't know how to reach the crate maintainer, who had apparently retired to a cabin in the woods with no internet access sometime around when Rust 1.0 was announced.
Then I did what any emotionally stable rustacean would do: I resurrected it. From the ashes. In modern Rust. With nostd. With Python bindings. With a build-time proc-macro codegen crate. With CI/CD, Debian packaging, RPM packaging, and a Ferris logo.
CRC-32 stands for Cyclic Redundancy Check, 32-bit variant. It's the checksum algorithm that: ZIP files use to verify archive integrity. Ethernet frames use to detect bit errors in transmission. FDDI, PKZIP, PNG and approximately half the binary protocols ever invented also use. Every zlib function you've ever called calls internally when you weren't paying attention.
In other words, CRC-32 is everywhere. It's the duct tape of data integrity. It's been computing checksums since before most of today's developers were handing in their first homework assignments.
The original Rust crc32 crate implemented the byte-at-a-time version, the classic, the grandaddy variant. Simple. Correct. Slow as a 56k modem by 2024 standards.
I didn't just port it. I rewrote the whole thing from the zlib source, added slicing-by-4, slicing-by-8, and slicing-by-16 table variants, pulled the table-generation itself into a build-time proc-macro subcrate, added a streaming Digest, a GF(2) matrix combine function, and shipped the whole thing as a nostd library with Python and Node.js bindings.
That's it. 3 files. A lookup table, a loop, a bitwise XOR. Beautiful in its simplicity, like a stone hut in a field. Functional. Zero frills. Absolutely not capable of 1 GiB/s throughput.
This is my personal favourite part of the whole project. And possibly the most unhinged.
CRC-32 slicing-by-16 requires 17 lookup tables: one 256-entry big-endian table, and 16 256-entry little-endian tables at different offsets. That's 17 × 256 × 4 = 17,408 bytes of lookup table data that needs to exist at runtime.
The naive way: write the tables by hand and commit 17,000 bytes of constants to git as a .rs file.
The way I did it: proc-macro subcrate (crc32-codegen) that is invoked from build.rs at compile time, generates all 17 tables using the CRC-32 polynomial arithmetic, writes them to $OUTDIR/crctables.rs, and the main crate include!s the output. Zero runtime initialization. Zero startup cost. The tables are baked into the binary at link time.
That's the entire build.rs. One function call. The codegen crate does all the polynomial math, formats the Rust source, and writes it to disk. The main crate wakes up with all 17 tables pre-computed, pre-verified, and pre-formatted.
This is the kind of thing that inspires either awe or a mildly concerned Slack message from your coworkers. There is no middle ground.
Let me paint you a picture of what "slicing-by-N" actually means, because it's one of those ideas that sounds arcane until you see the numbers and then you can't stop thinking about it.
One byte per iteration. One table lookup. On a 1 MiB payload that's 1,048,576 iterations. At ~2 ns per iteration (fast CPU, warm cache), that's ~2ms. Not terrible. Not great.
Slicing-by-4 says: what if we process four bytes per iteration instead? You pre-compute four separate 256-entry tables (one for each byte offset), and in each step you XOR together four table lookups instead of one. Four bytes per iteration, four table lookups, but modern CPUs can do all four lookups in parallel because there are no data dependencies between them. Result: roughly 2.4× faster.