compress
This package provides various compression algorithms.
- zstandard compression and decompression in pure Go.
- S2 is a high performance replacement for Snappy.
- Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
- snappy is a drop-in replacement for
github.com/golang/snappy offering better compression and concurrent streams. - huff0 and FSE implementations for raw entropy encoding.
- gzhttp Provides client and server wrappers for handling gzipped requests efficiently.
- pgzip is a separate package that provides a very fast parallel gzip implementation.
package usage
Use go get github.com/klauspost/compress@latest to add it to your project.
This package will support the current Go version and 2 versions back.
- Use the
nounsafe tag to disable all use of the “unsafe” package. - Use the
noasm tag to disable all assembly across packages.
Use the links above for more information on each.
changelog
Both compression and decompression now supports “synchronous” stream operations. This means that whenever “concurrency” is set to 1, they will operate without spawning goroutines.
Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.
While the release has been extensively tested, it is recommended to testing when upgrading.
Feb 22, 2022 (v1.14.4)
- flate: Fix rare huffman only (-2) corruption. #503
- zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in #502
- zip: don't read data descriptor early by @saracen in #501 #501
- huff0: Use static decompression buffer up to 30% faster #499 #500
Feb 17, 2022 (v1.14.3)
- flate: Improve fastest levels compression speed ~10% more throughput. #482 #489 #490 #491 #494 #478
- flate: Faster decompression speed, ~5-10%. #483
- s2: Faster compression with Go v1.18 and amd64 microarch level 3+. #484 #486
Jan 25, 2022 (v1.14.2)
- zstd: improve header decoder by @dsnet #476
- zstd: Add bigger default blocks #469
- zstd: Remove unused decompression buffer #470
- zstd: Fix logically dead code by @ningmingxiao #472
- flate: Improve level 7-9 #471 #473
- zstd: Add noasm tag for xxhash #475
Jan 11, 2022 (v1.14.1)
Aug 30, 2021 (v1.13.5)
- gz/zlib/flate: Alias stdlib errors #425
- s2: Add block support to commandline tools #413
- zstd: pooledZipWriter should return Writers to the same pool #426
- Removed golang/snappy as external dependency for tests #421
Aug 12, 2021 (v1.13.4)
Aug 3, 2021 (v1.13.3)
- zstd: Improve Best compression #404
- zstd: Fix WriteTo error forwarding #411
- gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. #406
- s2sx: Fix max size error #399
- zstd: Add optional stream content size on reset #401
- zstd: use SpeedBestCompression for level >= 10 #410
Jun 14, 2021 (v1.13.1)
- s2: Add full Snappy output support #396
- zstd: Add configurable Decoder window size #394
- gzhttp: Add header to skip compression #389
- s2: Improve speed with bigger output margin #395
Jun 3, 2021 (v1.13.0)
- Added gzhttp which allows wrapping HTTP servers and clients with GZIP compressors.
- zstd: Detect short invalid signatures #382
- zstd: Spawn decoder goroutine only if needed. #380
May 25, 2021 (v1.12.3)
- deflate: Better/faster Huffman encoding #374
- deflate: Allocate less for history. #375
- zstd: Forward read errors #373
Apr 27, 2021 (v1.12.2)
- zstd: Improve better/best compression #360 #364 #365
- zstd: Add helpers to compress/decompress zstd inside zip files #363
- deflate: Improve level 5+6 compression #367
- s2: Improve better/best compression #358 #359
- s2: Load after checking src limit on amd64. #362
- s2sx: Limit max executable size #368
Apr 14, 2021 (v1.12.1)
- snappy package removed. Upstream added as dependency.
- s2: Better compression in “best” mode #353
- s2sx: Add stdin input and detect pre-compressed from signature #352
- s2c/s2d: Add http as possible input #348
- s2c/s2d/s2sx: Always truncate when writing files #352
- zstd: Reduce memory usage further when using WithLowerEncoderMem #346
- s2: Fix potential problem with amd64 assembly and profilers #349
Mar 26, 2021 (v1.11.13)
- zstd: Big speedup on small dictionary encodes #344 #345
- zstd: Add WithLowerEncoderMem encoder option #336
- deflate: Improve entropy compression #338
- s2: Clean up and minor performance improvement in best #341
Mar 5, 2021 (v1.11.12)
Mar 1, 2021 (v1.11.9)
- s2: Add ARM64 decompression assembly. Around 2x output speed. #324
- s2: Improve “better” speed and efficiency. #325
- s2: Fix binaries.
Feb 25, 2021 (v1.11.8)
- s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended.
- s2: Add AMD64 assembly for better mode. 25-50% faster. #315
- s2: Less upfront decoder allocation. #322
- zstd: Faster “compression” of incompressible data. #314
- zip: Fix zip64 headers. #313
Jan 14, 2021 (v1.11.7)
- Use Bytes() interface to get bytes across packages. #309
- s2: Add ‘best’ compression option. #310
- s2: Add ReaderMaxBlockSize, changes
s2.NewReader signature to include varargs. #311 - s2: Fix crash on small better buffers. #308
- s2: Clean up decoder. #312
Jan 7, 2021 (v1.11.6)
- zstd: Make decoder allocations smaller #306
- zstd: Free Decoder resources when Reset is called with a nil io.Reader #305
Dec 20, 2020 (v1.11.4)
- zstd: Add Best compression mode #304
- Add header decoder #299
- s2: Add uncompressed stream option #297
- Simplify/speed up small blocks with known max size. #300
- zstd: Always reset literal dict encoder #303
Nov 15, 2020 (v1.11.3)
- inflate: 10-15% faster decompression #293
- zstd: Tweak DecodeAll default allocation #295
Oct 11, 2020 (v1.11.2)
- s2: Fix out of bounds read in “better” block compression #291
Oct 1, 2020 (v1.11.1)
- zstd: Set allLitEntropy true in default configuration #286
Sept 8, 2020 (v1.11.0)
- zstd: Add experimental compression dictionaries #281
- zstd: Fix mixed Write and ReadFrom calls #282
- inflate/gz: Limit variable shifts, ~5% faster decompression #274
- Jan 20,2020 (v1.9.8) Optimize gzip/deflate with better size estimates and faster table generation. #207 by luyu6056, #206.
- Jan 11, 2020: S2 Encode/Decode will use provided buffer if capacity is big enough. #204
- Jan 5, 2020: (v1.9.7) Fix another zstd regression in v1.9.5 - v1.9.6 removed.
- Jan 4, 2020: (v1.9.6) Regression in v1.9.5 fixed causing corrupt zstd encodes in rare cases.
- Jan 4, 2020: Faster IO in s2c + s2d commandline tools compression/decompression. #192
- Dec 29, 2019: Removed v1.9.5 since fuzz tests showed a compatibility problem with the reference zstandard decoder.
- Dec 29, 2019: (v1.9.5) zstd: 10-20% faster block compression. #199
- Dec 29, 2019: zip package updated with latest Go features
- Dec 29, 2019: zstd: Single segment flag condintions tweaked. #197
- Dec 18, 2019: s2: Faster compression when ReadFrom is used. #198
- Dec 10, 2019: s2: Fix repeat length output when just above at 16MB limit.
- Dec 10, 2019: zstd: Add function to get decoder as io.ReadCloser. #191
- Dec 3, 2019: (v1.9.4) S2: limit max repeat length. #188
- Dec 3, 2019: Add WithNoEntropyCompression to zstd #187
- Dec 3, 2019: Reduce memory use for tests. Check for leaked goroutines.
- Nov 28, 2019 (v1.9.3) Less allocations in stateless deflate.
- Nov 28, 2019: 5-20% Faster huff0 decode. Impacts zstd as well. #184
- Nov 12, 2019 (v1.9.2) Added Stateless Compression for gzip/deflate.
- Nov 12, 2019: Fixed zstd decompression of large single blocks. #180
- Nov 11, 2019: Set default s2c block size to 4MB.
- Nov 11, 2019: Reduce inflate memory use by 1KB.
- Nov 10, 2019: Less allocations in deflate bit writer.
- Nov 10, 2019: Fix inconsistent error returned by zstd decoder.
- Oct 28, 2019 (v1.9.1) ztsd: Fix crash when compressing blocks. #174
- Oct 24, 2019 (v1.9.0) zstd: Fix rare data corruption #173
- Oct 24, 2019 zstd: Fix huff0 out of buffer write #171 and always return errors #172
- Oct 10, 2019: Big deflate rewrite, 30-40% faster with better compression #105
- Oct 10, 2019: (v1.8.6) zstd: Allow partial reads to get flushed data. #169
- Oct 3, 2019: Fix inconsistent results on broken zstd streams.
- Sep 25, 2019: Added
-rm (remove source files) and -q (no output except errors) to s2c and s2d commands - Sep 16, 2019: (v1.8.4) Add
s2c and s2d commandline tools. - Sep 10, 2019: (v1.8.3) Fix s2 decoder Skip.
- Sep 7, 2019: zstd: Added WithWindowSize, contributed by ianwilkes.
- Sep 5, 2019: (v1.8.2) Add WithZeroFrames which adds full zero payload block encoding option.
- Sep 5, 2019: Lazy initialization of zstandard predefined en/decoder tables.
- Aug 26, 2019: (v1.8.1) S2: 1-2% compression increase in “better” compression mode.
- Aug 26, 2019: zstd: Check maximum size of Huffman 1X compressed literals while decoding.
- Aug 24, 2019: (v1.8.0) Added S2 compression, a high performance replacement for Snappy.
- Aug 21, 2019: (v1.7.6) Fixed minor issues found by fuzzer. One could lead to zstd not decompressing.
- Aug 18, 2019: Add fuzzit continuous fuzzing.
- Aug 14, 2019: zstd: Skip incompressible data 2x faster. #147
- Aug 4, 2019 (v1.7.5): Better literal compression. #146
- Aug 4, 2019: Faster zstd compression. #143 #144
- Aug 4, 2019: Faster zstd decompression. #145 #143 #142
- July 15, 2019 (v1.7.4): Fix double EOF block in rare cases on zstd encoder.
- July 15, 2019 (v1.7.3): Minor speedup/compression increase in default zstd encoder.
- July 14, 2019: zstd decoder: Fix decompression error on multiple uses with mixed content.
- July 7, 2019 (v1.7.2): Snappy update, zstd decoder potential race fix.
- June 17, 2019: zstd decompression bugfix.
- June 17, 2019: fix 32 bit builds.
- June 17, 2019: Easier use in modules (less dependencies).
- June 9, 2019: New stronger “default” zstd compression mode. Matches zstd default compression ratio.
- June 5, 2019: 20-40% throughput in zstandard compression and better compression.
- June 5, 2019: deflate/gzip compression: Reduce memory usage of lower compression levels.
- June 2, 2019: Added zstandard compression!
- May 25, 2019: deflate/gzip: 10% faster bit writer, mostly visible in lower levels.
- Apr 22, 2019: zstd decompression added.
- Aug 1, 2018: Added huff0 README.
- Jul 8, 2018: Added Performance Update 2018 below.
- Jun 23, 2018: Merged Go 1.11 inflate optimizations. Go 1.9 is now required. Backwards compatible version tagged with v1.3.0.
- Apr 2, 2018: Added huff0 en/decoder. Experimental for now, API may change.
- Mar 4, 2018: Added FSE Entropy en/decoder. Experimental for now, API may change.
- Nov 3, 2017: Add compression Estimate function.
- May 28, 2017: Reduce allocations when resetting decoder.
- Apr 02, 2017: Change back to official crc32, since changes were merged in Go 1.7.
- Jan 14, 2017: Reduce stack pressure due to array copies. See Issue #18625.
- Oct 25, 2016: Level 2-4 have been rewritten and now offers significantly better performance than before.
- Oct 20, 2016: Port zlib changes from Go 1.7 to fix zlib writer issue. Please update.
- Oct 16, 2016: Go 1.7 changes merged. Apples to apples this package is a few percent faster, but has a significantly better balance between speed and compression per level.
- Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
- Mar 24, 2016: Small speedup for level 1-3.
- Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
- Feb 19, 2016: Handle small payloads faster in level 1-3.
- Feb 19, 2016: Added faster level 2 + 3 compression modes.
- Feb 19, 2016: Rebalanced compression levels, so there is a more even progression in terms of compression. New default level is 5.
- Feb 14, 2016: Snappy: Merge upstream changes.
- Feb 14, 2016: Snappy: Fix aggressive skipping.
- Feb 14, 2016: Snappy: Update benchmark.
- Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
- Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
- Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
- Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
- Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
- Jan 8 2016: Merge CL 18317: fix reading, writing of zip64 archives.
- Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
- Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
- Dec 8 2015: Fixed rare one byte out-of bounds read. Please update!
- Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by @dsnet.
- Nov 20 2015: Small optimization to bit writer on 64 bit systems.
- Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See #15.
- Nov 12 2015: Added io.WriterTo support to gzip/inflate.
- Nov 11 2015: Merged CL 16669: archive/zip: enable overriding (de)compressors per file
- Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.
deflate usage
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
Typical speed is about 2x of the standard library packages.
| old import | new import | Documentation |
|---|
compress/gzip | github.com/klauspost/compress/gzip | gzip |
compress/zlib | github.com/klauspost/compress/zlib | zlib |
archive/zip | github.com/klauspost/compress/zip | zip |
compress/flate | github.com/klauspost/compress/flate | flate |
- Optimized deflate packages which can be used as a dropin replacement for gzip, zip and zlib.
You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.
Currently there is only minor speedup on decompression (mostly CRC32 calculation).
Memory usage is typically 1MB for a Writer. stdlib is in the same range. If you expect to have a lot of concurrently allocated Writers consider using the stateless compress described below.
For compression performance, see: this spreadsheet.
To disable all assembly add -tags=noasm. This works across all packages.
Stateless compression
This package offers stateless compression as a special option for gzip/deflate. It will do compression but without maintaining any state between Write calls.
This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.
This is only relevant in cases where you expect to run many thousands of compressors concurrently, but with very little activity. This is not intended for regular web servers serving individual requests.
Because of this, the size of actual Write calls will affect output size.
In gzip, specify level -3 / gzip.StatelessCompression to enable.
For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation
A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:
// replace 'ioutil.Discard' with your output.
gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
if err != nil {
return err
}
defer gzw.Close()
w := bufio.NewWriterSize(gzw, 4096)
defer w.Flush()
// Write to 'w'
This will only use up to 4KB in memory when the writer is idle.
Compression is almost always worse than the fastest compression level and each write will allocate (a little) memory.
Other packages
Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):
license
This code is licensed under the same conditions as the original Go code. See LICENSE file.
