Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ZSTD compression method for reading #256

Merged
merged 2 commits into from
Dec 20, 2024

Conversation

stschake
Copy link
Contributor

ZSTD is generally recognized as superior to LZW in size and speed, particularly for decompression.
We use the self-assigned tag first defined by libtiff and also used by GDAL and others.

ZSTD is generally recognized as superior to LZW in size and
speed, particularly for decompression.
We use the self-assigned tag first defined by libtiff and
also used by GDAL and others.
@stschake stschake force-pushed the feature/zstd-support branch from 5444ff9 to 663c19c Compare December 19, 2024 18:55
@stschake stschake changed the title Support the ZSTD compression method for reading Support ZSTD compression method for reading Dec 19, 2024
@fintelia
Copy link
Contributor

fintelia commented Dec 20, 2024

Thanks for this PR, zstd support is something I've wanted for a while!

My main concern is actually around build time. On my machine, the zstd crate takes around 10 seconds to compile and this change increases the main image crate's compile time by nearly 25%.

I see two possible options:

  • Check whether the ruzstd would work as an alternative.
  • Put zstd support behind a feature flag.

@stschake
Copy link
Contributor Author

Thanks for having a look! I looked into ruzstd but it seems to have a 1.5x-2x performance penalty over the reference implementation, which I felt would diminish the benefit of zstd too much. I added a feature flag now for the zstd support.

@fintelia fintelia merged commit a19cf37 into image-rs:master Dec 20, 2024
@fintelia
Copy link
Contributor

Thanks! If I have a chance, I might look into using ruzstd decoder by default and having the feature opt into the faster zstd bindings. Even the slower decoder may still significantly outperform LZW

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants