Environment Variables

The hardware zlib supports environment variables to switch between the hardware- and software implementation. Using the environment variables allows to enable/disable either hardware supported compression as well as decompression. User can analyse their application performance using any combination. It is possible to enable tracing output, helping to understand the applications behaviour and in particular the buffer sizes used for input and output buffers. Using environment variables, to control the hardware specific code, allows to keep the binary zlib interface as is.

• ZLIB_ACCELERATOR: Use CAPI or GENWQE (PCIe) cards. The default is GENWQE. In order to use a CAPI GZIP card with the accelerated zlib, set ZLIB_ACCELERATOR to CAPI.

NOTE: Most of our tools e.g. gzip/gunzip allow to switch between GenWQE and CAPI using the -A parameter. In this case using the environment variable is not needed. Same applies to the ZLIB_CARD setting described below.

• ZLIB_CARD: Card ID to be used. The library supports intelligent selection of cards to exploit multiple cards within a system/partition (currently only GenWQE). In addition this feature supports retrying failing requests on alternate cards. To use this ZLIB_CARD can be set to -1. Default setting is now -1.

• ZLIB_VERBOSE: Turn on low level code debug-info.

• ZLIB_TRACE: Turn on tracing bits:

• 0x1: General

• 0x2: Hardware

• 0x4: Software

• 0x8: Generate summary

• ZLIB_DEFLATE_IMPL: Default setting is to use hardware, so you normally do not need to set it as it was required for older versions.

• 0x00: SW, 0x01: HW The last 4 bits are used to switch between the hard- and software-implementation. The upper bits are used to control some internal optimizations. Since some of those are only useful for certain use-cases, we did not enable them by default.

• 0x20: Keep file-handles/worker threads to the card open/alive, even if no stream is in use anymore

• 0x40: Useful for cases like Genomics, or filesystems where the dictionary is not used after the operation completed, e.g. Z_FULL_FLUSH, etc

• 0x80: Use polling mode, only for CAPI

• ZLIB_INFLATE_IMPL: Default setting is to use hardware, so you normally do not need to set it as it was required for older versions.

• 0x00: SW, 0x01: HW Use software or hardware for inflate

• See above for control bits

• ZLIB_IBUF_TOTAL: Size of compression buffer, e. g. 768KiB. Setting this variable to 0 disables compression buffering support.

• ZLIB_OBUF_TOTAL: Size of decompression buffer e. g. 128KiB. Setting this variable to 0 disables decompression buffering support.

• ZLIB_INFLATE_THRESHOLD: Size of the threshold on the inflate output buffer on the first inflate call. If the given buffer is smaller than the threshold the library will fall back to software decompression to save overhead for trying to decompress with such small buffers. 16KiB is a good default value. You normally do not need to set that.

• ZLIB_LOGFILE: When printing out zlib traces on stderr is not appropriate e.g. if you are using zlib within a daemon, which closes stdin, stdout, and stderr. Note that if you start multiple instances of your application, you better define unique output filename for ZLIB_LOGFILE. Instead of globally setting it, set it only for the application you use it for. This is a general rule of thumb: use the environment variables with the application you tested it for, try not setting it globally to avoid unexpected/untested behaviour.

• LD_PRELOAD: This is a variable used by the program loader. It allows to load the hardware accelerated zlib before the loader tries to load the zlib software version installed on your system. Using this variable is a good way to replace the zlib software library with the hardware accelerated version. See our User's guide for an example how to use this.

• LD_LIBRARY_PATH: This variable is used by the loader to enhance the search path for the dynamically linked libraries. If you set this to a specific path, the loader tries to load any library from this location and only later tries the original load path. Since this trying might increase the load time a little, we think that using LD_PRELOAD instead works better and is less intrusive. We had one case were a specially setup program only worked with LD_PRELOAD and the trick with LD_LIBRARY_PATH failed. See loader documentation about the preferences it takes when processing those settings.

Examples

For samtools try setting ZLIB_INFLATE_IMPL=0x61 and ZLIB_DEFLATE_IMPL=0x61. ZLIB_ACCELERATOR=CAPI for the CAPI version or GENWQE for the PCIe version of the card. LD_PRELOAD should point to the genwqe version of zlib.

Piccard is using JAVA. The JAVA included in Ubuntu links dynamically against zlib, such that the settings from above should work too. Note that it might be that ldd does not show the dependency to libz, but we saw cases that other binaries were loaded, and those linked dynamically against zlib, such that setting LD_PRELOAD worked too. Try setting ZLIB_TRACE=0xf to see if the genwqe zlib is being used.

Consider setting ZLIB_LOGFILE such that you do not dump the accelerated zlib debug output to stderr, but into that file instead. This is also a good choice when using the accelerated zlib to speed up code running as daemon, since in that case mostly stdout, stderr and stdin are being closed by the application anyways.