Composability with sparse and quantization compressors

[rahul-tuli]

This PR enables accomplishes the following:

• Increases the sparsity threshold to 50%
• Allow sparse + quantized compression-decompression on llm-compressor side
• Adds a test for sparse+quantized compression-decompression

Needs: neuralmagic/compressed-tensors#241

Choices of Compressor's for different cases:

Quantization	Sparsity	Quant Compressor Format	Sparsity Compressor Format
W8A8 - int	None	int_quantized	Dense
W8A8 - float	None	float_quantized	Dense
W4A16 - int	None	pack_quantized	Dense
W8A16 - int	None	pack_quantized	Dense
W8A16 - float	None	naive_quantized	Dense
W8A8 - int	2:4	int_quantized	Sparse24
W8A8 - float	2:4	float_quantized	Sparse24
W4A16 - int	2:4	marlin_24	Dense
W8A16 - int	2:4	marlin_24	Dense
W8A16 - float	2:4	naive_quantized	Dense

Explanation

• Quantization Format: Specifies the quantization compression format used.
• Compressed Flag: Boolean flag indicating whether the model should be compressed on disk.
• Sparsity Structure: Type of sparsity structure inferred. Can be structured or unstructured.
• Sparse Compression Format: Resulting compression format based on the quantization_format, compressed_flag, and sparsity structure.

Notes

1. If the global sparsity (global_sparsity) is less than 0.05, no compression configuration is returned.
2. For CompressionFormat.marlin_24, the compression format defaults to dense regardless of sparsity structure.
3. For other quantization formats:
   • If compressed_flag is True and the sparsity structure is TWO_FOUR, the compression format is sparse_24_bitmask. (This is the only sparse compressor supported in vllm)
   • Otherwise, the compression format defaults to dense.
4. The SparsityThreshold (default 0.5) is used to determine targets and ignores in the model by evaluating parameter sparsity.

Additional Information

• Targets: Parameters selected for sparsification based on the SparsityThreshold.
• Ignores: Parameters excluded from sparsification based on the SparsityThreshold.
• These are inferred using the infer_sparse_targets_and_ignores function, considering the sparsity structure and threshold.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

[horheynm]

verified decompression works for sparse and quantized model

[dsikka]

This looks good. I have a few follow-up questions that I've noted. We can discuss during work session

[rahul-tuli]

Add case in the description where save_compressed is True but unstructured sparsity; which quantized compressor.

[dsikka]

W4A16 - float

W4A16 - float is not a relevant scheme
We're also missing W8A16 - no sparsity and W8A16 + 2:4 Sparsity

Please update the table

[dsikka]

Ran a ton of cases and everything seems well covered and integrated well with vLLM.

Few small comments

[rahul-tuli]

W4A16 - float

W4A16 - float is not a relevant scheme We're also missing W8A16 - no sparsity and W8A16 + 2:4 Sparsity

Please update the table

Done!

[dsikka]

Excellent work!

[dsikka]

@rahul-tuli infer_quantization_config had a signature change which is now failing tests

[tlrmchlsmth]

nit:
dense sparsity compressor is a kind of weird/confusing concept and could use some clarification

[rahul-tuli]

Is this better?

[tlrmchlsmth]

LGTM now once the tests are green

[mgoin]

Thank you!

[rahul-tuli]

Failing tests, are unrelated to this PR and also fail on main