vulkan: initial support for IQ1_S and IQ1_M quantizations #11528

remyoudompheng · 2025-01-30T21:26:13Z

This pull request implements basic support for the remaining I-quants (IQ1_S and IQ1_M).
Performance is not great but similar to IQ2 quantizations.

To avoid spamming shared memory, the IQ1S grid has been compressed to 2 bits per value (4kB shmem size).

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        248.08 ± 0.97 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.49 ± 0.22 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        233.05 ± 2.46 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.55 ± 0.17 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        234.95 ± 0.70 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         23.99 ± 0.04 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        182.14 ± 2.80 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.45 ± 0.19 |

Pull request is draft waiting for #11501 and #11502 to be merged

jeffbolznv

Works with coopmat2 enabled! Perf is a bit low, but I'll fix it after it's merged.

jeffbolznv · 2025-01-30T21:52:04Z

ggml/src/ggml-vulkan/vulkan-shaders/copy_to_quant.comp

@@ -217,7 +217,7 @@ void quantize(uint dst_idx, uint src_idx)
 #endif

 void main() {
-#if defined(DATA_A_IQ2_XXS) || defined(DATA_A_IQ2_XS) || defined(DATA_A_IQ2_S) || defined(DATA_A_IQ3_XXS) || defined(DATA_A_IQ3_S) || defined(DATA_A_IQ4_NL)
+#if defined(DATA_A_IQ1_S) || defined(DATA_A_IQ1_M) || defined(DATA_A_IQ2_XXS) || defined(DATA_A_IQ2_XS) || defined(DATA_A_IQ2_S) || defined(DATA_A_IQ3_XXS) || defined(DATA_A_IQ3_S) || defined(DATA_A_IQ4_XS) || defined(DATA_A_IQ4_NL)
    init_iq_shmem(gl_WorkGroupSize);


It would be nice to defined a "NEED_INIT_IQ_SHMEM" macro each place init_iq_shmem is defined, and then all the #ifs can be simple.

At this point this is a good idea, yeah.

remyoudompheng · 2025-02-01T19:06:27Z

I added MMV kernels for the new quants

Some performance figures on Radeon 780M (~70GB/s memory bandwidth). The LLVM/AMDGPU compiler does not like the generic code at all and behaves better with the specialized shader.

Before MMV kernels:

test-backend-ops:
  MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  3408 runs -   340.90 us/run - 117.44 MFLOP/run - 344.50 GFLOPS
  MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  2556 runs -   407.32 us/run - 117.44 MFLOP/run - 288.32 GFLOPS

legraphista/Qwen2.5-Coder-7B-Instruct-IMat-GGUF

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        248.08 ± 0.97 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.49 ± 0.22 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        233.05 ± 2.46 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.55 ± 0.17 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        234.95 ± 0.70 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         23.99 ± 0.04 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        182.14 ± 2.80 |
| qwen2 7B Q4_K - Medium         |   4.36 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         16.45 ± 0.19 |

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1 (LLVM 19.1.7)) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        133.94 ± 1.54 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         12.71 ± 0.05 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        121.18 ± 0.45 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         10.94 ± 0.14 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        115.58 ± 1.74 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         18.11 ± 0.16 |

After:

test-backend-ops:
  MUL_MAT(type_a=iq1_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  5964 runs -   192.37 us/run - 117.44 MFLOP/run - 610.48 GFLOPS
  MUL_MAT(type_a=iq1_m,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  5112 runs -   210.85 us/run - 117.44 MFLOP/run - 556.98 GFLOPS

legraphista/Qwen2.5-Coder-7B-Instruct-IMat-GGUF

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        243.66 ± 2.14 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         22.97 ± 0.12 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        225.84 ± 1.83 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         21.47 ± 0.22 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        204.98 ± 0.45 |
| qwen2 7B Q2_K - Medium         |   2.80 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         22.57 ± 0.33 |

ggml_vulkan: 0 = AMD Radeon 780M (RADV GFX1103_R1 (LLVM 19.1.7)) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        133.59 ± 1.04 |
| qwen2 7B IQ1_S - 1.5625 bpw    |   1.77 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         20.61 ± 0.07 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         pp512 |        121.55 ± 0.34 |
| qwen2 7B IQ1_M - 1.75 bpw      |   1.90 GiB |     7.62 B | Vulkan     |  99 |         tg128 |         17.40 ± 0.06 |

remyoudompheng · 2025-02-01T19:08:57Z

See branch https://github.com/remyoudompheng/llama.cpp/tree/vulkan-iq-mmv for MMV kernels for IQ2 and IQ3 quants

jeffbolznv · 2025-02-02T03:56:24Z

See branch https://github.com/remyoudompheng/llama.cpp/tree/vulkan-iq-mmv for MMV kernels for IQ2 and IQ3 quants

Very cool! I tested this and it's functionally correct and perf is better on RTX 4070.

I dug into the perf a bit and realized that a significant amount of time is spent in init_iq_shmem since the LUT is so large. I think I had suggested this before, but this more unrollable loop code helps:

    [[unroll]] for (uint i = 0; i < iq2s_grid.length(); i += wgsize.x) {
        iq2s_grid[i + gl_LocalInvocationIndex.x] = iq2s_grid_const[i + gl_LocalInvocationIndex.x];
    }

Even then, it's still expensive and that suggests we should be doing more work per workgroup to amortize the cost. The large shared memory allocation may also limit the number of workgroups that can run concurrently, which argues for using larger workgroups. I verified that increasing NUM_ROWS and workgroup size helps:

before
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 11076 runs -    95.35 us/run - 117.44 MFLOP/run -   1.23 TFLOPS
tot iq-mmv branch
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 12780 runs -    80.79 us/run - 117.44 MFLOP/run -   1.45 TFLOPS
unroll iq_shmem
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 14484 runs -    73.06 us/run - 117.44 MFLOP/run -   1.61 TFLOPS
rm_kq=4
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 17040 runs -    61.34 us/run - 117.44 MFLOP/run -   1.91 TFLOPS
rm_kq=4 and wgsize=64
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 17892 runs -    56.95 us/run - 117.44 MFLOP/run -   2.06 TFLOPS
rm_kq=8 and wgsize=64
  MUL_MAT(type_a=iq2_s,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                 19596 runs -    51.98 us/run - 117.44 MFLOP/run -   2.26 TFLOPS

You don't need to do all of this at once. I think the unrollable loops is a simple change and should help everywhere. For figuring out the best values for all the knobs we'll need to get more exhaustive data from different HW and also test with real models.

remyoudompheng · 2025-02-09T10:33:10Z

Rebased and added shmem sizes following #11502

0cc4m · 2025-02-10T06:25:17Z

ggml/src/ggml-vulkan/vulkan-shaders/copy_to_quant.comp

@@ -217,7 +217,7 @@ void quantize(uint dst_idx, uint src_idx)
 #endif

 void main() {
-#if defined(DATA_A_IQ2_XXS) || defined(DATA_A_IQ2_XS) || defined(DATA_A_IQ2_S) || defined(DATA_A_IQ3_XXS) || defined(DATA_A_IQ3_S) || defined(DATA_A_IQ4_NL)
+#if defined(DATA_A_IQ1_S) || defined(DATA_A_IQ1_M) || defined(DATA_A_IQ2_XXS) || defined(DATA_A_IQ2_XS) || defined(DATA_A_IQ2_S) || defined(DATA_A_IQ3_XXS) || defined(DATA_A_IQ3_S) || defined(DATA_A_IQ4_XS) || defined(DATA_A_IQ4_NL)
    init_iq_shmem(gl_WorkGroupSize);


At this point this is a good idea, yeah.

0cc4m · 2025-02-10T06:35:37Z

ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.comp

@@ -1,6 +1,9 @@
 #if !defined(DATA_A_F32) && !defined(DATA_A_F16)
 #extension GL_EXT_shader_explicit_arithmetic_types_int8 : require
 #endif
+#if defined(DATA_A_IQ1_M)
+#extension GL_EXT_shader_explicit_arithmetic_types_float16 : require
+#endif


This extension shouldn't be used due to hardware restrictions, otherwise IQ1_M is not going to work on hardware without 16-bit support.

0cc4m · 2025-02-10T06:43:36Z

ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs.comp

+vec2 get_dm(uint ib, uint a_offset) {
+    const uint16_t[4] scales = data_a[a_offset + ib].scales;
+    const u16vec4 s = u16vec4(scales[0], scales[1], scales[2], scales[3]) >> 12;
+    const float d = float(uint16BitsToHalf(s.x | (s.y << 4) | (s.z << 8) | (s.w << 12)));


Maybe you can work around the 16-bit requirement here with vec2 unpackHalf2x16(uint v)?

github-actions bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jan 30, 2025

jeffbolznv reviewed Jan 30, 2025

View reviewed changes

github-actions bot added the devops improvements to build systems and github actions label Feb 1, 2025

remyoudompheng mentioned this pull request Feb 2, 2025

vulkan: add specific MMV kernels for IQ2 and IQ3 quants + optimizations #11595

Draft

remyoudompheng force-pushed the vulkan-iq1 branch from 07820ef to 6a32664 Compare February 6, 2025 22:23

remyoudompheng added 3 commits February 9, 2025 11:31

vulkan: initial support for IQ1_S and IQ1_M quantizations

fa92caa

vulkan: define MMV kernels for IQ1 quantizations

ab2f56e

devops: increase timeout of Vulkan tests again

dfaa96c

remyoudompheng force-pushed the vulkan-iq1 branch from 6a32664 to dfaa96c Compare February 9, 2025 10:31

remyoudompheng marked this pull request as ready for review February 9, 2025 10:32

0cc4m reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vulkan: initial support for IQ1_S and IQ1_M quantizations #11528

vulkan: initial support for IQ1_S and IQ1_M quantizations #11528

remyoudompheng commented Jan 30, 2025

jeffbolznv left a comment

jeffbolznv Jan 30, 2025

0cc4m Feb 10, 2025

remyoudompheng commented Feb 1, 2025

remyoudompheng commented Feb 1, 2025

jeffbolznv commented Feb 2, 2025

remyoudompheng commented Feb 9, 2025

0cc4m Feb 10, 2025

0cc4m Feb 10, 2025

0cc4m Feb 10, 2025

vulkan: initial support for IQ1_S and IQ1_M quantizations #11528

Are you sure you want to change the base?

vulkan: initial support for IQ1_S and IQ1_M quantizations #11528

Conversation

remyoudompheng commented Jan 30, 2025

jeffbolznv left a comment

Choose a reason for hiding this comment

jeffbolznv Jan 30, 2025

Choose a reason for hiding this comment

0cc4m Feb 10, 2025

Choose a reason for hiding this comment

remyoudompheng commented Feb 1, 2025

remyoudompheng commented Feb 1, 2025

jeffbolznv commented Feb 2, 2025

remyoudompheng commented Feb 9, 2025

0cc4m Feb 10, 2025

Choose a reason for hiding this comment

0cc4m Feb 10, 2025

Choose a reason for hiding this comment

0cc4m Feb 10, 2025

Choose a reason for hiding this comment