FP8 / INT8 tensor core support

[Leikoe]

Hi, currently ggml uses cublas_v2 api for cublasGemmStridedBatchedEx and cublasGemmBatchedEx. Would it be possible to switch to cublasLt api and use their support for INT8 and (new) FP8 tensor cores gemm ?

[cmp-nct]

For a start you can take a look here: https://github.com/cmp-nct/ggllm.cpp/blob/ggfalcon_dev/ggml-cuda.cu
That's what I made for falcon inference project I had running, I couldn't keep up with development changes of llama.cpp but my project still has quite a few advantages, one of them is FP8 support.
Includes kernels for conversion for fp8 and a wrapper for (non batched) cublas, all was working without issues.

FP8 was a significant speed boost on my 4090 compared to FP16

[divisuals]

New here; thought of sharing a relevant development:

Intel is working on speeding-up GGML with lower-level optimizations:
https://arxiv.org/pdf/2311.00502.pdf

Code (LLM runtime is here):
https://github.com/intel/intel-extension-for-transformers

[2023/10] LLM runtime, an Intel-optimized GGML compatible runtime, demonstrates up to 15x performance gain in 1st token generation and 1.5x in other token generation over the default llama.cpp.