GPU-parallel inference

[vadimcn]

I have a 16-GPU machine available, and a model that fits on a single GPU. What is the best way to optimize vLLM performance in this situation?

I've tried tensor parallel, however it seems that it requires both the number of attention heads and the vocabulary size to be divisible by the number of GPUs, and in my case the GCD of these is 4 😞

[nicklausbrown]

Hi @vadimcn - I'm no expert, but I came across this reply which seems to possibly help with your question :) seems the answer is to maybe use pipeline parallelism as well.

##5500 (comment)

[vadimcn]

@nicklausbrown Yeah, tried it:
At first vLLM errored out with ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.
So I reduced pipeline_parallel_size to 8, and it appeared to initialize successfully... but eventually threw
NotImplementedError: Pipeline parallelism is only supported through AsyncLLMEngine as performance will be severely degraded otherwise.