Multi GPU

[zachNA2]

Does this work on a multi gpu setup? Something like LLaMA2 13B forces me to need 2 gpus. Is their a way to ensure I wont get the whole not all tensors are on same device? Is there any example scripts for multi gpu? Multi gpu as in the model is to big to load on 1 gpu.

[guillaumekln]

This is not possible at this time. See the existing issue #1052.