Multi node multi gpu distributed load

[rastinrastinii]

Hi
it seems we can load model in on one node then distribute model between node and train or inference but,
imagine we have 2 node each node 2 gpu with 24GB vram each gpu.
wanna loading model like gemma2 27B. one node can not load it. it need to distribute load between node from start till end without needing to load completely in one node.(imagine none of nodes can not offload for loading complete model on one node, like llama 3.1 405B).

is there a way to do this?

[tjruwase]

It sounds like you need zero stage 3 model initialization. The following links could be useful

1. https://deepspeed.readthedocs.io/en/latest/zero3.html#constructing-massive-models
2. HF: https://huggingface.co/docs/transformers/main/deepspeed#deepspeed

[rastinrastinii]

Thanks for your help