what is the minimum requirements to run in local?

[Amrosx]

  torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 7.69 GiB of which 13.94 MiB is free. Including non-PyTorch memory, this process has 7.64 GiB memory in use. Of the allocated memory 7.34 GiB is allocated by PyTorch, and 121.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. ```

[Amrosx]

I have increased the GPU to 24G memory but still out of memory

CUDA out of memory. Tried to allocate 112.00 MiB. GPU 0 has a total capacity of 23.58 GiB of which 3.38 MiB is free. Process 3060 has 16.51 GiB memory in use. Including non-PyTorch memory, this process has 7.02 GiB memory in use. Of the allocated memory 6.71 GiB is allocated by PyTorch, and 77.09 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[ashwinb]

@Amrosx What kind of GPU do you have? What model are you trying to use?

[Amrosx]

@ashwinb I am using NVIDIA RTX A5000 24G
the models
Llama3.1-8b-instruct
Llama-Guard-3-8B
same as in the README

[ashwinb]

What command are you using to start? I think you need to disable safety (i.e., avoid loading the Llama-Guard-3-8B model or maybe use the much lighter weight Llama-Guard-3-1B we just released) to be able to fit everything onto your card.

[Amrosx]

I have followed the instruction of llama stack build , configuration and run 8b-instruct
I run with safety and without but error still
python app/main.py --disabled-safety but no change
python app/main.py
when I sent Hello message
I got this error CUDA out of memory in server side

[yanxi0830]

You may start with Llama3.2-1B-Instruct model for inference. E.g.

inference: 
  - provider_id: meta-reference
    provider_type: meta-reference
    config:
      model: Llama3.2-1B-Instruct
      quantization: null
      torch_seed: null
      max_seq_len: 4096
      max_batch_size: 1