Skip to content

Commit

Permalink
Add a local-vllm template
Browse files Browse the repository at this point in the history
This is just like `local` using `meta-reference` for everything except
it uses `vllm` for inference.

Signed-off-by: Russell Bryant <[email protected]>
  • Loading branch information
russellb committed Oct 3, 2024
1 parent 31a0c51 commit b261242
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions llama_stack/distribution/templates/local-vllm-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
name: local-vllm
distribution_spec:
description: Like local, but use vLLM for running LLM inference
providers:
inference: vllm
memory: meta-reference
safety: meta-reference
agents: meta-reference
telemetry: meta-reference
image_type: docker

0 comments on commit b261242

Please sign in to comment.