text generation webui_en

Inference with text-generation-webui

A gradio web UI for running Large Language Models like LLaMA, llama.cpp and so on.

Step 1: install text-generation-webui

Text-generation-webui currently supports Windows/Linux/macOS/WSL systems. Refer to webui installation for more details.

Step 2: Preparing model weights

text-generation-webui currently supports huggingface and gguf models.

huggingface format

Copy the full weights of chinese-alpaca-2 to the text-generation-webui models folder, as shown below:

text-generation-webui
├── models
│   ├── chinese-alpaca-2-7b
│   │   ├── config.json
│   │   ├── generation_config.json
│   │   ├── pytorch_model-00001-of-00002.bin
│   │   ├── pytorch_model-00002-of-00002.bin
│   │   ├── pytorch_model.bin.index.json
│   │   ├── special_tokens_map.json
│   │   ├── tokenizer_config.json
│   │   └── tokenizer.model

GGUF format

GGUF models are a single file and should be placed directly into models. Example:

text-generation-webui
└── models
    └── chinese-alpaca-2-7b-q4_k-im.gguf

Step 3: Load the model and start the webui

Run the following command to talk to chinese-alpaca-2.

python server.py

On the Web page that opens, select the model you want to chat with. The webui will select the loading mode according to the model format. Then select Parameters -> Instruction template in order, then in Instruction template, choose Llama-v2 from the drop-down menu. Replace the prompt Answer the questions. in the Context input box with You are a helpful assistant. 你是一个乐于助人的助手。 Finally, go back to the Text generation interface, type your instruction in the chat box, and you can start a conversation with chinese-alpaca-2.

Please refer to webui using docs for more detailed official instructions. If you encounter installation or running problems, please go to the original repo to ask questions, the process has been run under commit-id 837bd88.