Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'oobabooga:main' into main
Browse files Browse the repository at this point in the history
IllogicalDesigns authored Jun 3, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
2 parents 871bf08 + 19f7868 commit bae8574
Showing 66 changed files with 841 additions and 550 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -22,6 +22,7 @@ venv/
*.log

settings.json
settings.yaml
notification.mp3
img_bot*
img_me*
55 changes: 30 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -25,7 +25,6 @@ Its goal is to become the [AUTOMATIC1111/stable-diffusion-webui](https://github.
* [4-bit GPTQ mode](docs/GPTQ-models-(4-bit-mode).md)
* [LoRA (loading and training)](docs/Using-LoRAs.md)
* [llama.cpp](docs/llama.cpp-models.md)
* [RWKV model](docs/RWKV-model.md)
* 8-bit and 4-bit through bitsandbytes
* Layers splitting across GPU(s), CPU, and disk
* CPU mode
@@ -53,8 +52,6 @@ Just download the zip above, extract it, and double-click on "start". The web UI

Recommended if you have some experience with the command line.

On Windows, I additionally recommend carrying out the installation on WSL instead of the base system: [WSL installation guide](https://github.com/oobabooga/text-generation-webui/blob/main/docs/WSL-installation-guide.md).

#### 0. Install Conda

https://docs.conda.io/en/latest/miniconda.html
@@ -81,6 +78,7 @@ conda activate textgen
| Linux/WSL | NVIDIA | `pip3 install torch torchvision torchaudio` |
| Linux | AMD | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2` |
| MacOS + MPS (untested) | Any | `pip3 install torch torchvision torchaudio` |
| Windows | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117` |

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

@@ -97,25 +95,25 @@ cd text-generation-webui
pip install -r requirements.txt
```

#### 4. Install GPTQ-for-LLaMa and the monkey patch
#### 4. Install GPTQ

The base installation covers [transformers](https://github.com/huggingface/transformers) models (`AutoModelForCausalLM` and `AutoModelForSeq2SeqLM` specifically) and [llama.cpp](https://github.com/ggerganov/llama.cpp) (GGML) models.

To use GPTQ models, the additional installation steps below are necessary:

[GPTQ models (4 bit mode)](https://github.com/oobabooga/text-generation-webui/blob/main/docs/GPTQ-models-(4-bit-mode).md)

#### Note about bitsandbytes
#### llama.cpp with GPU acceleration

Requires the additional compilation step described here: [GPU acceleration](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md#gpu-acceleration).

#### bitsandbytes

bitsandbytes >= 0.39 may not work on older NVIDIA GPUs. In that case, to use `--load-in-8bit`, you may have to downgrade like this:

* Linux: `pip install bitsandbytes==0.38.1`
* Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl`

### Alternative: manual Windows installation

As an alternative to the recommended WSL method, you can install the web UI natively on Windows using this guide. It will be a lot harder and the performance may be slower: [Windows installation guide](https://github.com/oobabooga/text-generation-webui/blob/main/docs/Windows-installation-guide.md).

### Alternative: Docker

```
@@ -126,7 +124,7 @@ docker compose up --build
```

* You need to have docker compose v2.17 or higher installed. See [this guide](https://github.com/oobabooga/text-generation-webui/blob/main/docs/Docker.md) for instructions.
* For additional docker files, check out [this repository](https://github.com/Atinoda/text-generation-webui/blob/docker-wrapper/docs/Docker.md#dedicated-docker-repository).
* For additional docker files, check out [this repository](https://github.com/Atinoda/text-generation-webui-docker).

### Updating the requirements

@@ -156,7 +154,9 @@ For example:

python download-model.py facebook/opt-1.3b

If you want to download a model manually, note that all you need are the json, txt, and pytorch\*.bin (or model*.safetensors) files. The remaining files are not necessary.
* If you want to download a model manually, note that all you need are the json, txt, and pytorch\*.bin (or model*.safetensors) files. The remaining files are not necessary.

* Set env vars `HF_USER` and `HF_PASS` to your Hugging Face username and password (or [User Access Token](https://huggingface.co/settings/tokens)) to download a protected model. The model's terms must first be accepted on the HF website.

#### GGML models

@@ -207,7 +207,7 @@ Optionally, you can use the following command-line flags:
| `--lora-dir LORA_DIR` | Path to directory with all the loras. |
| `--model-menu` | Show a model menu in the terminal when the web UI is first launched. |
| `--no-stream` | Don't stream the text output in real time. |
| `--settings SETTINGS_FILE` | Load the default interface settings from this json file. See `settings-template.json` for an example. If you create a file called `settings.json`, this file will be loaded by default without the need to use the `--settings` flag. |
| `--settings SETTINGS_FILE` | Load the default interface settings from this yaml file. See `settings-template.yaml` for an example. If you create a file called `settings.yaml`, this file will be loaded by default without the need to use the `--settings` flag. |
| `--extensions EXTENSIONS [EXTENSIONS ...]` | The list of extensions to load. If you want to load more than one extension, write the names separated by spaces. |
| `--verbose` | Print the prompts to the terminal. |

@@ -217,7 +217,7 @@ Optionally, you can use the following command-line flags:
|---------------------------------------------|-------------|
| `--cpu` | Use the CPU to generate text. Warning: Training on CPU is extremely slow.|
| `--auto-devices` | Automatically split the model across the available GPU(s) and CPU. |
| `--gpu-memory GPU_MEMORY [GPU_MEMORY ...]` | Maxmimum GPU memory in GiB to be allocated per GPU. Example: `--gpu-memory 10` for a single GPU, `--gpu-memory 10 5` for two GPUs. You can also set values in MiB like `--gpu-memory 3500MiB`. |
| `--gpu-memory GPU_MEMORY [GPU_MEMORY ...]` | Maximum GPU memory in GiB to be allocated per GPU. Example: `--gpu-memory 10` for a single GPU, `--gpu-memory 10 5` for two GPUs. You can also set values in MiB like `--gpu-memory 3500MiB`. |
| `--cpu-memory CPU_MEMORY` | Maximum CPU memory in GiB to allocate for offloaded weights. Same as above.|
| `--disk` | If the model is too large for your GPU(s) and CPU combined, send the remaining layers to the disk. |
| `--disk-cache-dir DISK_CACHE_DIR` | Directory to save the disk cache to. Defaults to `cache/`. |
@@ -226,7 +226,7 @@ Optionally, you can use the following command-line flags:
| `--no-cache` | Set `use_cache` to False while generating text. This reduces the VRAM usage a bit with a performance cost. |
| `--xformers` | Use xformer's memory efficient attention. This should increase your tokens/s. |
| `--sdp-attention` | Use torch 2.0's sdp attention. |
| `--trust-remote-code` | Set trust_remote_code=True while loading a model. Necessary for ChatGLM. |
| `--trust-remote-code` | Set trust_remote_code=True while loading a model. Necessary for ChatGLM and Falcon. |

#### Accelerate 4-bit

@@ -266,6 +266,14 @@ Optionally, you can use the following command-line flags:
| `--warmup_autotune` | (triton) Enable warmup autotune. |
| `--fused_mlp` | (triton) Enable fused mlp. |

#### AutoGPTQ

| Flag | Description |
|------------------|-------------|
| `--autogptq` | Use AutoGPTQ for loading quantized models instead of the internal GPTQ loader. |
| `--triton` | Use triton. |
|` --desc_act` | For models that don't have a quantize_config.json, this parameter is used to define whether to set desc_act or not in BaseQuantizeConfig. |

#### FlexGen

| Flag | Description |
@@ -308,6 +316,8 @@ Optionally, you can use the following command-line flags:
|---------------------------------------|-------------|
| `--api` | Enable the API extension. |
| `--public-api` | Create a public URL for the API using Cloudfare. |
| `--api-blocking-port BLOCKING_PORT` | The listening port for the blocking API. |
| `--api-streaming-port STREAMING_PORT` | The listening port for the streaming API. |

#### Multimodal

@@ -319,9 +329,9 @@ Out of memory errors? [Check the low VRAM guide](docs/Low-VRAM-guide.md).

## Presets

Inference settings presets can be created under `presets/` as text files. These files are detected automatically at startup.
Inference settings presets can be created under `presets/` as yaml files. These files are detected automatically at startup.

By default, 10 presets by NovelAI and KoboldAI are included. These were selected out of a sample of 43 presets after applying a K-Means clustering algorithm and selecting the elements closest to the average of each cluster.
By default, 10 presets based on NovelAI and KoboldAI presets are included. These were selected out of a sample of 43 presets after applying a K-Means clustering algorithm and selecting the elements closest to the average of each cluster.

[Visualization](https://user-images.githubusercontent.com/112222186/228956352-1addbdb9-2456-465a-b51d-089f462cd385.png)

@@ -333,18 +343,13 @@ https://github.com/oobabooga/text-generation-webui/tree/main/docs

## Contributing

Pull requests, suggestions, and issue reports are welcome.

You are also welcome to review open pull requests.

Before reporting a bug, make sure that you have:

1. Created a conda environment and installed the dependencies exactly as in the *Installation* section above.
2. [Searched](https://github.com/oobabooga/text-generation-webui/issues) to see if an issue already exists for the issue you encountered.
* Pull requests, suggestions, and issue reports are welcome.
* Make sure to carefully [search](https://github.com/oobabooga/text-generation-webui/issues) existing issues before starting a new one.
* If you have some experience with git, testing an open pull request and leaving a comment on whether it works as expected or not is immensely helpful.
* A simple way to contribute, even if you are not a programmer, is to leave a 👍 on an issue or pull request that you find relevant.

## Credits

- Gradio dropdown menu refresh button, code for reloading the interface: https://github.com/AUTOMATIC1111/stable-diffusion-webui
- Verbose preset: Anonymous 4chan user.
- NovelAI and KoboldAI presets: https://github.com/KoboldAI/KoboldAI-Client/wiki/Settings-Presets
- Code for early stopping in chat mode, code for some of the sliders: https://github.com/PygmalionAI/gradio-ui/
2 changes: 2 additions & 0 deletions api-example-chat-stream.py
Original file line number Diff line number Diff line change
@@ -39,6 +39,8 @@ async def run(user_input, history):
'typical_p': 1,
'epsilon_cutoff': 0, # In units of 1e-4
'eta_cutoff': 0, # In units of 1e-4
'tfs': 1,
'top_a': 0,
'repetition_penalty': 1.18,
'top_k': 40,
'min_length': 0,
2 changes: 2 additions & 0 deletions api-example-chat.py
Original file line number Diff line number Diff line change
@@ -33,6 +33,8 @@ def run(user_input, history):
'typical_p': 1,
'epsilon_cutoff': 0, # In units of 1e-4
'eta_cutoff': 0, # In units of 1e-4
'tfs': 1,
'top_a': 0,
'repetition_penalty': 1.18,
'top_k': 40,
'min_length': 0,
2 changes: 2 additions & 0 deletions api-example-stream.py
Original file line number Diff line number Diff line change
@@ -26,6 +26,8 @@ async def run(context):
'typical_p': 1,
'epsilon_cutoff': 0, # In units of 1e-4
'eta_cutoff': 0, # In units of 1e-4
'tfs': 1,
'top_a': 0,
'repetition_penalty': 1.18,
'top_k': 40,
'min_length': 0,
2 changes: 2 additions & 0 deletions api-example.py
Original file line number Diff line number Diff line change
@@ -18,6 +18,8 @@ def run(prompt):
'typical_p': 1,
'epsilon_cutoff': 0, # In units of 1e-4
'eta_cutoff': 0, # In units of 1e-4
'tfs': 1,
'top_a': 0,
'repetition_penalty': 1.18,
'top_k': 40,
'min_length': 0,
4 changes: 4 additions & 0 deletions characters/instruction-following/Bluemoon.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
user: "LEAD:"
bot: "ASSOCIATE:"
turn_template: "<|user|> <|user-message|>\n<|bot|> <|bot-message|></s>\n"
context: "A transcript of a roleplay between two players, LEAD and ASSOCIATE. LEAD sets up a scenario and the characters, from which ASSOCIATE then assumes a character role and continues the story for that role in response to description given by LEAD. The story and characters are developed by exchange of detailed event descriptions and character dialogs, successively given by both LEAD and ASSOCIATE.\n"
4 changes: 4 additions & 0 deletions css/chat.css
Original file line number Diff line number Diff line change
@@ -41,3 +41,7 @@ div.svelte-362y77>*, div.svelte-362y77>.form>* {
.sm.svelte-1ipelgc {
width: 100%;
}

#main button {
min-width: 0 !important;
}
Loading

0 comments on commit bae8574

Please sign in to comment.