Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add X-APIKEY support for llama.cpp AI backend #33

Merged
merged 2 commits into from
Dec 18, 2023
Merged

feat: add X-APIKEY support for llama.cpp AI backend #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
80a4e58
feat: add API key support for llama.cpp backend
ktos Dec 18, 2023
2595d14
docs: update docs about AI backend API key
ktos Dec 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Server/Fracture.Server.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<UserSecretsId>16f71bc9-60c6-466f-af95-a2818674e57c</UserSecretsId>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.9" />
Expand Down
3 changes: 0 additions & 3 deletions Server/appsettings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@
"Microsoft.AspNetCore": "Warning"
}
},
"AiEndpoint": {
"EndpointUrl": ""
},
"AiBackendProvider": "Fracture.Shared.External.Providers.Ai.LlamaCpp.LlamaCppBackendProvider, Fracture.Shared.External, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"AiPromptTemplateProvider": "Fracture.Shared.External.Providers.Ai.AlpacaPromptProvider, Fracture.Shared.External, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
"AllowedHosts": "*"
Expand Down
5 changes: 5 additions & 0 deletions Shared/Fracture.Shared.External/Providers/Ai/AiBackendConfig.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,10 @@ public class AiBackendConfig
/// The endpoint's URL.
/// </summary>
public string EndpointUrl { get; set; } = null!;

/// <summary>
/// The API key to be used during the communication with the AI backend.
/// </summary>
public string? ApiKey { get; set; } = null;
}
}
5 changes: 5 additions & 0 deletions Shared/Fracture.Shared.External/Providers/Ai/LlamaCpp/LlamaCppBackendProvider.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ public class LlamaCppBackendProvider : IAiBackendProvider
public LlamaCppBackendProvider(IOptions<AiBackendConfig> opts)
{
_client = new HttpClient { BaseAddress = new(opts.Value.EndpointUrl) };

if (opts.Value.ApiKey is not null)
{
_client.DefaultRequestHeaders.Add("X-ApiKey", opts.Value.ApiKey);
}
}

/// <summary>
Expand Down
20 changes: 18 additions & 2 deletions docs/ai.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,31 @@ aplikacji głównej.

## Konfiguracja backendu AI w aplikacji

W pliku `Server/appsettings.json`, możesz zdefiniować ścieżkę do URL serwera
llama.cpp, na przykład:
Konfiguracja backendu AI jest oparta o plik sekretów (`secrets.json`).
[Przeczytaj
dokumentację.](https://learn.microsoft.com/en-us/aspnet/core/security/app-secrets?view=aspnetcore-8.0&tabs=linux).

Sekrety aplikacji definiują URL serwera llama.cpp, na przykład:

```json
"AiEndpoint": {
"EndpointUrl": "http://127.0.0.1:8080/completion"
}
```

Opcjonalnie można również dostarczyć klucz API wykorzystywany do komunikacji:

```json
{
"AiEndpoint": {
"EndpointUrl": "http://127.0.0.1:8080/completion",
"ApiKey": "this-is-secret"
}
}
```

URL i klucz API serwera uczelnianego są na Discordzie.

Niezbędne jest również wybranie modułów odpowiedzialnych za komunikację z
backendem AI i przygotowaniem promptów, które to należy wybrać jako pełne nazwy
typów, włącznie z ich _assembly_, dla Mistral-7B należy wybrać
Expand Down