Skip to content

Latest commit

 

History

History
52 lines (44 loc) · 1.63 KB

README.md

File metadata and controls

52 lines (44 loc) · 1.63 KB

llm-api

Description

The repository invovles the FastAPI backend for interacting with Sagemaker endpoints for LLMs. We can create, stop, delete and call the endpoints. There's a websocket endpoint for listening the creation status of Sagemaker endpoints.

Local development

Prerequisites

  • Python 3.11
  • Docker

Start app

You can start the app in with or without Docker.

Without Docker

  • At the project's root folder, run
pip install -r requirements/dev.txt

export DEPLOY_ENV=local && uvicorn src.main:app --reload

With Docker

  • Build image: run docker build -t <image_name> -f Dockerfile-local . at the root folder.
  • Start the container in the interactive mode:
docker run -it --rm -v ~/.aws:/root/.aws  --name <container_name> -p 80:80 <image_name>

Call from frontend service

You may encounter the CORS error when calling the endpoints from a frontend application locally, for example from localhost:3000. Add this code snippet in the main.py file of this repo.

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

Run tests and coverage

  • Install pytest and coverage packages
  • To run tests only, use pytest at the root
  • To run tests with cover, use
coverage run --source=./src -m pytest

the option source limits the folders to be scanned for coverage. Execute coverage htmland go to the generated htmlcov folder inside the repository and find index.html to see details.