notebooks

Notebook Examples

In this folder, you will find various examples of how you can prompttools for various experimentation and testing. Often, you can simply change a few parameters and put in your own test data to make prompttools suitable for your use case.

If you have additional use case in mind or spot an issue, please open an issue and we will be happy to discuss.

We also welcome community contribution of usage examples! Please open a PR if you have something to share.

LLM

Single Model Examples

• OpenAI Chat Experiment shows how you can experiment with OpenAI with different parameters.
• LLaMA Cpp Experiment executes LLaMA locally with various parameters and see how it does.
• HuggingFace Hub compares different OSS models hosted on HuggingFace.

Head To Head Model Comparison

• Model Comparison shows how you can compare two OpenAI models.
• GPT4 vs LLaMA2 allows you understand if LLaMA might be enough for your use case.
• LLaMA Head To Head presents a match-up between LLaMA 1 and LLaMA 2!

Evaluation

• Auto Evaluation presents an example of how you can use another LLM to evaluate responses.
• Structured Output validates the model outputs adhere to your desired structured format.
• Semantic Similarity evaluates your model outputs compared to ideal outputs.
• Human Feedback allows you to provide human feedback to your outputs.

Vector Databases

• ChromaDB Experiment demonstrates how to experiment with different embedding functions and query parameters of Chroma. The example evaluates the results by computing the ranking correlation against an expected output.
• Weaviate Experiment shows how you can easily try different vectorizers, configuration, and query functions, and compare the final results.