New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: decrease cost per highlight #128

Open

MartinBernstorff opened this issue Mar 31, 2024 · 0 comments

Owner

MartinBernstorff commented Mar 31, 2024 •

edited

Loading

Instrumentation:

Track token-use and log it

Experiments:

Skip answer-generation
Generate questions for all new highlights in a document at once – avoids:
- Independent reasoning for each highlight
- Duplicate Q/A pairs
Generate reasoning for all highlights in a document at once, and re-use across highlights
Frame the reasoning more tightly, to use fewer tokens. E.g. must be bullet-points, 5 headers, no summary.
Integrate the reasoning and question generation steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment