Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix %tokens command crash #803

Merged
merged 2 commits into from
Dec 2, 2023

Conversation

CyanideByte
Copy link
Contributor

Describe the changes you have made:

  • Fixed %tokens command where count_tokens fails to find tokenizer because of openai/ in the model name
  • If tokenizer for a model isn't found, defaults to gpt-4 tokenizer

Reference any relevant issue (Fixes #000)

  • I have performed a self-review of my code:

I have tested the code on the following OS:

  • Windows
  • MacOS
  • Linux

AI Language Model (if applicable)

  • GPT4
  • GPT3
  • Llama 7B
  • Llama 13B
  • Llama 34B
  • Huggingface model (Please specify which one)

@KillianLucas
Copy link
Collaborator

Clean fix! Thanks @CyanideByte. Really like that it gives an estimation if it can't find it. Merging.

@KillianLucas KillianLucas merged commit 19b78b6 into OpenInterpreter:main Dec 2, 2023
@CyanideByte CyanideByte deleted the token_count_fix branch December 2, 2023 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants