Fix Google Gemini bug with finish_reason #281
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This continues with function/tool calling even if finish_reason == "stop".
Details:
I am using litellm OpenAI proxy to connect to none OpenAI models with extended_openai_conversation.
It is known that Google Gemini models respond with finish_reason=="stop" even if wanting to execute a tool or function. See BerriAI/litellm#2195 and BerriAI/litellm#6104
It appears more often with non-streaming.
Here is a log showing example response that causes the problem.
I tested and this happens with every Gemini model from 1.5 to latest 2.0 Flash.
2025-01-01 17:38:29.608 INFO (MainThread) [custom_components.extended_openai_conversation] Response {"id": "chatcmpl-ceb05157-2bb6-49d2-a43a-2cf2d0b81db4", "choices": [{"finish_reason": "stop", "index": 0, "message": {"role": "assistant", "tool_calls": [{"id": "call_64f0921a-b3b2-414a-b64f-095140cfd544", "function": {"arguments": "{\"list\": [{\"service\": \"close_cover\", \"domain\": \"cover\", \"service_data\": {\"entity_id\": \"cover.patio_door_shade\"}}]}", "name": "execute_services"}, "type": "function", "index": 0}]}}], "created": 1735774708, "model": "gemini-2.0-flash-exp", "object": "chat.completion", "usage": {"completion_tokens": 23, "prompt_tokens": 7557, "total_tokens": 7580}, "vertex_ai_grounding_metadata": [], "vertex_ai_safety_results": [[{"category": "HARM_CATEGORY_HATE_SPEECH", "probability": "NEGLIGIBLE"}, {"category": "HARM_CATEGORY_DANGEROUS_CONTENT", "probability": "NEGLIGIBLE"}, {"category": "HARM_CATEGORY_HARASSMENT", "probability": "NEGLIGIBLE"}, {"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT", "probability": "NEGLIGIBLE"}]], "vertex_ai_citation_metadata": []}