Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JS] Tool calling doesn't respect input schema #2022

Closed
gspencergoog opened this issue Feb 18, 2025 · 1 comment
Closed

[JS] Tool calling doesn't respect input schema #2022

gspencergoog opened this issue Feb 18, 2025 · 1 comment
Labels
bug Something isn't working js

Comments

@gspencergoog
Copy link

Describe the bug
When calling a tool with a defined input schema, the LLM ignores this schema and makes up its own. If I include the schema in the prompt, then tool calling works as expected. It seems as thought the LLM isn't being shown the schema for the tool.

To Reproduce
Unpack this tar file:

twenty_questions.tar.gz

  1. Run the app (I use npx genkit start -- npx tsx src/index.ts)
  2. Provide the twentyQuestionsFlow with the following input where "includeSchema" is false:
Input JSON
{
  "includeSchema": false,
  "previousQuestions": [
    {
      "question": {
        "questionId": "q1",
        "questionText": "Is it alive?",
        "questionType": "multiple_choice",
        "choices": {
          "minRequired": 1,
          "maxAllowed": 1,
          "choices": [
            {
              "value": "yes",
              "label": "Yes",
              "allowTextEntry": false
            },
            {
              "value": "no",
              "label": "No",
              "allowTextEntry": false
            }
          ]
        }
      },
      "response": {
        "questionId": "q1",
        "responses": [
          "no"
        ]
      }
    }
  ]
}

When you run it, you will see that you get errors similar to this trace:

Error Trace
{
  "traceId": "3c524c9c0617d17a79f11c8741bc6a7c",
  "displayName": "twentyQuestionsFlow",
  "startTime": 1739895486270,
  "endTime": 1739895487589.9666,
  "spans": {
    "4eed90ede4e9a7d7": {
      "spanId": "4eed90ede4e9a7d7",
      "traceId": "3c524c9c0617d17a79f11c8741bc6a7c",
      "parentSpanId": "f51662ca8718e3c7",
      "startTime": 1739895486290,
      "endTime": 1739895487588.4597,
      "attributes": {
        "genkit:type": "util",
        "genkit:name": "generate",
        "genkit:path": "/{twentyQuestionsFlow,t:flow}/{generate,t:util}",
        "genkit:input": "{\"model\":\"googleai/gemini-2.0-flash\",\"messages\":[{\"role\":\"system\",\"content\":[{\"text\":\"\\nYou are going to play the game of \\\"20 Questions\\\" with a user where they have something in mind and you are trying to guess what it is by asking a series of yes/no questions.\\n\\nYou'll receive a list of previous questions asked and their responses, and a list of your previous guesses.\\n\\nIf you have enough information to make a guess, just output a single string with your guess.\\n\\nIf you don't have enough information, formulate a question to ask the user to narrow down the possibilities using the askQuestionTool.\\n\\n\\nYou'll evaluate the result of asking the question with the askQuestionTool, and output a single string as a guess as to what the user is thinking of, or \\\"Still don't know\\\" if you still don't have enough information.\\n\\nPrevious questions:\\nQuestion: Is it alive?\\nResponse:\\n  - no\\n\"}]},{\"role\":\"user\",\"content\":[{\"text\":\"\\nWhat am I thinking of?\"}]}],\"tools\":[\"/tool/askQuestionTool\"],\"config\":{\"responseMimeType\":\"application/json\"},\"output\":{\"format\":\"text\",\"jsonSchema\":null},\"maxTurns\":20}",
        "genkit:state": "error"
      },
      "displayName": "generate",
      "links": [],
      "instrumentationLibrary": {
        "name": "genkit-tracer",
        "version": "v1"
      },
      "spanKind": "INTERNAL",
      "sameProcessAsParentSpan": {
        "value": true
      },
      "status": {
        "code": 2,
        "message": "INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}"
      },
      "timeEvents": {
        "timeEvent": [
          {
            "time": 1739895487588.3735,
            "annotation": {
              "attributes": {
                "exception.type": "400",
                "exception.message": "INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}",
                "exception.stacktrace": "GenkitError: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n    at parseSchema (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/schema.ts:133:21)\n    at AsyncFunction.actionFn.run (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/action.ts:282:13)\n    at actionFn (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/action.ts:266:28)\n    at resolveToolRequest (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:118:26)\n    at <anonymous> (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:174:55)\n    at Array.map (<anonymous>)\n    at resolveToolRequests (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:171:33)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at generate (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/action.ts:336:5)\n    at <anonymous> (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/action.ts:130:22)"
              },
              "description": "exception"
            }
          }
        ]
      }
    },
    "f51662ca8718e3c7": {
      "spanId": "f51662ca8718e3c7",
      "traceId": "3c524c9c0617d17a79f11c8741bc6a7c",
      "startTime": 1739895486270,
      "endTime": 1739895487589.9666,
      "attributes": {
        "genkit:type": "action",
        "genkit:metadata:subtype": "flow",
        "genkit:name": "twentyQuestionsFlow",
        "genkit:isRoot": true,
        "genkit:path": "/{twentyQuestionsFlow,t:flow}",
        "genkit:metadata:context": "{}",
        "genkit:input": "{\"includeSchema\":false,\"previousQuestions\":[{\"question\":{\"questionId\":\"q1\",\"questionText\":\"Is it alive?\",\"questionType\":\"multiple_choice\",\"choices\":{\"minRequired\":1,\"maxAllowed\":1,\"choices\":[{\"value\":\"yes\",\"label\":\"Yes\",\"allowTextEntry\":false},{\"value\":\"no\",\"label\":\"No\",\"allowTextEntry\":false}]}},\"response\":{\"questionId\":\"q1\",\"responses\":[\"no\"]}}]}",
        "genkit:state": "error"
      },
      "displayName": "twentyQuestionsFlow",
      "links": [],
      "instrumentationLibrary": {
        "name": "genkit-tracer",
        "version": "v1"
      },
      "spanKind": "INTERNAL",
      "sameProcessAsParentSpan": {
        "value": true
      },
      "status": {
        "code": 2,
        "message": "INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}"
      },
      "timeEvents": {
        "timeEvent": [
          {
            "time": 1739895487589.763,
            "annotation": {
              "attributes": {
                "exception.type": "400",
                "exception.message": "INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}",
                "exception.stacktrace": "GenkitError: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- questionType: must be equal to one of the allowed values\n\nProvided data:\n\n{\n  \"questionId\": \"1\",\n  \"questionText\": \"Is it man-made?\",\n  \"questionType\": \"boolean\"\n}\n\nRequired JSON schema:\n\n{\n  \"type\": \"object\",\n  \"properties\": {\n    \"questionId\": {\n      \"type\": \"string\",\n      \"description\": \"Unique identifier for this question.  Used to match responses. Must be unique across all questions within a session or interaction.\"\n    },\n    \"questionText\": {\n      \"type\": \"string\",\n      \"description\": \"The text of the question presented to the user.  Should be clear and concise.\"\n    },\n    \"questionType\": {\n      \"type\": \"string\",\n      \"enum\": [\n        \"multiple_choice\",\n        \"short_answer\"\n      ],\n      \"description\": \"The type of question. Determines the expected format of the response. Yes/No questions are defined as multiple_choice questions with only two choices, with a minAllowed and maxAllowed of one.\"\n    },\n    \"choices\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"minRequired\": {\n          \"type\": \"number\",\n          \"description\": \"The minimum number of answers required.  Must be an integer between 1 and the number of choices, inclusive.\"\n        },\n        \"maxAllowed\": {\n          \"type\": \"number\",\n          \"description\": \"The maximum number of answers allowed.  Must be an integer between 1 and the number of choices, inclusive. Must be greater than or equal to minRequired.\"\n        },\n        \"choices\": {\n          \"type\": \"array\",\n          \"items\": {\n            \"type\": \"object\",\n            \"properties\": {\n              \"value\": {\n                \"type\": \"string\",\n                \"description\": \"The value representing the choice (e.g., 'a', '1', 'option_one').  This value is used in the client's response to identify the selected choice(s). Should be concise and easily machine-readable.\"\n              },\n              \"label\": {\n                \"type\": \"string\",\n                \"description\": \"The human-readable label for the choice (e.g., 'Option One', 'Red', 'United States').  This is what's displayed to the user.\"\n              },\n              \"allowTextEntry\": {\n                \"type\": \"boolean\",\n                \"default\": false,\n                \"description\": \"Whether or not there should be a text input associated with this choice. Typically used for an 'other' option in a multiple-choice question. Defaults to false.\"\n              }\n            },\n            \"required\": [\n              \"value\",\n              \"label\"\n            ],\n            \"additionalProperties\": true\n          },\n          \"minItems\": 2,\n          \"description\": \"Available choices for questions with a questionType of multiple_choice.\"\n        }\n      },\n      \"required\": [\n        \"minRequired\",\n        \"maxAllowed\"\n      ],\n      \"additionalProperties\": true,\n      \"description\": \"The choices for multiple_choice questions. Should not be defined for short_answer questions.\"\n    }\n  },\n  \"required\": [\n    \"questionId\",\n    \"questionText\",\n    \"questionType\"\n  ],\n  \"additionalProperties\": true,\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n    at parseSchema (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/schema.ts:133:21)\n    at AsyncFunction.actionFn.run (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/action.ts:282:13)\n    at actionFn (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/core/src/action.ts:266:28)\n    at resolveToolRequest (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:118:26)\n    at <anonymous> (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:174:55)\n    at Array.map (<anonymous>)\n    at resolveToolRequests (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/resolve-tool-requests.ts:171:33)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at generate (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/action.ts:336:5)\n    at <anonymous> (/usr/local/google/home/gspencer/code/hil_test/node_modules/@genkit-ai/ai/src/generate/action.ts:130:22)"
              },
              "description": "exception"
            }
          }
        ]
      }
    }
  }
}
  1. If you run it again with includeSchema set to true, then the tool is called correctly.

Expected behavior
I would expect the schema definition for the tool input parameters to be given to the LLM automatically.

Runtime (please complete the following information):

  • OS: Linux
  • Version: 1.0.4

** Node version

  • v18.20.1
@pavelgj
Copy link
Collaborator

pavelgj commented Feb 19, 2025

This should be fixed in #2027
I plan to do a release tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working js
Projects
Status: Done
Development

No branches or pull requests

2 participants