Unable to generate long content using a.generation()

[amuresia]

Environment information

System:
  OS: macOS 15.2
  CPU: (8) arm64 Apple M1 Pro
  Memory: 147.38 MB / 16.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 22.0.0 - ~/.nvm/versions/node/v22.0.0/bin/node
  Yarn: 1.22.22 - /opt/homebrew/bin/yarn
  npm: 10.5.1 - ~/.nvm/versions/node/v22.0.0/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/auth-construct: 1.5.1
  @aws-amplify/backend: 1.13.0
  @aws-amplify/backend-auth: 1.4.2
  @aws-amplify/backend-cli: 1.4.6
  @aws-amplify/backend-data: 1.4.0
  @aws-amplify/backend-deployer: 1.1.13
  @aws-amplify/backend-function: 1.12.0
  @aws-amplify/backend-output-schemas: 1.4.0
  @aws-amplify/backend-output-storage: 1.1.4
  @aws-amplify/backend-secret: 1.1.5
  @aws-amplify/backend-storage: 1.2.4
  @aws-amplify/cli-core: 1.2.1
  @aws-amplify/client-config: 1.5.5
  @aws-amplify/deployed-backend-client: 1.5.0
  @aws-amplify/form-generator: 1.0.3
  @aws-amplify/model-generator: 1.0.12
  @aws-amplify/platform-core: 1.5.1
  @aws-amplify/plugin-types: 1.7.0
  @aws-amplify/sandbox: 1.2.9
  @aws-amplify/schema-generator: 1.2.6
  aws-amplify: 6.12.0
  aws-cdk: 2.174.1
  aws-cdk-lib: 2.172.0
  typescript: 5.5.4
No AWS environment variables
No CDK environment variables

Describe the bug

I am trying to generate content that is quite long. I never get responses that are 1000 words long using the prompt from below. The longest response I got was around 500 words and the shortest was 42 words long. I notice that from the moment I hit enter, 3 seconds elapse and then I get the response from the LLM. Whatever is generated during those 3 seconds (presumably before the execution of a lambda function times out?) is what I get.
This makes the parameter maxTokens in inferenceConfiguration a bit pointless IMHO, unless I am missing something.

Reproduction steps

Schema

Storyteller: a
      .generation({
        aiModel: a.ai.model('Claude 3 Haiku'),
        inferenceConfiguration: {
          maxTokens: 4000,
        },
        systemPrompt:
          'You are a master storyteller. You write stories that are exactly 1000 words long on the topic provided.'
      })
      .arguments({ topic: a.string() })
      .returns(
        a.customType({
          content: a.string(),
        })
      )
      .authorization((allow) => allow.authenticated()),

[atierian]

Thanks for opening this issue @amuresia.

Can you please provide logs for a request that generates less words than expected? If you don't have logging enabled for your AppSync API, you can do so with:

export const data = defineData({
  // existing ...
  logging: {
    fieldLogLevel: 'all'
  },
});

Generation routes have a 30 second timeout due to the AppSync resolver execution limit. Lambda functions aren't involved here; the requests go directly from AppSync to Bedrock. (source)

All APIs - Request execution time | Each supported Region: 30 Seconds

So something else is going on here. My hunch is that this is model specific and that using a newer model like Claude 3.5 Haiku or Claude 3.5 Sonnet / Sonnet v2 would have better results. But the logs will help us confirm what's going on here.

Thanks!

[amuresia]

Hi @atierian! Thanks for the prompt reply!
I enabled logging but I can't see any errors in the cloudwatch logs.

Logs

{
   "logType": "ResponseMapping",
   "path": [
       "***"
   ],
   "fieldName": "***",
   "resolverArn": "arn:aws:appsync:eu-central-1:***:apis/***/types/Query/resolvers/***",
   "functionName": "***",
   "requestId": "a27ffa2f-6e49-455b-8600-8bf5fac06a37",
   "fieldInError": false,
   "errors": [],
   "parentType": "Query",
   "graphQLAPIId": "***",
   "functionArn": "arn:aws:appsync:eu-central-1:***:apis/***/functions/***"
}

[atierian]

Thanks for that. There should be many more log statements generated for the that request, including the request and response to/from Bedrock.

If you haven't already, you'll need to redeploy once you enable logging via defineData. And once that deployment is complete, you'll need to make a new request.

If you've already taken those steps, you may be filtering the additional log statements out.

[amuresia]

Here are all the logs that get generated @atierian. I hope this helps!

CloudWatch Logs Live tail
Region: eu-central-1
Log group name(s): arn:aws:logs:eu-central-1:730335485068:log-group:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e:*
Log stream name(s):
Log stream prefix:
Filter pattern:

---

Event Time	Message	Log group	Log stream
2025-01-09T07:03:50.351Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd Begin Request	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:50.378Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd Begin Execution - Type Name: Query and Field Name: Storyteller	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:50.379Z	{"logType":"BeforeRequestFunctionEvaluation","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:50.379Z	{"logType":"RequestMapping","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","functionName":"QueryStorytellerauth0Function","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e","functionArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/functions/ncve3e2qyfcmnpmzbdqtrrcdjq"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:50.380Z	{"logType":"ResponseMapping","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","functionName":"QueryStorytellerauth0Function","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e","functionArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/functions/ncve3e2qyfcmnpmzbdqtrrcdjq"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:50.382Z	{"logType":"RequestFunctionEvaluation","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","functionName":"QueryStorytellerDataResolverFn","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e","functionArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/functions/oxh6lqwvdrfjrkv7m3kb4lmehi"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.467Z	{"logType":"ResponseFunctionEvaluation","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","functionName":"QueryStorytellerDataResolverFn","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e","functionArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/functions/oxh6lqwvdrfjrkv7m3kb4lmehi"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"logType":"AfterResponseFunctionEvaluation","path":["Storyteller"],"fieldName":"Storyteller","resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","fieldInError":false,"errors":[],"parentType":"Query","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd End Field Execution	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd Resolver Count: 1	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"duration":2090641621,"logType":"ExecutionSummary","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","startTime":"2025-01-09T07:03:50.377792667Z","endTime":"2025-01-09T07:03:52.468433279Z","parsing":{"startOffset":50272,"duration":38514},"version":1,"validation":{"startOffset":150514,"duration":96648},"graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd Begin Tracing	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"duration":2090349028,"logType":"Tracing","path":["Storyteller"],"fieldName":"Storyteller","startOffset":192368,"resolverArn":"arn:aws:appsync:eu-central-1:730335485068:apis/diw25wvasbddtjnspiowx7rs4e/types/Query/resolvers/Storyteller","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","parentType":"Query","returnType":"StorytellerReturnType","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"duration":7811,"logType":"Tracing","path":["Storyteller","title"],"fieldName":"title","startOffset":2090569433,"requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","parentType":"StorytellerReturnType","returnType":"String","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"duration":3020,"logType":"Tracing","path":["Storyteller","content"],"fieldName":"content","startOffset":2090585178,"requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","parentType":"StorytellerReturnType","returnType":"String","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e"}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd End Tracing	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	{"logType":"RequestSummary","requestId":"a984e83f-434f-4801-bfc6-ff3652a2c8fd","graphQLAPIId":"diw25wvasbddtjnspiowx7rs4e","statusCode":200,"latency":2117046872}	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd Tokens Consumed: 19	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff
2025-01-09T07:03:52.468Z	a984e83f-434f-4801-bfc6-ff3652a2c8fd End Request	730335485068:/aws/appsync/apis/diw25wvasbddtjnspiowx7rs4e	8276f776de378871ffbab3458f41381fac445787b78c65d4e09a6ef8baf6c1ff

---

[amuresia]

After changing to aiModel: a.ai.model('Claude 3.5 Sonnet')
results in

"CustomRuntimeException(message=Execution timeout: null, errorType=Http:ExecutionTimeoutException, data=null, errorInfo=null)"

after running for 30 seconds (AppSync resolver execution limit)
so I can't find a way to generate large text content.

[atierian]

Thanks for the logs. We'll take a look and follow up.