Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ikea): Batch Dataset.push calls to not hit API rate limit #2759

Merged
merged 1 commit into from
Jan 30, 2025
Merged

fix(ikea): Batch Dataset.push calls to not hit API rate limit #2759

merged 1 commit into from
Jan 30, 2025

Conversation

Jkuzz
Copy link
Contributor

@Jkuzz Jkuzz commented Jan 30, 2025

Should resolve #2608 by not calling Dataset.pushData individually for each item but the whole results batch. I observed all batches to be <500 items so this shouldn't hit the 5MB size limit either.

Issue is current logs for reference:

2025-01-30T01:07:53.747Z INFO  HttpCrawler: Finished! Total 39 requests: 2 succeeded, 37 failed. {"terminal":true}
2025-01-30T01:07:53.753Z INFO  Keboola: Uploading to table ikea_cz
2025-01-30T01:07:53.825Z INFO  Stats: saved
2025-01-30T01:07:53.829Z INFO  Stats: current {"items":6553,"totalCount":0,"failed":37}
2025-01-30T01:07:56.488Z WARN  ApifyClient: API request failed 4 times. Max attempts: 9.
2025-01-30T01:07:56.491Z Cause:ApifyApiError: You have exceeded the rate limit of 200 requests per second
2025-01-30T01:07:56.496Z   clientMethod: DatasetClient.pushItems
2025-01-30T01:07:56.501Z   statusCode: 429
2025-01-30T01:07:56.503Z   type: rate-limit-exceeded
2025-01-30T01:07:56.509Z   attempt: 4
2025-01-30T01:07:56.511Z   httpMethod: post
2025-01-30T01:07:56.513Z   path: /v2/datasets/XrwgwhSVe0aNwBXd2/items
2025-01-30T01:07:56.516Z   stack:
2025-01-30T01:07:56.520Z     at makeRequest (/usr/src/app/node_modules/apify-client/dist/http_client.js:184:30)
2025-01-30T01:07:56.522Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2025-01-30T01:07:56.525Z     at async DatasetClient.pushItems (/usr/src/app/node_modules/apify-client/dist/resource_clients/dataset.js:104:9)
2025-01-30T01:07:56.528Z     at async category (file:///usr/src/app/search.js:156:9)
2025-01-30T01:07:56.530Z     at async wrap (/usr/src/app/node_modules/@apify/timeout/cjs/index.cjs:54:21)

@Jkuzz Jkuzz mentioned this pull request Jan 30, 2025
Open
@rarous rarous merged commit a382619 into topmonks:trunk Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ikea_cz - lower number of results
2 participants
@Jkuzz @rarous