Add idempotency support for bulk request API #3625
Labels
Clients
Clients within the Core repository such as High level Rest client and low level client
distributed framework
enhancement
Enhancement or improvement to existing feature or request
Is your feature request related to a problem? Please describe.
Coming from #3000, where we say current bulk indexing API places a high configuration burden on users today to avoid RejectedExecutionException due to TOO_MANY_REQUESTS. This forces the user to "experiment" with bulk block sizes, multi-threading, refresh intervals, etc.
Describe the solution you'd like
Make it easier for clients to handle retries by making bulk requests idempotent by generating a request ID on the client, or assuming id=SHA(data), tracking pending bulk write tasks on the server and their progress, making it possible to safely retry the entire bulk operation. Clients receiving a 429 would have a built-in retry, and it would also be safe to retry from multiple instances of the client with the hope that one succeeds eventually when the server finally has capacity.
Some googling shows people are at least attempting implementing client-side retry, https://gist.github.com/henrikno/e0ebd6804cb62491343c or https://gitlab.com/gitlab-org/gitlab/-/issues/12372
Describe alternatives you've considered
Streaming API, #3000.
The text was updated successfully, but these errors were encountered: