[Fleet] Support bulk operation for more than 10k agents

[nchaulet]

Description

We currently use Elasticsearch search API to retrieve agents for bulk operation, we should change that to allow to support bulk operation (upgrade, unenroll) for more than 10k agents.

We can probably use the Elasticsearch scroll API to do this, and create multiple .fleet-actions document with the same action id.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

Good details in this related bug report as well: #133548

[joshdover]

Let's make sure to include reassigning policies as well and that we are using a single action for multiple agents rather than one action per agent

[joshdover]

FYI let’s make sure we are not using Elasticsearch scroll API for this or for #91562. Instead, we should be using the point in time API in conjunction with the search_after parameter. The scroll API does not preserve sorting across pagination requests which can result in duplicate or missing entries as the index data changes.

[juliaElastic]

@joshdover I tried out point in time search, and it seems to work with page and perPage options.

I think using search_after is not the best here, because on UI it is allowed to go to a random page e.g. page 5, however for search_after, you need to pass the last hit of the previous page.

[joshdover]

@juliaElastic Makes sense, PIT + page options should be fine. My main point was to avoid using the scroll API.

[juliaElastic]

@joshdover I noticed that the PIT request doesn't show latest agents enrolled since the opening of the PIT request, this means when keeping the Agent list UI open, newly added agents are not showing up, only when navigating away and coming back.
Do we want to handle this scenario?

EDIT: I managed to test with 10k+ agents, and I still get the same error as before when using PIT with page and pageSize.

search_phase_execution_exception: [illegal_argument_exception] Reason: Result window is too large, from + size must be less than or equal to: [10000] but was [10020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

[joshdover]

Hmm so maybe we shouldn't be using PIT for the pagination in the UI for this issue, only while iterating through a list on bulk actions on the backend when "select everything on all pages" is selected for a bulk action. In that case, we should be able to use search_after to avoid this and shouldn't have the problem that new agents weren't included since we'll iterate through the whole list quite quickly.

Pagination in the UI is separate but needs to be solved in #91562

[ablnk]

Hey @juliaElastic @joshdover I noticed a small UI issue, would you be mind to check it out? #135120

[amolnater-qasource]

Hi @juliaElastic @jen-huang
We have revalidated this feature on latest 8.4 Snapshot and found it working fine.
We are successfully able to perform below actions on more than 10k agents through bulk actions:

• Assign New Policy
• Add/remove tags
• Unenroll Agents
• Force Unenroll Agents
• Upgrade Agents
• Schedule Upgrade

Build details:
VERSION: 8.4.0 Snapshot
BUILD: 54194
COMMIT: f94d5ff

Screenshots:

Please let us know if we are missing anything here.
Thanks