[Feature Request] Better diagnostics on contents of query cache

[jhinch-at-atlassian-com]

Is your feature request related to a problem? Please describe

Currently you can get the hit, miss and eviction counts for the query cache via the nodes stats API but it can be difficult to diagnose problems with cache efficacy by identifying what gets uses regularly and what is getting evicted.

Describe the solution you'd like

There are two areas I think would help:

• The ability to enable logging of queries being evicted from the query cache. This likely might need to have some sort of additional threshold which could also be consider to only log when queries over a certain size are evicted
• The ability to retrieve the the top K queries in the cache and their size.

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[sandeshkr419]

@deshsidd @ansjcy Do you have any comments from Query Insights view point?

[msfroh]

The ability to enable logging of queries being evicted from the query cache. This likely might need to have some sort of additional threshold which could also be consider to only log when queries over a certain size are evicted

That would be pretty nice.

@jhinch-at-atlassian-com, what do you think of logging eviction of queries based on some minimum hit count, too? So if a query that's been served from the cache 1 million times gets evicted, we log it?

I think the Query Insights project might be a good sink if we wanted to make it a listener-based solution.