[BUG] Creating a report on Data Stream indices fails if one of the data stream indices are closed

[sastorsl]

What is the bug?
Creating a report on Data Stream indices fails if one of the data stream indices are closed.
This happens even if the actual data is on the other / open indices.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a data stream
2. Insert data into the data stream
3. Rollover the data stream index
4. Insert data into the data stream
5. Create an Index Template for the data stream
6. Save a Discover search for the data stream
7. Go to "Reporting", "Generate CSV"
8. Get an error message regarding "Error downloading"

Navigating to https://my.hostname/app/reports-dashboards#/ you see the report is created.
When you attempt to download the CSV you get the same error message.

NB! This does NOT apply to "regular" indices.
I went through the same steps with a set of regular indices, and closed one of them, created a report, and this worked fine.

What is the expected behavior?
The report is created and is available for Download.

What is your host/environment?

• OS: Kubernetes / docker
• Version 1.3.2
• Plugins

Do you have any screenshots?
Error message in OpenSearch Dashboards

Download error
There was an error generating this report.

Error message in OpenSearch Dashboard logs:

{"type":"log","@timestamp":"2022-07-01T09:26:45Z","tags":["error","plugins","reportsDashboards"],"pid":1,"message":"Failed to generate report: [index_closed_exception] closed, with { index=\".ds-logs-traffic-000001\" & index_uuid=\"5xhFtvNbT02Ong9cEmFATw\" } :: {\"path\":\"/logs-traffic/_count\",\"query\":{},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":{\\\"range\\\":{\\\"@timestamp\\\":{\\\"format\\\":\\\"epoch_millis\\\",\\\"gte\\\":1656664005068,\\\"lte\\\":1656667605070}}}}}}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"index_closed_exception\\\",\\\"reason\\\":\\\"closed\\\",\\\"index\\\":\\\".ds-logs-traffic-000001\\\",\\\"index_uuid\\\":\\\"5xhFtvNbT02Ong9cEmFATw\\\"}],\\\"type\\\":\\\"index_closed_exception\\\",\\\"reason\\\":\\\"closed\\\",\\\"index\\\":\\\".ds-logs-traffic-000001\\\",\\\"index_uuid\\\":\\\"5xhFtvNbT02Ong9cEmFATw\\\"},\\\"status\\\":400}\"}"}
{"type":"log","@timestamp":"2022-07-01T09:26:45Z","tags":["error","plugins","reportsDashboards"],"pid":1,"message":"{ Error: [index_closed_exception] closed, with { index=\".ds-logs-traffic-000001\" & index_uuid=\"5xhFtvNbT02Ong9cEmFATw\" }\n    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 400,\n  displayName: 'BadRequest',\n  message:\n   '[index_closed_exception] closed, with { index=\".ds-logs-traffic-000001\" & index_uuid=\"5xhFtvNbT02Ong9cEmFATw\" }',\n  path: '/logs-traffic/_count',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'index_closed_exception',\n        reason: 'closed',\n        index: '.ds-logs-traffic-000001',\n        index_uuid: '5xhFtvNbT02Ong9cEmFATw' },\n     status: 400 },\n  statusCode: 400,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"index_closed_exception\",\"reason\":\"closed\",\"index\":\".ds-logs-traffic-000001\",\"index_uuid\":\"5xhFtvNbT02Ong9cEmFATw\"}],\"type\":\"index_closed_exception\",\"reason\":\"closed\",\"index\":\".ds-logs-traffic-000001\",\"index_uuid\":\"5xhFtvNbT02Ong9cEmFATw\"},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }"}
{"type":"response","@timestamp":"2022-07-01T09:26:45Z","tags":[],"pid":1,"method":"post","statusCode":400,"req":{"url":"/api/reporting/generateReport?timezone=Europe%2FOslo&dateFormat=MMM%20D%2C%20YYYY%20%40%20HH%3Amm%3Ass.SSS&csvSeparator=%2C","method":"post","headers":{"content-length":"875","sec-ch-ua":"\"Chromium\";v=\"103\", \".Not/A)Brand\";v=\"99\"","pragma":"no-cache","accept-language":"en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36","content-type":"application/json","accept":"*/*","osd-xsrf":"reporting","sec-ch-ua-platform":"\"Linux\"","origin":"https://my.hostname","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://my.hostname/app/discover","accept-encoding":"gzip, deflate, br","dnt":"1","sec-gpc":"1","host":"my.hostname","x-forwarded-host":"my.hostname","x-forwarded-port":"443","x-forwarded-proto":"https","forwarded":"for=10.0.0.10;host=my.hostname;proto=https","x-forwarded-for":"10.0.0.10","securitytenant":""},"remoteAddress":"10.200.48.1","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36","referer":"https://my.hostname/app/discover"},"res":{"statusCode":400,"responseTime":180,"contentLength":9},"message":"POST /api/reporting/generateReport?timezone=Europe%2FOslo&dateFormat=MMM%20D%2C%20YYYY%20%40%20HH%3Amm%3Ass.SSS&csvSeparator=%2C 400 180ms - 9.0B"}

Do you have any additional context?
OpenSearch (backend) does log, but not an error message, so seems to be in Dashboards.

[2022-07-01T09:33:51,317][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-master-0] reports:ReportInstance-info nI0VuYEBqfHApcdZiFmK

We are closing data stream indices with ISM / Index State Management, and will have to stop closing indices as a workaround to support our customers.

[krisfreedain]

Catch All Triage - 1, 2, 3