[bug][storage] Make ES-Rollover idempotent by checking if the index or alias already exist

[Manik2708]

Which problem is this PR solving?

Fixes: #6203

Description of the changes

• Currently es-rollover checks for index existence through errors, it is mainly expecting the error:

{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"]"}],"type":"resource_already_exists_exception","reason":"request [/jaeger-*] contains unrecognized parameter: [help]"},"status":400}

But it can lead to inconsistent results as found in the issue, where init was failing due to the error:

Error: failed to create index: jaeger-span-000001, request failed, status code: 400, body: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"}],"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"},"status":400}

Here if we see carefully the error is coming due to existence of index but the reason is different. es-rollover is ready only for resource_already_exists_exception but there are other errors also which can be generated due to this (like the above).

The current way of marshalling error is unsafe, the safe way is: Check if index exists -> Create if not exists. This way the certained error (resource_already_exists is fixed) and the unavoidable error like index_name_exception is ignored.

How was this change tested?

• Unit and E2E Tests

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: npm run lint and npm run test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.03%. Comparing base (4b884bb) to head (a6db745).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6638   +/-   ##
=======================================
  Coverage   96.03%   96.03%           
=======================================
  Files         364      364           
  Lines       20675    20696   +21     
=======================================
+ Hits        19855    19876   +21     
  Misses        626      626           
  Partials      194      194           

Flag	Coverage Δ
badger_v1	9.76% <ø> (ø)
badger_v2	1.82% <ø> (ø)
cassandra-4.x-v1-manual	14.81% <ø> (ø)
cassandra-4.x-v2-auto	1.81% <ø> (ø)
cassandra-4.x-v2-manual	1.81% <ø> (ø)
cassandra-5.x-v1-manual	14.81% <ø> (ø)
cassandra-5.x-v2-auto	1.81% <ø> (ø)
cassandra-5.x-v2-manual	1.81% <ø> (ø)
elasticsearch-6.x-v1	19.15% <ø> (ø)
elasticsearch-7.x-v1	19.23% <ø> (ø)
elasticsearch-8.x-v1	19.40% <ø> (ø)
elasticsearch-8.x-v2	1.82% <ø> (ø)
grpc_v1	10.81% <ø> (ø)
grpc_v2	7.80% <ø> (ø)
kafka-3.x-v1	10.13% <ø> (ø)
kafka-3.x-v2	1.82% <ø> (ø)
memory_v2	1.82% <ø> (ø)
opensearch-1.x-v1	19.28% <ø> (ø)
opensearch-2.x-v1	19.28% <ø> (ø)
opensearch-2.x-v2	1.82% <ø> (ø)
tailsampling-processor	0.48% <ø> (ø)
unittests	94.92% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Manik2708]

@yurishkuro I think, even this approach can lead to the error described in the issue. An index might not exist but still pointed by an alias. Should we ignore that error in init (as that means rollover has taken place already)? I intially thought that index existence could fix this but I may be wrong. So I want to take your views over ignoring this error in init.

[Manik2708]

@mahadzaryab1 Can you please review this PR and solve this doubt #6638 (comment) ?

[Manik2708]

Fixing e2e tests

[yurishkuro]

please remove the binary cmd/esmapping-generator/esmapping-generator-linux-amd64

code coverage is below threshold (the ES error responses are not covered by the test)

[Manik2708]

@yurishkuro Is there something I can do here for failing checks in merge queue?

[yurishkuro]

you could add a fix to the label check workflow in a new PR #6712 (comment)

[Manik2708]

you could add a fix to the label check workflow in a new PR #6712 (comment)

@yurishkuro Will it be ok to disable it? Because I don't have any reason to enable it for the commit. It makes sense in the PR but I am doubtful for the commit! Also, DCO check is failing! But I think for that we need to find a fix rather than removing it!