Fix SPI flaky Integ tests

[prudhvigodithi]

Description

Fix flaky SPI Integ tests with retry plugin. After some debugging found that the exact issue is NodeDisconnectedException[[integTest-0][127.0.0.1:41811][indices:admin/delete] disconnected] which is coming from OpenSearch Integ test framework used and thus causing the error:

    java.lang.AssertionError: Failed to release lock.
        at __randomizedtesting.SeedInfo.seed([C9B40C426EAB7723]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.jobscheduler.spi.utils.LockServiceIT.lambda$testRenewLock$71(LockServiceIT.java:471)|

Initially I have tested with re-try logic and not seen any issue with LockServiceIT.testRenewLock after multiple runs.

public void testRenewLock() throws Exception {
       String uniqSuffix = "_lock_renew";
       String lockID = randomAlphaOfLengthBetween(6, 15);
       int maxAttempts = 3;
       int attempt = 0;
       boolean lockAcquired = false;

       while (!lockAcquired && attempt < maxAttempts) {
           attempt++;

           LockService lockService = new LockService(client(), this.clusterService);
           final JobExecutionContext context = new JobExecutionContext(
                   Instant.now(),
                   new JobDocVersion(0, 0, 0),
                   lockService,
                   JOB_INDEX_NAME + uniqSuffix,
                   JOB_ID + uniqSuffix
           );

           CountDownLatch latch = new CountDownLatch(1);
           AtomicBoolean lockReleased = new AtomicBoolean(false);

           lockService.acquireLockWithId(context.getJobIndexName(), LOCK_DURATION_SECONDS, lockID, ActionListener.wrap(lock -> {
               assertNotNull("Expected to successfully grab lock", lock);
               Instant now = Instant.now();
               lockService.setTime(now);
               lockService.renewLock(lock, ActionListener.wrap(renewedLock -> {
                   assertNotNull("Expected to successfully renew lock", renewedLock);
                   assertEquals("lock_time is expected to be the renewal time.", now, renewedLock.getLockTime());
                   assertEquals("lock_duration is expected to be unchanged.", lock.getLockDurationSeconds(), renewedLock.getLockDurationSeconds());
                   lockService.release(lock, ActionListener.wrap(released -> {
                       assertTrue("Failed to release lock.", released);
                       lockReleased.set(true);
                       lockService.deleteLock(lock.getLockId(), ActionListener.wrap(deleted -> {
                           assertTrue("Failed to delete lock.", deleted);
                           latch.countDown();
                       }, exception -> {
                           fail(exception.getMessage());
                           latch.countDown();
                       }));
                   }, exception -> {
                       fail(exception.getMessage());
                       latch.countDown();
                   }));
               }, exception -> {
                   fail(exception.getMessage());
                   latch.countDown();
               }));
           }, exception -> {
               fail(exception.getMessage());
               latch.countDown();
           }));

           latch.await(5L, TimeUnit.SECONDS);

           if (lockReleased.get()) {
               lockAcquired = true;
           } else {
               // Wait before retrying
               Thread.sleep(1000);
           }
       }
       assertTrue("Failed to acquire lock after " + maxAttempts + " retries.", lockAcquired);
   }

But @reta suggested to use org.gradle.test-retry that does the exact some thing for all tests.

Issues Resolved

#56

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[prudhvigodithi]

Adding @cwperks @reta @peternied @joshpalis @gaiksaya @peterzhuamazon @bbarani.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (761c9bb) 29.17% compared to head (0115587) 29.17%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #556   +/-   ##
=========================================
  Coverage     29.17%   29.17%           
  Complexity       98       98           
=========================================
  Files            22       22           
  Lines          1186     1186           
  Branches        109      109           
=========================================
  Hits            346      346           
  Misses          819      819           
  Partials         21       21           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cwperks]

@prudhvigodithi Can the issues upload coverage be resolved by an update to the custom github runner? Looks like the Linux runner can't run Node v20.

Run codecov/codecov-action@v3
/usr/bin/docker exec  a[7](https://github.com/opensearch-project/job-scheduler/actions/runs/7707716909/job/21005389855?pr=556#step:6:8)67427a45009[8](https://github.com/opensearch-project/job-scheduler/actions/runs/7707716909/job/21005389855?pr=556#step:6:9)4029259d3e6553598197a4e845e5a015141b7a032eec537ea7 sh -c "cat /etc/*release | grep ^ID"
/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.[9](https://github.com/opensearch-project/job-scheduler/actions/runs/7707716909/job/21005389855?pr=556#step:6:10)' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

[prudhvigodithi]

Hey @cwperks looks like this is coming from public.ecr.aws/opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v3 which is used for runs-on: ubuntu-latest.

[prudhvigodithi]

Can I get one more approval please?
@joshpalis @saratvemulapalli @dbwiddis @kaituo @vibrantvarun