Arrow Flight Server bootstrap logic

[rishabhmaurya]

Description

This PR adds server bootstrap logic for Arrow Flight. This is in continuation to #16691

• Flight server bootstrap logic and integration with Auxiliary Transport.
• new plugin for StreamManager implementation behind feature flag with default behaviour as disabled.
• support for SslContext in Flight server and client.
• update in SecureTransportSettingsProvider to build SslContext outside of security plugin.
• ClientManager for creating a pool of flight clients for data nodes only.
• custom event loop group and thread pool for server and client channel.
• Default netty server and allocator configuration defined in ServerConfig.
• 2 new gradle projects - arrow-memory-shaded and flight-core-shaded to share dependencies to avoid polluting server classpath.
• Flight info API to get the publish and bound address for each of the node.
• Integration test for FlightServer client handshake.
• StreamManager wrapper for integration with task API. moved to next PR (Base Flight producer integration with Flight Server #17066)
• integration of StreamManager with server module. moved to next PR (Base Flight producer integration with Flight Server #17066)

TODOs:

• Security plugin changes to expose parameters to build SslContext using SecureTransportSettingsProvider [FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin security#5011
• Handle certificate hot reload [Feature Request] Handle certificate hot reload for Flight server #16965
• Tuning up default values for configurations defined in ServerConfig - [Feature Request] Tune up server config for Flight server #16966

I will handle both of them as part of follow up PRs.

Related Issues

Resolves #16963

#16679

Check List

• Functionality includes testing.
  - [ ] API changes companion pull request created, if applicable.
  - [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 53ad56f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f360c80: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for b38f302: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

@rishabhmaurya I know that this is work in progress but please take a look at [1] where the notion of auxiliary transports (and plugins implementing ones, like gRPC), has been introduced. We may (and very likely - should) consider Apache Arrow Flight as one of such transports, it should be optional plugin and not part of the core server. Thank you

[1] #16534

[rishabhmaurya]

@reta thanks for the pointer, I will update the PR and try to use aux transport.

[github-actions]

❌ Gradle check result for 60f82e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f9623e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 93df0aa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 63bc0f4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d364c1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rishabhmaurya]

@reta do you know how can i address the gradle check failure related to ApiAnnotationProcessorTests ? I know i have messed something up but unable to troubleshoot.

java.lang.AssertionError: 
Expected: an instance of org.opensearch.common.annotation.processor.CompilerSupport$Failure
     but: <org.opensearch.common.annotation.processor.CompilerSupport$Success@1b7367d4> is a org.opensearch.common.annotation.processor.CompilerSupport$Success
	at __randomizedtesting.SeedInfo.seed([188B34A9BBC5FD49:D29516146CC3C7B0]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.common.annotation.processor.ApiAnnotationProcessorTests.testPublicApiMethodReturnNotAnnotatedWildcardGenerics(ApiAnnotationProcessorTests.java:310)

ref: https://build.ci.opensearch.org/job/gradle-check/51929/testReport/junit/org.opensearch.common.annotation.processor/ApiAnnotationProcessorTests/testPublicApiMethodReturnNotAnnotatedWildcardGenerics/

[reta]

@reta do you know how can i address the gradle check failure related to ApiAnnotationProcessorTests ? I know i have messed something up but unable to troubleshoot.

@rishabhmaurya It is caused by https://github.com/opensearch-project/OpenSearch/pull/16962/files#diff-756d8dcb1dfd9f8fa9bf974880af0b001cef12541eb17857ff30d4aa922bc05eR200 (change in server/src/main/resources/org/opensearch/bootstrap/security.policy

[rishabhmaurya]

@reta ah, thanks. How did you find that out as I couldn't find any error message in the logs

[reta]

@reta ah, thanks. How did you find that out as I couldn't find any error message in the logs

@rishabhmaurya needed some debugging, afair some tests were required some more security permissions but those could be ignored (since tests are focused on diagnostics)

[rishabhmaurya]

@reta I have made the necessary changes suggested by you. Also, i'm unsure how to proceed with the ApiAnnotationProcessorTests with lack of troubleshooting details.

[reta]

Thanks @rishabhmaurya

Also, i'm unsure how to proceed with the ApiAnnotationProcessorTests with lack of troubleshooting details.

I will fix these shortly, thank you

[github-actions]

❌ Gradle check result for 1f2b3f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 96fd0dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[rishabhmaurya]

@reta Thanks for fixing the annotation processor test.
It would be nice if you and @andrross can review this PR, thank you.

[github-actions]

❌ Gradle check result for 93157cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

It would be nice if you and @andrross can review this PR, thank you.

Sure, I will try to wrap it up this week, thanks @rishabhmaurya