Skip to content

Commit

Permalink
Fix Issue 16890: Hide Stracktrace while translog transfer
Browse files Browse the repository at this point in the history
Signed-off-by: meetvm <[email protected]>

Bump com.nimbusds:oauth2-oidc-sdk from 11.19.1 to 11.20.1 in /plugins/repository-azure (opensearch-project#16895)

* Bump com.nimbusds:oauth2-oidc-sdk in /plugins/repository-azure

Bumps [com.nimbusds:oauth2-oidc-sdk](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions) from 11.19.1 to 11.20.1.
- [Changelog](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/branches/compare/11.20.1..11.19.1)

---
updated-dependencies:
- dependency-name: com.nimbusds:oauth2-oidc-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Bump com.netflix.nebula.ospackage-base from 11.10.0 to 11.10.1 in /distribution/packages (opensearch-project#16896)

* Bump com.netflix.nebula.ospackage-base in /distribution/packages

Bumps com.netflix.nebula.ospackage-base from 11.10.0 to 11.10.1.

---
updated-dependencies:
- dependency-name: com.netflix.nebula.ospackage-base
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Bump ch.qos.logback:logback-classic from 1.5.12 to 1.5.15 in /test/fixtures/hdfs-fixture (opensearch-project#16898)

* Bump ch.qos.logback:logback-classic in /test/fixtures/hdfs-fixture

Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.5.12 to 1.5.15.
- [Commits](qos-ch/logback@v_1.5.12...v_1.5.15)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Bump lycheeverse/lychee-action from 2.1.0 to 2.2.0 (opensearch-project#16897)

* Bump lycheeverse/lychee-action from 2.1.0 to 2.2.0

Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/lycheeverse/lychee-action/releases)
- [Commits](lycheeverse/lychee-action@v2.1.0...v2.2.0)

---
updated-dependencies:
- dependency-name: lycheeverse/lychee-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Create sub directories for ThirdPartyAudit dependency metadata (opensearch-project#16844)

* Extract jars to sub dirs during thirdPartyAudit task.

Signed-off-by: Finn Carroll <[email protected]>

* Change regex to split on '-'/'.'. Ignore version.

Signed-off-by: Finn Carroll <[email protected]>

* Split on .jar for sub folder prefix.

Signed-off-by: Finn Carroll <[email protected]>

---------

Signed-off-by: Finn Carroll <[email protected]>

Retrieve value from DocValues in a flat_object filed (opensearch-project#16802)

Bump com.microsoft.azure:msal4j from 1.17.2 to 1.18.0 in /plugins/repository-azure (opensearch-project#16918)

* Bump com.microsoft.azure:msal4j in /plugins/repository-azure

Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.17.2 to 1.18.0.
- [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases)
- [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt)
- [Commits](AzureAD/microsoft-authentication-library-for-java@v1.17.2...v1.18.0)

---
updated-dependencies:
- dependency-name: com.microsoft.azure:msal4j
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Bump org.apache.commons:commons-text from 1.12.0 to 1.13.0 in /test/fixtures/hdfs-fixture (opensearch-project#16919)

* Bump org.apache.commons:commons-text in /test/fixtures/hdfs-fixture

Bumps org.apache.commons:commons-text from 1.12.0 to 1.13.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Add gRPC server as transport-grpc plugin (opensearch-project#16534)

Introduce auxiliary transport to NetworkPlugin and add gRPC plugin.

Auxiliary transports are optional lifecycle components provided by
network plugins which run in parallel to the http server/native
transport. They are distinct from the existing NetworkPlugin
interfaces of 'getTransports' and 'getHttpTransports' as auxiliary
transports are optional. Each AuxTransport implements it's own
'aux.transport.type' and 'aux.transport.<type>.ports' setting. Since
Security.java initializes previous to Node.java during bootstrap
socket binding permissions are granted based on
'aux.transport.<type>.ports' for each enabled 'aux.transport.type',
falling back to a default if no ports are specified.

Signed-off-by: Finn Carroll <[email protected]>

Update script supports java.lang.String.sha1() and java.lang.String.sha256() methods (opensearch-project#16923)

* Update script supports java.lang.String.sha1() and java.lang.String.sha256() methods

Signed-off-by: Gao Binlong <[email protected]>

* Modify change log

Signed-off-by: Gao Binlong <[email protected]>

---------

Signed-off-by: Gao Binlong <[email protected]>

Workflow benchmark-pull-request.yml fix (opensearch-project#16925)

Signed-off-by: Prudhvi Godithi <[email protected]>

Add benchmark confirm for lucene-10 big5 index snapshot (opensearch-project#16940)

Signed-off-by: Rishabh Singh <[email protected]>

Remove duplicate DCO check (opensearch-project#16942)

Signed-off-by: Andriy Redko <[email protected]>

Allow extended plugins to be optional (opensearch-project#16909)

* Make extended plugins optional

Signed-off-by: Craig Perkins <[email protected]>

* Make extended plugins optional

Signed-off-by: Craig Perkins <[email protected]>

* Load extensions for classpath plugins

Signed-off-by: Craig Perkins <[email protected]>

* Ensure only single instance for each classpath extension

Signed-off-by: Craig Perkins <[email protected]>

* Add test for classpath plugin extended plugin loading

Signed-off-by: Craig Perkins <[email protected]>

* Modify test to allow optional extended plugin

Signed-off-by: Craig Perkins <[email protected]>

* Only optional extended plugins

Signed-off-by: Craig Perkins <[email protected]>

* Add additional warning message

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Add tag to make extended plugin optional

Signed-off-by: Craig Perkins <[email protected]>

* Only send plugin names when serializing PluginInfo

Signed-off-by: Craig Perkins <[email protected]>

* Keep track of optional extended plugins in separate set

Signed-off-by: Craig Perkins <[email protected]>

* Include in ser/de of PluginInfo

Signed-off-by: Craig Perkins <[email protected]>

* Change to 3_0_0

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>

Change version in PluginInfo to V_2_19_0 after backport to 2.x merged (opensearch-project#16947)

Signed-off-by: Craig Perkins <[email protected]>

Support object fields in star-tree index (opensearch-project#16728)

---------

Signed-off-by: bharath-techie <[email protected]>

Bump ch.qos.logback:logback-core from 1.5.12 to 1.5.16 in /test/fixtures/hdfs-fixture (opensearch-project#16951)

* Bump ch.qos.logback:logback-core in /test/fixtures/hdfs-fixture

Bumps [ch.qos.logback:logback-core](https://github.com/qos-ch/logback) from 1.5.12 to 1.5.16.
- [Commits](qos-ch/logback@v_1.5.12...v_1.5.16)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

[Workload Management] Add Workload Management IT (opensearch-project#16359)

* add workload management IT
Signed-off-by: Ruirui Zhang <[email protected]>

* address comments
Signed-off-by: Ruirui Zhang <[email protected]>

---------

Signed-off-by: Ruirui Zhang <[email protected]>

Add new benchmark config for nested workload (opensearch-project#16956)

Signed-off-by: Rishabh Singh <[email protected]>

Bump com.azure:azure-core-http-netty from 1.15.5 to 1.15.7 in /plugins/repository-azure (opensearch-project#16952)

* Bump com.azure:azure-core-http-netty in /plugins/repository-azure

Bumps [com.azure:azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.15.5 to 1.15.7.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.15.5...azure-core-http-netty_1.15.7)

---
updated-dependencies:
- dependency-name: com.azure:azure-core-http-netty
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Always use constant_score query for match_only_text (opensearch-project#16964)

In some cases, when we create a term query over a `match_only_text`
field, it may still try to compute scores, which prevents early
termination. We should *always* use a constant score query when
querying `match_only_text`, since we don't have the statistics
required to compute scores.

---------

Signed-off-by: Michael Froh <[email protected]>

Changes to support unmapped fields in metric aggregation (opensearch-project#16481)

Avoids exception when querying unmapped field when star tree experimental
feature is enables.

---------

Signed-off-by: expani <[email protected]>

Use async client for delete blob or path in S3 Blob Container (opensearch-project#16788)

* Use async client for delete blob or path in S3 Blob Container

Signed-off-by: Ashish Singh <[email protected]>

* Fix UTs

Signed-off-by: Ashish Singh <[email protected]>

* Fix failures in S3BlobStoreRepositoryTests

Signed-off-by: Ashish Singh <[email protected]>

* Fix S3BlobStoreRepositoryTests

Signed-off-by: Ashish Singh <[email protected]>

* Fix failures in S3RepositoryThirdPartyTests

Signed-off-by: Ashish Singh <[email protected]>

* Fix failures in S3RepositoryPluginTests

Signed-off-by: Ashish Singh <[email protected]>

---------

Signed-off-by: Ashish Singh <[email protected]>

Fix Shallow copy snapshot failures on closed index (opensearch-project#16868)

* Fix shallow v1 snapshot failures on closed index

Signed-off-by: Shubh Sahu <[email protected]>

* UT fix

Signed-off-by: Shubh Sahu <[email protected]>

* Adding UT

Signed-off-by: Shubh Sahu <[email protected]>

* small fix

Signed-off-by: Shubh Sahu <[email protected]>

* Addressing comments

Signed-off-by: Shubh Sahu <[email protected]>

* Addressing comments

Signed-off-by: Shubh Sahu <[email protected]>

* Modifying IT to restore snapshot

Signed-off-by: Shubh Sahu <[email protected]>

---------

Signed-off-by: Shubh Sahu <[email protected]>
Co-authored-by: Shubh Sahu <[email protected]>

Add Response Status Number in http trace logs. (opensearch-project#16978)

Signed-off-by: Rishikesh1159 <[email protected]>

support termQueryCaseInsensitive/termQuery can search from doc_value in flat_object/keyword field (opensearch-project#16974)

Signed-off-by: kkewwei <[email protected]>
Signed-off-by: kkewwei <[email protected]>

use the correct type to widen the sort fields when merging top docs (opensearch-project#16881)

* use the correct type to widen the sort fields when merging top docs

Signed-off-by: panguixin <[email protected]>

* fix

Signed-off-by: panguixin <[email protected]>

* apply commments

Signed-off-by: panguixin <[email protected]>

* changelog

Signed-off-by: panguixin <[email protected]>

* add more tests

Signed-off-by: panguixin <[email protected]>

---------

Signed-off-by: panguixin <[email protected]>

Fix multi-value sort for unsigned long (opensearch-project#16732)

* Fix multi-value sort for unsigned long

Signed-off-by: panguixin <[email protected]>

* Add initial rest-api-spec tests

Signed-off-by: Andriy Redko <[email protected]>

* add more rest tests

Signed-off-by: panguixin <[email protected]>

* fix

Signed-off-by: panguixin <[email protected]>

* fix

Signed-off-by: panguixin <[email protected]>

* Extend MultiValueMode with dedicated support of unsigned_long doc values

Signed-off-by: Andriy Redko <[email protected]>

* Add CHANGELOG.md, minor cleanups

Signed-off-by: Andriy Redko <[email protected]>

* Correct the license headers

Signed-off-by: Andriy Redko <[email protected]>

* Correct the @publicapi version

Signed-off-by: Andriy Redko <[email protected]>

* Replace SingletonSortedNumericUnsignedLongValues with LongToSortedNumericUnsignedLongValues (as per review comments)

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: panguixin <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>

Update Gradle to 8.12 (opensearch-project#16884)

Signed-off-by: Andriy Redko <[email protected]>

`phone-search` analyzer: don't emit sip/tel prefix, int'l prefix, extension & unformatted input (opensearch-project#16993)

* `phone-search` analyzer: don't emit int'l prefix

this was an oversight in the initial implementation: if the tokenizer
emits the international calling prefix in the search analyzer then all
documents with the same international calling prefix will match.

e.g. when searching for `+1-555-123-4567` not only documents with this
number would match but also any other document with a `1` token (i.e.
any other number with this prefix).

thus the search functionality is currently broken for this analyzer,
making it useless.

the test coverage has now been extended to cover these and other
use-cases.

Signed-off-by: Ralph Ursprung <[email protected]>

* `phone-search` analyzer: don't emit extension & unformatted input

if these tokens are emitted it meant that phone numbers with other
international dialling prefixes still matched.

e.g. searching for `+1 1234` would also match a number stored as
`+2 1234`, which was wrong.

the tokens still need to be emited for the `phone` analyzer, e.g. when
the user only enters the extension / local number it should still match,
the same is with the other ngrams: these are needed for
search-as-you-type style queries where the user input needs to match
against partial phone numbers.

Signed-off-by: Ralph Ursprung <[email protected]>

* `phone-search` analyzer: don't emit sip/tel prefix

in line with the previous two commits, this is something else the search
analyzer shouldn't emit since otherwise searching for any number with
such a prefix will match _any_ document with the same prefix.

Signed-off-by: Ralph Ursprung <[email protected]>

---------

Signed-off-by: Ralph Ursprung <[email protected]>

Limit RW separation to remote store enabled clusters and update recovery flow (opensearch-project#16760)

* Update search only replica recovery flow

This PR includes multiple changes to search replica recovery.
1. Change search only replica copies to recover as empty store instead of PEER. This will run a store recovery that syncs segments from remote store directly and eliminate any primary communication.
2. Remove search replicas from the in-sync allocation ID set and update routing table to exclude them from allAllocationIds.  This ensures primaries aren't tracking or validating the routing table for any search replica's presence.
3. Change search replica validation to require remote store.  There are versions of the above changes that are still possible with primary based node-node replication, but I don't think they are worth making  at this time.

Signed-off-by: Marc Handalian <[email protected]>

* more coverage

Signed-off-by: Marc Handalian <[email protected]>

* add changelog entry

Signed-off-by: Marc Handalian <[email protected]>

* add assertions that Search Replicas are not in the in-sync id set nor the AllAllocationIds set in the routing table

Signed-off-by: Marc Handalian <[email protected]>

* update async task to only run if the FF is enabled and we are a remote store cluster.

This check had previously only checked for segrep

Signed-off-by: Marc Handalian <[email protected]>

* clean up max shards logic

Signed-off-by: Marc Handalian <[email protected]>

* remove search replicas from check during renewPeerRecoveryRetentionLeases

Signed-off-by: Marc Handalian <[email protected]>

* Revert "update async task to only run if the FF is enabled and we are a remote store cluster."

reverting this, we already check for remote store earlier.

This reverts commit 48ca1a3.

Signed-off-by: Marc Handalian <[email protected]>

* Add more tests for failover case

Signed-off-by: Marc Handalian <[email protected]>

* Update remotestore restore logic and add test ensuring we can restore only writers when red

Signed-off-by: Marc Handalian <[email protected]>

* Fix Search replicas to honor node level recovery limits

Signed-off-by: Marc Handalian <[email protected]>

* Fix translog UUID mismatch on existing store recovery.

This commit adds PR feedback and recovery tests post node restart.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless

Signed-off-by: Marc Handalian <[email protected]>

* Fix bug with remote restore and add more tests

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Marc Handalian <[email protected]>

Fix case insensitive and escaped query on wildcard (opensearch-project#16827)

* fix case insensitive and escaped query on wildcard

Signed-off-by: gesong.samuel <[email protected]>

* add changelog

Signed-off-by: gesong.samuel <[email protected]>

---------

Signed-off-by: gesong.samuel <[email protected]>
Signed-off-by: Michael Froh <[email protected]>
Co-authored-by: gesong.samuel <[email protected]>
Co-authored-by: Michael Froh <[email protected]>

Bump opentelemetry from 1.41.0 to 1.46.0 and opentelemetry-semconv from 1.27.0-alpha to 1.29.0-alpha (opensearch-project#17000)

Signed-off-by: Andriy Redko <[email protected]>

TransportBulkAction.doRun() (opensearch-project#16950)

Signed-off-by: kkewwei <[email protected]>
Signed-off-by: kkewwei <[email protected]>

Show only intersecting buckets to the Adjacency matrix aggregation (opensearch-project#11733)

Signed-off-by: Ivan Brusic <[email protected]>

Bump com.google.re2j:re2j from 1.7 to 1.8 in /plugins/repository-hdfs (opensearch-project#17012)

* Bump com.google.re2j:re2j from 1.7 to 1.8 in /plugins/repository-hdfs

Bumps [com.google.re2j:re2j](https://github.com/google/re2j) from 1.7 to 1.8.
- [Release notes](https://github.com/google/re2j/releases)
- [Commits](google/re2j@re2j-1.7...re2j-1.8)

---
updated-dependencies:
- dependency-name: com.google.re2j:re2j
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bump com.nimbusds:oauth2-oidc-sdk from 11.20.1 to 11.21 in /plugins/repository-azure (opensearch-project#17010)

* Bump com.nimbusds:oauth2-oidc-sdk in /plugins/repository-azure

Bumps [com.nimbusds:oauth2-oidc-sdk](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions) from 11.20.1 to 11.21.
- [Changelog](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/branches/compare/11.21..11.20.1)

---
updated-dependencies:
- dependency-name: com.nimbusds:oauth2-oidc-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

improve `PhoneNumberAnalyzerTests#testTelPrefixSearch` (opensearch-project#17016)

this way we ensure that it doesn't include any additional tokens which
we don't want.

this is a follow-up to commit 4d94399 / opensearch-project#16993.

Signed-off-by: Ralph Ursprung <[email protected]>

Filter shards for sliced search at coordinator (opensearch-project#16771)

* Filter shards for sliced search at coordinator

Prior to this commit, a sliced search would fan out to every shard,
then apply a MatchNoDocsQuery filter on shards that don't correspond
to the current slice. This still creates a (useless) search context
on each shard for every slice, though. For a long-running sliced
scroll, this can quickly exhaust the number of available scroll
contexts.

This change avoids fanning out to all the shards by checking at the
coordinator if a shard is matched by the current slice. This should
reduce the number of open scroll contexts to max(numShards, numSlices)
instead of numShards * numSlices.

---------

Signed-off-by: Michael Froh <[email protected]>

Upgrade HttpCore5/HttpClient5 to support ExtendedSocketOption in HttpAsyncClient (opensearch-project#16757)

* upgrade httpcore5/httpclient5 to support ExtendedSocketOption in HttpAsyncClient

Signed-off-by: kkewwei <[email protected]>
Signed-off-by: kkewwei <[email protected]>

* Use the Upgrade flow by default

Signed-off-by: Andriy Redko <[email protected]>

* Update Reactor Netty to 1.1.26.Final

Signed-off-by: Andriy Redko <[email protected]>

* Add SETTING_H2C_MAX_CONTENT_LENGTH to configure h2cMaxContentLength for reactor-netty4 transport

Signed-off-by: Andriy Redko <[email protected]>

* Update Apache HttpCore5 to 5.3.2

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: kkewwei <[email protected]>
Signed-off-by: kkewwei <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>

Update version checks for backport (opensearch-project#17030)

Signed-off-by: Michael Froh <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: Michael Froh <[email protected]>

Fix versions and breaking API changes (opensearch-project#17031)

Signed-off-by: Andriy Redko <[email protected]>

Bump com.nimbusds:nimbus-jose-jwt from 9.47 to 10.0.1 in /test/fixtures/hdfs-fixture (opensearch-project#17011)

* Bump com.nimbusds:nimbus-jose-jwt in /test/fixtures/hdfs-fixture

Bumps [com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.47 to 10.0.1.
- [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/10.0.1..9.47)

---
updated-dependencies:
- dependency-name: com.nimbusds:nimbus-jose-jwt
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Remove user data from logs when not in debug/trace mode (opensearch-project#17007)

* Remove user data from logs when not in debug/trace mode

Signed-off-by: Mohit Godwani <[email protected]>

Remove user data from logs when not in debug/trace mode (opensearch-project#17007)

* Remove user data from logs when not in debug/trace mode

Signed-off-by: Mohit Godwani <[email protected]>
Signed-off-by: meetvm <[email protected]>
  • Loading branch information
meetvm committed Jan 17, 2025
1 parent 634b84a commit 1c5f519
Show file tree
Hide file tree
Showing 9 changed files with 89 additions and 17 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
match_all: {}

- match:
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\],.request:.+/
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\]/

- do:
indices.refresh: {}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
dest:
index: dest
- match:
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)\..Timeout\:.\[1s\],.request:.+/
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)\..Timeout\:.\[1s\]/

- do:
reindex:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
wait_for_active_shards: 4
timeout: 1s
- match:
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\],.request:.+/
failures.0.cause.reason: /Not.enough.active.copies.to.meet.shard.count.of.\[4\].\(have.1,.needed.4\)..Timeout\:.\[1s\]/

- do:
update_by_query:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ public void testReplicationWaitsForActiveShardCount() throws Exception {
assertThat(e.status(), equalTo(RestStatus.SERVICE_UNAVAILABLE));
assertThat(
e.getMessage(),
startsWith("[test][0] Not enough active copies to meet shard count of [2] (have 1, needed 2). Timeout: [100ms], request:")
startsWith("[test][0] Not enough active copies to meet shard count of [2] (have 1, needed 2). Timeout: [100ms]")
);
// but really, all is well
}
Expand Down Expand Up @@ -120,7 +120,7 @@ public void testReplicationWaitsForActiveShardCount() throws Exception {
startsWith(
"[test][0] Not enough active copies to meet shard count of ["
+ ActiveShardCount.ALL
+ "] (have 2, needed 3). Timeout: [100ms], request:"
+ "] (have 2, needed 3). Timeout: [100ms]"
)
);
// but really, all is well
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,15 +141,7 @@ public void execute() throws Exception {
final ShardRouting primaryRouting = primary.routingEntry();
final ShardId primaryId = primaryRouting.shardId();
if (activeShardCountFailure != null) {
finishAsFailed(
new UnavailableShardsException(
primaryId,
"{} Timeout: [{}], request: [{}]",
activeShardCountFailure,
request.timeout(),
request
)
);
finishAsFailed(new UnavailableShardsException(primaryId, "{} Timeout: [{}]", activeShardCountFailure, request.timeout()));
return;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1246,7 +1246,7 @@ void finishOnSuccess(Response response) {
}

void retryBecauseUnavailable(ShardId shardId, String message) {
retry(new UnavailableShardsException(shardId, "{} Timeout: [{}], request: [{}]", message, request.timeout(), request));
retry(new UnavailableShardsException(shardId, "{} Timeout: [{}]", message, request.timeout()));
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
import org.opensearch.index.shard.IndexShard;
import org.opensearch.script.Script;
import org.opensearch.script.ScriptService;
import org.opensearch.script.ScriptType;
import org.opensearch.script.UpdateScript;
import org.opensearch.search.lookup.SourceLookup;

Expand Down Expand Up @@ -128,7 +129,11 @@ Tuple<UpdateOpType, Map<String, Object>> executeScriptedUpsert(Map<String, Objec

if (operation != UpdateOpType.CREATE && operation != UpdateOpType.NONE) {
// Only valid options for an upsert script are "create" (the default) or "none", meaning abort upsert
logger.warn("Invalid upsert operation [{}] for script [{}], doing nothing...", operation, script.getIdOrCode());
if (logger.isDebugEnabled() || ScriptType.STORED.equals(script.getType())) {
logger.warn("Invalid upsert operation [{}] for script [{}], doing nothing...", operation, script.getIdOrCode());
} else {
logger.warn("Invalid upsert operation [{}] for given script", operation);
}
operation = UpdateOpType.NONE;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,8 @@ public boolean transferSnapshot(TransferSnapshot transferSnapshot, TranslogTrans
} catch (Exception ex) {
logger.error(() -> new ParameterizedMessage("Transfer failed for snapshot {}", transferSnapshot), ex);
captureStatsOnUploadFailure();
translogTransferListener.onUploadFailed(transferSnapshot, ex);
Exception exWithoutSuppressed = new TranslogUploadFailedException(ex.getMessage());
translogTransferListener.onUploadFailed(transferSnapshot, exWithoutSuppressed);
return false;
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,80 @@ public void onUploadFailed(TransferSnapshot transferSnapshot, Exception ex) {
assertEquals(4, fileTransferTracker.allUploaded().size());
}

public void testTransferSnapshotOnFileTransferUploadFail() throws Exception {
AtomicInteger fileTransferSucceeded = new AtomicInteger();
AtomicInteger fileTransferFailed = new AtomicInteger();
AtomicInteger translogTransferSucceeded = new AtomicInteger();
AtomicInteger translogTransferFailed = new AtomicInteger();

doAnswer(invocationOnMock -> {
ActionListener<TransferFileSnapshot> listener = (ActionListener<TransferFileSnapshot>) invocationOnMock.getArguments()[2];
Set<TransferFileSnapshot> transferFileSnapshots = (Set<TransferFileSnapshot>) invocationOnMock.getArguments()[0];

TransferFileSnapshot actualFileSnapshot = transferFileSnapshots.iterator().next();
FileTransferException testException = new FileTransferException(
actualFileSnapshot,
new RuntimeException("FileTransferUploadNeedsToFail-Exception")
);

listener.onFailure(testException);
transferFileSnapshots.stream().skip(1).forEach(listener::onResponse);
return null;
}).when(transferService).uploadBlobs(anySet(), anyMap(), any(ActionListener.class), any(WritePriority.class));

FileTransferTracker fileTransferTracker = new FileTransferTracker(
new ShardId("index", "indexUUid", 0),
remoteTranslogTransferTracker
) {
@Override
public void onSuccess(TransferFileSnapshot fileSnapshot) {
fileTransferSucceeded.incrementAndGet();
super.onSuccess(fileSnapshot);
}

@Override
public void onFailure(TransferFileSnapshot fileSnapshot, Exception e) {
fileTransferFailed.incrementAndGet();
super.onFailure(fileSnapshot, e);
}
};

TranslogTransferManager translogTransferManager = new TranslogTransferManager(
shardId,
transferService,
remoteBaseTransferPath.add(TRANSLOG.getName()),
remoteBaseTransferPath.add(METADATA.getName()),
fileTransferTracker,
remoteTranslogTransferTracker,
DefaultRemoteStoreSettings.INSTANCE,
isTranslogMetadataEnabled
);

SetOnce<Exception> exception = new SetOnce<>();
assertFalse(translogTransferManager.transferSnapshot(createTransferSnapshot(), new TranslogTransferListener() {
@Override
public void onUploadComplete(TransferSnapshot transferSnapshot) {
translogTransferSucceeded.incrementAndGet();
}

@Override
public void onUploadFailed(TransferSnapshot transferSnapshot, Exception ex) {
translogTransferFailed.incrementAndGet();
exception.set(ex);
}
}));

assertNotNull(exception.get());
assertTrue(exception.get() instanceof TranslogUploadFailedException);
assertEquals("Failed to upload 1 files during transfer", exception.get().getMessage());
assertEquals(0, exception.get().getSuppressed().length);
assertEquals(3, fileTransferSucceeded.get());
assertEquals(1, fileTransferFailed.get());
assertEquals(0, translogTransferSucceeded.get());
assertEquals(1, translogTransferFailed.get());
assertEquals(3, fileTransferTracker.allUploaded().size());
}

public void testTransferSnapshotOnUploadTimeout() throws Exception {
doAnswer(invocationOnMock -> {
Set<TransferFileSnapshot> transferFileSnapshots = invocationOnMock.getArgument(0);
Expand Down

0 comments on commit 1c5f519

Please sign in to comment.