Merge main

dev-docs/releasing.adoc

@@ -0,0 +1,82 @@
+
+= Releasing Solr
+:toc: left
+
+== Motivated?
+So you're of the opinion that there are unreleased features or bugfixes committed to the Solr repository that the world needs?
+Are you so convinced of this that you are willing to volunteer to make it happen?
+Good! This document tells you how to get started.
+
+== Overview
+The following is an overview of the artifacts you will be publishing. Although the release wizard should be your primary guide, having a picture of what's going on may help you understand and validate what the release wizard is asking you to do.
+
+There are five major parts of a release. They all become available to the public in slightly different ways, and it helps to understand these differences.
+
+IMPORTANT: All of these publications experience a time delay while background infrastructure detects and propagates your changes. There will be points in the release process when you just need to wait.
+
+=== Source Code and Binaries (downloads.apache.org)
+
+The distribution of a specific version of the source code is the theoretical center of any release by the Apache Software Foundation. As a convenience, precompiled binaries are also provided on downloads.apache.org. The mechanics of this process start with an SVN commit. The result of that commit is automatically synced to the downloads server (~5m?), and then on a longer time frame (6h?) anything on downloads is synced to archives.apache.org. See https://infra.apache.org/release-publishing.html#timeline for more detail
+
+=== Maven Artifacts (repository.apache.org)
+The compiled jar files, source jars, javadoc and pom need to be distributed to the maven ecosystem. This happens via repository.apache.org which is then automatically synced into maven central (~2h, upto 24h for maven central search). If you have released software to maven central via Sonatype, you will see that Apache uses the same repository manager software (Nexus), and this interface will look familiar to you.
+
+=== The Docker Image (hub.docker.com/_/solr)
+The official solr docker image will be published for use by the world. The latency for this publication is unknown to me, (it went smoothly, and I forgot to make a note of it) but someone who figures that out can update this section.
+
+=== The Web Site (solr.apache.org)
+Our website must be updated with each release. It is based on the content of the solr-site git repository and checking in changes to the `main` branch there will automatically become available for preview within a few minutes at https://solr.staged.apache.org/[https://solr.staged.apache.org/]. Merging main into a branch named `production` publishes your updates to the live website also within a few minutes. Note that it is normal for the staging site links to javadocs and ref guide to return 404 because these are published by a different process.
+
+=== The Reference Guide (solr.apache.org/guide)
+The ref guide is published once every 24h by a Jenkins build found at https://ci-builds.apache.org/job/Solr/job/solr-reference-guide-official/[https://ci-builds.apache.org/job/Solr/job/solr-reference-guide-official/]. This is a complicated process that has been automated for you. It is primarily influenced by the publication of an `antora.yaml` file during the steps the release wizard will guide you through. If the Jenkins build runs while there is a branch, but no updated antora.yaml file, an `M.N-beta` reference may be seen on the live ref guide, but there will be no `M.N-beta` ref guide pages and selecting it will merely browse the latest. After the Jenkins build runs, it takes several hours for the new version to become visible in a browser (possibly due to a caching layer?). There's a window of 6-30h after the release where this is not of concern, so don't panic. You can check the above jenkins job while you wait to estimate when the changes will be expected to become visible.
+
+== Step 0 - Become a Committer
+To do a release you must become a https://community.apache.org/contributors/becomingacommitter.html[committer] on the project. Additionally, if you are not on the https://apache.org/foundation/how-it-works/#pmc[Project Management Committee (PMC)]  you will also need to take special steps, and you will need to partner with a PMC member for at least one step. See https://www.apache.org/legal/release-policy.html#upload-ci
+
+== Step 1 - Run the Release Wizard!
+
+But wait! don't I need community approval?! (you exclaim)
+
+Yes! But there are some details to take care of that don't require anyone's approval, and getting comfortable with the release wizard will make it smoother, so you should run through the early sections of the release wizard first.
+
+=== Where to find the release wizard
+
+The release wizard is found in a checkout of solr at `dev-tools/scripts/releaseWizard.py`
+
+=== What working copy to use
+
+The release wizard is meant to be used from any working copy you like, with the expectation that you will check out the PARENT branch. In other words you should check out main to publish M.0.0 and branch_Mx to publish M.N.0, and branch M.N.x-1 for M.N.x. The wizard will (eventually) guide you into creating a fresh, clean checkout where your intended release is built, but you don't need that at the start.
+
+=== How to run the release wizard
+
+1. Make sure you have python 3.4+ installed (if a higher version becomes required, the wizard should complain and tell you what it is)
+2. Install dependencies with `pip3 install -r requirements.txt`, from the `dev-tools/scripts` folder.
+3. Run the command you see documented at `dev-tools/scripts/README.md`. Using `--dry-run` initially is fine.
+
+NOTE: `--dry-run` does still create `~/.solr_releases` and `~/.solrrc` files and record the release version you intend for future reference so one might say it's really better described as a "slightly damp" run :).  It should however not execute other commands (in theory). Also, checklist elements approved during a dry run are remembered, but this may change, see https://issues.apache.org/jira/browse/SOLR-17246[SOLR-17246]
+
+
+== Step 2 - Complete the First two Checklists
+CAUTION: The release wizard is software, it may have bugs or not cover every situation. You need to think about the commands you are given. Some of the known pitfalls can be found with https://issues.apache.org/jira/issues/?jql=project%20%3D%20SOLR%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20release-scripts%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC[this jira search].
+
+The release wizard is organized into checklists and the first two checklists are preparation/planning related, and are good to complete before proposing the release.
+
+== Step 3 - Propose the Release
+
+Mail [email protected] with a subject like `[DISCUSS] Solr X.Y release` and wax poetic about the wonderful features and awful bugs that are fixed but not yet released and volunteer to do the release. If you are sufficiently inspiring most of the PMC will quietly say to themselves "Whew! I don't have to do it" and then some of them will respond with a +1. Then at the last minute right before the time you proposed to start the release 1-4 people will suddenly mail the list saying something like "Can we wait for X amount of time so that I can get SOLR-XXXXXX in?" Let the community discuss it, and give them some time because you knew this would happen (because you read this guide!)
+
+== Step 4 - Continue Following the Release Wizard
+
+It's supposed to tell you all you need from here... Mail the list or post on `#solr-dev` Slack if you have questions, and file Jira issues for things that could be improved for the next person (please set component to `release-scripts`).
+
+== Step 5 - Complete the After Release Steps in the Wizard too.
+
+There are some things that need to be done after the release is officially published and announced. The wizard will guide you through that (things like updating Jira etc.)
+
+== Dive deeper
+
+The https://infra.apache.org/release-publishing.html[Release Creation Process] we page gives more in-depth explanation of an Apache release.
+
+The origins of the ReleaseWizard is described in https://www.linkedin.com/pulse/releasing-lucene-just-61-steps-jan-h%C3%B8ydahl/[this blog post] from 2019. The same tool is currently used by Lucene, Solr and Solr-Operator.
+
+

gradlew

@@ -157,21 +157,23 @@ if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
 fi
 
 GRADLE_WRAPPER_JAR="$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
-"$JAVACMD" $JAVA_OPTS --source 11 "$APP_HOME/buildSrc/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java" "$GRADLE_WRAPPER_JAR"
-WRAPPER_STATUS=$?
-if [ "$WRAPPER_STATUS" -eq 1 ]; then
-    echo "ERROR: Something went wrong. Make sure you're using Java version between 11 and 21."
-    exit $WRAPPER_STATUS
-elif [ "$WRAPPER_STATUS" -ne 0 ]; then
-    exit $WRAPPER_STATUS
+if [ ! -e "$GRADLE_WRAPPER_JAR" ]; then
+    "$JAVACMD" $JAVA_OPTS "$APP_HOME/buildSrc/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java" "$GRADLE_WRAPPER_JAR"
+    WRAPPER_STATUS=$?
+    if [ "$WRAPPER_STATUS" -eq 1 ]; then
+        echo "ERROR: Something went wrong. Make sure you're using Java version between 11 and 21."
+        exit $WRAPPER_STATUS
+    elif [ "$WRAPPER_STATUS" -ne 0 ]; then
+        exit $WRAPPER_STATUS
+    fi
 fi
 
 CLASSPATH=$GRADLE_WRAPPER_JAR
 
 # START OF LUCENE CUSTOMIZATION
 # Generate gradle.properties if they don't exist
 if [ ! -e "$APP_HOME/gradle.properties" ]; then
-    "$JAVACMD" $JAVA_OPTS --source 11 "$APP_HOME/buildSrc/src/main/java/org/apache/lucene/gradle/GradlePropertiesGenerator.java" "$APP_HOME/gradle/template.gradle.properties" "$APP_HOME/gradle.properties"
+    "$JAVACMD" $JAVA_OPTS "$APP_HOME/buildSrc/src/main/java/org/apache/lucene/gradle/GradlePropertiesGenerator.java" "$APP_HOME/gradle/template.gradle.properties" "$APP_HOME/gradle.properties"
     GENERATOR_STATUS=$?
     if [ "$GENERATOR_STATUS" -ne 0 ]; then
         exit $GENERATOR_STATUS

gradlew.bat

@@ -75,9 +75,11 @@ goto fail
 
 @rem LUCENE-9266: verify and download the gradle wrapper jar if we don't have one.
 set GRADLE_WRAPPER_JAR=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
-"%JAVA_EXE%" %JAVA_OPTS% --source 11 "%APP_HOME%/buildSrc/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java" "%GRADLE_WRAPPER_JAR%"
-IF %ERRORLEVEL% EQU 1 goto failWithJvmMessage
-IF %ERRORLEVEL% NEQ 0 goto fail
+IF NOT EXIST "%GRADLE_WRAPPER_JAR%" (
+    "%JAVA_EXE%" %JAVA_OPTS% "%APP_HOME%/buildSrc/src/main/java/org/apache/lucene/gradle/WrapperDownloader.java" "%GRADLE_WRAPPER_JAR%"
+    IF %ERRORLEVEL% EQU 1 goto failWithJvmMessage
+    IF %ERRORLEVEL% NEQ 0 goto fail
+)
 
 @rem Setup the command line
 set CLASSPATH=%GRADLE_WRAPPER_JAR%
@@ -87,7 +89,7 @@ set CLASSPATH=%GRADLE_WRAPPER_JAR%
 IF NOT EXIST "%APP_HOME%\gradle.properties" (
   @rem local expansion is needed to check ERRORLEVEL inside control blocks.
   setlocal enableDelayedExpansion
-  "%JAVA_EXE%" %JAVA_OPTS% --source 11 "%APP_HOME%/buildSrc/src/main/java/org/apache/lucene/gradle/GradlePropertiesGenerator.java" "%APP_HOME%\gradle\template.gradle.properties" "%APP_HOME%\gradle.properties"
+  "%JAVA_EXE%" %JAVA_OPTS% "%APP_HOME%/buildSrc/src/main/java/org/apache/lucene/gradle/GradlePropertiesGenerator.java" "%APP_HOME%\gradle\template.gradle.properties" "%APP_HOME%\gradle.properties"
   IF %ERRORLEVEL% NEQ 0 goto fail
   endlocal
 )

solr/CHANGES.txt

@@ -65,6 +65,7 @@ Deprecation Removals
 
 * SOLR-14115: Remove deprecated zkcli script in favour of equivalent bin/solr sub commmands. (Eric Pugh)
 
+* SOLR-17284: Remove deprecated BlobRepository in favour of FileStore. (Eric Pugh)
 
 Dependency Upgrades
 ---------------------
@@ -95,6 +96,8 @@ Other Changes
 
 * SOLR-17205: De-couple SolrJ required Java version from server Java version (janhoy)
 
+* SOLR-17279: Introduce SecurityJson.java file to Test Framework to consolidate setting up authentication in tests. (Rudy Seitz via Eric Pugh)
+
 ==================  9.7.0 ==================
 New Features
 ---------------------
@@ -111,6 +114,12 @@ Improvements
 
 * SOLR-14115: Add linkconfig, cluster, and updateacls as commands to SolrCLI.  Allow bin/solr commands to have parity with zkcli.sh commands. (Eric Pugh)
 
+* SOLR-17274: Allow JSON atomic updates to use multiple modifiers or a modifier like 'set' as a field name
+  if child docs are not enabled. (Calvin Smith, David Smiley)
+
+* SOLR-17300: Http2SolrClient.Builder.withHttpClient now copies HttpListenerFactory (e.g. for auth, metrics, traces, etc.)
+  (Sanjay Dutt, David Smiley)
+
 Optimizations
 ---------------------
 * SOLR-17257: Both Minimize Cores and the Affinity replica placement strategies would over-gather
@@ -122,6 +131,17 @@ Bug Fixes
 
 * SOLR-17261: Remove unintended timeout of 60 seconds for core loading. (Houston Putman)
 
+* SOLR-17049: Actually mark all replicas down at startup and truly wait for them.
+  This includes replicas that might not exist anymore locally. (Houston Putman, Vincent Primault)
+
+* SOLR-17263: Fix 'Illegal character in query' exception in HttpJdkSolrClient (Andy Webb)
+
+* SOLR-17275: SolrJ ZkClientClusterStateProvider, revert SOLR-17153 for perf regression when aliases are used. (Aparna Suresh)
+
+* PR-2475: Fixed node listing bug in Admin UI when different hostnames start with the same front part. (@hgdharold via Eric Pugh)
+
+* SOLR-16659: Properly construct V2 base urls instead of replacing substring "/solr" with "/api" (Andrey Bozhko via Eric Pugh)
+
 Dependency Upgrades
 ---------------------
 (No changes)
@@ -133,8 +153,19 @@ Other Changes
 * SOLR-16505: Use Jetty HTTP2 for index replication and other "recovery" operations
   (Sanjay Dutt, David Smiley)
 
+* GITHUB#2454: Refactor preparePutOrPost method in HttpJdkSolrClient (Andy Webb)
+
+* SOLR-16503: Use Jetty HTTP2 for SyncStrategy and PeerSyncWithLeader for "recovery" operations (Sanjay Dutt, David Smiley)
+
 * SOLR-16503: Introduce new default Http2SolrClient in UpdateShardHandler and deprecated old default HttpClient (Sanjay Dutt, David Smiley)
 
+==================  9.6.1 ==================
+Bug Fixes
+---------------------
+* SOLR-17296: Remove (broken) singlePass attempt when reRankScale + debug is used, and fix underlying NPE. (hossman)
+
+* SOLR-17307: Use the system file separator instead of an explicit '/' in CachingDirectoryFactory (Houston Putman, hossman)
+
 ==================  9.6.0 ==================
 New Features
 ---------------------

solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java

@@ -164,7 +164,8 @@ public NamedList<Object> request(SolrRequest<?> request, String coreName)
     if (handler != null) {
       try {
         SolrQueryRequest req =
-            _parser.buildRequestFrom(null, request.getParams(), getContentStreams(request));
+            _parser.buildRequestFrom(
+                null, request.getParams(), getContentStreams(request), request.getUserPrincipal());
         req.getContext().put("httpMethod", request.getMethod().name());
         req.getContext().put(PATH, path);
         SolrQueryResponse resp = new SolrQueryResponse();

solr/core/src/java/org/apache/solr/cloud/SyncStrategy.java

@@ -24,10 +24,9 @@
 import java.util.List;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
-import org.apache.http.client.HttpClient;
 import org.apache.solr.client.solrj.SolrClient;
 import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
 import org.apache.solr.client.solrj.request.CoreAdminRequest.RequestRecovery;
 import org.apache.solr.common.cloud.ZkCoreNodeProps;
 import org.apache.solr.common.cloud.ZkNodeProps;
@@ -55,7 +54,7 @@ public class SyncStrategy {
 
   private volatile boolean isClosed;
 
-  private final HttpClient client;
+  private final Http2SolrClient solrClient;
 
   private final ExecutorService updateExecutor;
 
@@ -69,7 +68,7 @@ private static class RecoveryRequest {
 
   public SyncStrategy(CoreContainer cc) {
     UpdateShardHandler updateShardHandler = cc.getUpdateShardHandler();
-    client = updateShardHandler.getDefaultHttpClient();
+    solrClient = updateShardHandler.getRecoveryOnlyHttpClient();
     shardHandler = cc.getShardHandlerFactory().getShardHandler();
     updateExecutor = updateShardHandler.getUpdateExecutor();
   }
@@ -361,12 +360,11 @@ private void requestRecovery(
               RequestRecovery recoverRequestCmd = new RequestRecovery();
               recoverRequestCmd.setAction(CoreAdminAction.REQUESTRECOVERY);
               recoverRequestCmd.setCoreName(coreName);
-
               try (SolrClient client =
-                  new HttpSolrClient.Builder(baseUrl)
-                      .withHttpClient(SyncStrategy.this.client)
+                  new Http2SolrClient.Builder(baseUrl)
+                      .withHttpClient(solrClient)
                       .withConnectionTimeout(30000, TimeUnit.MILLISECONDS)
-                      .withSocketTimeout(120000, TimeUnit.MILLISECONDS)
+                      .withIdleTimeout(120000, TimeUnit.MILLISECONDS)
                       .build()) {
                 client.request(recoverRequestCmd);
               } catch (Throwable t) {