Skip to content

Commit

Permalink
GEODE-8899: Upgrade to Gradle v6.8.3 (#6280)
Browse files Browse the repository at this point in the history
* GEODE-8899: Upgrade to Gradle v6.8.3

Updated Geode's build system to use Gradle 6.8.3:

- Updated the gradle wrapper to use Gradle 6.8.3.
- Updated geode-assembly/build.gradle to make the ivy "repository"
  compatible with Gradle 6.8.3.
- Changed ./gradle.properties to specify 'minimumGradleVersion=6.8'.
- Updated the expected-pom.xml file to the format that Gradle 6.8.3
  produces.
- Updated all Gradle plugins to the latest versions.
- In buildSrc/build.gradle, added groovy source sets for testing, to
  allow running tests of plugin code.
- Updated common "test isolation" code to be compatible with Gradle
  6.8.3. See below for details.
- Updated the RepeatTest task to be compatible with Gradle 6.8.3. See
  below for details.
- Rewrote the Dockerized test plugin to be compatible with Gradle 6.8.3.
  See below for details.
- Added a new 'dunitDockerJVM' project property to allow developers run
  tests in Docker on macOS. To do this, use -PtestJVM to identify the
  mac's JVM, and use -PdunitDockerJVM to specify the JVM for the test
  worker process running in Docker.
- Updated the BUILDING.md file for Gradle 6.8.3. As part of this I added
  two versions of the Apache copyright notice to the etc dir: A plain
  text version and an xml version that IntelliJ IDEA can import into its
  copyright settings.

===== Test isolation code =====

Some code for isolating tests is used by the RepeatTest task and the
Dockerized test plugin. This code was updated to be compatible with
Gradle v6.8.3, and to somewhat reduce its dependence on Gradle internal
implementation details:

- WorkingDirectoryIsolator adjusts a given ProcessBuilder to give it a
  unique working directory.
- AdjustableProcessLauncher applies a given Consumer (such as the
  isolator, above) to adjust each ProcessBuilder before launching the
  process.
- LauncherProxyWorkerProcessBuilder and
  LauncherProcessWorkerProcessFactory forcefully update each Gradle exec
  handle to use a given ProcessLauncher to launch test worker processes.
- Executers and Workers offer convenience methods for constructing
  TestExecuters and ProcessWorker builders and factories.

===== RepeatTest task update =====

Geode's RepeatTest task uses a custom TestExecuter that was copied from
Gradle 5.5 (or earlier) and modified. This custom class is now updated
to be compatible with Gradle 6.8.3, and renamed from
OverriddenTestExecutor to RepeatableTestExecuter.

===== Dockerized test plugin rewrite =====

Removed the modified copy of pedjak's Dockerized test plugin.

Configuration
-------------

- Rewrote gradle/docker.gradle and renamed it as
  gradle/multi-process-test.gradle.
- Moved the code that translates project properties into configuration
  settings from the plugin to the config class. Renamed the config class
  as DockerTestWorkerConfig.
- Moved code that configures command lines and environments for Docker
  test workers based on project properties. Thise code was defiend as a
  closure in a build script. It is now a method in
  DockerTestWorkerConfig.
- Changed command line options that CI scripts pass to Gradle to run
  tests in Docker:
    - Specify the --max-workers Gradle option to configure the maximum
      number of workers.
    - Specify the 'testMaxParallelForks' project property to configure
      the maximum number of forks for each test task.
    - Remove the 'dunitParallelForks' option.

Implementation
--------------

The Dockerized test plugin uses these classes to launches processes in
Docker containers:

- DockerProcess overrides java.lang.Process to represent a test worker
  process running in Docker.
- DockerProcessLauncher launches each test worker process in a Docker
  container, and creates a DockerProcess to represent and manage it.
- Other classes described in the "Test isolation code" section, above.

The plugin overrides several of Gradle's internal classes to allow test
worker processes running in Docker containers to communicate with
Gradle:

- DockerConnectionAcceptor implements Gradle's ConnectionAcceptor, which
  (among other responsibilities), produces a "multi-choice address" that
  a test worker process can use to connect to Gradle's messaging server.
  Gradle's default implementation produces multi-choice addresses that
  processes in Docker containers cannot use.
- DockerMessagingServer implements Gradle's MessagingServer using
  DockerConnectionAcceotpr.
- WildcardBindingInetAddressFactory overrides Gradle's
  InetAddressFactory to instruct messaging servers to listen on an
  address that processes in Dockerized containers can connect to.

Timeouts
--------

Due to a recent surge of CI problems due to delays in Docker operations,
I rewrote the plugins's timeout mechanism to make it configurable and
much more robust:

- Added a 'dockerTimeout' project property, defaulting to 5 minutes.
  Each Docker operation will throw an exception if its duration exceeds
  this value.
- Log a warning if any Docker operation takes longer than 1 minute. This
  can help us to see how often significant delays occur in CI, while
  still allowing tests to proceed.
- These exceptions and warnings identify the project for which test
  worker process is being launched. This can help us to see whether
  there are project-specific factors that affect the delays.

* Fix problems identified by rhoughton-pivot
  • Loading branch information
demery-pivotal authored Apr 12, 2021
1 parent 8dada2f commit 75bb0d2
Show file tree
Hide file tree
Showing 50 changed files with 1,916 additions and 2,332 deletions.
139 changes: 75 additions & 64 deletions BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

All platforms require a Java installation, with JDK 1.8 or more recent version.

Set the JAVA\_HOME environment variable. For example:
Set the JAVA\_HOME environment variable. For example:

| Platform | Command |
| :---: | --- |
Expand All @@ -14,16 +14,17 @@ Download the project source from the Releases page at
[Apache Geode](http://geode.apache.org/releases/), and unpack the source code.

Within the directory containing the unpacked source code, run the gradle build:

```console
$ ./gradlew build
```

Once the build completes, the project files will be installed at
`geode-assembly/build/install/apache-geode`. The distribution archives will be
created in `geode-assembly/build/distributions/`.
`geode-assembly/build/install/apache-geode`. The distribution archives will be created
in `geode-assembly/build/distributions/`.

Verify the installation by invoking the `gfsh` shell command to print version information:

Verify the installation by invoking the `gfsh` shell command to print version
information:
```console
$ ./geode-assembly/build/install/apache-geode/bin/gfsh version
v1.1.0
Expand All @@ -33,64 +34,74 @@ Note: on Windows invoke the `gfsh.bat` script to print the version string.

## Setting up IntelliJ

The following steps have been tested with:

* **IntelliJ IDEA 2018.3.5**

1. Run `./gradlew --parallel generate` from Geode repository root to create compiler generated source.
1. Alternatively (and to ensure these sources stay up-to-date):
- Perform step 2
- In the **Gradle** dockable, expand the **geode** (the root project) -> **Tasks** -> **build**
- Right-click **generate** task, select **Execute Before Sync** and **Execute Before Build**
- Click **Refresh All Gradle Projects** at the top of the Gradle dockable

![After](https://cwiki.apache.org/confluence/download/attachments/103096184/geode-generate-before-sync.png?api=v2)

2. Import project into IntelliJ IDEA.

From the **Welcome to IntelliJ IDEA** window:

1. **Import Project ->** select *build.gradle* file from Geode repository root and press **Open**.
2. Optionally, enable **Use auto-import**
3. Enable **Create separate module per source set**
4. Select **Use Project JDK 1.8.0_*nnn*** where *nnn* is latest build required for Geode

3. Change Code Style Scheme to GeodeStyle.

Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Code Style**. Select *GeodeStyle* in Scheme drop-down box if it already exists.

To define the *GeodeStyle* in **Scheme**, select the gear icon next to the drop-down box, click **Import Scheme ->** and select **IntelliJ IDEA code style XML**. Select *etc/intellij-java-modified-google-style.xml* from Geode repository root, enter **To:** *GeodeStyle*, check **Current scheme** and press **OK**.

4. Make Apache the default Copyright.

Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright**. Select *Apache* in drop-down box **Default project copyright**.

To define *Apache* in **Copyright**, navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright -> Copyright Profiles**. Click **+** to add a new project. Enter *Apache* as the **Name** and enter the following block without asterisks or leading spaces:

```text
Licensed to the Apache Software Foundation (ASF) under one or more contributor license
agreements. See the NOTICE file distributed with this work for additional information regarding
copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance with the License. You may obtain a
copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License
is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
or implied. See the License for the specific language governing permissions and limitations under
the License.
```

...then return to **Copyright** and select *Apache* in drop-down box **Default project copyright**.

Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright -> Formatting**. Uncheck **Add blank line after** and press **OK**.

5. Rebuild the Project.

Navigate to **Build -> Rebuild Project** and the full project should compile without errors.
The following steps have been tested with **IntelliJ IDEA 2020.3.3**

1. Run `./gradlew --parallel generate` from Geode repository root to create compiler generated
source.

1. Import the project into IntelliJ IDEA.

1. Select **File -> Open...** from the menu.
1. Select the `build.gradle` file in the Geode repository root and select **Open**.
1. In the **Open Project?** popup, select **Open Project**.
1. In the **Trust and Open Gradle Project?** popup, select **Trust Project**.
1. Wait for IntelliJ to import the project and complete its background tasks.

1. Configure IntelliJ IDEA to build and run the project and tests.
* Set the Java SDK for the project.
1. Select **File -> Project Structure...** from the menu.
1. Open the **Project Settings -> Project** section.
1. Set **Project SDK** to your most recent Java 1.8 JDK.

* To automatically re-generate sources when needed (recommended).
1. Select **View -> Tool Windows -> Gradle** from the menu.
1. In the Gradle dockable, open **geode -> Tasks -> build**.
1. Right click the **generate** task and select **Execute Before Sync**.
1. Right click the **generate** task and select **Execute Before Build**.

* To reload the project when build scripts change (recommended).
1. Select **IntelliJ IDEA -> Preferences...** from the menu.
1. Open the **Build, Execution, Deployment -> Build Tools** section.
1. Set **Reload project after changes in the build scripts:** to **Any changes**.

* To build and run with Gradle (recommended).
1. Select **IntelliJ IDEA -> Preferences...** from the menu.
1. Open the **Build, Execution, Deployment -> Build Tools -> Gradle** section.
1. Set **Build and run using:** to **Gradle**.
1. Set **Run tests using:** to **Gradle**.

1. Set the Code Style Scheme to GeodeStyle.

1. Select **IntelliJ IDEA -> Preferences...**
1. Open the **Editor -> Code Style** section.
1. If *GeodeStyle* style does not appear in the **Scheme** drop-down box
1. Select the gear icon next to the drop-down.
1. Select **Import Scheme -> IntelliJ IDEA code style XML**.
1. Select `etc/intellij-java-modified-google-style.xml` from the Geode repository root.
1. Enter **To:** *GeodeStyle*, check **Current scheme**, and press **OK**.
1. Select *GeodeStyle* in **Scheme** drop-down box.

1. Make Apache the default Copyright.

1. Select **IntelliJ IDEA -> Preferences...** from the menu.
1. Open the **Editor -> Copyright** section.
1. If *Apache* does not appear in the **Default project copyright** drop-down box:
1. Open the **Copyright Profiles** subsection.
1. Select the "import" icon (the small arrow pointing down and to the left) from the
Copyright Profiles section's toolbar.
1. Select `etc/intellij-apache-copyright-notice.xml` from the Geode repository root.
1. Return to the **Copyright** section.
1. Select *Apache* in the **Default project copyright** drop-down box.
1. Open the **Formatting** subsection.
1. Uncheck **Add blank line after** and select **OK**.

1. Rebuild the Project.

1. Select **Build -> Rebuild Project** from the menu. The full project should compile without
errors.

Some optional sanity tests to make sure things are working properly:
* Try looking up classes using **Navigate -> Class...**
* Open and run a distributed test such as BasicDistributedTest in geode-core.
* Create a new java class and ensure the Apache license is automatically added to the top of the file with no blank line before the package line.
* Try looking up classes using **Navigate -> Class...**
* Open and run a distributed test such as BasicDistributedTest in geode-core.
* Create a new java class and ensure the Apache license is automatically added to the top of the
file with no blank line before the package line.
Loading

0 comments on commit 75bb0d2

Please sign in to comment.