diff --git a/BUILDING.md b/BUILDING.md index ca7b7ff30333..ebcb84204bd4 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -2,7 +2,7 @@ All platforms require a Java installation, with JDK 1.8 or more recent version. -Set the JAVA\_HOME environment variable. For example: +Set the JAVA\_HOME environment variable. For example: | Platform | Command | | :---: | --- | @@ -14,16 +14,17 @@ Download the project source from the Releases page at [Apache Geode](http://geode.apache.org/releases/), and unpack the source code. Within the directory containing the unpacked source code, run the gradle build: + ```console $ ./gradlew build ``` Once the build completes, the project files will be installed at -`geode-assembly/build/install/apache-geode`. The distribution archives will be -created in `geode-assembly/build/distributions/`. +`geode-assembly/build/install/apache-geode`. The distribution archives will be created +in `geode-assembly/build/distributions/`. + +Verify the installation by invoking the `gfsh` shell command to print version information: -Verify the installation by invoking the `gfsh` shell command to print version -information: ```console $ ./geode-assembly/build/install/apache-geode/bin/gfsh version v1.1.0 @@ -33,64 +34,74 @@ Note: on Windows invoke the `gfsh.bat` script to print the version string. ## Setting up IntelliJ -The following steps have been tested with: - -* **IntelliJ IDEA 2018.3.5** - -1. Run `./gradlew --parallel generate` from Geode repository root to create compiler generated source. - 1. Alternatively (and to ensure these sources stay up-to-date): - - Perform step 2 - - In the **Gradle** dockable, expand the **geode** (the root project) -> **Tasks** -> **build** - - Right-click **generate** task, select **Execute Before Sync** and **Execute Before Build** - - Click **Refresh All Gradle Projects** at the top of the Gradle dockable - - ![After](https://cwiki.apache.org/confluence/download/attachments/103096184/geode-generate-before-sync.png?api=v2) - -2. Import project into IntelliJ IDEA. - - From the **Welcome to IntelliJ IDEA** window: - - 1. **Import Project ->** select *build.gradle* file from Geode repository root and press **Open**. - 2. Optionally, enable **Use auto-import** - 3. Enable **Create separate module per source set** - 4. Select **Use Project JDK 1.8.0_*nnn*** where *nnn* is latest build required for Geode - -3. Change Code Style Scheme to GeodeStyle. - - Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Code Style**. Select *GeodeStyle* in Scheme drop-down box if it already exists. - - To define the *GeodeStyle* in **Scheme**, select the gear icon next to the drop-down box, click **Import Scheme ->** and select **IntelliJ IDEA code style XML**. Select *etc/intellij-java-modified-google-style.xml* from Geode repository root, enter **To:** *GeodeStyle*, check **Current scheme** and press **OK**. - -4. Make Apache the default Copyright. - - Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright**. Select *Apache* in drop-down box **Default project copyright**. - - To define *Apache* in **Copyright**, navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright -> Copyright Profiles**. Click **+** to add a new project. Enter *Apache* as the **Name** and enter the following block without asterisks or leading spaces: - - ```text - Licensed to the Apache Software Foundation (ASF) under one or more contributor license - agreements. See the NOTICE file distributed with this work for additional information regarding - copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance with the License. You may obtain a - copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software distributed under the License - is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - or implied. See the License for the specific language governing permissions and limitations under - the License. - ``` - - ...then return to **Copyright** and select *Apache* in drop-down box **Default project copyright**. - - Navigate to **IntelliJ IDEA -> Preferences... -> Editor -> Copyright -> Formatting**. Uncheck **Add blank line after** and press **OK**. - -5. Rebuild the Project. - - Navigate to **Build -> Rebuild Project** and the full project should compile without errors. +The following steps have been tested with **IntelliJ IDEA 2020.3.3** + +1. Run `./gradlew --parallel generate` from Geode repository root to create compiler generated + source. + +1. Import the project into IntelliJ IDEA. + + 1. Select **File -> Open...** from the menu. + 1. Select the `build.gradle` file in the Geode repository root and select **Open**. + 1. In the **Open Project?** popup, select **Open Project**. + 1. In the **Trust and Open Gradle Project?** popup, select **Trust Project**. + 1. Wait for IntelliJ to import the project and complete its background tasks. + +1. Configure IntelliJ IDEA to build and run the project and tests. + * Set the Java SDK for the project. + 1. Select **File -> Project Structure...** from the menu. + 1. Open the **Project Settings -> Project** section. + 1. Set **Project SDK** to your most recent Java 1.8 JDK. + + * To automatically re-generate sources when needed (recommended). + 1. Select **View -> Tool Windows -> Gradle** from the menu. + 1. In the Gradle dockable, open **geode -> Tasks -> build**. + 1. Right click the **generate** task and select **Execute Before Sync**. + 1. Right click the **generate** task and select **Execute Before Build**. + + * To reload the project when build scripts change (recommended). + 1. Select **IntelliJ IDEA -> Preferences...** from the menu. + 1. Open the **Build, Execution, Deployment -> Build Tools** section. + 1. Set **Reload project after changes in the build scripts:** to **Any changes**. + + * To build and run with Gradle (recommended). + 1. Select **IntelliJ IDEA -> Preferences...** from the menu. + 1. Open the **Build, Execution, Deployment -> Build Tools -> Gradle** section. + 1. Set **Build and run using:** to **Gradle**. + 1. Set **Run tests using:** to **Gradle**. + +1. Set the Code Style Scheme to GeodeStyle. + + 1. Select **IntelliJ IDEA -> Preferences...** + 1. Open the **Editor -> Code Style** section. + 1. If *GeodeStyle* style does not appear in the **Scheme** drop-down box + 1. Select the gear icon next to the drop-down. + 1. Select **Import Scheme -> IntelliJ IDEA code style XML**. + 1. Select `etc/intellij-java-modified-google-style.xml` from the Geode repository root. + 1. Enter **To:** *GeodeStyle*, check **Current scheme**, and press **OK**. + 1. Select *GeodeStyle* in **Scheme** drop-down box. + +1. Make Apache the default Copyright. + + 1. Select **IntelliJ IDEA -> Preferences...** from the menu. + 1. Open the **Editor -> Copyright** section. + 1. If *Apache* does not appear in the **Default project copyright** drop-down box: + 1. Open the **Copyright Profiles** subsection. + 1. Select the "import" icon (the small arrow pointing down and to the left) from the + Copyright Profiles section's toolbar. + 1. Select `etc/intellij-apache-copyright-notice.xml` from the Geode repository root. + 1. Return to the **Copyright** section. + 1. Select *Apache* in the **Default project copyright** drop-down box. + 1. Open the **Formatting** subsection. + 1. Uncheck **Add blank line after** and select **OK**. + +1. Rebuild the Project. + + 1. Select **Build -> Rebuild Project** from the menu. The full project should compile without + errors. Some optional sanity tests to make sure things are working properly: - * Try looking up classes using **Navigate -> Class...** - * Open and run a distributed test such as BasicDistributedTest in geode-core. - * Create a new java class and ensure the Apache license is automatically added to the top of the file with no blank line before the package line. + * Try looking up classes using **Navigate -> Class...** + * Open and run a distributed test such as BasicDistributedTest in geode-core. + * Create a new java class and ensure the Apache license is automatically added to the top of the + file with no blank line before the package line. diff --git a/boms/geode-all-bom/src/test/resources/expected-pom.xml b/boms/geode-all-bom/src/test/resources/expected-pom.xml index c7d889068c90..93e322913458 100644 --- a/boms/geode-all-bom/src/test/resources/expected-pom.xml +++ b/boms/geode-all-bom/src/test/resources/expected-pom.xml @@ -16,6 +16,11 @@ See the License for the specific language governing permissions and limitations under the License. --> + + + + + 4.0.0 org.apache.geode geode-all-bom @@ -41,823 +46,686 @@ antlr antlr 2.7.7 - compile cglib cglib 3.3.0 - compile com.arakelian java-jq 1.1.0 - compile com.carrotsearch.randomizedtesting randomizedtesting-runner 2.7.8 - compile com.github.davidmoten geo 0.7.7 - compile com.github.stefanbirkner system-rules 1.19.0 - compile com.github.stephenc.findbugs findbugs-annotations 1.3.9-1 - compile com.google.code.findbugs jsr305 3.0.2 - compile com.google.guava guava 30.1-jre - compile com.healthmarketscience.rmiio rmiio 2.1.2 - compile com.mockrunner mockrunner-servlet 2.0.6 - compile com.nimbusds nimbus-jose-jwt 8.11 - compile com.nimbusds oauth2-oidc-sdk 8.9 - compile com.sun.istack istack-commons-runtime 4.0.0 - compile com.sun.mail javax.mail 1.6.2 - compile com.sun.xml.bind jaxb-impl 2.3.2 - compile com.tngtech.archunit archunit-junit4 0.15.0 - compile com.zaxxer HikariCP 4.0.3 - compile commons-beanutils commons-beanutils 1.9.4 - compile commons-codec commons-codec 1.15 - compile commons-collections commons-collections 3.2.2 - compile commons-configuration commons-configuration 1.10 - compile commons-digester commons-digester 2.1 - compile commons-fileupload commons-fileupload 1.4 - compile commons-io commons-io 2.8.0 - compile commons-logging commons-logging 1.2 - compile commons-modeler commons-modeler 2.0.1 - compile commons-validator commons-validator 1.7 - compile io.github.classgraph classgraph 4.8.52 - compile io.micrometer micrometer-core 1.6.5 - compile io.netty netty-all 4.1.59.Final - compile io.swagger swagger-annotations 1.6.2 - compile it.unimi.dsi fastutil 8.5.2 - compile javax.annotation javax.annotation-api 1.3.2 - compile javax.annotation jsr250-api 1.0 - compile javax.ejb ejb-api 3.0 - compile javax.mail javax.mail-api 1.6.2 - compile javax.resource javax.resource-api 1.7.1 - compile javax.servlet javax.servlet-api 3.1.0 - compile javax.xml.bind jaxb-api 2.3.1 - compile joda-time joda-time 2.10.9 - compile junit junit 4.13.2 - compile mx4j mx4j-tools 3.0.1 - compile mysql mysql-connector-java 5.1.46 - compile net.java.dev.jna jna 5.7.0 - compile net.java.dev.jna jna-platform 5.7.0 - compile net.openhft compiler 2.4.1 - compile net.sf.jopt-simple jopt-simple 5.0.4 - compile net.sourceforge.pmd pmd-java 6.32.0 - compile net.sourceforge.pmd pmd-test 6.32.0 - compile net.spy spymemcached 2.12.3 - compile org.apache.bcel bcel 6.5.0 - compile org.apache.commons commons-lang3 3.12.0 - compile org.apache.commons commons-text 1.9 - compile org.apache.derby derby 10.14.2.0 - compile org.apache.httpcomponents httpclient 4.5.13 - compile org.apache.httpcomponents httpcore 4.4.14 - compile org.apache.shiro shiro-core 1.7.1 - compile org.assertj assertj-core 3.19.0 - compile org.awaitility awaitility 4.0.3 - compile org.bouncycastle bcpkix-jdk15on 1.68 - compile org.codehaus.cargo cargo-core-uberjar 1.9.2 - compile org.eclipse.jetty jetty-server 9.4.39.v20210325 - compile org.eclipse.jetty jetty-webapp 9.4.39.v20210325 - compile org.eclipse.persistence javax.persistence 2.2.1 - compile org.httpunit httpunit 1.7.3 - compile org.iq80.snappy snappy 0.4 - compile org.jboss.modules jboss-modules 1.11.0.Final - compile org.jgroups jgroups 3.6.14.Final - compile org.mockito mockito-core 3.8.0 - compile org.mortbay.jetty servlet-api 3.0.20100224 - compile org.openjdk.jmh jmh-core 1.26 - compile org.postgresql postgresql 42.2.8 - compile org.skyscreamer jsonassert 1.5.0 - compile org.slf4j slf4j-api 1.7.30 - compile org.springframework.hateoas spring-hateoas 1.2.5 - compile org.springframework.ldap spring-ldap-core 2.3.2.RELEASE - compile org.springframework.shell spring-shell 1.2.0.RELEASE - compile org.testcontainers testcontainers 1.14.3 - compile pl.pragmatists JUnitParams 1.1.0 - compile redis.clients jedis 3.5.2 - compile io.lettuce lettuce-core 6.0.3.RELEASE - compile xerces xercesImpl 2.12.0 - compile com.fasterxml.jackson.core jackson-annotations 2.12.2 - compile com.fasterxml.jackson.core jackson-core 2.12.2 - compile com.fasterxml.jackson.core jackson-databind 2.12.2 - compile com.fasterxml.jackson.datatype jackson-datatype-joda 2.12.2 - compile com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.12.2 - compile com.jayway.jsonpath json-path-assert 2.5.0 - compile com.jayway.jsonpath json-path 2.5.0 - compile com.palantir.docker.compose docker-compose-rule-core 0.31.1 - compile com.palantir.docker.compose docker-compose-rule-junit4 0.31.1 - compile com.pholser junit-quickcheck-core 1.0 - compile com.pholser junit-quickcheck-generators 1.0 - compile io.springfox springfox-swagger-ui 2.9.2 - compile io.springfox springfox-swagger2 2.9.2 - compile mx4j mx4j-remote 3.0.2 - compile mx4j mx4j 3.0.2 - compile org.apache.logging.log4j log4j-api 2.14.1 - compile org.apache.logging.log4j log4j-core 2.14.1 - compile org.apache.logging.log4j log4j-jcl 2.14.1 - compile org.apache.logging.log4j log4j-jul 2.14.1 - compile org.apache.logging.log4j log4j-slf4j-impl 2.14.1 - compile org.apache.lucene lucene-analyzers-common 6.6.6 - compile org.apache.lucene lucene-analyzers-phonetic 6.6.6 - compile org.apache.lucene lucene-core 6.6.6 - compile org.apache.lucene lucene-queryparser 6.6.6 - compile org.apache.lucene lucene-test-framework 6.6.6 - compile org.hamcrest hamcrest 2.2 - compile org.seleniumhq.selenium selenium-api 3.141.59 - compile org.seleniumhq.selenium selenium-chrome-driver 3.141.59 - compile org.seleniumhq.selenium selenium-remote-driver 3.141.59 - compile org.seleniumhq.selenium selenium-support 3.141.59 - compile org.springframework.security spring-security-config 5.4.5 - compile org.springframework.security spring-security-core 5.4.5 - compile org.springframework.security spring-security-ldap 5.4.5 - compile org.springframework.security spring-security-test 5.4.5 - compile org.springframework.security spring-security-web 5.4.5 - compile org.springframework.security spring-security-oauth2-core 5.4.5 - compile org.springframework.security spring-security-oauth2-client 5.4.5 - compile org.springframework.security spring-security-oauth2-jose 5.4.5 - compile org.springframework spring-aspects 5.3.5 - compile org.springframework spring-beans 5.3.5 - compile org.springframework spring-context 5.3.5 - compile org.springframework spring-core 5.3.5 - compile org.springframework spring-expression 5.3.5 - compile org.springframework spring-oxm 5.3.5 - compile org.springframework spring-test 5.3.5 - compile org.springframework spring-tx 5.3.5 - compile org.springframework spring-web 5.3.5 - compile org.springframework spring-webmvc 5.3.5 - compile org.springframework.boot spring-boot-starter 2.4.3 - compile org.springframework.boot spring-boot-starter-jetty 2.4.3 - compile org.springframework.boot spring-boot-starter-web 2.4.3 - compile org.springframework.boot spring-boot-starter-data-redis 2.4.3 - compile org.springframework.session spring-session-data-redis 2.4.2 - compile org.jetbrains annotations 20.1.0 - compile org.apache.geode @@ -895,12 +763,6 @@ ${version} compile - - org.apache.geode - geode-deployment-legacy - ${version} - compile - org.apache.geode geode-dunit @@ -1045,6 +907,12 @@ ${version} compile + + org.apache.geode + geode-deployment-legacy + ${version} + compile + org.apache.geode geode-lucene-test diff --git a/build.gradle b/build.gradle index 66e8644ca2b0..349f664fc21e 100755 --- a/build.gradle +++ b/build.gradle @@ -21,16 +21,16 @@ plugins { id "base" id "idea" id "eclipse" - id "com.diffplug.spotless" version "5.10.2" apply false - id "com.github.ben-manes.versions" version "0.36.0" apply false - id "nebula.lint" version "16.17.0" apply false + id "com.diffplug.spotless" version "5.11.1" apply false + id "com.github.ben-manes.versions" version "0.38.0" apply false + id "nebula.lint" version "16.17.1" apply false id "com.palantir.docker" version "0.26.0" apply false id "io.spring.dependency-management" version "1.0.11.RELEASE" apply false id "org.ajoberstar.grgit" version "4.1.0" apply false - id "org.nosphere.apache.rat" version "0.6.0" apply false + id "org.nosphere.apache.rat" version "0.7.0" apply false id "org.sonarqube" version "3.1.1" apply false id "me.champeau.gradle.japicmp" apply false // Version defined in buildSrc/build.gradle - id 'me.champeau.gradle.jmh' version '0.5.2' apply false + id 'me.champeau.gradle.jmh' version '0.5.3' apply false } diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DefaultWorkerSemaphore.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DefaultWorkerSemaphore.groovy deleted file mode 100644 index a42c4a1d74df..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DefaultWorkerSemaphore.groovy +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import org.gradle.api.Project -import org.gradle.api.tasks.testing.Test - -import java.util.concurrent.Semaphore - -class DefaultWorkerSemaphore implements WorkerSemaphore { - private int maxWorkers = Integer.MAX_VALUE - private Semaphore semaphore - private logger - - @Override - void acquire() { - semaphore().acquire() - logger.debug("Semaphore acquired, available: {}/{}", semaphore().availablePermits(), maxWorkers) - } - - @Override - void release() { - semaphore().release() - logger.debug("Semaphore released, available: {}/{}", semaphore().availablePermits(), maxWorkers) - } - - @Override - synchronized void applyTo(Project project) { - if (semaphore) return - if (!logger) { - logger = project.logger - } - - maxWorkers = project.tasks.withType(Test).findAll { - it.extensions.docker?.image != null - }.collect { - def v = it.maxParallelForks - it.maxParallelForks = 10000 - v - }.min() ?: 1 - semaphore() - } - - private synchronized setMaxWorkers(int num) { - if (this.@maxWorkers > num) { - this.@maxWorkers = num - } - } - - private synchronized Semaphore semaphore() { - if (semaphore == null) { - semaphore = new Semaphore(maxWorkers) - logger.lifecycle("Do not allow more than {} test workers", maxWorkers) - } - semaphore - } -} diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedJavaExecHandleBuilder.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedJavaExecHandleBuilder.groovy deleted file mode 100644 index 8f4e04c433d1..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedJavaExecHandleBuilder.groovy +++ /dev/null @@ -1,108 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import com.pedjak.gradle.plugins.dockerizedtest.DockerizedTestExtension -import com.pedjak.gradle.plugins.dockerizedtest.ExitCodeTolerantExecHandle -import com.pedjak.gradle.plugins.dockerizedtest.WorkerSemaphore -import org.gradle.api.internal.file.FileCollectionFactory -import org.gradle.api.internal.file.FileResolver -import org.gradle.initialization.BuildCancellationToken -import org.gradle.process.internal.* -import org.gradle.process.internal.streams.OutputStreamsForwarder - -import java.util.concurrent.Executor - -class DockerizedJavaExecHandleBuilder extends JavaExecHandleBuilder { - protected final FileCollectionFactory fileCollectionFactory - - def streamsHandler - def executor - def buildCancellationToken - private final DockerizedTestExtension extension - - private final WorkerSemaphore workersSemaphore - - DockerizedJavaExecHandleBuilder(DockerizedTestExtension extension, - FileResolver fileResolver, - FileCollectionFactory fileCollectionFactory, - Executor executor, - BuildCancellationToken buildCancellationToken, - WorkerSemaphore workersSemaphore) { - super(fileResolver, fileCollectionFactory, executor, buildCancellationToken) - this.fileCollectionFactory = fileCollectionFactory - this.extension = extension - this.executor = executor - this.buildCancellationToken = buildCancellationToken - this.workersSemaphore = workersSemaphore - } - - def StreamsHandler getStreamsHandler() { - StreamsHandler effectiveHandler; - if (this.streamsHandler != null) { - effectiveHandler = this.streamsHandler; - } else { - boolean shouldReadErrorStream = !redirectErrorStream; - effectiveHandler = new OutputStreamsForwarder(standardOutput, errorOutput, shouldReadErrorStream); - } - return effectiveHandler; - } - - ExecHandle build() { - - return new ExitCodeTolerantExecHandle(new DockerizedExecHandle(extension, getDisplayName(), - getWorkingDir(), - 'java', - allArguments, - getActualEnvironment(), - getStreamsHandler(), - inputHandler, - listeners, - redirectErrorStream, - timeoutMillis, - daemon, - executor, - buildCancellationToken), - workersSemaphore) - - } - - def timeoutMillis = Integer.MAX_VALUE - - @Override - AbstractExecHandleBuilder setTimeout(int timeoutMillis) { - this.timeoutMillis = timeoutMillis - return super.setTimeout(timeoutMillis) - } - - boolean redirectErrorStream - - @Override - AbstractExecHandleBuilder redirectErrorStream() { - redirectErrorStream = true - return super.redirectErrorStream() - } - - def listeners = [] - - @Override - AbstractExecHandleBuilder listener(ExecHandleListener listener) { - listeners << listener - return super.listener(listener) - } - -} diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestExtension.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestExtension.groovy deleted file mode 100644 index 8a007f117fae..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestExtension.groovy +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import com.github.dockerjava.api.DockerClient - -class DockerizedTestExtension { - - String image - Map volumes - String user - - Closure beforeContainerCreate - - Closure afterContainerCreate - - Closure beforeContainerStart - - Closure afterContainerStart - - Closure afterContainerStop = { containerId, client -> - try { - client.removeContainerCmd(containerId).exec(); - } catch (Exception e) { - // ignore any error - } - } - - // could be a DockerClient instance or a closure that returns a DockerClient instance - private def clientOrClosure - - void setClient(clientOrClosure) { - this.clientOrClosure = clientOrClosure - } - - DockerClient getClient() { - if (clientOrClosure == null) return null - if (DockerClient.class.isAssignableFrom(clientOrClosure.getClass())) { - return (DockerClient) clientOrClosure; - } else { - return (DockerClient) ((Closure) clientOrClosure).call(); - } - } -} diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestPlugin.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestPlugin.groovy deleted file mode 100644 index 176fa58cf930..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/DockerizedTestPlugin.groovy +++ /dev/null @@ -1,188 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import com.github.dockerjava.api.DockerClient -import com.github.dockerjava.core.DefaultDockerClientConfig -import com.github.dockerjava.core.DockerClientBuilder -import com.github.dockerjava.netty.NettyDockerCmdExecFactory -import org.apache.commons.lang3.SystemUtils -import org.apache.maven.artifact.versioning.ComparableVersion -import org.gradle.api.Action -import org.gradle.api.GradleException -import org.gradle.api.Plugin -import org.gradle.api.Project -import org.gradle.api.internal.file.DefaultFileCollectionFactory -import org.gradle.api.tasks.testing.Test -import org.gradle.initialization.DefaultBuildCancellationToken -import org.gradle.internal.concurrent.DefaultExecutorFactory -import org.gradle.internal.concurrent.ExecutorFactory -import org.gradle.internal.operations.BuildOperationExecutor -import org.gradle.internal.remote.Address -import org.gradle.internal.remote.ConnectionAcceptor -import org.gradle.internal.remote.MessagingServer -import org.gradle.internal.remote.ObjectConnection -import org.gradle.internal.remote.internal.ConnectCompletion -import org.gradle.internal.remote.internal.IncomingConnector -import org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection -import org.gradle.internal.remote.internal.inet.MultiChoiceAddress -import org.gradle.internal.time.Clock -import org.gradle.process.internal.JavaExecHandleFactory -import org.gradle.process.internal.worker.DefaultWorkerProcessFactory - -import javax.inject.Inject - -class DockerizedTestPlugin implements Plugin { - - def supportedVersion = '4.8' - def currentUser - def messagingServer - def static workerSemaphore = new DefaultWorkerSemaphore() - def memoryManager = new com.pedjak.gradle.plugins.dockerizedtest.NoMemoryManager() - - @Inject - DockerizedTestPlugin(MessagingServer messagingServer) { - this.currentUser = SystemUtils.IS_OS_WINDOWS ? "0" : "id -u".execute().text.trim() - this.messagingServer = new MessageServer(messagingServer.connector, messagingServer.executorFactory) - } - - void configureTest(project, test) { - def ext = test.extensions.create("docker", DockerizedTestExtension, [] as Object[]) - def startParameter = project.gradle.startParameter - ext.volumes = ["$startParameter.gradleUserHomeDir": "$startParameter.gradleUserHomeDir", - "$project.projectDir" : "$project.projectDir"] - ext.user = currentUser - test.doFirst { - def extension = test.extensions.docker - - if (extension?.image) { - - workerSemaphore.applyTo(test.project) - test.testExecuter = new com.pedjak.gradle.plugins.dockerizedtest.TestExecuter(newProcessBuilderFactory(project, extension, test.processBuilderFactory), actorFactory, moduleRegistry, services.get(BuildOperationExecutor), services.get(Clock)); - - if (!extension.client) { - extension.client = createDefaultClient() - } - } - - } - } - - DockerClient createDefaultClient() { - DockerClientBuilder.getInstance(DefaultDockerClientConfig.createDefaultConfigBuilder()) - .withDockerCmdExecFactory(new NettyDockerCmdExecFactory()) - .build() - } - - void apply(Project project) { - - boolean unsupportedVersion = new ComparableVersion(project.gradle.gradleVersion).compareTo(new ComparableVersion(supportedVersion)) < 0 - if (unsupportedVersion) throw new GradleException("dockerized-test plugin requires Gradle ${supportedVersion}+") - - project.tasks.withType(Test).each { test -> configureTest(project, test) } - project.tasks.whenTaskAdded { task -> - if (task instanceof Test) configureTest(project, task) - } - } - - def newProcessBuilderFactory(project, extension, defaultProcessBuilderFactory) { - - def executorFactory = new DefaultExecutorFactory() - def executor = executorFactory.create("Docker container link") - def buildCancellationToken = new DefaultBuildCancellationToken() - - def defaultfilecollectionFactory = new DefaultFileCollectionFactory(project.fileResolver, null) - def execHandleFactory = [newJavaExec: { -> - new DockerizedJavaExecHandleBuilder( - extension, project.fileResolver, defaultfilecollectionFactory, - executor, buildCancellationToken, workerSemaphore) - }] as JavaExecHandleFactory - new DefaultWorkerProcessFactory(defaultProcessBuilderFactory.loggingManager, - messagingServer, - defaultProcessBuilderFactory.workerImplementationFactory.classPathRegistry, - defaultProcessBuilderFactory.idGenerator, - defaultProcessBuilderFactory.gradleUserHomeDir, - defaultProcessBuilderFactory.workerImplementationFactory.temporaryFileProvider, - execHandleFactory, - defaultProcessBuilderFactory.workerImplementationFactory.jvmVersionDetector, - defaultProcessBuilderFactory.outputEventListener, - memoryManager - ) - } - - class MessageServer implements MessagingServer { - def IncomingConnector connector; - def ExecutorFactory executorFactory; - - public MessageServer(IncomingConnector connector, ExecutorFactory executorFactory) { - this.connector = connector; - this.executorFactory = executorFactory; - } - - public ConnectionAcceptor accept(Action action) { - return new ConnectionAcceptorDelegate(connector.accept(new ConnectEventAction(action, executorFactory), true)) - } - - - } - - class ConnectEventAction implements Action { - def action; - def executorFactory; - - public ConnectEventAction(Action action, executorFactory) { - this.executorFactory = executorFactory - this.action = action - } - - public void execute(ConnectCompletion completion) { - action.execute(new MessageHubBackedObjectConnection(executorFactory, completion)); - } - } - - class ConnectionAcceptorDelegate implements ConnectionAcceptor { - - MultiChoiceAddress address - - @Delegate - ConnectionAcceptor delegate - - ConnectionAcceptorDelegate(ConnectionAcceptor delegate) { - this.delegate = delegate - } - - Address getAddress() { - synchronized (delegate) - { - if (address == null) { - def remoteAddresses = NetworkInterface.networkInterfaces.findAll { - try { - return it.up && !it.loopback - } catch (SocketException ex) { - logger.warn("Unable to inspect interface " + it) - return false - } - }*.inetAddresses*.collect { it }.flatten() - def original = delegate.address - address = new MultiChoiceAddress(original.canonicalAddress, original.port, remoteAddresses) - } - } - address - } - } - -} diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/ExitCodeTolerantExecHandle.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/ExitCodeTolerantExecHandle.groovy deleted file mode 100644 index ab1c6f119c2f..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/ExitCodeTolerantExecHandle.groovy +++ /dev/null @@ -1,92 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import com.pedjak.gradle.plugins.dockerizedtest.WorkerSemaphore -import org.gradle.process.ExecResult -import org.gradle.process.internal.ExecException -import org.gradle.process.internal.ExecHandle -import org.gradle.process.internal.ExecHandleListener - -/** - * All exit codes are normal - */ -class ExitCodeTolerantExecHandle implements ExecHandle { - - private final WorkerSemaphore testWorkerSemaphore - - @Delegate - private final ExecHandle delegate - - ExitCodeTolerantExecHandle(ExecHandle delegate, WorkerSemaphore testWorkerSemaphore) { - this.delegate = delegate - this.testWorkerSemaphore = testWorkerSemaphore - delegate.addListener(new ExecHandleListener() { - - @Override - void executionStarted(ExecHandle execHandle) { - // do nothing - } - - @Override - void executionFinished(ExecHandle execHandle, ExecResult execResult) { - testWorkerSemaphore.release() - } - }) - } - - ExecHandle start() { - testWorkerSemaphore.acquire() - try { - delegate.start() - } catch (Exception e) { - testWorkerSemaphore.release() - throw e - } - } - - private static class ExitCodeTolerantExecResult implements ExecResult { - - @Delegate - private final ExecResult delegate - - ExitCodeTolerantExecResult(ExecResult delegate) { - this.delegate = delegate - } - - ExecResult assertNormalExitValue() throws ExecException { - // no op because we are perfectly ok if the exit code is anything - // because Docker can complain about not being able to remove the used image - // although the tests completed fine - this - } - } - - private static class ExecHandleListenerFacade implements ExecHandleListener { - - @Delegate - private final ExecHandleListener delegate - - ExecHandleListenerFacade(ExecHandleListener delegate) { - this.delegate = delegate - } - - void executionFinished(ExecHandle execHandle, ExecResult execResult) { - delegate.executionFinished(execHandle, new ExitCodeTolerantExecResult(execResult)) - } - } -} diff --git a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/WorkerSemaphore.groovy b/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/WorkerSemaphore.groovy deleted file mode 100644 index 0268088a434a..000000000000 --- a/buildSrc/src/main/groovy/com/pedjak/gradle/plugins/dockerizedtest/WorkerSemaphore.groovy +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest - -import org.gradle.api.Project - -interface WorkerSemaphore { - - void acquire() - - void release() - - void applyTo(Project project) -} \ No newline at end of file diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/RepeatTest.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/RepeatTest.groovy deleted file mode 100644 index 71231016b400..000000000000 --- a/buildSrc/src/main/groovy/org/apache/geode/gradle/RepeatTest.groovy +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more contributor license - * agreements. See the NOTICE file distributed with this work for additional information regarding - * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. You may obtain a - * copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ -package org.apache.geode.gradle - -import org.gradle.StartParameter -import org.gradle.api.file.FileTree -import org.gradle.api.internal.DocumentationRegistry -import org.gradle.api.internal.tasks.testing.JvmTestExecutionSpec -import org.gradle.api.internal.tasks.testing.TestExecuter -import org.gradle.api.internal.tasks.testing.detection.DefaultTestExecuter -import org.gradle.api.internal.tasks.testing.filter.DefaultTestFilter -import org.gradle.api.tasks.testing.Test -import org.gradle.internal.operations.BuildOperationExecutor -import org.gradle.internal.time.Clock -import org.gradle.internal.work.WorkerLeaseRegistry - -class RepeatTest extends Test { - int times = 1 - - @Override - FileTree getCandidateClassFiles() { - FileTree candidates = super.getCandidateClassFiles() - int additionalRuns = times - 1 - for (int i = 0; i < additionalRuns; i++) { - candidates = candidates.plus(super.getCandidateClassFiles()) - } - - return candidates - } - - /* - * We have to override gradles default test executor, because that uses {@link RunPreviousFailedFirstTestClassProcessor} - * Which deduplicates the test specs we're passing in - */ - @Override - protected TestExecuter createTestExecuter() { - def oldExecutor = super.createTestExecuter() - - //Use the previously set worker process factory. If the test is - //being run using the parallel docker plugin, this will be a docker - //process factory - def workerProcessFactory = oldExecutor.workerFactory - - return new OverriddenTestExecutor(workerProcessFactory, getActorFactory(), - getModuleRegistry(), - getServices().get(WorkerLeaseRegistry.class), - getServices().get(BuildOperationExecutor.class), - getServices().get(StartParameter.class).getMaxWorkerCount(), - getServices().get(Clock.class), - getServices().get(DocumentationRegistry.class), - (DefaultTestFilter) getFilter()) - } - -} diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Executers.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Executers.groovy new file mode 100644 index 000000000000..01d33ca16aca --- /dev/null +++ b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Executers.groovy @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing + +import org.apache.geode.gradle.testing.process.AdjustableProcessLauncher +import org.gradle.StartParameter +import org.gradle.api.internal.DocumentationRegistry +import org.gradle.api.internal.tasks.testing.JvmTestExecutionSpec +import org.gradle.api.internal.tasks.testing.TestExecuter +import org.gradle.api.internal.tasks.testing.detection.DefaultTestExecuter +import org.gradle.internal.time.Clock +import org.gradle.internal.work.WorkerLeaseRegistry + +class Executers { + /** + * Creates a {@code TestExecuter} that applies an adjustment to each test worker + * {@link ProcessBuilder} just before launching the process. + */ + static TestExecuter withAdjustment(testTask, adjustment) { + def gradleWorkerProcessFactory = testTask.createTestExecuter().workerFactory + def isolatingWorkerProcessFactory = Workers.createWorkerProcessFactory( + gradleWorkerProcessFactory, + new AdjustableProcessLauncher(adjustment), + gradleWorkerProcessFactory.server) + return withFactory(testTask, isolatingWorkerProcessFactory) + } + + /** + * Creates a {@code TestExecuter} that uses the given factory to create test worker processes. + */ + static TestExecuter withFactory(testTask, workerProcessFactory) { + def services = testTask.services + return new DefaultTestExecuter( + workerProcessFactory, + testTask.actorFactory, + testTask.moduleRegistry, + services.get(WorkerLeaseRegistry), + services.get(StartParameter).getMaxWorkerCount(), + services.get(Clock), + services.get(DocumentationRegistry), + testTask.filter + ) + } + +} diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Workers.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Workers.groovy new file mode 100644 index 000000000000..5d0b46fd2091 --- /dev/null +++ b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/Workers.groovy @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing + +import org.apache.geode.gradle.testing.process.LauncherProxyWorkerProcessFactory +import org.apache.geode.gradle.testing.process.ProcessLauncher +import org.gradle.internal.remote.MessagingServer +import org.gradle.process.internal.worker.WorkerProcessFactory + +class Workers { + /** + * Creates a worker process factory that borrows most components from the donor, but uses the + * given process launcher and messaging server to launch worker processes. + */ + static WorkerProcessFactory createWorkerProcessFactory( + WorkerProcessFactory donor, + ProcessLauncher processLauncher, + MessagingServer messagingServer) { + def workerImplementationFactory = donor.workerImplementationFactory + return new LauncherProxyWorkerProcessFactory( + donor.loggingManager, + messagingServer, + workerImplementationFactory.classPathRegistry, + donor.idGenerator, + workerImplementationFactory.gradleUserHomeDir, + workerImplementationFactory.temporaryFileProvider, + donor.execHandleFactory, + workerImplementationFactory.jvmVersionDetector, + donor.outputEventListener, + donor.memoryManager, + processLauncher) + } +} diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerTestWorkerConfig.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerTestWorkerConfig.groovy new file mode 100644 index 000000000000..61cca0615c8e --- /dev/null +++ b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerTestWorkerConfig.groovy @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.dockerized + +import org.gradle.api.Project + +class DockerTestWorkerConfig { + static long durationWarningThreshold = 60_000 + String image + String javaHome + String localUserID + String name + int timeoutMillis = 300_000 + String user + Map volumes = new HashMap<>() + + DockerTestWorkerConfig(Project project) { + name = project.path + image = project.dunitDockerImage + user = project.dunitDockerUser + + if (project.hasProperty('dunitDockerJVM') && !project.dunitDockerJVM.trim().isEmpty()) { + javaHome = project.dunitDockerJVM as String + } else if (project.hasProperty('testJVM') && !project.testJVM.trim().isEmpty()) { + javaHome = project.testJVM as String + } + + // Mount the user's Gradle home dir, the Geode project root directory, and any + // user-specified volumes. + def gradleUserHomeDir = project.gradle.startParameter.gradleUserHomeDir.getAbsolutePath() as String + def geodeDir = new File(System.getenv('PWD')).getCanonicalPath() + volumes = [(geodeDir) : geodeDir, + (gradleUserHomeDir): gradleUserHomeDir] + + if (project.hasProperty('dunitDockerVolumes')) { + volumes.putAll(project.dunitDockerVolumes) + } + + if (project.hasProperty("dunitDockerTimeout")) { + timeoutMillis = Integer.parseUnsignedInt(project.dunitDockerTimeout) + } + + // Unfortunately this snippet of code is here and is required by + // dev-tools/docker/base/entrypoint.sh. This allows preserving the outer user inside the + // running container. Required for Jenkins and other environments. There doesn't seem to be + // a way to pass this environment variable in from a Jenkins Gradle job. + if (System.env['LOCAL_USER_ID'] == null) { + def username = System.getProperty("user.name") + localUserID = ['id', '-u', username].execute().text.trim() as String + } + } + + /** + * Adjust the process builder's command and environment to run in a Docker container. + */ + def dockerize(processBuilder) { + def command = processBuilder.command() + def environment = processBuilder.environment() + + // The JAVA_HOME and PATH environment variables set by Gradle are meaningless inside a + // Docker container. Remove them. + if (environment['JAVA_HOME']) { + environment.remove 'JAVA_HOME' + environment['JAVA_HOME_REMOVED'] = "" + } + if (environment['PATH']) { + environment.remove 'PATH' + environment['PATH_REMOVED'] = "" + } + + if (javaHome) { + environment['JAVA_HOME'] = javaHome + command.set(0, "${javaHome}/bin/java" as String) + } + + if (localUserID) { + environment['LOCAL_USER_ID'] = localUserID + } + } +} diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerizedTestPlugin.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerizedTestPlugin.groovy new file mode 100644 index 000000000000..6f7c570caddd --- /dev/null +++ b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/dockerized/DockerizedTestPlugin.groovy @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.dockerized + + +import org.apache.geode.gradle.testing.Executers +import org.apache.geode.gradle.testing.Workers +import org.apache.geode.gradle.testing.isolation.WorkingDirectoryIsolator +import org.gradle.api.Plugin +import org.gradle.api.Project +import org.gradle.api.tasks.testing.Test +import org.gradle.internal.remote.MessagingServer +import org.gradle.internal.remote.internal.inet.TcpIncomingConnector +import org.gradle.process.internal.worker.WorkerProcessFactory + +import javax.inject.Inject + +/** + * Extends each test task with a configuration that it can use to launch its test workers in Docker + * containers. The /gradle/multi-process-test.gradle file defines which test + * tasks actually apply the configuration to run test workers. + */ +class DockerizedTestPlugin implements Plugin { + /** + * A custom {@link MessagingServer} that supports communication between Gradle and processes + * running in Docker containers. + */ + def static dockerMessagingServer + /** + * The singleton {@link MessagingServer} created by Gradle. This plugin borrows the server's + * internal components to build a custom {@code MessagingServer} that can communicate with + * processes running in Docker containers. + */ + def static gradleMessagingServer + /** + * The singleton {@link WorkerProcessFactory} created by Gradle. This plugin borrows the + * factory's internal components to build a custom {@code WorkerProcessFactory} that launches + * processes in Docker containers. + */ + def static gradleWorkerProcessFactory + + /** + * The injected values are singletons. Gradle injects the same instances into each instance of + * this plugin. + *

+ * CAVEAT: The types of these parameters are declared internal by Gradle v6.8.3. Future + * versions of Gradle may not include these types, or may change their implementation. + *

+ * CAVEAT: The Gradle v6.8.3 documentation does not list these types among the services that + * Gradle will inject into plugins. Future versions of Gradle may not inject these values. + */ + @Inject + DockerizedTestPlugin(MessagingServer gradleMessagingServer, + WorkerProcessFactory gradleWorkerProcessFactory) { + initializeGradleWorkerProcessFactory(gradleWorkerProcessFactory) + initializeGradleMessagingServer(gradleMessagingServer) + } + + @Override + void apply(Project project) { + if (!project.hasProperty('parallelDunit')) { + return + } + + initializeMessagingServer() + + def dockerTestWorkerConfig = new DockerTestWorkerConfig(project) + def dockerProcessLauncher = new DockerProcessLauncher(dockerTestWorkerConfig, new WorkingDirectoryIsolator()) + def dockerWorkerProcessFactory = Workers.createWorkerProcessFactory( + gradleWorkerProcessFactory, + dockerProcessLauncher, + dockerMessagingServer) + + def useDockerTestWorker = { + it.doFirst { + testExecuter = Executers.withFactory(it, dockerWorkerProcessFactory) + } + } + + project.tasks.withType(Test).each(useDockerTestWorker) + project.tasks.whenTaskAdded() { + if (it instanceof Test) { + it.configure(useDockerTestWorker) + } + } + } + + synchronized static initializeGradleMessagingServer(server) { + if (!gradleMessagingServer) { + gradleMessagingServer = server + } + } + + synchronized static void initializeGradleWorkerProcessFactory(factory) { + if (!gradleWorkerProcessFactory) { + gradleWorkerProcessFactory = factory + } + } + + synchronized static void initializeMessagingServer() { + if (dockerMessagingServer) { + return + } + + def gradleConnector = gradleMessagingServer.connector + def gradleExecutorFactory = gradleConnector.executorFactory + def gradleIdGenerator = gradleConnector.idGenerator + + /** + * Use a custom {@link WildcardBindingInetAddressFactory} to allow connections from + * processes in Docker containers. + */ + def wildcardAddressFactory = new WildcardBindingInetAddressFactory() + def dockerConnector = new TcpIncomingConnector( + gradleExecutorFactory, + wildcardAddressFactory, + gradleIdGenerator + ) + /** + * Use a custom {@link DockerMessagingServer} that yields connection addresses usable + * by processes in Docker containers. + */ + dockerMessagingServer = new DockerMessagingServer(dockerConnector, gradleExecutorFactory) + } +} diff --git a/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/repeat/RepeatTest.groovy b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/repeat/RepeatTest.groovy new file mode 100644 index 000000000000..bafeab9f4f52 --- /dev/null +++ b/buildSrc/src/main/groovy/org/apache/geode/gradle/testing/repeat/RepeatTest.groovy @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.repeat + +import org.gradle.StartParameter +import org.gradle.api.file.FileTree +import org.gradle.api.internal.DocumentationRegistry +import org.gradle.api.internal.tasks.testing.JvmTestExecutionSpec +import org.gradle.api.internal.tasks.testing.TestExecuter +import org.gradle.api.internal.tasks.testing.filter.DefaultTestFilter +import org.gradle.api.tasks.testing.Test +import org.gradle.internal.time.Clock +import org.gradle.internal.work.WorkerLeaseRegistry + +class RepeatTest extends Test { + int times = 1 + + /** + * Submit each test class for processing multiple times. + */ + @Override + FileTree getCandidateClassFiles() { + FileTree candidates = super.getCandidateClassFiles() + int additionalRuns = times - 1 + for (int i = 0; i < additionalRuns; i++) { + candidates = candidates.plus(super.getCandidateClassFiles()) + } + + return candidates + } + + /** + * Use a custom {@link TestExecuter} that processes each test class as many times as submitted. + */ + @Override + protected TestExecuter createTestExecuter() { + return new RepeatableTestExecuter( + super.createTestExecuter().workerFactory, + getActorFactory(), + getModuleRegistry(), + getServices().get(WorkerLeaseRegistry.class), + getServices().get(StartParameter.class).getMaxWorkerCount(), + getServices().get(Clock.class), + getServices().get(DocumentationRegistry.class), + (DefaultTestFilter) getFilter()) + } +} diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandle.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandle.java deleted file mode 100755 index a3cf93426ce8..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandle.java +++ /dev/null @@ -1,689 +0,0 @@ -/* - * Copyright 2010 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import static java.lang.String.format; - -import java.io.File; -import java.io.InputStream; -import java.io.OutputStream; -import java.io.PipedInputStream; -import java.io.PipedOutputStream; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.concurrent.CountDownLatch; -import java.util.concurrent.Executor; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.atomic.AtomicInteger; -import java.util.concurrent.locks.Condition; -import java.util.concurrent.locks.Lock; -import java.util.concurrent.locks.ReentrantLock; - -import javax.annotation.Nullable; - -import com.github.dockerjava.api.DockerClient; -import com.github.dockerjava.api.command.CreateContainerCmd; -import com.github.dockerjava.api.model.Frame; -import com.github.dockerjava.api.model.StreamType; -import com.github.dockerjava.api.model.WaitResponse; -import com.github.dockerjava.core.command.AttachContainerResultCallback; -import com.github.dockerjava.core.command.WaitContainerResultCallback; -import com.github.dockerjava.api.model.Bind; -import com.github.dockerjava.api.model.Volume; -import com.google.common.base.Joiner; -import groovy.lang.Closure; -import org.gradle.api.logging.Logger; -import org.gradle.api.logging.Logging; -import org.gradle.initialization.BuildCancellationToken; -import org.gradle.internal.UncheckedException; -import org.gradle.internal.event.ListenerBroadcast; -import org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable; -import org.gradle.process.ExecResult; -import org.gradle.process.internal.ExecException; -import org.gradle.process.internal.ExecHandle; -import org.gradle.process.internal.ExecHandleListener; -import org.gradle.process.internal.ExecHandleShutdownHookAction; -import org.gradle.process.internal.ExecHandleState; -import org.gradle.process.internal.ProcessSettings; -import org.gradle.process.internal.StreamsHandler; -import org.gradle.process.internal.shutdown.ShutdownHooks; - -/** - * Default implementation for the ExecHandle interface. - * - *

State flows

- * - *
    - *
  • INIT -> STARTED -> [SUCCEEDED|FAILED|ABORTED|DETACHED]
  • - *
  • INIT -> FAILED
  • - *
  • INIT -> STARTED -> DETACHED -> ABORTED
  • - *
- * - * State is controlled on all control methods: - *
    - *
  • {@link #start()} allowed when state is INIT
  • - *
  • {@link #abort()} allowed when state is STARTED or DETACHED
  • - *
- */ -public class DockerizedExecHandle implements ExecHandle, ProcessSettings { - - private static final Logger LOGGER = Logging.getLogger(DockerizedExecHandle.class); - - private final String displayName; - - /** - * The working directory of the process. - */ - private final File directory; - - /** - * The executable to run. - */ - private final String command; - - /** - * Arguments to pass to the executable. - */ - private final List arguments; - - /** - * The variables to set in the environment the executable is run in. - */ - private final Map environment; - private final StreamsHandler outputHandler; - private final StreamsHandler inputHandler; - private final boolean redirectErrorStream; - private int timeoutMillis; - private boolean daemon; - - /** - * Lock to guard all mutable state - */ - private final Lock lock; - private final Condition stateChanged; - - private final Executor executor; - - /** - * State of this ExecHandle. - */ - private ExecHandleState state; - - /** - * When not null, the runnable that is waiting - */ - private DockerizedExecHandleRunner execHandleRunner; - - private ExecResultImpl execResult; - - private final ListenerBroadcast broadcast; - - private final ExecHandleShutdownHookAction shutdownHookAction; - - private final BuildCancellationToken buildCancellationToken; - - private final DockerizedTestExtension testExtension; - - public DockerizedExecHandle(DockerizedTestExtension testExtension, String displayName, - File directory, String command, List arguments, - Map environment, StreamsHandler outputHandler, - StreamsHandler inputHandler, - List listeners, boolean redirectErrorStream, - int timeoutMillis, boolean daemon, - Executor executor, BuildCancellationToken buildCancellationToken) { - this.displayName = displayName; - this.directory = directory; - this.command = command; - this.arguments = arguments; - this.environment = environment; - this.outputHandler = outputHandler; - this.inputHandler = inputHandler; - this.redirectErrorStream = redirectErrorStream; - this.timeoutMillis = timeoutMillis; - this.daemon = daemon; - this.executor = executor; - this.buildCancellationToken = buildCancellationToken; - this.testExtension = testExtension; - lock = new ReentrantLock(); - stateChanged = lock.newCondition(); - state = ExecHandleState.INIT; - shutdownHookAction = new ExecHandleShutdownHookAction(this); - broadcast = new ListenerBroadcast<>(ExecHandleListener.class); - broadcast.addAll(listeners); - } - - @Override - public File getDirectory() { - return directory; - } - - @Override - public String getCommand() { - return command; - } - - public boolean isDaemon() { - return daemon; - } - - @Override - public String toString() { - return displayName; - } - - @Override - public List getArguments() { - return Collections.unmodifiableList(arguments); - } - - @Override - public Map getEnvironment() { - return Collections.unmodifiableMap(environment); - } - - @Override - public ExecHandleState getState() { - lock.lock(); - try { - return state; - } finally { - lock.unlock(); - } - } - - private void setState(ExecHandleState state) { - lock.lock(); - try { - LOGGER.debug("Changing state to: {}", state); - this.state = state; - stateChanged.signalAll(); - } finally { - lock.unlock(); - } - } - - private boolean stateIn(ExecHandleState... states) { - lock.lock(); - try { - return Arrays.asList(states).contains(state); - } finally { - lock.unlock(); - } - } - - private void setEndStateInfo(ExecHandleState newState, int exitValue, Throwable failureCause) { - ShutdownHooks.removeShutdownHook(shutdownHookAction); - buildCancellationToken.removeCallback(shutdownHookAction); - ExecHandleState currentState; - lock.lock(); - try { - currentState = state; - } finally { - lock.unlock(); - } - - ExecResultImpl - newResult = - new ExecResultImpl(exitValue, execExceptionFor(failureCause, currentState), displayName); - if (!currentState.isTerminal() && newState != ExecHandleState.DETACHED) { - try { - broadcast.getSource().executionFinished(this, newResult); - } catch (Exception e) { - newResult = new ExecResultImpl(exitValue, execExceptionFor(e, currentState), displayName); - } - } - - lock.lock(); - try { - setState(newState); - execResult = newResult; - } finally { - lock.unlock(); - } - - LOGGER.debug("Process '{}' finished with exit value {} (state: {})", displayName, exitValue, - newState); - } - - @Nullable - private ExecException execExceptionFor(Throwable failureCause, ExecHandleState currentState) { - return failureCause != null - ? new ExecException(failureMessageFor(currentState), failureCause) - : null; - } - - private String failureMessageFor(ExecHandleState currentState) { - return currentState == ExecHandleState.STARTING - ? format("A problem occurred starting process '%s'", displayName) - : format("A problem occurred waiting for process '%s' to complete.", displayName); - } - - @Override - public ExecHandle start() { - LOGGER.info("Starting process '{}'. Working directory: {} Command: {}", - displayName, directory, command + ' ' + Joiner.on(' ').useForNull("null").join(arguments)); - if (LOGGER.isDebugEnabled()) { - LOGGER.debug("Environment for process '{}': {}", displayName, environment); - } - lock.lock(); - try { - if (!stateIn(ExecHandleState.INIT)) { - throw new IllegalStateException( - format("Cannot start process '%s' because it has already been started", displayName)); - } - setState(ExecHandleState.STARTING); - - execHandleRunner = - new DockerizedExecHandleRunner(this, new CompositeStreamsHandler(), executor); - executor.execute(new CurrentBuildOperationPreservingRunnable(execHandleRunner)); - - while (stateIn(ExecHandleState.STARTING)) { - LOGGER.debug("Waiting until process started: {}.", displayName); - try { - if (!stateChanged.await(10, TimeUnit.MINUTES)) { - execHandleRunner.abortProcess(); - throw new RuntimeException("Giving up on " + execHandleRunner); - } - } catch (InterruptedException e) { - //ok, wrapping up - } - } - - if (execResult != null) { - execResult.rethrowFailure(); - } - - LOGGER.info("Successfully started process '{}'", displayName); - } finally { - lock.unlock(); - } - return this; - } - - @Override - public void abort() { - lock.lock(); - try { - if (stateIn(ExecHandleState.SUCCEEDED, ExecHandleState.FAILED, ExecHandleState.ABORTED)) { - return; - } - if (!stateIn(ExecHandleState.STARTED, ExecHandleState.DETACHED)) { - throw new IllegalStateException( - format("Cannot abort process '%s' because it is not in started or detached state. It is currently in state: '%s'", - displayName,state)); - } - execHandleRunner.abortProcess(); - waitForFinish(); - } finally { - lock.unlock(); - } - } - - @Override - public ExecResult waitForFinish() { - lock.lock(); - try { - while (!state.isTerminal()) { - try { - stateChanged.await(); - } catch (InterruptedException e) { - //ok, wrapping up... - throw UncheckedException.throwAsUncheckedException(e); - } - } - } finally { - lock.unlock(); - } - - // At this point: - // If in daemon mode, the process has started successfully and all streams to the process have been closed - // If in fork mode, the process has completed and all cleanup has been done - // In both cases, all asynchronous work for the process has completed and we're done - - return result(); - } - - private ExecResult result() { - lock.lock(); - try { - return execResult.rethrowFailure(); - } finally { - lock.unlock(); - } - } - - void detached() { - setEndStateInfo(ExecHandleState.DETACHED, 0, null); - } - - void started() { - ShutdownHooks.addShutdownHook(shutdownHookAction); - buildCancellationToken.addCallback(shutdownHookAction); - setState(ExecHandleState.STARTED); - broadcast.getSource().executionStarted(this); - } - - void finished(int exitCode) { - if (exitCode != 0) { - setEndStateInfo(ExecHandleState.FAILED, exitCode, null); - } else { - setEndStateInfo(ExecHandleState.SUCCEEDED, 0, null); - } - } - - void aborted(int exitCode) { - if (exitCode == 0) { - // This can happen on Windows - exitCode = -1; - } - setEndStateInfo(ExecHandleState.ABORTED, exitCode, null); - } - - void failed(Throwable failureCause) { - setEndStateInfo(ExecHandleState.FAILED, -1, failureCause); - } - - @Override - public void addListener(ExecHandleListener listener) { - broadcast.add(listener); - } - - @Override - public void removeListener(ExecHandleListener listener) { - broadcast.remove(listener); - } - - public String getDisplayName() { - return displayName; - } - - @Override - public boolean getRedirectErrorStream() { - return redirectErrorStream; - } - - public int getTimeout() { - return timeoutMillis; - } - - public Process runContainer() { - try { - DockerClient client = testExtension.getClient(); - CreateContainerCmd createCmd = client.createContainerCmd(testExtension.getImage()) - .withTty(false) - .withStdinOpen(true) - .withWorkingDir(directory.getAbsolutePath()); - - createCmd.withEnv(getEnv()); - - String user = testExtension.getUser(); - if (user != null) { - createCmd.withUser(user); - } - bindVolumes(createCmd); - List cmdLine = new ArrayList<>(); - cmdLine.add(command); - cmdLine.addAll(arguments); - createCmd.withCmd(cmdLine); - - invokeIfNotNull(testExtension.getBeforeContainerCreate(), createCmd, client); - String containerId = createCmd.exec().getId(); - invokeIfNotNull(testExtension.getAfterContainerCreate(), containerId, client); - - invokeIfNotNull(testExtension.getBeforeContainerStart(), containerId, client); - client.startContainerCmd(containerId).exec(); - invokeIfNotNull(testExtension.getAfterContainerStart(), containerId, client); - - if (!client.inspectContainerCmd(containerId).exec().getState().getRunning()) { - throw new RuntimeException("Container " + containerId + " not running!"); - } - - Process - proc = - new DockerizedProcess(client, containerId, testExtension.getAfterContainerStop()); - - return proc; - } catch (Exception e) { - e.printStackTrace(); - throw new RuntimeException(e); - } - } - - private void invokeIfNotNull(Closure closure, Object... args) { - if (closure != null) { - int l = closure.getParameterTypes().length; - Object[] nargs; - if (l < args.length) { - nargs = new Object[l]; - System.arraycopy(args, 0, nargs, 0, l); - } else { - nargs = args; - } - closure.call(nargs); - } - } - - private List getEnv() { - List env = new ArrayList<>(); - for (Map.Entry e : environment.entrySet()) { - env.add(e.getKey() + "=" + e.getValue()); - } - return env; - } - - private void bindVolumes(CreateContainerCmd cmd) { - List volumes = new ArrayList<>(); - List binds = new ArrayList<>(); - for (Object o : testExtension.getVolumes().entrySet()) { - @SuppressWarnings("unchecked") - Map.Entry e = (Map.Entry) o; - Volume volume = new Volume(e.getValue().toString()); - Bind bind = new Bind(e.getKey().toString(), volume); - binds.add(bind); - volumes.add(volume); - } - cmd.withVolumes(volumes).withBinds(binds); - } - - private static class ExecResultImpl implements ExecResult { - private final int exitValue; - private final ExecException failure; - private final String displayName; - - ExecResultImpl(int exitValue, ExecException failure, String displayName) { - this.exitValue = exitValue; - this.failure = failure; - this.displayName = displayName; - } - - @Override - public int getExitValue() { - return exitValue; - } - - @Override - public ExecResult assertNormalExitValue() throws ExecException { - return this; - } - - @Override - public ExecResult rethrowFailure() throws ExecException { - if (failure != null) { - throw failure; - } - return this; - } - - @Override - public String toString() { - return "{exitValue=" + exitValue + ", failure=" + failure + "}"; - } - } - - private class CompositeStreamsHandler implements StreamsHandler { - @Override - public void connectStreams(Process process, String processName, Executor executor) { - inputHandler.connectStreams(process, processName, executor); - outputHandler.connectStreams(process, processName, executor); - } - - @Override - public void start() { - inputHandler.start(); - outputHandler.start(); - } - - @Override - public void stop() { - inputHandler.stop(); - outputHandler.stop(); - } - - @Override - public void disconnect() { - inputHandler.disconnect(); - outputHandler.disconnect(); - } - } - - private class DockerizedProcess extends Process { - - private final DockerClient dockerClient; - private final String containerId; - private final Closure afterContainerStop; - - private final PipedOutputStream stdInWriteStream = new PipedOutputStream(); - private final PipedInputStream stdOutReadStream = new PipedInputStream(); - private final PipedInputStream stdErrReadStream = new PipedInputStream(); - private final PipedInputStream stdInReadStream = new PipedInputStream(stdInWriteStream); - private final PipedOutputStream stdOutWriteStream = new PipedOutputStream(stdOutReadStream); - private final PipedOutputStream stdErrWriteStream = new PipedOutputStream(stdErrReadStream); - - private final CountDownLatch finished = new CountDownLatch(1); - private AtomicInteger exitCode = new AtomicInteger(); - private final AttachContainerResultCallback - attachContainerResultCallback = - new AttachContainerResultCallback() { - @Override - public void onNext(Frame frame) { - try { - if (frame.getStreamType().equals(StreamType.STDOUT)) { - stdOutWriteStream.write(frame.getPayload()); - } else if (frame.getStreamType().equals(StreamType.STDERR)) { - stdErrWriteStream.write(frame.getPayload()); - } - } catch (Exception e) { - LOGGER.error("Error while writing to stream:", e); - } - super.onNext(frame); - } - }; - - private final WaitContainerResultCallback - waitContainerResultCallback = - new WaitContainerResultCallback() { - @Override - public void onNext(WaitResponse waitResponse) { - exitCode.set(waitResponse.getStatusCode()); - try { - attachContainerResultCallback.close(); - attachContainerResultCallback.awaitCompletion(); - stdOutWriteStream.close(); - stdErrWriteStream.close(); - } catch (Exception e) { - LOGGER.debug("Error by detaching streams", e); - } finally { - try { - invokeIfNotNull(afterContainerStop, containerId, dockerClient); - } catch (Exception e) { - LOGGER.debug("Exception thrown at invoking afterContainerStop", e); - } finally { - finished.countDown(); - } - - } - - - } - }; - - public DockerizedProcess(final DockerClient dockerClient, final String containerId, - final Closure afterContainerStop) throws Exception { - this.dockerClient = dockerClient; - this.containerId = containerId; - this.afterContainerStop = afterContainerStop; - attachStreams(); - dockerClient.waitContainerCmd(containerId).exec(waitContainerResultCallback); - } - - private void attachStreams() throws Exception { - dockerClient.attachContainerCmd(containerId) - .withFollowStream(true) - .withStdOut(true) - .withStdErr(true) - .withStdIn(stdInReadStream) - .exec(attachContainerResultCallback); - if (!attachContainerResultCallback.awaitStarted(2, TimeUnit.MINUTES)) { - LOGGER.warn("Not attached to container " + containerId + " within 10secs"); - throw new RuntimeException("Not attached to container " + containerId + " within 10secs"); - } - } - - @Override - public OutputStream getOutputStream() { - return stdInWriteStream; - } - - @Override - public InputStream getInputStream() { - return stdOutReadStream; - } - - @Override - public InputStream getErrorStream() { - return stdErrReadStream; - } - - @Override - public int waitFor() throws InterruptedException { - finished.await(); - return exitCode.get(); - } - - @Override - public int exitValue() { - if (finished.getCount() > 0) { - throw new IllegalThreadStateException("docker process still running"); - } - return exitCode.get(); - } - - @Override - public void destroy() { - dockerClient.killContainerCmd(containerId).exec(); - } - - @Override - public String toString() { - return "Container " + containerId + " on " + dockerClient.toString(); - } - } - -} diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandleRunner.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandleRunner.java deleted file mode 100644 index cfc2fb8b0107..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/DockerizedExecHandleRunner.java +++ /dev/null @@ -1,102 +0,0 @@ -/* - * Copyright 2010 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import java.util.concurrent.Executor; -import java.util.concurrent.locks.Lock; -import java.util.concurrent.locks.ReentrantLock; - -import org.gradle.api.logging.Logger; -import org.gradle.api.logging.Logging; -import org.gradle.process.internal.StreamsHandler; - -public class DockerizedExecHandleRunner implements Runnable { - private static final Logger - LOGGER = - Logging.getLogger(org.gradle.process.internal.ExecHandleRunner.class); - - private final DockerizedExecHandle execHandle; - private final Lock lock = new ReentrantLock(); - private final Executor executor; - - private Process process; - private boolean aborted; - private final StreamsHandler streamsHandler; - - public DockerizedExecHandleRunner(DockerizedExecHandle execHandle, StreamsHandler streamsHandler, - Executor executor) { - this.executor = executor; - if (execHandle == null) { - throw new IllegalArgumentException("execHandle == null!"); - } - this.streamsHandler = streamsHandler; - this.execHandle = execHandle; - } - - public void abortProcess() { - lock.lock(); - try { - aborted = true; - if (process != null) { - LOGGER.debug("Abort requested. Destroying process: {}.", execHandle.getDisplayName()); - process.destroy(); - } - } finally { - lock.unlock(); - } - } - - @Override - public void run() { - try { - process = execHandle.runContainer(); - streamsHandler.connectStreams(process, execHandle.getDisplayName(), executor); - - execHandle.started(); - - LOGGER.debug("waiting until streams are handled..."); - streamsHandler.start(); - if (execHandle.isDaemon()) { - streamsHandler.stop(); - detached(); - } else { - int exitValue = process.waitFor(); - streamsHandler.stop(); - completed(exitValue); - } - } catch (Throwable t) { - execHandle.failed(t); - } - } - - private void completed(int exitValue) { - if (aborted) { - execHandle.aborted(exitValue); - } else { - execHandle.finished(exitValue); - } - } - - private void detached() { - execHandle.detached(); - } - - public String toString() { - return "Handler for " + process.toString(); - } -} - diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForciblyStoppableTestWorker.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForciblyStoppableTestWorker.java deleted file mode 100644 index fbbe48e50264..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForciblyStoppableTestWorker.java +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import java.util.Timer; -import java.util.TimerTask; -import java.util.concurrent.TimeUnit; - -import org.gradle.api.internal.tasks.testing.WorkerTestClassProcessorFactory; -import org.gradle.api.internal.tasks.testing.worker.TestWorker; - -public class ForciblyStoppableTestWorker extends TestWorker { - private static final int SHUTDOWN_TIMEOUT = 60; // secs - - public ForciblyStoppableTestWorker(WorkerTestClassProcessorFactory factory) { - super(factory); - } - - @Override - public void stop() { - new Timer(true).schedule(new TimerTask() { - @Override - public void run() { - System.err.println("Worker process did not shutdown gracefully within " + SHUTDOWN_TIMEOUT - + "s, forcing it now"); - Runtime.getRuntime().halt(-100); - } - }, TimeUnit.SECONDS.toMillis(SHUTDOWN_TIMEOUT)); - super.stop(); - } -} diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForkingTestClassProcessor.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForkingTestClassProcessor.java deleted file mode 100644 index d6f409b461ab..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/ForkingTestClassProcessor.java +++ /dev/null @@ -1,153 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import java.io.File; -import java.net.URL; -import java.util.List; - -import org.gradle.api.Action; -import org.gradle.api.internal.classpath.ModuleRegistry; -import org.gradle.api.internal.tasks.testing.TestClassProcessor; -import org.gradle.api.internal.tasks.testing.TestClassRunInfo; -import org.gradle.api.internal.tasks.testing.TestResultProcessor; -import org.gradle.api.internal.tasks.testing.WorkerTestClassProcessorFactory; -import org.gradle.api.internal.tasks.testing.worker.RemoteTestClassProcessor; -import org.gradle.api.internal.tasks.testing.worker.TestEventSerializer; -import org.gradle.internal.remote.ObjectConnection; -import org.gradle.process.JavaForkOptions; -import org.gradle.process.internal.worker.WorkerProcess; -import org.gradle.process.internal.worker.WorkerProcessBuilder; -import org.gradle.process.internal.worker.WorkerProcessFactory; -import org.gradle.util.CollectionUtils; - -public class ForkingTestClassProcessor implements TestClassProcessor { - private final WorkerProcessFactory workerFactory; - private final WorkerTestClassProcessorFactory processorFactory; - private final JavaForkOptions options; - private final Iterable classPath; - private final Action buildConfigAction; - private final ModuleRegistry moduleRegistry; - private RemoteTestClassProcessor remoteProcessor; - private WorkerProcess workerProcess; - private TestResultProcessor resultProcessor; - - public ForkingTestClassProcessor(WorkerProcessFactory workerFactory, - WorkerTestClassProcessorFactory processorFactory, - JavaForkOptions options, Iterable classPath, - Action buildConfigAction, - ModuleRegistry moduleRegistry) { - this.workerFactory = workerFactory; - this.processorFactory = processorFactory; - this.options = options; - this.classPath = classPath; - this.buildConfigAction = buildConfigAction; - this.moduleRegistry = moduleRegistry; - } - - @Override - public void startProcessing(TestResultProcessor resultProcessor) { - this.resultProcessor = resultProcessor; - } - - @Override - public void processTestClass(TestClassRunInfo testClass) { - int i = 0; - RuntimeException exception = null; - while (remoteProcessor == null && i < 10) { - try { - remoteProcessor = forkProcess(); - exception = null; - break; - } catch (RuntimeException e) { - exception = e; - i++; - } - } - - if (exception != null) { - throw exception; - } - remoteProcessor.processTestClass(testClass); - } - - RemoteTestClassProcessor forkProcess() { - WorkerProcessBuilder - builder = - workerFactory.create(new ForciblyStoppableTestWorker(processorFactory)); - builder.setBaseName("Gradle Test Executor"); - builder.setImplementationClasspath(getTestWorkerImplementationClasspath()); - builder.applicationClasspath(classPath); - options.copyTo(builder.getJavaCommand()); - buildConfigAction.execute(builder); - - workerProcess = builder.build(); - workerProcess.start(); - - ObjectConnection connection = workerProcess.getConnection(); - connection.useParameterSerializers(TestEventSerializer.create()); - connection.addIncoming(TestResultProcessor.class, resultProcessor); - RemoteTestClassProcessor - remoteProcessor = - connection.addOutgoing(RemoteTestClassProcessor.class); - connection.connect(); - remoteProcessor.startProcessing(); - return remoteProcessor; - } - - List getTestWorkerImplementationClasspath() { - return CollectionUtils.flattenCollections(URL.class, - moduleRegistry.getModule("gradle-core-api").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-core").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-logging").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-messaging").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-base-services").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-cli").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-native").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-testing-base").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-testing-jvm").getImplementationClasspath().getAsURLs(), - moduleRegistry.getModule("gradle-process-services").getImplementationClasspath() - .getAsURLs(), - moduleRegistry.getExternalModule("slf4j-api").getImplementationClasspath().getAsURLs(), - moduleRegistry.getExternalModule("jul-to-slf4j").getImplementationClasspath().getAsURLs(), - moduleRegistry.getExternalModule("native-platform").getImplementationClasspath() - .getAsURLs(), - moduleRegistry.getExternalModule("kryo").getImplementationClasspath().getAsURLs(), - moduleRegistry.getExternalModule("commons-lang").getImplementationClasspath().getAsURLs(), - moduleRegistry.getExternalModule("junit").getImplementationClasspath().getAsURLs(), - ForkingTestClassProcessor.class.getProtectionDomain().getCodeSource().getLocation() - ); - } - - @Override - public void stop() { - if (remoteProcessor != null) { - try { - remoteProcessor.stop(); - workerProcess.waitForStop(); - } finally { - // do nothing - } - } - } - - @Override - public void stopNow() { - stop(); // TODO need anything else ?? - } - -} diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/NoMemoryManager.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/NoMemoryManager.java deleted file mode 100644 index 3a8be604f006..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/NoMemoryManager.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import org.gradle.process.internal.health.memory.JvmMemoryStatusListener; -import org.gradle.process.internal.health.memory.MemoryHolder; -import org.gradle.process.internal.health.memory.MemoryManager; -import org.gradle.process.internal.health.memory.OsMemoryStatusListener; - -public class NoMemoryManager implements MemoryManager { - @Override - public void addListener(JvmMemoryStatusListener jvmMemoryStatusListener) { - - } - - @Override - public void addListener(OsMemoryStatusListener osMemoryStatusListener) { - - } - - @Override - public void removeListener(JvmMemoryStatusListener jvmMemoryStatusListener) { - - } - - @Override - public void removeListener(OsMemoryStatusListener osMemoryStatusListener) { - - } - - @Override - public void addMemoryHolder(MemoryHolder memoryHolder) { - - } - - @Override - public void removeMemoryHolder(MemoryHolder memoryHolder) { - - } - - @Override - public void requestFreeMemory(long l) { - - } -} diff --git a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/TestExecuter.java b/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/TestExecuter.java deleted file mode 100644 index b99e124a7473..000000000000 --- a/buildSrc/src/main/java/com/pedjak/gradle/plugins/dockerizedtest/TestExecuter.java +++ /dev/null @@ -1,119 +0,0 @@ -/* - * Copyright 2015 the original author or authors. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.pedjak.gradle.plugins.dockerizedtest; - -import java.io.File; -import java.util.Set; -import java.util.UUID; - -import com.google.common.collect.ImmutableSet; -import org.gradle.api.file.FileTree; -import org.gradle.api.internal.classpath.ModuleRegistry; -import org.gradle.api.internal.tasks.testing.JvmTestExecutionSpec; -import org.gradle.api.internal.tasks.testing.TestClassProcessor; -import org.gradle.api.internal.tasks.testing.TestFramework; -import org.gradle.api.internal.tasks.testing.TestResultProcessor; -import org.gradle.api.internal.tasks.testing.WorkerTestClassProcessorFactory; -import org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner; -import org.gradle.api.internal.tasks.testing.detection.TestFrameworkDetector; -import org.gradle.api.internal.tasks.testing.processors.MaxNParallelTestClassProcessor; -import org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor; -import org.gradle.api.internal.tasks.testing.processors.TestMainAction; -import org.gradle.internal.Factory; -import org.gradle.internal.actor.ActorFactory; -import org.gradle.internal.operations.BuildOperationExecutor; -import org.gradle.internal.time.Clock; -import org.gradle.process.internal.worker.WorkerProcessFactory; - -public class TestExecuter - implements org.gradle.api.internal.tasks.testing.TestExecuter { - private final WorkerProcessFactory workerFactory; - private final ActorFactory actorFactory; - private final ModuleRegistry moduleRegistry; - private final BuildOperationExecutor buildOperationExecutor; - private final Clock clock; - private TestClassProcessor processor; - - public TestExecuter(WorkerProcessFactory workerFactory, ActorFactory actorFactory, - ModuleRegistry moduleRegistry, BuildOperationExecutor buildOperationExecutor, - Clock clock) { - this.workerFactory = workerFactory; - this.actorFactory = actorFactory; - this.moduleRegistry = moduleRegistry; - this.buildOperationExecutor = buildOperationExecutor; - this.clock = clock; - } - - @Override - public void execute(final JvmTestExecutionSpec testExecutionSpec, - TestResultProcessor testResultProcessor) { - final TestFramework testFramework = testExecutionSpec.getTestFramework(); - final WorkerTestClassProcessorFactory testInstanceFactory = testFramework.getProcessorFactory(); - final Set classpath = ImmutableSet.copyOf(testExecutionSpec.getClasspath()); - final Factory forkingProcessorFactory = new Factory() { - @Override - public TestClassProcessor create() { - return new ForkingTestClassProcessor(workerFactory, testInstanceFactory, - testExecutionSpec.getJavaForkOptions(), - classpath, testFramework.getWorkerConfigurationAction(), moduleRegistry); - } - }; - Factory reforkingProcessorFactory = new Factory() { - @Override - public TestClassProcessor create() { - return new RestartEveryNTestClassProcessor(forkingProcessorFactory, - testExecutionSpec.getForkEvery()); - } - }; - - processor = new MaxNParallelTestClassProcessor(testExecutionSpec.getMaxParallelForks(), - reforkingProcessorFactory, actorFactory); - - final FileTree testClassFiles = testExecutionSpec.getCandidateClassFiles(); - - Runnable detector; - if (testExecutionSpec.isScanForTestClasses()) { - TestFrameworkDetector - testFrameworkDetector = - testExecutionSpec.getTestFramework().getDetector(); - testFrameworkDetector.setTestClasses(testExecutionSpec.getTestClassesDirs().getFiles()); - testFrameworkDetector.setTestClasspath(classpath); - detector = new DefaultTestClassScanner(testClassFiles, testFrameworkDetector, processor); - } else { - detector = new DefaultTestClassScanner(testClassFiles, null, processor); - } - - Object testTaskOperationId; - - try { - testTaskOperationId = buildOperationExecutor.getCurrentOperation().getParentId(); - } catch (Exception e) { - testTaskOperationId = UUID.randomUUID(); - } - - new TestMainAction(detector, processor, testResultProcessor, clock, testTaskOperationId, - testExecutionSpec.getPath(), "Gradle Test Run " + testExecutionSpec.getIdentityPath()) - .run(); - } - - @Override - public void stopNow() { - if (processor != null) { - processor.stopNow(); - } - } -} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/RunInSubdirectoryTestFramework.java b/buildSrc/src/main/java/org/apache/geode/gradle/RunInSubdirectoryTestFramework.java deleted file mode 100644 index 546ae1e7b5c6..000000000000 --- a/buildSrc/src/main/java/org/apache/geode/gradle/RunInSubdirectoryTestFramework.java +++ /dev/null @@ -1,102 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more contributor license - * agreements. See the NOTICE file distributed with this work for additional information regarding - * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance with the License. You may obtain a - * copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software distributed under the License - * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express - * or implied. See the License for the specific language governing permissions and limitations under - * the License. - */ - -package org.apache.geode.gradle; - -import static java.nio.file.StandardCopyOption.COPY_ATTRIBUTES; - -import java.io.IOException; -import java.io.UncheckedIOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.concurrent.atomic.AtomicLong; - -import org.gradle.api.Action; -import org.gradle.api.internal.tasks.testing.TestFramework; -import org.gradle.api.internal.tasks.testing.WorkerTestClassProcessorFactory; -import org.gradle.api.internal.tasks.testing.detection.TestFrameworkDetector; -import org.gradle.api.tasks.testing.TestFrameworkOptions; -import org.gradle.process.internal.JavaExecHandleBuilder; -import org.gradle.process.internal.worker.WorkerProcessBuilder; - -/** - * Wraps a test framework to run each test worker in a separate working directory. - */ -public class RunInSubdirectoryTestFramework implements TestFramework { - private static final String GEMFIRE_PROPERTIES = "gemfire.properties"; - private final AtomicLong workerId = new AtomicLong(); - private final TestFramework delegate; - - public RunInSubdirectoryTestFramework(TestFramework delegate) { - this.delegate = delegate; - } - - @Override - public TestFrameworkDetector getDetector() { - return delegate.getDetector(); - } - - @Override - public TestFrameworkOptions getOptions() { - return delegate.getOptions(); - } - - @Override - public WorkerTestClassProcessorFactory getProcessorFactory() { - return delegate.getProcessorFactory(); - } - - /** - * Return an action that configures the test worker builder to run the test worker in a unique - * subdirectory of the task's working directory. - */ - @Override - public Action getWorkerConfigurationAction() { - return workerProcessBuilder -> { - delegate.getWorkerConfigurationAction().execute(workerProcessBuilder); - JavaExecHandleBuilder javaCommand = workerProcessBuilder.getJavaCommand(); - - Path taskWorkingDir = javaCommand.getWorkingDir().toPath(); - String workerWorkingDirName = String.format("test-worker-%06d", workerId.incrementAndGet()); - Path workerWorkingDir = taskWorkingDir.resolve(workerWorkingDirName); - - createWorkingDir(workerWorkingDir); - copyGemFirePropertiesFile(taskWorkingDir, workerWorkingDir); - - javaCommand.setWorkingDir(workerWorkingDir); - }; - } - - private void copyGemFirePropertiesFile(Path taskWorkingDir, Path workerWorkingDir) { - Path taskPropertiesFile = taskWorkingDir.resolve(GEMFIRE_PROPERTIES); - if (!Files.exists(taskPropertiesFile)) { - return; - } - Path workerPropertiesFile = workerWorkingDir.resolve(taskPropertiesFile.getFileName()); - try { - Files.copy(taskPropertiesFile, workerPropertiesFile, COPY_ATTRIBUTES); - } catch (IOException e) { - throw new UncheckedIOException(e); - } - } - - private void createWorkingDir(Path workerWorkingDir) { - try { - Files.createDirectories(workerWorkingDir); - } catch (IOException e) { - throw new UncheckedIOException(e); - } - } -} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerConnectionAcceptor.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerConnectionAcceptor.java new file mode 100644 index 000000000000..8e38d266d363 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerConnectionAcceptor.java @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.dockerized; + +import static java.util.Collections.list; +import static java.util.stream.Collectors.toList; + +import java.io.IOException; +import java.net.InetAddress; +import java.net.NetworkInterface; +import java.net.SocketException; +import java.util.List; + +import org.gradle.api.UncheckedIOException; +import org.gradle.internal.remote.Address; +import org.gradle.internal.remote.ConnectionAcceptor; +import org.gradle.internal.remote.internal.inet.MultiChoiceAddress; + +/** + * Wraps a {@link ConnectionAcceptor} to give it a {@link MultiChoiceAddress} that processes in + * Docker containers can use to connect to Gradle's messaging server. + */ +class DockerConnectionAcceptor implements ConnectionAcceptor { + private static final List DOCKER_ACCEPTABLE_ADDRESSES; + + static { + try { + DOCKER_ACCEPTABLE_ADDRESSES = list(NetworkInterface.getNetworkInterfaces()).stream() + .filter(DockerConnectionAcceptor::isAcceptable) + .flatMap(i -> list(i.getInetAddresses()).stream()) + .filter(DockerConnectionAcceptor::isAcceptable) + .collect(toList()); + } catch (SocketException e) { + throw new UncheckedIOException("Unable to identify usable addresses", e); + } + } + + private final MultiChoiceAddress address; + private final ConnectionAcceptor delegate; + + /** + * Creates a {@code MultiChoiceAddress} whose candidate addresses are acceptable for processes in + * Docker containers to use to attempt to connect to Gradle's messaging server. The messaging + * server will accept connections from one of those candidate addresses. + */ + DockerConnectionAcceptor(ConnectionAcceptor delegate) { + this.delegate = delegate; + // The delegate's candidates are the host's loopback addresses, which processes in Docker + // containers cannot use. + MultiChoiceAddress original = (MultiChoiceAddress) delegate.getAddress(); + // Replace the delegate's unacceptable candidate addresses with acceptable ones. + address = new MultiChoiceAddress(original.getCanonicalAddress(), original.getPort(), + DOCKER_ACCEPTABLE_ADDRESSES); + } + + @Override + public Address getAddress() { + return address; + } + + @Override + public void requestStop() { + delegate.requestStop(); + } + + @Override + public void stop() { + delegate.stop(); + } + + /** + * Reports whether the candidate interface is acceptable for processes in Docker containers to use + * to connect to Gradle's messaging server. An interface is acceptable if it satisfies all of: + *
    + *
  • it is up
  • + *
  • it is not a loopback interface
  • + *
  • it is not a point to point interface (e.g. VPN)
  • + */ + private static boolean isAcceptable(NetworkInterface candidate) { + try { + return !candidate.isLoopback() + && !candidate.isPointToPoint() + && candidate.isUp(); + } catch (SocketException ignored) { + return false; + } + } + + /** + * Reports whether the candidate address is acceptable for processes in Docker containers to use + * to connect to Gradle's messaging server. An address is acceptable if it is reachable. + */ + private static boolean isAcceptable(InetAddress candidate) { + try { + return candidate.isReachable(2000); + } catch (IOException ignored) { + return false; + } + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerMessagingServer.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerMessagingServer.java new file mode 100644 index 000000000000..69a28c7a0d2a --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerMessagingServer.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.dockerized; + +import org.gradle.api.Action; +import org.gradle.internal.concurrent.ExecutorFactory; +import org.gradle.internal.remote.ConnectionAcceptor; +import org.gradle.internal.remote.MessagingServer; +import org.gradle.internal.remote.ObjectConnection; +import org.gradle.internal.remote.internal.ConnectCompletion; +import org.gradle.internal.remote.internal.IncomingConnector; +import org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection; + +/** + * A copy of MessageHubBackedServer from Gradle v6.8.3, modified to accept connections from + * processes running in Docker containers. + */ +public class DockerMessagingServer implements MessagingServer { + private final IncomingConnector connector; + private final ExecutorFactory executorFactory; + + public DockerMessagingServer(IncomingConnector connector, ExecutorFactory executorFactory) { + this.connector = connector; + this.executorFactory = executorFactory; + } + + /** + * Transforms Gradle's standard connection acceptor into one that will accept connections from + * worker processes in Docker containers. A connection acceptor reports a list of candidate + * addresses for worker processes to try to connect to. Gradle's standard acceptor reports the + * host's loopback addresses, which processes in Docker containers cannot use. The transformed + * acceptor instead reports a list of non-loopback addresses, and the Dockerized process will be + * able to use at least one of those to connect to this server. + */ + @Override + public ConnectionAcceptor accept(Action action) { + ConnectEventAction connectEventAction = new ConnectEventAction(action); + ConnectionAcceptor originalConnectionAcceptor = connector.accept(connectEventAction, true); + return new DockerConnectionAcceptor(originalConnectionAcceptor); + } + + /** + * An unmodified copy of MessageHubBackedServer.ConnectionEvent from Gradle v6.8.3. + */ + private class ConnectEventAction implements Action { + private final Action action; + + public ConnectEventAction(Action action) { + this.action = action; + } + + @Override + public void execute(ConnectCompletion completion) { + action.execute(new MessageHubBackedObjectConnection(executorFactory, completion)); + } + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcess.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcess.java new file mode 100644 index 000000000000..d799b29d5f67 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcess.java @@ -0,0 +1,287 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.dockerized; + +import static java.util.concurrent.TimeUnit.MILLISECONDS; +import static org.apache.geode.gradle.testing.dockerized.DockerTestWorkerConfig.getDurationWarningThreshold; + +import java.io.Closeable; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.PipedInputStream; +import java.io.PipedOutputStream; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.atomic.AtomicInteger; + +import com.github.dockerjava.api.DockerClient; +import com.github.dockerjava.api.async.ResultCallback; +import com.github.dockerjava.api.async.ResultCallbackTemplate; +import com.github.dockerjava.api.model.Frame; +import com.github.dockerjava.api.model.StreamType; +import com.github.dockerjava.api.model.WaitResponse; +import org.gradle.api.logging.Logger; +import org.gradle.api.logging.Logging; +import org.gradle.internal.UncheckedException; + +/** + * Represents a process running in a Docker container. + */ +public class DockerProcess extends Process { + private static final Logger LOGGER = Logging.getLogger(DockerProcess.class); + + private final String owner; + private final DockerClient client; + private final String containerId; + private final int timeoutMillis; + private final Runnable onCompletion; + private final PipedOutputStream stdIn = new PipedOutputStream(); + private final PipedInputStream stdOut = new PipedInputStream(); + private final PipedInputStream stdErr = new PipedInputStream(); + private final PipedInputStream stdInToContainer = new PipedInputStream(); + private final PipedOutputStream stdOutFromContainer = new PipedOutputStream(); + private final PipedOutputStream stdErrFromContainer = new PipedOutputStream(); + private final AtomicInteger exitCode = new AtomicInteger(); + private final CountDownLatch finished = new CountDownLatch(1); + private final OutputListener outputListener = new OutputListener(); + private final TerminationListener terminationListener = new TerminationListener(); + + /** + * Creates a {@link Process} that represents a process running in a Docker container. + * + * @param owner the name of this process's owner (used for diagnostics) + * @param client a Docker client to use to listen for process output and termination + * @param containerId the ID of the container in which the process is running + * @param timeoutMillis duration to wait for each listener to start + * @param onCompletion a runnable to run when this process completes + * @return a Process that represents the process in the container + */ + public static Process attachedTo(String owner, DockerClient client, String containerId, + int timeoutMillis, Runnable onCompletion) { + DockerProcess process = new DockerProcess( + owner, client, containerId, timeoutMillis, onCompletion); + try { + process.attach(); + } catch (Exception e) { + UncheckedException.throwAsUncheckedException(e); + } + return process; + } + + private DockerProcess(String owner, DockerClient client, String containerId, int timeoutMillis, + Runnable onCompletion) { + this.owner = owner; + this.client = client; + this.containerId = containerId; + this.timeoutMillis = timeoutMillis; + this.onCompletion = onCompletion; + } + + @Override + public OutputStream getOutputStream() { + return stdIn; + } + + @Override + public InputStream getInputStream() { + return stdOut; + } + + @Override + public InputStream getErrorStream() { + return stdErr; + } + + @Override + public int waitFor() throws InterruptedException { + finished.await(); + return exitValue(); + } + + @Override + public int exitValue() { + if (finished.getCount() != 0) { + throw new IllegalThreadStateException(toString() + " is still running"); + } + return exitCode.get(); + } + + @Override + public void destroy() { + finish(); + } + + @Override + public String toString() { + return String.format("DockerProcess{%s:%s}", owner, containerId); + } + + /** + * Attach this {@code DockerProcess}'s input and output streams to the container's, and set a + * callback for when the containerized process finishes. + * + * @throws Exception if an error occurs while attaching to the container + */ + private void attach() throws Exception { + listenForTermination(); + connectStreams(); + listenForOutput(); + } + + private void listenForOutput() throws InterruptedException { + LOGGER.debug("{} installing {}", this, outputListener); + try { + long startTime = System.currentTimeMillis(); + client.attachContainerCmd(containerId) + .withFollowStream(true) + .withStdOut(true) + .withStdErr(true) + .withStdIn(stdInToContainer) + .exec(outputListener); + LOGGER.debug("{} installed {}", this, outputListener); + long duration = System.currentTimeMillis() - startTime; + if(duration > getDurationWarningThreshold()) { + LOGGER.warn("{} {} installation took {}ms", this, outputListener, duration); + } + } catch (RuntimeException e) { + String message = String.format("%s error while installing %s", this, outputListener); + throw new RuntimeException(message, e); + } + waitUntilStarted(outputListener); + } + + private void connectStreams() throws IOException { + stdInToContainer.connect(stdIn); + stdOutFromContainer.connect(stdOut); + stdErrFromContainer.connect(stdErr); + } + + private void listenForTermination() throws InterruptedException { + LOGGER.debug("{} installing {}", this, terminationListener); + try { + long startTime = System.currentTimeMillis(); + client.waitContainerCmd(containerId) + .exec(terminationListener); + LOGGER.debug("{} installed {}", this, terminationListener); + long duration = System.currentTimeMillis() - startTime; + if(duration > getDurationWarningThreshold()) { + LOGGER.warn("{} {} installation took {}ms", this, terminationListener, duration); + } + } catch (RuntimeException e) { + String message = String.format("%s error while installing %s", this, terminationListener); + throw new RuntimeException(message, e); + } + waitUntilStarted(terminationListener); + } + + private void waitUntilStarted(ResultCallbackTemplate listener) + throws InterruptedException { + if (timeoutMillis > 0) { + LOGGER.debug("{} waiting {}ms for {} to start", this, timeoutMillis, listener); + if (!listener.awaitStarted(timeoutMillis, MILLISECONDS)) { + String message = String.format( + "%s timed out after %dms waiting for %s to start", this, timeoutMillis, listener); + throw new RuntimeException(message); + } + } else { + LOGGER.debug("{} waiting for {} to start", this, listener); + listener.awaitStarted(); + } + LOGGER.debug("{} {} started", this, listener); + } + + private void finish() { + close("stdin", stdIn); + close("stdout", stdOut); + close("stderr", stdErr); + close("stdin to container", stdInToContainer); + close("stdout from container", stdOutFromContainer); + close("stderr from container", stdErrFromContainer); + close("client", client); + finished.countDown(); + onCompletion.run(); + } + + private void close(String name, Closeable closeable) { + try { + closeable.close(); + LOGGER.debug("{} closed {}", this, name); + } catch (IOException e) { + String message = String.format("%s error while closing %s", this, name); + LOGGER.warn(message, e); + } + } + + /** + * A listener for Docker to notify whenever the containerized process writes new output. The + * listener copies each frame of the process's output to this DockerProcess's stdout or stderr. + */ + private class OutputListener extends ResultCallback.Adapter { + @Override + public void onNext(Frame frame) { + byte[] payload = frame.getPayload(); + StreamType streamType = frame.getStreamType(); + try { + switch (streamType) { + case STDOUT: + stdOutFromContainer.write(payload); + break; + case STDERR: + stdErrFromContainer.write(payload); + break; + default: + } + } catch (IOException e) { + String message = String.format("%s %s error while writing to %s", + DockerProcess.this.toString(), this, streamType); + LOGGER.error(message, e); + } + } + @Override + public String toString() { + return "output listener"; + } + + } + + /** + * A listener for Docker to notify when the containerized process terminates. The listener records + * the process's exit code, stops watching its streams, closes this {@code DockerProcess}'s output + * streams, and removes the Docker container. + */ + private class TerminationListener extends ResultCallback.Adapter { + @Override + public void onNext(WaitResponse response) { + Integer statusCode = response.getStatusCode(); + LOGGER.debug("{} {} called: process exited with status code {}", + DockerProcess.this, this, statusCode); + exitCode.set(statusCode); + try { + outputListener.close(); + outputListener.awaitCompletion(); + } catch (Exception e) { + String message = String.format("%s error while removing %s", DockerProcess.this, this); + LOGGER.warn(message, e); + } finally { + finish(); + } + } + + @Override + public String toString() { + return "termination listener"; + } + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcessLauncher.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcessLauncher.java new file mode 100644 index 000000000000..eb190d4fa3f4 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/DockerProcessLauncher.java @@ -0,0 +1,227 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.geode.gradle.testing.dockerized; + +import static java.util.stream.Collectors.toList; +import static org.apache.geode.gradle.testing.dockerized.DockerTestWorkerConfig.getDurationWarningThreshold; + +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.function.Consumer; + +import com.github.dockerjava.api.DockerClient; +import com.github.dockerjava.api.command.CreateContainerCmd; +import com.github.dockerjava.api.command.InspectContainerResponse; +import com.github.dockerjava.api.model.Bind; +import com.github.dockerjava.api.model.Volume; +import com.github.dockerjava.core.AbstractDockerCmdExecFactory; +import com.github.dockerjava.core.DockerClientBuilder; +import com.github.dockerjava.netty.NettyDockerCmdExecFactory; +import org.gradle.api.logging.Logger; +import org.gradle.api.logging.Logging; +import org.gradle.internal.UncheckedException; + +import org.apache.geode.gradle.testing.process.ProcessLauncher; + +/** + * A process launcher that launches each process in a Docker container. + */ +public class DockerProcessLauncher implements ProcessLauncher { + private static final Logger LOGGER = Logging.getLogger(DockerProcessLauncher.class); + private final DockerTestWorkerConfig config; + private final Consumer adjustment; + + /** + * @param config the configuration of the docker containers + * @param adjustment configures the process builder before it is dockerized + */ + public DockerProcessLauncher(DockerTestWorkerConfig config, Consumer adjustment) { + this.config = config; + this.adjustment = adjustment; + } + + /** + * Launches the specified process in a Docker container. + * + * @param processBuilder a builder that specifies the process to launch + * @return a Process that represents the process running in the Docker container + */ + @Override + public Process start(ProcessBuilder processBuilder) { + adjustment.accept(processBuilder); + config.dockerize(processBuilder); + int timeout = config.getTimeoutMillis(); + + // For synchronous Docker operations (create, start, inspect), time out if connecting or + // reading takes too long. + DockerClient clientForSynchronousOperations = dockerClient(timeout, timeout); + + String containerId = createContainer(processBuilder, clientForSynchronousOperations); + + try { + startContainer(clientForSynchronousOperations, containerId); + // For asynchronous Docker operations, time out only on connects. The DockerProcess uses the + // async client to listen for process output and process termination. Because the client must + // listen indefinitely for these events, do not time out on reads. + DockerClient clientForAsynchronousOperations = dockerClient(timeout, 0); + return DockerProcess.attachedTo( + config.getName(), clientForAsynchronousOperations, containerId, timeout, + () -> removeContainer(clientForSynchronousOperations, containerId)); + } catch (Exception e) { + removeContainer(clientForSynchronousOperations, containerId); + UncheckedException.throwAsUncheckedException(e); + return null; // Unreachable + } + } + + /** + * Creates a docker client with the given timeouts. + * + * @param connectTimeout timeout for connecting, or 0 to disable connect timeouts + * @param readTimeout timeout for reading, or 0 to disable read timeouts + */ + private static DockerClient dockerClient(int connectTimeout, int readTimeout) { + AbstractDockerCmdExecFactory cmdExecFactory = new NettyDockerCmdExecFactory(); + if (connectTimeout > 0) { + cmdExecFactory.withConnectTimeout(connectTimeout); + } + if (readTimeout > 0) { + cmdExecFactory.withReadTimeout(readTimeout); + } + // Must use the deprecated withDockerCmdExecFactory() because it is currently the only way to + // use Netty, and Netty is currently the only transport that supports timeouts. + @SuppressWarnings("deprecation") + DockerClient client = DockerClientBuilder.getInstance() + .withDockerCmdExecFactory(cmdExecFactory) + .build(); + return client; + } + + private String createContainer(ProcessBuilder processBuilder, DockerClient client) { + CreateContainerCmd createContainerCommand = client.createContainerCmd(config.getImage()) + .withTty(false) + .withStdinOpen(true) + .withWorkingDir(processBuilder.directory().getAbsolutePath()) + .withEnv(asStrings(processBuilder.environment())) + .withCmd(processBuilder.command()); + setUser(createContainerCommand); + setVolumes(createContainerCommand); + LOGGER.debug("{} creating container", this); + try { + long startTime = System.currentTimeMillis(); + String containerId = createContainerCommand.exec().getId(); + long duration = System.currentTimeMillis() - startTime; + if (duration > getDurationWarningThreshold()) { + LOGGER.warn("{} create took {}ms", this, duration); + } + LOGGER.debug("{} created container {}", this, containerId); + return containerId; + } catch (RuntimeException e) { + String message = String.format("%s error while creating container", this); + throw new RuntimeException(message, e); + } + } + + private void startContainer(DockerClient client, String containerId) { + LOGGER.debug("{} starting container {}", this, containerId); + try { + long startTime = System.currentTimeMillis(); + client.startContainerCmd(containerId).exec(); + LOGGER.debug("{} started container {}", this, containerId); + long duration = System.currentTimeMillis() - startTime; + if (duration > getDurationWarningThreshold()) { + LOGGER.warn("{} start {} took {}ms", this, containerId, duration); + } + } catch (RuntimeException e) { + String message = String.format("%s error while starting container %s", this, containerId); + throw new RuntimeException(message, e); + } + InspectContainerResponse report; + try { + long startTime = System.currentTimeMillis(); + report = client.inspectContainerCmd(containerId).exec(); + long duration = System.currentTimeMillis() - startTime; + if (duration > getDurationWarningThreshold()) { + LOGGER.warn("{} inspect {} took {}ms", this, containerId, duration); + } + } catch (RuntimeException e) { + String message = String.format("%s error while inspecting container %s", this, containerId); + throw new RuntimeException(message, e); + } + InspectContainerResponse.ContainerState state = report.getState(); + LOGGER.debug("{} container {} state is {}", this, containerId, state); + Boolean isRunning = state.getRunning(); + if (isRunning == null || !isRunning) { + String message = String.format("%s cannot attach to container %s because it is %s", + this, containerId, state.getStatus()); + throw new RuntimeException(message); + } + } + + private void removeContainer(DockerClient client, String containerId) { + LOGGER.debug("{} removing container {}", this, containerId); + try { + long startTime = System.currentTimeMillis(); + client.removeContainerCmd(containerId) + .withForce(true) + .exec(); + long duration = System.currentTimeMillis() - startTime; + if (duration > getDurationWarningThreshold()) { + LOGGER.warn("{} remove {} took {}ms", this, containerId, duration); + } + LOGGER.debug("{} removed container {}", this, containerId); + } catch (Exception e) { + String message = String.format("%s error while removing container %s", this, containerId); + LOGGER.warn(message, e); + } + try { + client.close(); + LOGGER.debug("{} closed client", this); + } catch (IOException e) { + String message = String.format("%s error while closing client", this); + LOGGER.warn(message, e); + } + } + + private void setUser(CreateContainerCmd command) { + String user = config.getUser(); + if (user != null) { + command.withUser(user); + } + } + + private void setVolumes(CreateContainerCmd command) { + List binds = config.getVolumes().entrySet().stream() + .map(e -> new Bind(e.getKey(), new Volume(e.getValue()))) + .collect(toList()); + List volumes = binds.stream() + .map(Bind::getVolume) + .collect(toList()); + command.withVolumes(volumes); + command.getHostConfig().withBinds(binds); + } + + @Override + public String toString() { + return "DockerProcessLauncher{" + config.getName() + "}"; + } + + private static List asStrings(Map map) { + return map.entrySet().stream() + .map(e -> e.getKey() + "=" + e.getValue()) + .collect(toList()); + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/WildcardBindingInetAddressFactory.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/WildcardBindingInetAddressFactory.java new file mode 100644 index 000000000000..b1fdad9c24d6 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/dockerized/WildcardBindingInetAddressFactory.java @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.dockerized; + +import java.net.InetAddress; + +import org.gradle.internal.remote.internal.inet.InetAddressFactory; + +/** + * Overrides Gradle's standard {@link InetAddressFactory} to report the host's wildcard address as + * the factory's local binding address. + */ +class WildcardBindingInetAddressFactory extends InetAddressFactory { + @Override + public InetAddress getLocalBindingAddress() { + return super.getWildcardBindingAddress(); + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/isolation/WorkingDirectoryIsolator.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/isolation/WorkingDirectoryIsolator.java new file mode 100644 index 000000000000..0a892fb872c2 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/isolation/WorkingDirectoryIsolator.java @@ -0,0 +1,95 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.isolation; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; +import java.util.OptionalInt; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.Consumer; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.IntStream; + +import org.gradle.api.UncheckedIOException; + +public class WorkingDirectoryIsolator implements Consumer { + private static final AtomicInteger WORKER_ID = new AtomicInteger(); + private static final Pattern GRADLE_WORKER_CLASSPATH_FILE_PATTERN = + Pattern.compile("^@.*gradle-worker-classpath.*txt$"); + private static final String PROPERTIES_FILE_NAME = "gemfire.properties"; + + /** + * Each test task gives all of its test workers the same working directory. Because + * Geode tests cannot tolerate this when run in parallel, we give each test worker its + * own unique working directory. + */ + @Override + public void accept(ProcessBuilder processBuilder) { + String subdirectory = String.format("test-worker-%06d", WORKER_ID.getAndIncrement()); + Path originalWorkingDirectory = processBuilder.directory().toPath(); + Path newWorkingDirectory = originalWorkingDirectory.resolve(subdirectory); + + try { + Files.createDirectories(newWorkingDirectory); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + processBuilder.directory(newWorkingDirectory.toFile()); + + Path originalPropertiesFile = originalWorkingDirectory.resolve(PROPERTIES_FILE_NAME); + if (Files.exists(originalPropertiesFile)) { + Path newPropertiesFile = newWorkingDirectory.resolve(PROPERTIES_FILE_NAME); + copy(originalPropertiesFile, newPropertiesFile); + } + + // If the command specifies a gradle worker classpath file that exists, copy the file to the + // unique working directory and update the command line argument to refer to the new location. + List command = processBuilder.command(); + findGradleWorkerClasspathArg(command) + .ifPresent(i -> updateGradleWorkerClasspathFile(command, i, newWorkingDirectory)); + } + + private void updateGradleWorkerClasspathFile(List command, int argIndex, Path directory) { + String originalClasspathFileArg = command.get(argIndex); + Matcher matcher = GRADLE_WORKER_CLASSPATH_FILE_PATTERN + .matcher(originalClasspathFileArg); + matcher.matches(); + Path originalClasspathFile = Paths.get(matcher.group().substring(1)); + Path newClasspathFile = directory.resolve("gradle-worker-classpath.txt"); + copy(originalClasspathFile, newClasspathFile); + String newClasspathFileArg = "@" + newClasspathFile.toString(); + command.set(argIndex, newClasspathFileArg); + } + + private static void copy(Path source, Path dest) { + try { + Files.copy(source, dest); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private static OptionalInt findGradleWorkerClasspathArg(List command) { + return IntStream.range(0, command.size()) + .filter(i -> GRADLE_WORKER_CLASSPATH_FILE_PATTERN.matcher(command.get(i)).matches()) + .findFirst(); + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/AdjustableProcessLauncher.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/AdjustableProcessLauncher.java new file mode 100644 index 000000000000..4c6753d7f547 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/AdjustableProcessLauncher.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.process; + +import java.io.IOException; +import java.util.function.Consumer; + +import org.gradle.api.UncheckedIOException; + +/** + * A process launcher that applies an adjustment to the {@link ProcessBuilder} before launching the + * process. + */ +public class AdjustableProcessLauncher implements ProcessLauncher { + private final Consumer adjustment; + + public AdjustableProcessLauncher(Consumer adjustment) { + this.adjustment = adjustment; + } + + @Override + public Process start(ProcessBuilder processBuilder) { + adjustment.accept(processBuilder); + try { + return processBuilder.start(); + } catch (IOException e) { + throw new UncheckedIOException("Cannot launch process", e); + } + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessBuilder.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessBuilder.java new file mode 100644 index 000000000000..a18f087b445b --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessBuilder.java @@ -0,0 +1,160 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.process; + +import static org.apache.geode.gradle.testing.process.Reflection.getField; +import static org.apache.geode.gradle.testing.process.Reflection.getFieldValue; +import static org.apache.geode.gradle.testing.process.Reflection.setFieldValue; + +import java.io.File; +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.Proxy; +import java.net.URL; +import java.util.List; +import java.util.Set; + +import org.gradle.api.Action; +import org.gradle.api.logging.LogLevel; +import org.gradle.process.internal.JavaExecHandleBuilder; +import org.gradle.process.internal.worker.WorkerProcess; +import org.gradle.process.internal.worker.WorkerProcessBuilder; +import org.gradle.process.internal.worker.WorkerProcessContext; + +/** + * Wraps a worker process builder to make it use the given process launcher. + */ +public class LauncherProxyWorkerProcessBuilder implements WorkerProcessBuilder { + private final WorkerProcessBuilder delegate; + private final ProcessLauncher processLauncher; + private static Object processLauncherProxy; + + public LauncherProxyWorkerProcessBuilder(WorkerProcessBuilder delegate, + ProcessLauncher processLauncher) { + this.delegate = delegate; + this.processLauncher = processLauncher; + } + + @Override + public WorkerProcessBuilder applicationClasspath(Iterable files) { + return delegate.applicationClasspath(files); + } + + @Override + public Set getApplicationClasspath() { + return delegate.getApplicationClasspath(); + } + + @Override + public WorkerProcessBuilder applicationModulePath(Iterable files) { + return delegate.applicationModulePath(files); + } + + @Override + public Set getApplicationModulePath() { + return delegate.getApplicationModulePath(); + } + + @Override + public WorkerProcessBuilder setBaseName(String baseName) { + return delegate.setBaseName(baseName); + } + + @Override + public String getBaseName() { + return delegate.getBaseName(); + } + + @Override + public WorkerProcessBuilder setLogLevel(LogLevel logLevel) { + return delegate.setLogLevel(logLevel); + } + + @Override + public WorkerProcessBuilder sharedPackages(Iterable packages) { + return delegate.sharedPackages(packages); + } + + @Override + public Set getSharedPackages() { + return delegate.getSharedPackages(); + } + + @Override + public JavaExecHandleBuilder getJavaCommand() { + return delegate.getJavaCommand(); + } + + @Override + public LogLevel getLogLevel() { + return delegate.getLogLevel(); + } + + @Override + public WorkerProcessBuilder sharedPackages(String... packages) { + return delegate.sharedPackages(packages); + } + + @Override + public Action getWorker() { + return delegate.getWorker(); + } + + @Override + public void setImplementationClasspath(List implementationClasspath) { + delegate.setImplementationClasspath(implementationClasspath); + } + + @Override + public void setImplementationModulePath(List implementationModulePath) { + delegate.setImplementationModulePath(implementationModulePath); + } + + @Override + public void enableJvmMemoryInfoPublishing(boolean shouldPublish) { + delegate.enableJvmMemoryInfoPublishing(shouldPublish); + } + + /** + * Replaces the standard worker process's process launcher with this builder's launcher. + */ + @Override + public WorkerProcess build() { + WorkerProcess workerProcess = delegate.build(); + Object workerProcessDelegate = getFieldValue(workerProcess, "delegate"); + Object execHandle = getFieldValue(workerProcessDelegate, "execHandle"); + Class processLauncherType = getField(execHandle, "processLauncher").getType(); + setFieldValue(execHandle, "processLauncher", assignableProcessLauncher(processLauncherType)); + return workerProcess; + } + + /** + * Because the exec handle created by Gradle uses a classloader different from ours, we can't + * simply construct a Gradle {@code ProcessLauncher} to assign. Instead we create proxy, using the + * exec handle's classloader. + */ + private synchronized Object assignableProcessLauncher(Class requiredType) { + if (processLauncherProxy == null) { + // Assume that only start() will be called, and simply delegate to this builder's launcher + InvocationHandler handler = + (proxy, method, args) -> processLauncher.start((ProcessBuilder) args[0]); + ClassLoader classLoader = requiredType.getClassLoader(); + Class[] interfaces = {requiredType}; + processLauncherProxy = Proxy.newProxyInstance(classLoader, interfaces, handler); + } + return processLauncherProxy; + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessFactory.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessFactory.java new file mode 100644 index 000000000000..fcee360ad19f --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/LauncherProxyWorkerProcessFactory.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.process; + +import java.io.File; + +import org.gradle.api.Action; +import org.gradle.api.internal.ClassPathRegistry; +import org.gradle.api.internal.file.TemporaryFileProvider; +import org.gradle.api.logging.LoggingManager; +import org.gradle.internal.id.IdGenerator; +import org.gradle.internal.jvm.inspection.JvmVersionDetector; +import org.gradle.internal.logging.events.OutputEventListener; +import org.gradle.internal.remote.MessagingServer; +import org.gradle.process.internal.JavaExecHandleFactory; +import org.gradle.process.internal.health.memory.MemoryManager; +import org.gradle.process.internal.worker.DefaultWorkerProcessFactory; +import org.gradle.process.internal.worker.WorkerProcessBuilder; +import org.gradle.process.internal.worker.WorkerProcessContext; + +/** + * Overrides Gradle's {@link DefaultWorkerProcessFactory} to return an {@link WorkerProcessBuilder} + * that uses the given {@link ProcessLauncher} to launch worker processes. + */ +public class LauncherProxyWorkerProcessFactory extends DefaultWorkerProcessFactory { + private final ProcessLauncher processLauncher; + + public LauncherProxyWorkerProcessFactory(LoggingManager loggingManager, + MessagingServer server, ClassPathRegistry classPathRegistry, + IdGenerator idGenerator, File gradleUserHomeDir, + TemporaryFileProvider temporaryFileProvider, + JavaExecHandleFactory execHandleFactory, + JvmVersionDetector jvmVersionDetector, + OutputEventListener outputEventListener, + MemoryManager memoryManager, ProcessLauncher processLauncher) { + super(loggingManager, server, classPathRegistry, idGenerator, gradleUserHomeDir, + temporaryFileProvider, execHandleFactory, jvmVersionDetector, outputEventListener, + memoryManager); + this.processLauncher = processLauncher; + } + + @Override + public WorkerProcessBuilder create(Action workerAction) { + return new LauncherProxyWorkerProcessBuilder(super.create(workerAction), processLauncher); + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/ProcessLauncher.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/ProcessLauncher.java new file mode 100644 index 000000000000..a5adbc5ad873 --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/ProcessLauncher.java @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.process; + +public interface ProcessLauncher { + Process start(ProcessBuilder processBuilder); +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/Reflection.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/Reflection.java new file mode 100644 index 000000000000..18af487dc1dc --- /dev/null +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/process/Reflection.java @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more contributor license + * agreements. See the NOTICE file distributed with this work for additional information regarding + * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. You may obtain a + * copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed under the License + * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + * or implied. See the License for the specific language governing permissions and limitations under + * the License. + * + */ + +package org.apache.geode.gradle.testing.process; + +import java.lang.reflect.Field; +import java.util.Objects; +import java.util.function.BiFunction; + +/** + * Utility methods to retrieve and set otherwise inaccessible fields via reflection. + */ +public class Reflection { + /** + * Returns a {@link Field} that describes the named field in the given owner. + */ + public static Field getField(Object owner, String fieldName) { + Objects.requireNonNull(owner); + try { + return owner.getClass().getDeclaredField(fieldName); + } catch (NoSuchFieldException e) { + String message = String.format("Getting %s declaration for %s", fieldName, owner); + throw new RuntimeException(message, e); + } + } + + /** + * Returns the value of the named field from the given owner. + */ + public static Object getFieldValue(Object owner, String fieldName) { + return withAccessibleField(owner, fieldName, getValue()); + } + + /** + * Sets the value of the named field in the given owner. + */ + public static void setFieldValue(Object owner, String fieldName, Object value) { + withAccessibleField(owner, fieldName, setValue(value)); + } + + /** + * Makes a field temporarily accessible and applies the operation to it. + */ + private static Object withAccessibleField(Object owner, String fieldName, + BiFunction operation) { + Field field = getField(owner, fieldName); + boolean accessible = field.isAccessible(); + try { + field.setAccessible(true); + return operation.apply(owner, field); + } finally { + field.setAccessible(accessible); + } + } + + /** + * Creates a function that extracts the value of a field from an owner. + */ + private static BiFunction getValue() { + return (owner, field) -> { + try { + return field.get(owner); + } catch (IllegalAccessException | IllegalArgumentException e) { + String message = String.format("Getting %s (%s) value for %s", + field.getName(), field.isAccessible(), owner); + throw new RuntimeException(message, e); + } + }; + } + + /** + * Creates a function that sets a field of an owner to the given value. + */ + private static BiFunction setValue(Object value) { + return (owner, field) -> { + try { + field.set(owner, value); + } catch (IllegalAccessException | IllegalArgumentException e) { + String message = String.format("Setting %s (%s) value for %s", + field.getName(), field.isAccessible(), owner); + throw new RuntimeException(message, e); + } + return null; + }; + } +} diff --git a/buildSrc/src/main/java/org/apache/geode/gradle/OverriddenTestExecutor.java b/buildSrc/src/main/java/org/apache/geode/gradle/testing/repeat/RepeatableTestExecuter.java similarity index 63% rename from buildSrc/src/main/java/org/apache/geode/gradle/OverriddenTestExecutor.java rename to buildSrc/src/main/java/org/apache/geode/gradle/testing/repeat/RepeatableTestExecuter.java index 63434b2983df..1536ea57a276 100644 --- a/buildSrc/src/main/java/org/apache/geode/gradle/OverriddenTestExecutor.java +++ b/buildSrc/src/main/java/org/apache/geode/gradle/testing/repeat/RepeatableTestExecuter.java @@ -12,9 +12,10 @@ * or implied. See the License for the specific language governing permissions and limitations under * the License. */ -package org.apache.geode.gradle; +package org.apache.geode.gradle.testing.repeat; import java.io.File; +import java.util.List; import java.util.Set; import com.google.common.collect.ImmutableSet; @@ -28,6 +29,7 @@ import org.gradle.api.internal.tasks.testing.TestResultProcessor; import org.gradle.api.internal.tasks.testing.WorkerTestClassProcessorFactory; import org.gradle.api.internal.tasks.testing.detection.DefaultTestClassScanner; +import org.gradle.api.internal.tasks.testing.detection.DefaultTestExecuter; import org.gradle.api.internal.tasks.testing.detection.TestFrameworkDetector; import org.gradle.api.internal.tasks.testing.filter.DefaultTestFilter; import org.gradle.api.internal.tasks.testing.processors.MaxNParallelTestClassProcessor; @@ -40,38 +42,44 @@ import org.gradle.api.logging.Logging; import org.gradle.internal.Factory; import org.gradle.internal.actor.ActorFactory; -import org.gradle.internal.operations.BuildOperationExecutor; import org.gradle.internal.time.Clock; import org.gradle.internal.work.WorkerLeaseRegistry; import org.gradle.process.internal.worker.WorkerProcessFactory; /** - * Test executor that is used to replace gradles DefaultTestExecutor and does - * not include a {@link RunPreviousFailedFirstTestClassProcessor} in the processor - * chain. This is used by the RepeatTest task. + * A copy of {@link DefaultTestExecuter} from Gradle v6.8.3, modified to process each test class + * as many times as it was submitted. This is required by our {@link RepeatTest} task, because: + *
      + *
    • Geode's {@code RepeatTest} task operates by submitting each test class for processing + * multiple times.
    • + *
    • Gradle's {@code DefaultTestExecuter} includes a + * {@link RunPreviousFailedFirstTestClassProcessor}, which de-duplicates the submitted test + * classes, preventing the {@code RepeatTest} from repeating the tests.
    • + *
    + *

    + * This class omits the {@code RunPreviousFailedFirstTestClassProcessor}, and so each test class + * is processed as many times as {@code RepeatTest} submits it. See the comment in {@link #execute}. */ -class OverriddenTestExecutor implements TestExecuter { - private static final Logger LOGGER = Logging.getLogger(OverriddenTestExecutor.class); +public class RepeatableTestExecuter implements TestExecuter { + private static final Logger LOGGER = Logging.getLogger(RepeatableTestExecuter.class); private final WorkerProcessFactory workerFactory; private final ActorFactory actorFactory; private final ModuleRegistry moduleRegistry; private final WorkerLeaseRegistry workerLeaseRegistry; - private final BuildOperationExecutor buildOperationExecutor; private final int maxWorkerCount; private final Clock clock; private final DocumentationRegistry documentationRegistry; private final DefaultTestFilter testFilter; private TestClassProcessor processor; - public OverriddenTestExecutor(WorkerProcessFactory workerFactory, ActorFactory actorFactory, ModuleRegistry moduleRegistry, - WorkerLeaseRegistry workerLeaseRegistry, BuildOperationExecutor buildOperationExecutor, int maxWorkerCount, - Clock clock, DocumentationRegistry documentationRegistry, DefaultTestFilter testFilter) { + public RepeatableTestExecuter(WorkerProcessFactory workerFactory, ActorFactory actorFactory, + ModuleRegistry moduleRegistry, WorkerLeaseRegistry workerLeaseRegistry, int maxWorkerCount, + Clock clock, DocumentationRegistry documentationRegistry, DefaultTestFilter testFilter) { this.workerFactory = workerFactory; this.actorFactory = actorFactory; this.moduleRegistry = moduleRegistry; this.workerLeaseRegistry = workerLeaseRegistry; - this.buildOperationExecutor = buildOperationExecutor; this.maxWorkerCount = maxWorkerCount; this.clock = clock; this.documentationRegistry = documentationRegistry; @@ -79,27 +87,42 @@ public OverriddenTestExecutor(WorkerProcessFactory workerFactory, ActorFactory a } @Override - public void execute(final JvmTestExecutionSpec testExecutionSpec, TestResultProcessor testResultProcessor) { + public void execute(final JvmTestExecutionSpec testExecutionSpec, + TestResultProcessor testResultProcessor) { final TestFramework testFramework = testExecutionSpec.getTestFramework(); final WorkerTestClassProcessorFactory testInstanceFactory = testFramework.getProcessorFactory(); - final WorkerLeaseRegistry.WorkerLease currentWorkerLease = workerLeaseRegistry.getCurrentWorkerLease(); + final WorkerLeaseRegistry.WorkerLease + currentWorkerLease = + workerLeaseRegistry.getCurrentWorkerLease(); final Set classpath = ImmutableSet.copyOf(testExecutionSpec.getClasspath()); + final Set modulePath = ImmutableSet.copyOf(testExecutionSpec.getModulePath()); + final List + testWorkerImplementationModules = + testFramework.getTestWorkerImplementationModules(); final Factory forkingProcessorFactory = new Factory() { @Override public TestClassProcessor create() { - return new ForkingTestClassProcessor(currentWorkerLease, workerFactory, testInstanceFactory, testExecutionSpec.getJavaForkOptions(), - classpath, testFramework.getWorkerConfigurationAction(), moduleRegistry, documentationRegistry); - } - }; - final Factory reforkingProcessorFactory = new Factory() { - @Override - public TestClassProcessor create() { - return new RestartEveryNTestClassProcessor(forkingProcessorFactory, testExecutionSpec.getForkEvery()); + return new ForkingTestClassProcessor(currentWorkerLease, workerFactory, testInstanceFactory, + testExecutionSpec.getJavaForkOptions(), + classpath, modulePath, testWorkerImplementationModules, + testFramework.getWorkerConfigurationAction(), moduleRegistry, documentationRegistry); } }; + final Factory + reforkingProcessorFactory = + new Factory() { + @Override + public TestClassProcessor create() { + return new RestartEveryNTestClassProcessor(forkingProcessorFactory, + testExecutionSpec.getForkEvery()); + } + }; + // Create the chain of test class processors, omitting the + // RunPreviousFailedFirstTestClassProcessor that Gradle's DefaultTestExecuter creates. processor = new PatternMatchTestClassProcessor(testFilter, - new MaxNParallelTestClassProcessor(getMaxParallelForks(testExecutionSpec), reforkingProcessorFactory, actorFactory)); + new MaxNParallelTestClassProcessor(getMaxParallelForks(testExecutionSpec), + reforkingProcessorFactory, actorFactory)); final FileTree testClassFiles = testExecutionSpec.getCandidateClassFiles(); @@ -113,9 +136,8 @@ public TestClassProcessor create() { detector = new DefaultTestClassScanner(testClassFiles, null, processor); } - final Object testTaskOperationId = buildOperationExecutor.getCurrentOperation().getParentId(); - - new TestMainAction(detector, processor, testResultProcessor, clock, testTaskOperationId, testExecutionSpec.getPath(), "Gradle Test Run " + testExecutionSpec.getIdentityPath()).run(); + new TestMainAction(detector, processor, testResultProcessor, clock, testExecutionSpec.getPath(), + "Gradle Test Run " + testExecutionSpec.getIdentityPath()).run(); } @Override @@ -128,7 +150,8 @@ public void stopNow() { private int getMaxParallelForks(JvmTestExecutionSpec testExecutionSpec) { int maxParallelForks = testExecutionSpec.getMaxParallelForks(); if (maxParallelForks > maxWorkerCount) { - LOGGER.info("{}.maxParallelForks ({}) is larger than max-workers ({}), forcing it to {}", testExecutionSpec.getPath(), maxParallelForks, maxWorkerCount, maxWorkerCount); + LOGGER.info("{}.maxParallelForks ({}) is larger than max-workers ({}), forcing it to {}", + testExecutionSpec.getPath(), maxParallelForks, maxWorkerCount, maxWorkerCount); maxParallelForks = maxWorkerCount; } return maxParallelForks; diff --git a/buildSrc/src/main/resources/META-INF/gradle-plugins/com.github.pedjak.dockerized-test.properties b/buildSrc/src/main/resources/META-INF/gradle-plugins/com.github.pedjak.dockerized-test.properties deleted file mode 100644 index 1cfe2cb87aab..000000000000 --- a/buildSrc/src/main/resources/META-INF/gradle-plugins/com.github.pedjak.dockerized-test.properties +++ /dev/null @@ -1 +0,0 @@ -implementation-class = com.pedjak.gradle.plugins.dockerizedtest.DockerizedTestPlugin \ No newline at end of file diff --git a/buildSrc/src/main/resources/META-INF/gradle-plugins/geode-dockerized-test.properties b/buildSrc/src/main/resources/META-INF/gradle-plugins/geode-dockerized-test.properties new file mode 100644 index 000000000000..1ea705cb1680 --- /dev/null +++ b/buildSrc/src/main/resources/META-INF/gradle-plugins/geode-dockerized-test.properties @@ -0,0 +1 @@ +implementation-class = org.apache.geode.gradle.testing.dockerized.DockerizedTestPlugin \ No newline at end of file diff --git a/ci/scripts/execute_build.sh b/ci/scripts/execute_build.sh index ebb7ed92687d..cd91a9a42f16 100755 --- a/ci/scripts/execute_build.sh +++ b/ci/scripts/execute_build.sh @@ -77,7 +77,7 @@ scp ${SSH_OPTIONS} ${SCRIPTDIR}/capture-call-stacks.sh geode@${INSTANCE_IP_ADDRE if [[ -n "${PARALLEL_DUNIT}" && "${PARALLEL_DUNIT}" == "true" ]]; then PARALLEL_DUNIT="-PparallelDunit -PdunitDockerUser=geode" if [ -n "${DUNIT_PARALLEL_FORKS}" ]; then - DUNIT_PARALLEL_FORKS="-PdunitParallelForks=${DUNIT_PARALLEL_FORKS} --max-workers=${DUNIT_PARALLEL_FORKS} -PtestMaxParallelForks=${DUNIT_PARALLEL_FORKS}" + DUNIT_PARALLEL_FORKS="--max-workers=${DUNIT_PARALLEL_FORKS} -PtestMaxParallelForks=${DUNIT_PARALLEL_FORKS}" fi else PARALLEL_DUNIT="" diff --git a/ci/scripts/execute_tests.sh b/ci/scripts/execute_tests.sh index 52148c79e562..916cc305f672 100755 --- a/ci/scripts/execute_tests.sh +++ b/ci/scripts/execute_tests.sh @@ -64,7 +64,7 @@ scp ${SSH_OPTIONS} ${SCRIPTDIR}/capture-call-stacks.sh geode@${INSTANCE_IP_ADDRE if [[ -n "${PARALLEL_DUNIT}" && "${PARALLEL_DUNIT}" == "true" ]]; then PARALLEL_DUNIT="-PparallelDunit -PdunitDockerUser=geode -PdunitDockerImage=\$(docker images --format '{{.Repository}}:{{.Tag}}')" if [ -n "${DUNIT_PARALLEL_FORKS}" ]; then - DUNIT_PARALLEL_FORKS="-PdunitParallelForks=${DUNIT_PARALLEL_FORKS} --max-workers=${DUNIT_PARALLEL_FORKS} -PtestMaxParallelForks=${DUNIT_PARALLEL_FORKS}" + DUNIT_PARALLEL_FORKS="--max-workers=${DUNIT_PARALLEL_FORKS} -PtestMaxParallelForks=${DUNIT_PARALLEL_FORKS}" fi else PARALLEL_DUNIT="" diff --git a/etc/apache-copyright-notice.txt b/etc/apache-copyright-notice.txt new file mode 100644 index 000000000000..58fd5efd3dac --- /dev/null +++ b/etc/apache-copyright-notice.txt @@ -0,0 +1,12 @@ +Licensed to the Apache Software Foundation (ASF) under one or more contributor license +agreements. See the NOTICE file distributed with this work for additional information regarding +copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance with the License. You may obtain a +copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed under the License +is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +or implied. See the License for the specific language governing permissions and limitations under +the License. \ No newline at end of file diff --git a/etc/intellij-apache-copyright-notice.xml b/etc/intellij-apache-copyright-notice.xml new file mode 100644 index 000000000000..940ad4fd6ec0 --- /dev/null +++ b/etc/intellij-apache-copyright-notice.xml @@ -0,0 +1,6 @@ + + + + \ No newline at end of file diff --git a/geode-assembly/build.gradle b/geode-assembly/build.gradle index 3919cbb7a317..a92d8735cd4b 100755 --- a/geode-assembly/build.gradle +++ b/geode-assembly/build.gradle @@ -125,14 +125,16 @@ artifacts { compositeTarget distTar } -//This "repository" only exists to download tomcat-6, because the zip for tomcat 6 is -//not in a maven repo. Later versions of tomcat are. repositories { + //This "repository" only exists to download tomcat-6, because the zip for tomcat 6 is + //not in a maven repo. Later versions of tomcat are. ivy { url 'https://archive.apache.org/' patternLayout { artifact '/dist/tomcat/tomcat-6/v6.0.37/bin/[organisation]-[module]-[revision].[ext]' } + // Infer the metadata from the presence of the artifact + metadataSources { artifact() } } // For gradle tooling dependencies maven { diff --git a/gradle.properties b/gradle.properties index a6c23cd904b9..72924f6b682c 100755 --- a/gradle.properties +++ b/gradle.properties @@ -47,7 +47,7 @@ buildId = 0 productName = Apache Geode productOrg = Apache Software Foundation (ASF) -minimumGradleVersion = 5.5 +minimumGradleVersion = 6.8 # Set this on the command line with -P or in ~/.gradle/gradle.properties # to change the buildDir location. Use an absolute path. buildRoot= @@ -55,8 +55,6 @@ buildRoot= # We want signing to be on by default. Signing requires GPG to be set up. nexusSignArchives = true -# Control how many concurrent dunit (using docker) tests will be run -dunitParallelForks = 8 # This is the name of the Docker image for running parallel dunits dunitDockerImage = apachegeode/geode-build # Docker user for parallel dunit tests diff --git a/gradle/docker.gradle b/gradle/docker.gradle deleted file mode 100644 index 77541fd80a19..000000000000 --- a/gradle/docker.gradle +++ /dev/null @@ -1,149 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/* - * Configuration for running (dunit) tests in parallel in Docker containers. - * The container used must hava JAVA_HOME set in it's environment and must - * have 'java' defined on the path. For example, the relevant Dockerfile - * content could be: - * - * ENV JAVA_HOME=/opt/jdk1.8.0_u101 - * ENV PATH=$PATH:$JAVA_HOME/bin - * - * In addition, the container must have docker installed. - * - * The plugin can be activated with the Gradle property 'parallelDunit'. - * Additional properties that can be set are: - * - * dunitDockerImage - The docker image used for running parallel dunits. The - * default image is 'bellsoft/liberica-openjdk-debian:8'. The image is required to - * have 'JAVA_HOME' set as an environment variable. - * dunitParallelForks - The number of parallel containers that will be - * launched. The default is 8. - * dunitDockerUser - The user used within the docker container to run tests. - * The default is 'root'. - */ - -apply plugin: 'com.github.pedjak.dockerized-test' - - -if (project.hasProperty('parallelDunit')) { - def pwd = System.getenv('PWD') - def geodeDir = new File(pwd).getCanonicalPath() - project.ext.dunitDockerVolumes = ["${geodeDir}":geodeDir] -} - -project.ext.dockerConfig = { - maxParallelForks = dunitParallelForks.toInteger() - - docker { - // base image for creating docker containers that execute the tests - image = dunitDockerImage - - // volumes mounted to the containers - // in a form: host_dir : container_dir - def gradleHome = System.getenv('GRADLE_USER_HOME') ?: "${System.getenv('HOME')}/.gradle" - volumes = ["${gradleHome}":gradleHome] - - // Add volumes configured by top-level build script - volumes << project.dunitDockerVolumes - - // specify the user for starting Gradle test worker within the container. - user = dunitDockerUser - - beforeContainerCreate = { cmd, client -> - def javaHomeIdx = -1 - def pathIdx = -1 - def tmpEnv = [] - cmd.getEnv().each { tmpEnv << it } - - tmpEnv.eachWithIndex { x, j -> - if (x.startsWith('JAVA_HOME')) { - javaHomeIdx = j - } - if (x.startsWith('PATH')) { - pathIdx = j - } - } - - // Remove JAVA_HOME and PATH env variables - they might not be the same as the container needs - if (javaHomeIdx >= 0) { - tmpEnv[javaHomeIdx] = 'JAVA_HOME_REMOVED=' - } - if (pathIdx >= 0) { - tmpEnv[pathIdx] = 'PATH_REMOVED=' - } - - if (project.hasProperty('testJVM') && !testJVM.trim().isEmpty()) { - // Docker command is just 'java' so set to full path - tmpEnv << ("JAVA_HOME=${project.testJVM}" as String) - } - - - // Unfortunately this snippet of code is here and is required by dev-tools/docker/base/entrypoint.sh. - // This allows preserving the outer user inside the running container. Required for Jenkins - // and other environments. There doesn't seem to be a way to pass this environment variable - // in from a Jenkins Gradle job. - if (System.env['LOCAL_USER_ID'] == null) { - def username = System.getProperty("user.name") - def uid = ['id', '-u', username].execute().text.trim() - tmpEnv << ("LOCAL_USER_ID=${uid}" as String) - } - - cmd.withEnv(tmpEnv) - - // Infer the index of this invocation - def cmdList = cmd.getCmd() - - if (project.hasProperty('testJVM') && !testJVM.trim().isEmpty()) { - // Docker command is just 'java' so set to full path - cmdList[0] = ("${project.testJVM}/bin/java" as String) - } - - // copy the classpath file to the working dir - def classPathFileIndex = cmdList.findIndexOf { it =~ /^@.*gradle-worker-classpath.*txt$/ } - if (classPathFileIndex > 0) { - def dst = new File(cmd.getWorkingDir(), "gradle-worker-classpath.txt") - if (!dst.exists()) { - def src = new File(cmdList[classPathFileIndex].substring(1)) - dst.write(src.text) - } - cmdList[classPathFileIndex] = '@'+dst.toString() - } - - //println cmd - } - } -} - - -if (project.hasProperty('parallelDunit')) { - uiTest.configure(project.ext.dockerConfig) - repeatUnitTest.configure(project.ext.dockerConfig) - - integrationTest.configure(project.ext.dockerConfig) - repeatIntegrationTest.configure(project.ext.dockerConfig) - - distributedTest.configure(project.ext.dockerConfig) - repeatDistributedTest.configure(project.ext.dockerConfig) - - upgradeTest.configure(project.ext.dockerConfig) - repeatUpgradeTest.configure(project.ext.dockerConfig) - - acceptanceTest.configure(project.ext.dockerConfig) - repeatAcceptanceTest.configure(project.ext.dockerConfig) -} diff --git a/gradle/multi-process-test.gradle b/gradle/multi-process-test.gradle new file mode 100644 index 000000000000..90144a22216c --- /dev/null +++ b/gradle/multi-process-test.gradle @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Configuration for running multi-process tests in parallel. See the 'multiProcessTestTasks' + * variable for the list of multi-process test tasks. + * + * To run multi-process tests in Docker, set the Gradle property 'parallelDunit'. + * The container used must have JAVA_HOME set in its environment and must + * have 'java' defined on the path. For example, the relevant Dockerfile + * content could be: + * + * ENV JAVA_HOME=/opt/jdk1.8.0_u101 + * ENV PATH=$PATH:$JAVA_HOME/bin + * + * In addition, the container must have docker installed. + * + * Additional configuration properties for running multi-process tests in Docker are: + * + * dunitDockerImage - The docker image used for running parallel dunits. The default image is + * 'bellsoft/liberica-openjdk-debian:8'. The image is required to have + * 'JAVA_HOME' set as an environment variable. + * dunitDockerUser - The user used within the docker container to run tests. + * The default is 'root'. + * dunitDockerVolumes - Docker volumes to mount in the docker container in addition to the ones + * mounted by the plugin. The plugin always mounts the directory of the + * top-level Geode project and the current user's Gradle home directory. + * dunitDockerJVM - The JVM to use to launch test worker processes inside Docker. The default + * is the value of the testJVM property if that property is set. + * + * The following properties apply to multi-process tests, whether running in Docker or not: + * + * --max-workers The maximum number of workers processes for Gradle to run in parallel. + * (Note that Gradle applies this value to all tasks, not just test tasks.) + * + * testMaxParallelForks The maximum number of tests for each multi-process test task to process in + * parallel. If 'parallelDunit' us defined, `testMaxParallelForks` defaults to + * 1/4 of the machine's available processors. If 'parallelDunit' is not + * defined, the default 'testMaxParallelForks' is 1. (Note that test.gradle + * also applies this property to unit test tasks, with different defaults.) + */ + +import org.apache.geode.gradle.testing.Executers +import org.apache.geode.gradle.testing.isolation.WorkingDirectoryIsolator + +def multiProcessTestTasks = [acceptanceTest, repeatAcceptanceTest, + distributedTest, repeatDistributedTest, + integrationTest, repeatIntegrationTest, + upgradeTest, repeatUpgradeTest, + uiTest, repeatUnitTest] + +if (project.hasProperty('parallelDunit')) { + def parallelForks = project.hasProperty('testMaxParallelForks') + ? Integer.parseUnsignedInt(project.testMaxParallelForks) + : Runtime.runtime.availableProcessors().intdiv(4) ?: 1 + for (task in multiProcessTestTasks) { + task.maxParallelForks = parallelForks + } + apply plugin: 'geode-dockerized-test' +} else { + for (task in multiProcessTestTasks) { + if (project.hasProperty('testMaxParallelForks')) { + task.maxParallelForks = Integer.parseUnsignedInt(project.testMaxParallelForks) + } + task.doFirst { + testExecuter = Executers.withAdjustment(it, new WorkingDirectoryIsolator()) + } + } +} diff --git a/gradle/standard-subproject-configuration.gradle b/gradle/standard-subproject-configuration.gradle index 49916eb65992..e3e944a87ca8 100644 --- a/gradle/standard-subproject-configuration.gradle +++ b/gradle/standard-subproject-configuration.gradle @@ -19,7 +19,7 @@ apply from: "${rootDir}/${scriptDir}/java.gradle" apply from: "${rootDir}/${scriptDir}/dependency-resolution.gradle" apply from: "${rootDir}/${scriptDir}/test.gradle" apply from: "${rootDir}/${scriptDir}/code-analysis.gradle" -apply from: "${rootDir}/${scriptDir}/docker.gradle" +apply from: "${rootDir}/${scriptDir}/multi-process-test.gradle" apply from: "${rootDir}/${scriptDir}/spotless.gradle" apply from: "${rootDir}/${scriptDir}/ide.gradle" apply plugin: 'com.github.ben-manes.versions' diff --git a/gradle/test.gradle b/gradle/test.gradle index e601f5c28872..7c0fb78cac98 100644 --- a/gradle/test.gradle +++ b/gradle/test.gradle @@ -1,7 +1,7 @@ import org.apache.geode.gradle.TestPropertiesWriter -import org.apache.geode.gradle.RepeatTest + import org.apache.geode.gradle.plugins.DependencyConstraints -import org.apache.geode.gradle.RunInSubdirectoryTestFramework +import org.apache.geode.gradle.testing.repeat.RepeatTest /* * Licensed to the Apache Software Foundation (ASF) under one or more @@ -41,6 +41,7 @@ test { } } + apply plugin: 'nebula.facet' facets { integrationTest { @@ -139,17 +140,6 @@ configure([integrationTest, distributedTest, performanceTest, acceptanceTest, ui } } -configure([acceptanceTest, distributedTest, integrationTest, uiTest, upgradeTest, repeatAcceptanceTest, repeatDistributedTest, repeatIntegrationTest, repeatUnitTest, repeatUpgradeTest]) { - doFirst { - // Wrap the task's test framework in a wrapper that runs each test worker JVM in a unique - // subdirectory. - def subdirFramework = new RunInSubdirectoryTestFramework(testFramework) - // This call works for now, but the Test class declares useTestFramework() as protected, so - // this could become troublesome in a future version of Gradle/Groovy. - useTestFramework subdirFramework - } -} - configure([repeatDistributedTest, repeatIntegrationTest, repeatUpgradeTest, repeatUnitTest, repeatAcceptanceTest]) { times = Integer.parseInt(repeat) useJUnit {} @@ -212,7 +202,7 @@ gradle.taskGraph.whenReady({ graph -> // The distributed tests seem to need to use /tmp directly, // so exclude them from using the supplied temp directory. - if (!test.name.contains("distributed")) { + if (!test.name.contains("distributed") && !test.name.contains("repeatDistributed")) { systemProperty 'java.io.tmpdir', System.getProperty('java.io.tmpdir') } @@ -238,7 +228,7 @@ gradle.taskGraph.whenReady({ graph -> acceptanceTest { // Acceptance tests may reach out and run a gfsh command from the assembled gfsh. // If the test JVM version is specified, we must have the correct JAVA_HOME set for gfsh to use. - // See also environment configuration for parallel testing in docker.gradle + // See also environment configuration for parallel testing in multi-process-test.gradle if (project.hasProperty('testJVM') && !testJVM.trim().isEmpty()) { environment "JAVA_HOME", "${project.testJVM}" } diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar index 87b738cbd051..e708b1c023ec 100644 Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index e0c4de36dddb..8cf6eb5ad222 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,5 +1,5 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-5.5-all.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.3-all.zip zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists diff --git a/gradlew b/gradlew index cccdd3d517fc..4f906e0c811f 100755 --- a/gradlew +++ b/gradlew @@ -1,5 +1,21 @@ #!/usr/bin/env sh +# +# Copyright 2015 the original author or authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + ############################################################################## ## ## Gradle start up script for UN*X @@ -28,7 +44,7 @@ APP_NAME="Gradle" APP_BASE_NAME=`basename "$0"` # Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. -DEFAULT_JVM_OPTS="" +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' # Use the maximum available, or set MAX_FD != -1 to use that value. MAX_FD="maximum" @@ -66,6 +82,7 @@ esac CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + # Determine the Java command to use to start the JVM. if [ -n "$JAVA_HOME" ] ; then if [ -x "$JAVA_HOME/jre/sh/java" ] ; then @@ -109,10 +126,11 @@ if $darwin; then GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\"" fi -# For Cygwin, switch paths to Windows format before running java -if $cygwin ; then +# For Cygwin or MSYS, switch paths to Windows format before running java +if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then APP_HOME=`cygpath --path --mixed "$APP_HOME"` CLASSPATH=`cygpath --path --mixed "$CLASSPATH"` + JAVACMD=`cygpath --unix "$JAVACMD"` # We build the pattern for arguments to be converted via cygpath @@ -138,19 +156,19 @@ if $cygwin ; then else eval `echo args$i`="\"$arg\"" fi - i=$((i+1)) + i=`expr $i + 1` done case $i in - (0) set -- ;; - (1) set -- "$args0" ;; - (2) set -- "$args0" "$args1" ;; - (3) set -- "$args0" "$args1" "$args2" ;; - (4) set -- "$args0" "$args1" "$args2" "$args3" ;; - (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; - (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; - (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; - (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; - (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; + 0) set -- ;; + 1) set -- "$args0" ;; + 2) set -- "$args0" "$args1" ;; + 3) set -- "$args0" "$args1" "$args2" ;; + 4) set -- "$args0" "$args1" "$args2" "$args3" ;; + 5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; + 6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; + 7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; + 8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; + 9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; esac fi @@ -159,14 +177,9 @@ save () { for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done echo " " } -APP_ARGS=$(save "$@") +APP_ARGS=`save "$@"` # Collect all arguments for the java command, following the shell quoting and substitution rules eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS" -# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong -if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then - cd "$(dirname "$0")" -fi - exec "$JAVACMD" "$@" diff --git a/gradlew.bat b/gradlew.bat index e95643d6a2ca..ac1b06f93825 100644 --- a/gradlew.bat +++ b/gradlew.bat @@ -1,3 +1,19 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem + @if "%DEBUG%" == "" @echo off @rem ########################################################################## @rem @@ -13,15 +29,18 @@ if "%DIRNAME%" == "" set DIRNAME=. set APP_BASE_NAME=%~n0 set APP_HOME=%DIRNAME% +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. -set DEFAULT_JVM_OPTS= +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" @rem Find java.exe if defined JAVA_HOME goto findJavaFromJavaHome set JAVA_EXE=java.exe %JAVA_EXE% -version >NUL 2>&1 -if "%ERRORLEVEL%" == "0" goto init +if "%ERRORLEVEL%" == "0" goto execute echo. echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. @@ -35,7 +54,7 @@ goto fail set JAVA_HOME=%JAVA_HOME:"=% set JAVA_EXE=%JAVA_HOME%/bin/java.exe -if exist "%JAVA_EXE%" goto init +if exist "%JAVA_EXE%" goto execute echo. echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% @@ -45,28 +64,14 @@ echo location of your Java installation. goto fail -:init -@rem Get command-line arguments, handling Windows variants - -if not "%OS%" == "Windows_NT" goto win9xME_args - -:win9xME_args -@rem Slurp the command line arguments. -set CMD_LINE_ARGS= -set _SKIP=2 - -:win9xME_args_slurp -if "x%~1" == "x" goto execute - -set CMD_LINE_ARGS=%* - :execute @rem Setup the command line set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + @rem Execute Gradle -"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS% +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* :end @rem End local scope for the variables with windows NT shell