Skip to content

Commit

Permalink
Configure evaluation module in local environment
Browse files Browse the repository at this point in the history
  • Loading branch information
@javiertuya
javiertuya committed Jul 13, 2024
1 parent 7e28a73 commit 4d0a414
Show file tree
Hide file tree
Showing 9 changed files with 156 additions and 125 deletions.
39 changes: 23 additions & 16 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,42 @@

This project contains the system and integration tests of *Test Data Generation for APIs* (TDG).
Includes the required test classes, result files and the SUT projects,
along with an additional project (to convert into module) to evaluate the mutation
along with an additional module to evaluate the mutation
score of the tests for one of the SUTs.

## Systems Under Test (SUTs)

- Swagger Petstore (`swagger-petstore-main-fork`).
- Swagger Petstore (`sut-petstore` folder).
This is a detached fork of https://github.com/swagger-api/swagger-petstore at v1.0.17 (2022-09-30) with some additions
- Market (`swagger-market-fork`).
- Market (`sut-market` folder).
This is a detached fork of https://github.com/aleksey-lukyanets/market (2022-02-07) with some additions
- Gestao Hospital (`swagger-gestaoHospital-fork`).
- Gestao Hospital (`sut-gestaoHospital` folder).
This is a detached fork of https://github.com/ValchanOficial/GestaoHospital (2023-07-05) with some additions

## Structure of this project

- Module `st-tdg-test`:
- Java tests (`src/test/java`) for each SUT.
- Files for test result comparison (`src/test/resources`)
- A folder for each of the SUTs that are evaluated in this replication package.
- A folder `setup` with scripts to execute each SUT (backend and databases).
- Clasess `*Petstore0*` contain some commented examples to illustrate
some details of the TDG approach.
- There is a common base class for all tests (`BaseAll`) and
Each namespace has an additional base class
with the particular configuration of each SUT.
- A folder for each of the SUTs that are tested in this projects.
- A folder `setup` with scripts to run each SUT (backend and databases).
- Module `st-tdg-eval`: To evaluate the effectiveness of the test data my measuring
the mutation score of the tests using PIT (pitest.org).
the mutation score of the tests using PIT (https://pitest.org/).

## How to execute the System Tests

All experiments are enclosed in the Java tests.
The tests are in the `st-tdg-test` module.

To run the tests of a SUT in your local development environment:
- Ensure that the port that uses the SUT is not used by other SUT
- Ensure that the port bindings of each SUT are not used by other application
- Run the server from the `setup` folder and wait until it is up
- Execute the tests in the namespace that corresponds with the project
- To execute from maven (e.g. for the petstore): `mvn -pl st-tdg-test`
- Execute the tests in the namespace that corresponds with the SUT
- To execute this module from maven: `mvn -pl st-tdg-test test`

Port reservation and endpoints: To avoid conflicts when running in local,
different host port are reserved for each SUT.
Expand All @@ -41,7 +46,7 @@ Below are the ports and the main endpoint urls to check that SUTs are working:
[api example](http://localhost:8081/api/v3/pet/findByStatus?status=available) |
[swagger-ui](http://localhost:8081/) |
[api-docs](http://localhost:8081/api/v3/openapi.json)
- Market: 8082 (web), 8083 (api), 8084 (db)
- Market: 8082 (web), 8083 (api), 8087 (db)
[api example](http://localhost:8083/products) |
[swagger-ui](http://localhost:8083/swagger-ui/index.html) |
[api-docs](http://localhost:8083/v2/api-docs)
Expand All @@ -51,14 +56,16 @@ Below are the ports and the main endpoint urls to check that SUTs are working:

The whole sequence of tests for all SUTs is run when executed in CI

## How to evaluate the mutation score
## How to evaluate the Mutation Score

At `st-tdg-eval` run this maven command:
The tests are located in the `st-tdg-eval` module.
Run this maven command:
```
mvn test-compile org.pitest:pitest-maven:mutationCoverage
mvn -pl st-tdg-eval test-compile org.pitest:pitest-maven:mutationCoverage
```

Test results are in the `target` folder, and mutation report in `target/pit-reports`
Test results of the module are in the `target` folder,
and the mutation report in `target/pit-reports`

## Graphical TDM models

Expand Down
15 changes: 1 addition & 14 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

<modules>
<module>st-tdg-test</module>
<module>st-tdg-eval</module>
</modules>

<dependencyManagement>
Expand Down Expand Up @@ -89,20 +90,6 @@
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<!-- Sets the VM argument line used when unit tests are run under JaCoCo -->
<argLine>${surefireArgLine}</argLine>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<!-- evita fallo con jenkins slave linux y openjdk: https://stackoverflow.com/questions/23260057/the-forked-vm-terminated-without-saying-properly-goodbye-vm-crash-or-system-exi/53070605 -->
<useSystemClassLoader>false</useSystemClassLoader>
<skipTests>${skipTests}</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
Expand Down
48 changes: 48 additions & 0 deletions st-tdg-eval/.classpath
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
<?xml version="1.0" encoding="UTF-8"?>
<classpath>
<classpathentry excluding="**/io/swagger/petstore/controller/**|**/io/swagger/petstore/exception/**|**/io/swagger/petstore/utils/**" kind="src" output="target/classes" path="src/main/java">
<attributes>
<attribute name="optional" value="true"/>
<attribute name="maven.pomderived" value="true"/>
</attributes>
</classpathentry>
<classpathentry kind="src" path="petstoreMainResources"/>
<classpathentry excluding="io/swagger/petstore/controller/|io/swagger/petstore/exception/|io/swagger/petstore/utils/" kind="src" path="petstoreMainJava"/>
<classpathentry kind="src" output="target/test-classes" path="src/test/java">
<attributes>
<attribute name="test" value="true"/>
<attribute name="optional" value="true"/>
<attribute name="maven.pomderived" value="true"/>
</attributes>
</classpathentry>
<classpathentry excluding="**" kind="src" output="target/test-classes" path="src/test/resources">
<attributes>
<attribute name="test" value="true"/>
<attribute name="maven.pomderived" value="true"/>
<attribute name="optional" value="true"/>
</attributes>
</classpathentry>
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.8">
<attributes>
<attribute name="maven.pomderived" value="true"/>
</attributes>
</classpathentry>
<classpathentry kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER">
<attributes>
<attribute name="maven.pomderived" value="true"/>
</attributes>
</classpathentry>
<classpathentry excluding="**/io/swagger/petstore/controller/**|**/io/swagger/petstore/exception/**|**/io/swagger/petstore/utils/**" kind="src" output="target/classes" path="TUYA/IN2TEST/github-giis/tdrules-st-tdg/sut-petstore/src/main/java">
<attributes>
<attribute name="optional" value="true"/>
<attribute name="maven.pomderived" value="true"/>
</attributes>
</classpathentry>
<classpathentry excluding="**" kind="src" output="target/classes" path="src/main/resources">
<attributes>
<attribute name="maven.pomderived" value="true"/>
<attribute name="optional" value="true"/>
</attributes>
</classpathentry>
<classpathentry kind="output" path="target/classes"/>
</classpath>
35 changes: 35 additions & 0 deletions st-tdg-eval/.project
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>st-tdg-eval</name>
<comment></comment>
<projects>
</projects>
<buildSpec>
<buildCommand>
<name>org.eclipse.jdt.core.javabuilder</name>
<arguments>
</arguments>
</buildCommand>
<buildCommand>
<name>org.eclipse.m2e.core.maven2Builder</name>
<arguments>
</arguments>
</buildCommand>
</buildSpec>
<natures>
<nature>org.eclipse.jdt.core.javanature</nature>
<nature>org.eclipse.m2e.core.maven2Nature</nature>
</natures>
<linkedResources>
<link>
<name>petstoreMainJava</name>
<type>2</type>
<locationURI>PARENT-1-PROJECT_LOC/sut-petstore/src/main/java</locationURI>
</link>
<link>
<name>petstoreMainResources</name>
<type>2</type>
<locationURI>PARENT-1-PROJECT_LOC/sut-petstore/src/main/resources</locationURI>
</link>
</linkedResources>
</projectDescription>
6 changes: 6 additions & 0 deletions st-tdg-eval/.settings/org.eclipse.core.resources.prefs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
eclipse.preferences.version=1
encoding//TUYA/IN2TEST/github-giis/tdrules-st-tdg/sut-petstore/src/main/java=UTF-8
encoding//src/main/java=UTF-8
encoding//src/test/java=UTF-8
encoding//src/test/resources=UTF-8
encoding/<project>=UTF-8
8 changes: 8 additions & 0 deletions st-tdg-eval/.settings/org.eclipse.jdt.core.prefs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
eclipse.preferences.version=1
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8
org.eclipse.jdt.core.compiler.compliance=1.8
org.eclipse.jdt.core.compiler.problem.enablePreviewFeatures=disabled
org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning
org.eclipse.jdt.core.compiler.problem.reportPreviewFeatures=ignore
org.eclipse.jdt.core.compiler.release=disabled
org.eclipse.jdt.core.compiler.source=1.8
115 changes: 31 additions & 84 deletions st-tdg-eval/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -1,77 +1,55 @@
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>giis</groupId>
<artifactId>swagger-petstore-eval</artifactId>
<version>1.0.0-SNAPSHOT</version>

<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<tdrules.version>4.3.0</tdrules.version>
<qagrow.version>1.4.266</qagrow.version>

<!--
<swagger-parser-version>2.0.27</swagger-parser-version>
<swagger-core-version>2.1.10</swagger-core-version>
<swagger-inflector-version>2.0.5</swagger-inflector-version>
-->
</properties>

<parent>
<groupId>io.github.giis-uniovi</groupId>
<artifactId>tdrules-st-tdg</artifactId>
<version>1.5.0-SNAPSHOT</version>
</parent>
<artifactId>st-tdg-eval</artifactId>
<packaging>jar</packaging>

<name>st-tdg-eval</name>
<description>TDG - Test Data Generation for APIs - Petstore evaluation</description>
<url>http://github.com/giis-uniovi/tdrules-st-tdg</url>
<organization>
<name>Software Engineering Research Group (GIIS) - Universidad de Oviedo, ES</name>
<url>http://giis.uniovi.es/</url>
</organization>

<dependencies>
<dependency>
<groupId>io.github.giis-uniovi</groupId>
<artifactId>tdrules-client</artifactId>
<version>${tdrules.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.github.giis-uniovi</groupId>
<artifactId>tdrules-client-oa</artifactId>
<version>${tdrules.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.github.giis-uniovi</groupId>
<artifactId>tdrules-store-loader</artifactId>
<version>${tdrules.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>giis</groupId>
<artifactId>qagrow</artifactId>
<version>${qagrow.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.36</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.13</version>
<scope>test</scope>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</dependency>

<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.github.javiertuya</groupId>
<artifactId>visual-assert</artifactId>
<version>2.4.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
Expand All @@ -81,10 +59,11 @@
</dependency>

<!--
Not latest versions, but we would needto compile the petstore files.
To avoid conficts with dependencies we drop these dependencies and:
only use the model and data folders from the petstore sources
and create some dummy annotation classes to allow compile the model.
Dependencies required by swagger inflector are not latest versions,
but we would need to compile the petstore files.
To prevent potential conficts with dependencies we drop these dependencies and:
- only use the model and data folders from the petstore sources.
- create some dummy annotation classes to allow compile the model.
<dependency>
<groupId>io.swagger.core.v3</groupId>
Expand Down Expand Up @@ -122,7 +101,7 @@
<configuration>
<sources>
<source>
../swagger-petstore-main-fork/src/main/java
../sut-petstore/src/main/java
</source>
</sources>
</configuration>
Expand All @@ -132,7 +111,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.13.0</version>
<configuration>
<excludes>
<exclude>**/io/swagger/petstore/controller/**</exclude>
Expand Down Expand Up @@ -161,41 +139,10 @@
-->
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>giis</id>
<url>https://in2test.lsi.uniovi.es/xlib/maven/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
<repository>
<id>giis-snapshots</id>
<url>https://in2test.lsi.uniovi.es/xlib/maven-snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
<!-- Repositorios de snapshots en GitHub -->
<repository>
<id>github-giis-uniovi</id>
<url>https://maven.pkg.github.com/giis-uniovi/*</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
</repositories>

</project>
</project>
Loading

0 comments on commit 4d0a414

Please sign in to comment.