We support several testsuite execution report types: JUnit, JSON, HTML, and HTML with screenshots and console.
You can select which reporter to use in the reporters
config file property.
If you don't need to configure the reporters, you can simply set them as an array of strings, specifying the path to the reporter within the UIVeri5 folder. You can pass along options to the reporter instance when the reporters
option is a plain object or an array of plain objects. For example, to set the myProp
property of the reporter myReporter
to myValue
use:
reporters: [{name: 'myReporter', myProp: 'myValue'}]
These are the default settings for each testing profile:
- The console reporter is enabled for all profiles.
- No other reporters are currently enabled for the visual profile.
- The screenshot reporter is enabled for the integration profile.
The console reporter is activated by including the name ./reporter/consoleReporter
in the reporters
config file option. It is enabled by default for all testing profiles and has no further configuration. It adds execution summary in the process console logs. The console reporter respects the selected verbosity level.
Logs are added on spec end, on suite end and on complete execution end. When a spec ends, the failed expectations status and details are printed and in verbose
mode including the failure stack trace. When a suite ends, a statistic of the execution is printed with respect to specs and expectations, for example, successful and failed spec count, execution times, and expectation failure counts by type. When the entire test run completes, an overall success and failure statistic is printed along with the statistic for each suite.
The JUnit reporter is activated by including the name ./reporter/junitReporter
in the reporters
config file option.
It creates an XML file in the JUnit format which contains testsuite details and information about disabled or failed expectations. This type of report is commonly integrated in various tools, for example you can publish it in Jenkins using the JUnit plugin.
In order to make published results more readable, we've introduced two parameters - prefix
and postfix
for the suite name, which are empty strings by default. One use case for them is when you have multiple executions of the same suite with different test data or on different browsers. You can differentiate the executions by having separate config files and setting the prefix and/or postfix. The default location of the report file is target/report/junitReport.xml
. It can be changed by the reportName
parameter.
Example configuration:
reporters: [{
name: './reporter/junitReporter',
reportName: 'target/report/myReport.xml',
prefix: 'mySuitePrefix',
postfix: 'mySuitePostfix'
}]
The JSON reporter is activated by including the name ./reporter/jsonReporter
in the reporters
config file option.
It creates a JSON file containing the full overview of test execution. The information is similar to the console reporter output and includes suites and specs statistics and failed expectation details. By default the location of the report file is target/report/report.json
but it can be changed by setting the reportName
parameter:
reporters: [
{name: './reporter/jsonReporter', reportName: 'target/report/myReport.json'}
]
The HTML reporter is activated by including the name ./reporter/htmlReporter
in the reporters
config file option.
It is inteded to be used with the visual testing profile. It contains suite statistics and failed expectation details similar to the JSON report. In addition, it displays the images for each visual comparison: reference, actual and diff images for each failure and reference image for each success. You can use a different HTML template if you need to visually customize the report. By default, the location of the report file is target/report/report.html
and the template HTML is ./reporter/report.tpl.html
. They can be changed by setting respectively the reportName
and templateName
parameters:
reporters: [{
name: './reporter/htmlReporter',
reportName: 'target/report/myReport.html',
templateName: 'myTemplate.tpl.html'
}]
The HTML screenshot reporter is activated by including the name ./reporter/screenshotReporter
in the reporters
config file option. Screenshot reporting is enabled for application testing by default.
The screenshot reporter creates an HTML file with a simple representation of the test execution flow. It creates and stores screenshots on specified moments of test execution:
- on expectation completion (either successful or failed)
- before action execution (after syncing with the application and immediately before interaction). The currently supported actions are:
click
andsendKeys
.
Screenshots are in png format and follow the naming convention:
- for expectations: fullTestName_indexOfTestExpectation_status_screenshotCreationTime
- for actions: actionName_elementID_screenshotCreationTime
The default location for storing the report and screenshots is the subfolder target/report/screenshots/
of the current execution directory. It can be changed by the screenshotsRoot
parameter:
reporters: [
{name: './reporter/screenshotReporter', screenshotsRoot: 'myScreenshots/'}
]
You can configure when screenshots are taken. Currently the available options are: on expectation failure, on expectation success, and on action (including both click
and sendKeys
).
Reports can be configured by adding the screenshotReporter
and setting the takeScreenshot
option.
Example:
takeScreenshot: {
onExpectFailure: true,
onExpectSuccess: false,
onAction: false
}
- add reporter:
--confKeys=reporters[1].reportName:"target/report/jsonReport.json";reporters[1].name:"./reporter/jsonReporter";