Set exit code status according to sitespeed test command errors criteria

[vs-odessa]

Currently, by default sitespeed test always has the successful exit status code 0 despite any errors that occurred during commands execution (navigate, click, wait by, etc.).
So, if something went wrong (for example, some web element can't be found or clicked) - eventually we get a successful test run anyway.
As I see, the exit status code can be managed by different criteria using budget configuration only:
https://www.sitespeed.io/documentation/sitespeed.io/performance-budget/
But there's no option for how to handle command errors themselves.
Am I missing anything in the documentation?

In total, I'd like to:

• Set eventual exit status code depending on errors in test execution.

• Stop tests execution on getting a specified number or percentage of errors.

Is it possible with existing features?

[soulgalore]

Hi @vs-odessa if you create an error in your scripting https://www.sitespeed.io/documentation/sitespeed.io/scripting/#error-handling sitespeed.io will exit with an error (I need to update the documentation), is that something that works for you?

[vs-odessa]

Hi @soulgalore Hmm, it works, but somehow it was suppressed by the following code and turned again to 0, will investigate it...
What is the expected behavior after setting a custom error - should it continue next steps in iteration, start a new iteration, or completely stop the test run?

[soulgalore]

Aha I see. It actually continues on with the next iteration, maybe we should make that configurable.

[soulgalore]

This took some time, sorry. With #4044 I've changed so that if the exit code is set within sitespeed.io it doesn't change. So in your script if you want to make the run as a failure you can set process.exitCode = 1;

That would at least help to get CI:s to fail.

[soulgalore]

Since the latest release you can use https://www.sitespeed.io/documentation/sitespeed.io/scripting/Commands.html#markAsFailure and the exit code will be an error or you can choose to set the process.exitCode in your script.