Questions about Flacoco

[danglotb]

Hello,

I'm using Flacoco in the context of detecting energy regressions, based on the energy consumption of the tests.

The idea is the following:

First, we have a program in two versions P and P' where P is the program before applying a commit, and P' is the program after applying the commit.

I take the tests that execute the lines modified and compute their respective Software Energy Consumption (SEC) of both versions of the program.

Then, for each test t that has SEC(t, P') - SEC(t, P) > 0, meaning that the commit increases the SEC of the test t, I make them fail by inserting a junit.framework.Assert.fail() at the end of their body.

Then, I give to Flacoco only the tests that are now failing. I take the ranked list of lines, and filter it by keeping only the modified lines.

In your opinion, is it a proper way to use Flacoco? Should I also use the "passing" tests in Flacoco? Does the junit.framework.Assert.fail() significant regarding the fault localization accuracy of Flacoco?

Thank you very much!

[andre15silva]

Hello,

Hi @danglotb ,

I'm using Flacoco in the context of detecting energy regressions, based on the energy consumption of the tests.

Nice!

The idea is the following:

First, we have a program in two versions P and P' where P is the program before applying a commit, and P' is the program after applying the commit.

I take the tests that execute the lines modified and compute their respective Software Energy Consumption (SEC) of both versions of the program.

Then, for each test t that has SEC(t, P') - SEC(t, P) > 0, meaning that the commit increases the SEC of the test t, I make them fail by inserting a junit.framework.Assert.fail() at the end of their body.

Then, I give to Flacoco only the tests that are now failing. I take the ranked list of lines, and filter it by keeping only the modified lines.

If I understood correctly, what you are doing is essentially computing the code coverage of the tests that you implicitly fail. Correct?

In your opinion, is it a proper way to use Flacoco?

You can use Flacoco to compute code coverage in this way, however it might make more sense to just use a code coverage tool directly.

Should I also use the "passing" tests in Flacoco?

If you don't use the passing test cases, all executed lines will have a suspiciousness value of 1.0.

If you pass the test cases the suspiciousness values will take them into account. This means that lines predominantly executed by failing test cases will have higher values than others.

Please note that we currently have a limitation #57, which means we can only select test classes.
If you want to run specific test methods within a class, test-runner supports this and it can be easily done through flacoco. I can setup a branch for you if that's the case. You'll, ofc, be limited by the problem in #57.

Does the junit.framework.Assert.fail() significant regarding the fault localization accuracy of Flacoco?

What do you mean by "fault localization accuracy"? By implicitly failing a test case, it will be counted as a failing test case like any other.

[danglotb]

Hi @andre15silva,

Thank you for your answers.

If I understood correctly, what you are doing is essentially computing the code coverage of the tests that you implicitly fail. Correct?

Not really, what I want is to compute a score for each modified line that corresponds to their suspiciousness regarding the energy regression.

This scenario can be seen as classical regression, where you want to find out which lines seem to be responsible for the bug, that is detected by a failing test case.
However, in my case, no test case is failing, so I make artificially the ones that reveal the energy regression failing, and then I can apply Flacoco to have the ranking.

What do you mean by "fault localization accuracy"? By implicitly failing a test case, it will be counted as a failing test case like any other.

What I want to know is that does the error that makes the test failing has an impact on the result of Flacoco? This junit.framework.Assert.fail() provides a stack trace without any link with the "production" code. I am afraid that it would result in a less accurate result.

Best

[andre15silva]

Not really, what I want is to compute a score for each modified line that corresponds to their suspiciousness regarding the energy regression.

Ah okay. Then in that case you want to pass the entire test suite. The more tests you pass the more accurate results should be (depending on test quality, coverage, etc).

What I want to know is that does the error that makes the test failing has an impact on the result of Flacoco? This junit.framework.Assert.fail() provides a stack trace without any link with the "production" code. I am afraid that it would result in a less accurate result.

I see. You shouldn't worry about that. Since the stack-trace won't include any lines in the code that is being analyzed, no lines from it will be added to the result. We only add lines from the stack-trace that correspond to lines in classes that are present in the coverage report from Jacoco.

[danglotb]

Ok thank you very much! This is much clearer.

You can close this issue since you answered all my interrogations, for now! 😃