Session is reset after failing test?

[luklapp]

I have multiple test methods with shared sessionStrategy. In the first test, i visit my URL. If this test is passing, the other tests work as well. Otherwise, the session is reset to a blank page. Maybe it's a feature, not a bug? But I think it should be saved.

Code:

class ExampleTest extends BrowserTestCase
{

    public static $browsers = array(
        array(
            'driver' => 'selenium2',
            'host' => 'localhost',
            'port' => 4444,
            'browserName' => 'firefox',
            'baseUrl' => 'http://localhost',
            'sessionStrategy' => 'shared'
        ),
    );

    public function test1()
    {   
        $session = $this->getSession();
        $session->visit( 'http://localhost/test' );
        $this->assertTrue( 1 == 2 );
    }

    public function test2()
    {
        // blank page if test1 is failing
        file_put_contents('/tmp/screenshot.png', $this->getSession()->getDriver()->getScreenshot());

    }

}

[aik099]

It's a feature. When using shared session strategy the cookies are shared between all tests in a test case. This allows to have complex test cases, that are combined of several commonly used tests (e.g. testLogin, testAddProductToCart, testPayWithCreditCard). If one of tests failed, then possibly there are some left-overs in the browser that might interfere with other tests. For that reason browser window is closed.

What you though (not sure if I've explained that in docs) the shared session strategy is for?

[luklapp]

I know what the shared session strategy is for, I just thougt that the tab isn't closed if one test fails. Is there any other way than calling $session->visit() in every test? I'm just searching for different strings, but don't want them to be in just one test.

Thanks for the fast answer!

[stof]

@luklapp be careful that it is possible to have tests running without running the others. If they depend on the outcome of the previous test, you should use the @depend feature of PHPUnit, which will skip the second test if the first one failed, recognizing that it depends on the work done in the first one

[aik099]

Failing test is bad in any case, so if window/tab is closed, that shouldn't bother anybody really.

Also one thing about @depends annotation: the tests that use it must be declared below ones, that they depend upon. This is technical limitation of PHPUnit.

Now, once @stof mentioned @depends annotation this kind of defeats the purpose of shared session strategy all together, because you need to manually specify which tests require which other test(-s) to pass to ensure specific test execution order. In contract in Behat you got the chain automatically by defining scenario. But what's bad in Behat is that you've got feature file as a mix of totally unrelated steps which turns out maintenance of the tests into a nightmare.

[luklapp]

@stof @aik099 Thanks for clarification. They don't depend on each other, they search just different strings, but on the same page (without any interaction). Should I put them in the same test, altough I am testing different things? It's possible that one test fails, but the others are working.

[aik099]

If test structure is similar, then I tend to have 1 test + specify all strings via data provider method.

[luklapp]

OK, thanks for support!

[stof]

@aik099 Behat will still reset sessions. It will simply do a soft-reset rather than a hard reset. It does not share the session state between scenarios (and this would be rejected as Behat considers that scenarios are meant to be fully isolated).

And if the test relies on the state generated by the previous one, it is precisely the use case for @depends

[aik099]

Behat will still reset sessions. It will simply do a soft-reset rather than a hard reset.

Yeah, I know and that's cool. I'm also planning to switch to soft reset permanently (at some point in #5) or make it configurable, but I'm not sure how to technically implement that knowing architecture of the library.

And if the test relies on the state generated by the previous one, it is precisely the use case for @depends

Then all is fine and works as expected. That's good to know.