testing-in-go.slide

Testing in Go
Test Composition Patterns, Fixtures & Helpers

Golang Bangalore - Meetup 36
8th Sept 2018

Tags: golang, patterns, testing, unittest, helpers, fixtures, suite

Praveen G Shirali
praveengshirali@gmail.com

https://github.com/pshirali/testing-in-go



* This talk covers ...

1. Unittests, and other forms of automated tests
2. Testing with Go: Tools, packages, other tools etc
3. Test code organization patterns, with examples
4. Guidelines & Best practices
5. Test fixtures & Helpers
6. Common fixture patterns



* PART 1 - Unittests, and other forms of automated tests



* Unittests

1. Pieces of code meant to test other code.
2. Unittests should follow these principles:

    Fast            --- Milliseconds per test. Second(s) (or less) for all tests
    Isolated        --- Order agnostic. No dependency on state, system, test environments etc
    Repeatable      --- Same results anywhere, anytime, any number of times
    Self-Validating --- A Test can determine by itself whether it failed or passed.
    Timely          --- Tests are written just before the the code they test



* Other automated tests

1. Integration, acceptance tests, etc.
2. They have dependencies. The dependencies are real.
3. Tests may involve managing a system state or isolated environments

    Setup    - Automatically setup the environment and dependencies
    Teardown - Clean up everything. Restore initial test state

4. Reminder: Its 2018.

: Reminder: Its 2018.
:   Infrastructure automation has been improving heavily over the last few years.
:   This has greatly influenced test patterns as well. Today, it is easier to
:   automate virtual infrastructure, test deployments, and execute tests against
:   the `real thing`. Containers also provide for easy-to-spin-and-discard
:   environments which contribute to running repeatable isolated environemnts
:   on all kinds of hosts.
:   In testing, integration tests are gaining a wider ground with the above
:   facilities. This area could expect further growth.



* PART 2 - Testing with Go



* The Go Test Runner & Tests in Go

Or simply 'go test' is the tool used to discover and execute tests in Go.

    > go help test      // description of of what 'go test' does
    > go help tesfunc   // description of the function spec
    > go test -h        // CLI flags and their description
    > go test           // discovers and executes tests in your current package

A function having the following signature:

    func TestXxx(*testing.T)

`Xxx` - Must begin with a capital letter

    TestInvalidLoginReturnsError     // valid test name
    TestarossaFerrari                // invalid test name

Test function are discovered from filenames ending with *`_test.go`*



* Test APIs, Bench Implementation

1. Go supports `exported` and `private` variable identifiers.
2. Test files which belong to a package have access to both `exported` and `private` code
3. Test files can also exist in a special package with the name `<package>_test`.
4. Files from `<package>` and `<package>_test` can co-exist in the same directory.
5. Code in `<package>_test` has access only to `exported` code from `<package>`.

-- WHY ??

1. Packages should be tested by invoking their exported API.
2. This is what an external package would use to `call-in` to the package under test.
3. Tests residing in the internal `<package>` can be used to test finer details of the implementation.
4. Benchmarks are suited for use in internal packages as they are aimed at quantifying implementation performance.


* Not covered in this talk

Go provides comprehensive set of tools to track code-coverage, benchmark, analyze,
and profile go code. These are not covered in this talk.



* PART 3 - Test code organization patterns & examples



* [EX1] A 'hello world' through tests



* [EX1] Code and results

Contents of 'samples/ex1/sample_test.go'
.code samples/ex1/sample_test.go /START OMIT/,/END OMIT/

Running 'go test' results in:
.code samples/ex1/testOutput_normal

Running 'go test -v' results in verbose output:
.code samples/ex1/testOutput_verbose



* The TB interface

Shared by both T and B test types: [[https://golang.org/pkg/testing/#TB]]

Skip

    Skip the test from the point where it's called

Log

    Log a message. (go test '-v')

Error

    Log an error. Marks the test FAIL, but continues execution.

Fatal

    - Log a fatal error.
    - Mark the test FAIL, and stop execution of the current test.
    - Execute any deferred functions.
    - Proceed to the next test.



* [EX2] Table Driven Tests & Subtests



* [EX2] Table Driven Tests - Code under test

  Can be used when a test logic needs to be executed
  with multiple sets of inputs and corresponding results.

Example: The 'hello world' of table driven tests!
Add an arbitrary number of integers and return their sum.

.code samples/ex2/adder.go /START OMIT/,/END OMIT/



* [EX2] Table Driven Tests - Test code

Iterate over test parameters and feed them into the test logic.

.code samples/ex2/adder_test.go /^func TestAdderUsingTable/,/^}/



* [EX2] But, it's still a single test!

Output of 'go test -v'

.code samples/ex2/testOutput_tables_singletest

Notice the use of t.Errorf, not t.Fatalf

.code samples/ex2/adder_test.go /START OMIT/,/END OMIT/

In order to ensure that we continue to test other paramters, should one of them fail,
't.Errorf' has been used.



* Is there a better way? -- Yes ---> Subtests

Subtests are tests within a test.

Test are named `<ParentTest>/<SubTest>`, with slash (/) separating parents from children

Ref: [[https://golang.org/pkg/testing/#hdr-Subtests_and_Sub_benchmarks]]



* [EX2] Table Driven Tests - Using subtests

.code samples/ex2/adder_test.go /^func TestAdderUsingSubtests/,/^}/



* [EX2] Table Driven Tests - Using subtests - Output

.code samples/ex2/testOutput_tables_subtest



* Try your own helper functions ...

Do experiment with simple helper functions before settling on a 3rd-party lib.
The ones below are not perfect, they are minimal (on purpose)

    // SkipIf skips the test if the condition is true
    func SkipIf(t *testing.T, condition bool, args ...interface{}) {
        if condition { t.Skip(args...) }
    }

    // Assert fatally fails the test if a condition is false
    func Assert(t *testing.T, condition bool, args ...interface{}) {
        if !condition { t.Fatal(args...) }
    }

    // Equal deeply compares two types and fatally fails if they are unequal
    import "reflect"
    func Equal(t *testing.T, lhs, rhs interface{}, args ...interface{}) {
        if !reflect.DeepEqual(lhs, rhs) { t.Fatal(args...) }
    }

The implementation above is used in code samples in rest of the slides.



* Some more test features ...

t.Helper() -- Ref: [[https://golang.org/src/testing/testing.go?s=24302:24327#L669]]

t.Parallel() -- Ref: [[https://golang.org/src/testing/testing.go?s=25187:25209#L696]]

t.Parallel() in subtests -- Ref: [[https://blog.golang.org/subtests]]

testdata -- Ref: [[https://golang.org/cmd/go/#hdr-Description_of_package_lists]]

    Dirs and files that begin with "." or "_" are ignored by go tool
    Dirs named "testdata" are ignored



* [EX3] Test Suites



* [EX3] Example: An Integer counter

    1. A counter has an initial value of 0.
    2. Exposes method to increment value. Implicit increment by 1.
    3. Exposes method to retrieve current value.
    4. Exposes method to reset value to 0.

Interface (for reference)

.code samples/ex3/counter/ifc.go /START OMIT/,/END OMIT/



* [EX3] An 'unsafe' Counter implementation

Goroutine safety not guaranteed

.code samples/ex3/unsafe_counter/counter.go /START OMIT/,/END OMIT/



* [EX3] Some (non-comprehensive) test code ...

.code samples/ex3/test_normal/counter_test.go /START OMIT/,/END OMIT/

Counters are stateful. We need a fresh instance in each test.

    c := NewUnsafeCounter()



* [EX3] A 'safe' Counter implementation

.code samples/ex3/safe_counter/counter.go /START OMIT/,/END OMIT/



* [EX3] Desired solution

(X set of tests) * (Y set of inputs)

In our example:

    X = All tests which test the behavior of 'Counter' interface
    Y = Multiple implementations which satisfy 'Counter'
        Thus, each implementation must satisfy all tests in X



* [EX3] Problem: Constructors with varying signatures

    NewUnsafeCounter() *unsafeCounter
    NewSafeCounter() *safeCounter

Lets assume:

    - We only have access to the constructor function, not the structs
    - We only get pointers to respective structs
    - Constructor signatures are (and also assumed to be) different for
      each implementation
    - The only 'commonality' is that the respective structs satisfy 
      a common (in this case 'Counter') interface

We can't natively pass the constructor of each implementation to
any common test executor function.



* [EX3] Create a 'Builder' for each implementation

The builder encapsulates the construction of each implementation, and
its dependencies.

It exposes a uniform interface through which new
instances of each implementation can be generated.

    type CounterBuilder func() Counter

This could be achieved by a function.

    func <Interface>Builder() <Interface> {
        // instantiate dependencies here
        // return a new instance here
        return <Constructor>() // returns a pointer
    }



* [EX3] Applying this to our counters

.code samples/ex3/test_suite/counter_suite_test.go /BUILDER START OMIT/,/BUILDER END OMIT/

Now both Builders satisfy the signature

    func() Counter



* [EX3] Lets create a Suite

.code samples/ex3/test_suite/counter_suite_test.go /SUITE START OMIT/,/SUITE END OMIT/



* [EX3] Add tests ...

.code samples/ex3/test_suite/counter_suite_test.go /TESTS START OMIT/,/TESTS END OMIT/

Changes to the first two lines

    1. Tests are now methods implemented on Suite struct
    2. Each test gets a fresh instance of Counter supplied by `s.builder()`



* [EX3] Add the runner ...

.code samples/ex3/test_suite/counter_suite_test.go /RUNALL START OMIT/,/RUNALL END OMIT/

Note:

    The method 'RunAllTests' also matches the test signature 'func(*testing.T)'
    Step 6 exists to exclude 'RunAllTests' and avoid a recursive loop



* [EX3] Finally, the TestCounterSuite

.code samples/ex3/test_suite/counter_suite_test.go /RUN_TEST_SUITE START OMIT/,/RUN_TEST_SUITE END OMIT/



* [EX3] Suite - Output

.code samples/ex3/test_suite/suite_output



* [EX3] Suite - Addresses of *testing.T and counters

.code samples/ex3/test_suite/suite_debug



* [EX3] Before and After functions for Suite and Tests

.code samples/ex3/test_suite/counter_suite_test.go /RUNNER_FIXTURE START OMIT/,/RUNNER_FIXTURE END OMIT/

Use them more for test agnostic checks like timing, log, leak detection etc.
Test code and its dependencies should remain within the test



* [EX3] Suite - Wrapper behavior

.code samples/ex3/test_suite/suite_wrap



* Suite - Advantages

1. Ability to define Suite local helper methods

    func (s *suite) GenerateTestData()
    func (s *suite) DoSomethingAwesomeWith(c Counter)

2. Can be designed to accept multi-dimension inputs

    Suite(c TestConfig, builder ()func Interface).RunAllTests
    - TestConfig := LocalFileSystem, InMemoryFileSystemAbstraction,
                    InMemoryDatabase, RealDatabase, etc.
    - builder    := Implementations under test

3. Custom test runner

    Suite(..).RunAllTests
    Suite(..).RunSpecificTests
    Suite(..).RunPrivateTests
              \_ RunPrivateTests exposes the runner, but hides the 'testMethods'
                 1. The suite could be published in a package
                 2. Consumer cannot modify the tests



* Suite usage [1/3]

The 'Counter' example was contrived and overdone on purpose of this presentation

    INGREDIENTS:
    2 tests
    2 implementations
    1 fat 'Counter' interface (which happened to have all methods from the implementations)

The 'Counter' type of suite can work well when:

    1. Large X -- Large number of tests to validate one input-set/implementation
    2. The tests effectively use all methods of the interface

Interfaces must be small and lean

    1. If a Suite requires a fat interface, but clusters within the suite use a subset of interfaces.
       
       Problem  : THE SUITE IS TOO MIXED. It breaks single-responsibility-principle
       Solution : Break the suite into smaller suites where:
                  - The interface footprint is smaller
                  - The cluser of tests now effectively use all methods


* Suite usage [2/3]

Reflection to iterate identify Test* methods and run them is not a necessity.
The runner could also invoke 't.Run' multiple times.

    PROS of reflection vs a manual list of 't.Run'
    - Works great for large number of tests
    - Proof against misspelt test names (strings, not TestFunction names)
    - Proof against maintenance of the 'Run' list

    CONS
    - Overkill



* Suite usage [3/3] - With 'Counter' as an example:

Suites satisfy necessity:

    Example:
    1. TestCounterSuite tests basic counter functionality
    2. TestCounterSuite DID NOT test 'goroutine safety' in SafeCounter
       Testing the goroutine safety of counters would be a different suite by itself

Interface to Suite is not 1:1:

    Example:
    An implementation which returns numbers from the fibonacci series
    on 'Increment()' could still satisfy the 'Counter' interface, but
    fail TestCounterSuite



* Guidelines & best practices



* Avoid ...

    1. MUST NOT:
    Share state between tests
    - Stateful Suite members with testdata
    - State dependent on order of test execution

    2. SHOULD NOT:
    - Perform excessive Setup or Teardown outside the test function
    - This should be invoked from within each test function per test.
    - Copy-paste is not considered (as) bad in testing (but don't overdo)

    3. SHOULD NOT:
    Make the Suite complex any more than it should be.
    - Core Suite responsiblity: (single responsibility principle)
        a) Encapsulate a collection of tests
        b) Provide runner(s) to execute those tests
    - Extend responsibly
    - "Test" is the king. "Suite" is the helper.



* Application code vs Test code [1/2]

            Handler {                   |  Runner {
                Middleware1 {           |      TestFunction {
                    Middleware2 {       |          Env Setup+ defer Teardown
                        BusinessCode {} |          Test Setup+ defer Teardown
                    }                   |          Test Logic
                }                       |      }
            }                           |  }
            ----------------------------+-------------------------------------
             This is a common pattern   |  Everything that happens in a test,
                in application code     |       remains within the test!
            ----------------------------+-------------------------------------

+

    - Setup and Teardown should be invoked from within the test.
    - If a test fails, you should only have to look into the code
      within the failing test



* Application code vs Test code [1/2]

Good

    <--------- code under test ---------->    \
                      ^                        \
                      |                         | Keep this distance minimum. Ideally next hop.
                      |                        /
    <------------ test code ------------->    /

Try to avoid

    <--------- code under test ---------->    \
                      ^                        \
             calls something else               | Affects readability. Increases test code footprint
                calls something                /
    <------------ test code ------------->    /

    1. Test code nested through many calls can affect readability
    2. Larger test code footprint -> More chances of bugs in test code
    3. If distributed across multiple files, then fragmented test code
       affects readability further.

    >> Simple vs Easy <<

: While it is recommended that all test-code must reside within the test function
: ideally in a flat way, there comes a point when the function becomes too long.
: A test with simplicity, yet adverse readability would be one with hundred+
: lines of code. When there are multiple tests of similar nature where 80%+ of
: content between these tests is copy-paste and looks very similar, it can become
: hard to distinguish between tests. The actual test logic, which forms a small
: portion of the quantity of code doesn't stand out.



* Examples from go and stdlib

Some examples which I found interesting (related to subtests & test data)

Ref: [[https://golang.org/src/cmd/go/go_test.go]]

    testgoData
    \__ Use of helper functions and their usage in tests

Ref: [[https://golang.org/src/net/http/response_test.go]]

    Data driven

Ref: [[https://golang.org/src/cmd/gofmt/gofmt_test.go]]

    Use of golden files



* PART II - Test Fixtures & Helpers



* Test Fixtures and Helpers

Help you prepare the environment and test data to run your test.

    Setup     - Stuff you do before the test logic begins
    Teardown  - Stuff you do after a test has PASSED or FAILED.
                The teardown will 'undo' what Setup did.

When to use it?

    1. If you need to read/write to temp files on the filesystem
    2. Talk to a database
    3. Talk to a server over the network
    4. Assemble a complex piece of testdata to test with
    5. Test resilience or error handling in failure scenarios
    etc.



* [F1] Setup-Only

Example: The Builders from the 'Counter' Suite

Everything happens in-memory. No persistent state changes anywhere.
Use-and-throw. Nothing to teardown.

Generalized example:

    func BuildSomethingComplex(<args>) <someType> {
        //
        //  Assemble dependencies,
        //  Generate randomized data, templates, etc
        //  specifically tuned as an input for testing
        //
        return <someType>
    }

A good practice is to assemble a new instance of every ingredient.
Each test gets a fresh copy of incredients.
Lowers the risk of errors due to ingredients having some past state.



* [F2] Idempotent Teardowns

1. Teardown (cleanup) functions which can be run anywhere.
2. Can be run both before and after tests.
3. If state is clean, Teardown does nothing.
4. If not, Teardown will clean it up.
5. If an error occurs with Teardown, its a catastrophic failure. Future tests may be invalid.
   (if the setup/teardown involves global-scope environment changes)



* Fixture example and its usage

Ref: [[https://golang.org/src/syscall/syscall_linux_test.go]]

    [1] chtmpdir
    [2] Usage of chtmpdir in: TestFaccessat



* [F3] Setup; return Teardown func() on success

    func fixture(t *testing.T) func() {
        // setup
        if err != nil {
            t.Fatal(..errorMsg..)
        }
        return func() {
            // teardown
        }
    }

Usage:

    func TestFunction(t *testing.T) {
        defer fixture(t)()
          ^      ^      ^
          |      |      +____ This () is for the returned teardown func()
          |      |
          |      +___________ Fixture does setup and returns a teardown func()
          |
          +__________________ Deferred: Hence, guaranteed execution after the
        ...                   TestFunction completes execution
    }



* [F4, F5] Some other variants

[F4] Return data, resources for the test along with teardown func

    func TestFunction(t *testing.T) {
        resource, teardown := fixture(t)
        defer teardown()
        //
        //  test code uses resource
        //
        ...
    }

[F5] Return a struct on which teardown is a method (amongst other fields & methods)

    func TestFunction(t *testing.T) {
        strukt := fixture(t)        // returns a struct with extended functionality
        defer strukt.teardown()     // cleanup
        //
        //  strukt.<fields> and struct.<methods> get used in the test
        //
        ...
    }



* [G1] Gotcha: Teardown func() is returned only on successful setup

    func LeakingFixture(t *testing.T) func() {
        var err error
        err = Step1()
        if err != nil { t.Fatalf("Failed Step 1") }
        err = Step2()
        if err != nil { t.Fatalf("Failed Step 2") }

        return func() { ..teardown.. }
    }

If Step2 fails fatally, and Step1 has made a system-scope
state change, that change leaks. Test and teardown are skipped.



* [G1] Solution

Fixture functions which setup first and return a teardown func() must:

    1. Raise t.Fatal against the first state change causing code.
    2. No more state change causing code must be part of that fixture.
    3. A teardown func() would thus not have to run on t.Fatal as:
        - The error was caused while making the first state change
    4. When the setup does succeed, the teardown concerns itself with
       reverting the one state change that succeeded.

Tests can stack multiple individual fixtures of this nature:

    func TestSomethingInIsolation(t *testing.T) func {
        defer requireContainer(t)()             // idempotent setup & teardown
        defer requireSystemTestConfig(t)()      // idempotent setup & teardown
        defer requireSwitchToContainer(t)()     // idempotent setup & teardown
        //
        //  subprocess go test cmd to re-run this test inside
        //  a container
        //
    }



* [G2] Gotcha: Forget parantheses on deferred call

If you skip the () in defer, then Setup runs after the test!
The code is still valid if you miss the parantheses. So, be vigilant.

    defer AyyoFixture(t)()
                         \__ Don't miss this 


Alternatives -> Fixture formats [F4] and [F5]

They return values, which ask for an explicit defer call of the teardown func
on the subsequent line. This makes it readable.



* Q&A