Skip to content

Latest commit

 

History

History
141 lines (99 loc) · 3.85 KB

flaky.md

File metadata and controls

141 lines (99 loc) · 3.85 KB

Fight the flakiness

Tips on detecting and solving flaky tests in Rails apps.

Make sure tests run in random order

For example, config.order :random for RSpec.

Make sure you use transactional tests

For example, config.transactional_tests = true for RSpec.

Avoid before(:all) (unless you sure it's safe)

  • Use rubocop-rspec RSpec/BeforeAfterAll cop to find before(:all) usage
  • Consider replacing with before or before_all

Travel through time and always return back

  • Find leaking time traveling with TimecopLinter
  • Add config.after { Timecop.return }
  • If you rely on time zones in the app, randomize the current time zone in tests (e.g. with zonebie) to make sure your tests don't depend on it.

Clear cache / in-memory stores after each test

For example, for ActiveJob (to avoid have_enqueued_job matcher catching jobs from other tests):

RSpec.configure do |config|
  config.after do
    # Clear ActiveJob jobs
    if defined?(ActiveJob) && ActiveJob::QueueAdapters::TestAdapter === ActiveJob::Base.queue_adapter
      ActiveJob::Base.queue_adapter.enqueued_jobs.clear
      ActiveJob::Base.queue_adapter.performed_jobs.clear
    end
  end
end

Generated data must be random enough

  • Respect DB uniqueness constraint in your factories (check with FactoryLinter)

Make sure tests pass offline

Tests should not depend on the unknown outside world.

  • Wrap dependencies into testable modules/classes:
# Make Resolv testable
module Resolver
  class << self
    def getaddress(host)
      return "1.2.3.4" if test?
      Resolv.getaddress(host)
    end

    def test!
      @test = true
    end

    def test?
      @test == true
    end
  end
end

# rspec_helper.rb

Resolver.test!
  • Provide mock implementations:
# App-specific wrapper over S3
class S3Object
  attr_reader :key, :bucket
  def initialize(bucket_name, key = SecureRandom.hex)
    @key = key
    @bucket = bucket_name
  end

  def get
    S3Client.get_object(bucket: @bucket, key: @key).body.read
  end

  def put!(file)
    S3Client.put_object(bucket: @bucket, key: @key, body: file)
  end
end

# Mock for S3Object to avoid calling real AWS
class S3ObjectMock < S3Object
  def get
    @file.rewind
    @file.read.force_encoding(Encoding::UTF_8)
  end

  def put!(file)
    @file = file
  end
end

# in test
before { stub_const "S3Object", S3ObjectMock }

Do not sleep in tests

When writing System Tests avoid indeterministic sleep 1 and use have_xyz matchers instead–they keep internal timeout and could wait for event to happened.

Remember: Time is relative (Einstein).

Match arrays with match_array

If you don't need to the exact ordering, use match_array matcher instead of eq([...]).

Testing NotFound with random IDs

If you test not-found-like behaviour you can make up non-existent IDs like this:

expect { User.find(1234) }.to raise_error(ActiveRecord::RecordNotFound)

There is a change that the record with this ID exists (if you have before/before(:all) or fixtures).

A better "ID" for this purposes is "-1":

expect { User.find(-1) }.to raise_error(ActiveRecord::RecordNotFound)

Read more