Greater Test Control With RSpec’s Tag Filters

We test a lot at Sharethrough. One of our projects had significant complex behavior that required numerous integration tests. These tests were inherently slow, so our test suite’s running time kept creeping up. They had value and they also had drawbacks.

Since we didn’t need to run them all the time, we decided to use RSpec’s tag filtering to separate the integration tests from the unit tests. This gave us a lot of speed improvements, but we needed to be careful to make sure we could still easily run the entire suite.

tl;dr

  • RSpec tags are great for test isolation when you want to keep contextually similar tests together (and not in another location)
  • Automatically excluding tests based on their tags is convenient and dangerous
  • rake runs everything; let people opt-in to the speedups
  • Create additional rake tasks to load a tagless options file to run all your tests
  • Check your options files into the repository to share the love

Setting Up Your Filters

Note the integration: true and slow: true sections. This is how we tag example groups and examples.

1
2
3
4
5
6
7
8
9
10
11
describe 'a slow example group', integration: true do
  it 'does some complex calculations' do
    expect(Universe.answer).to eq(42)
  end
end

describe 'a potentially slow example group' do
  it 'does something slow just in this example', slow: true do
    expect(Sloth.look_smug).to be_true
  end
end

That :true is a bit pesky since it’s always true inherently. Note that we can set an option to give us a cleaner syntax:

In spec/spec_helper.rb:

1
2
3
RSpec.configure do |config|
  config.treat_symbols_as_metadata_keys_with_true_values = true
end

Now tests look like this:

1
2
3
4
5
describe 'a tagged test', :integration do
  it 'does some complex calculations' do
    expect(Universe.answer).to eq(42)
  end
end

Tag-Based Example Execution

Run only :integration tests:

1
rspec --tag integration spec/

Run everything, except for :slow tests:

1
rspec --tag ~slow spec/

Tags let us focus on only the tests we care about right now. Our feedback loop for running tests is minimized because the tests will run faster (since there are less of them running).

Excluding Slow Specs By Default?

RSpec lets us create an options file that will get automatically evaluated when we run our tests, so we don’t even need to worry about specifying the tags on each test run.

In ./.rspec:

1
2
3
4
5
# Run the tests with color
--color
# Skip all tests tagged with "integration" or "slow"
--tag ~integration
--tag ~slow

There is a danger here in that we’ve changed the default behaviour of rake away from running all examples, meaning you’re blind to failures in your integration suite. This is not good. Our previous solution was to not worry about it because the CI would not have .rspec and so would run all the tests. This worked, but it would take longer to get feedback on the failing tests since they would have to get pushed to the repo (so the CI machine would kick off a build) before we would see the failures.

Option 1 - Include All Known Tags

We could also run the test suite with all the appropriate tags, but then we end up running our tests multiple times and have to keep track of all the tags to make sure we didn’t miss one.

Option 2 - Use Good Defaults

The power of tagging was great, but we also needed an easy override to run the entire test suite. We ended up writing a separate rake task that will use a different options files that does not contain any tags.

In lib/tasks/run_all_specs.rake:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
begin
  require 'rspec/core/rake_task'

  desc 'Run all tests regardless of tags'
  RSpec::Core::RakeTask.new('spec:all') do |task|
    task.pattern = './spec/**/*_spec.rb'
    # Load the tagless options file
    task.rspec_opts = '-O .rspec-no-tags'
  end

  task 'spec:all' => 'test:prepare'

rescue LoadError => e
  desc 'Run all tests regardless of tags'
  task 'spec:all' do
    abort 'spec:all rake task is not available.'
  end
end

In ./.rspec-no-tags:

1
--color

Now we have the power of tags without the overhead of keeping track of them or the risk of missing failing tests.

Automate All The Things!

There is a change in our workflow, and change can be hard. Let’s make this super easy to run by overriding the default rake task to use our new tagless rake task.

In ./Rakefile:

1
2
3
4
5
6
# Clear out default spec task from running automatically
task(:default).clear

# Rebuild the rake task, including all tasks you want
# to run automatically
task default: %w[spec:all spec:fast]

We can now add tags to our tests without having to worry about excluding tests and pushing broken code, and we’re able to focus our test runs to reduce the development feedback loop.

A Final Note for Capybara and Webmock Interactions

Capybara runs a local server, and webmock blocks outgoing HTTP calls, blocking capybara from running properly. To fix this, make sure webmock is not blocking HTTP calls from localhost:

1
WebMock.disable_net_connect!(allow_localhost: true)

You can customize this as needed.