We test a lot at Sharethrough. One of our projects had significant complex behavior that required numerous integration tests. These tests were inherently slow, so our test suite’s running time kept creeping up. They had value and they also had drawbacks.
Since we didn’t need to run them all the time, we decided to use RSpec’s tag filtering to separate the integration tests from the unit tests. This gave us a lot of speed improvements, but we needed to be careful to make sure we could still easily run the entire suite.
- RSpec tags are great for test isolation when you want to keep contextually similar tests together (and not in another location)
- Automatically excluding tests based on their tags is convenient and dangerous
rakeruns everything; let people opt-in to the speedups
- Create additional rake tasks to load a tagless options file to run all your tests
- Check your options files into the repository to share the love
Setting Up Your Filters
integration: true and
slow: true sections. This is how we tag example groups and examples.
1 2 3 4 5 6 7 8 9 10 11 describe 'a slow example group', integration: true do it 'does some complex calculations' do expect(Universe.answer).to eq(42) end end describe 'a potentially slow example group' do it 'does something slow just in this example', slow: true do expect(Sloth.look_smug).to be_true end end
:true is a bit pesky since it’s always true inherently. Note that we can set an option to give us a cleaner syntax:
1 2 3 RSpec.configure do |config| config.treat_symbols_as_metadata_keys_with_true_values = true end
Now tests look like this:
1 2 3 4 5 describe 'a tagged test', :integration do it 'does some complex calculations' do expect(Universe.answer).to eq(42) end end
Tag-Based Example Execution
1 rspec --tag integration spec/
Run everything, except for
1 rspec --tag ~slow spec/
Tags let us focus on only the tests we care about right now. Our feedback loop for running tests is minimized because the tests will run faster (since there are less of them running).
Excluding Slow Specs By Default?
RSpec lets us create an options file that will get automatically evaluated when we run our tests, so we don’t even need to worry about specifying the tags on each test run.
1 2 3 4 5 # Run the tests with color --color # Skip all tests tagged with "integration" or "slow" --tag ~integration --tag ~slow
There is a danger here in that we’ve changed the default behaviour of
rake away from running all examples, meaning you’re blind to failures in your integration suite. This is not good. Our previous solution was to not worry about it because the CI would not have
.rspec and so would run all the tests. This worked, but it would take longer to get feedback on the failing tests since they would have to get pushed to the repo (so the CI machine would kick off a build) before we would see the failures.
Option 1 - Include All Known Tags
We could also run the test suite with all the appropriate tags, but then we end up running our tests multiple times and have to keep track of all the tags to make sure we didn’t miss one.
Option 2 - Use Good Defaults
The power of tagging was great, but we also needed an easy override to run the entire test suite. We ended up writing a separate rake task that will use a different options files that does not contain any tags.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 begin require 'rspec/core/rake_task' desc 'Run all tests regardless of tags' RSpec::Core::RakeTask.new('spec:all') do |task| task.pattern = './spec/**/*_spec.rb' # Load the tagless options file task.rspec_opts = '-O .rspec-no-tags' end task 'spec:all' => 'test:prepare' rescue LoadError => e desc 'Run all tests regardless of tags' task 'spec:all' do abort 'spec:all rake task is not available.' end end
Now we have the power of tags without the overhead of keeping track of them or the risk of missing failing tests.
Automate All The Things!
There is a change in our workflow, and change can be hard. Let’s make this super easy to run by overriding the default rake task to use our new tagless rake task.
1 2 3 4 5 6 # Clear out default spec task from running automatically task(:default).clear # Rebuild the rake task, including all tasks you want # to run automatically task default: %w[spec:all spec:fast]
We can now add tags to our tests without having to worry about excluding tests and pushing broken code, and we’re able to focus our test runs to reduce the development feedback loop.
A Final Note for Capybara and Webmock Interactions
1 WebMock.disable_net_connect!(allow_localhost: true)
You can customize this as needed.