This post will demonstrate how the Fullscript DevOps team integrated the Crystalball gem and Knapsack_Pro to achieve the goal of applying RTS (Regression Test Selection concept) with our RSpec tests without sacrificing RSpec parallelization.

Image from: Integrating Crystalball and Knapsack_Pro to achieve RTS with RSpec

What is RTS?

RTS stands for Regression Testing Selection. It is a way of selecting only the most appropriate test cases for a change (from a full test suite) while ensuring a CI pipeline still delivers a correct result.

Context

By the time I am writing this documentation, we have 2178 RSpec test files in our main project. We were running them every time developers committed code on feature branches. We had approximately 80 RSpec runs on feature branches daily. Each Rspec run was parallelized across 13 GitLab runners, which is equal to an average of 6.5 AWS EC2 C5 instances running for 10 minutes. As Fullscript’s Engineering team is growing fast, these numbers will increase exponentially.

However more than half of the Git commits didn’t touch Ruby code, meaning that we didn’t have to run RSpec tests at all. And since some commits only changed a few lines of Ruby, we could run one or two specs only to cover it. We have been working on this project to spend fewer resources running RSpec tests and still deliver the same or even BETTER results than the full test suite.

Solution

After research and evaluation, we found an open-source project Crystalball which can predictively select a group of RSpec tests based on modified Ruby files and collected spec examples. However, we had to modify and extend this gem in order to better fit our needs. Based on the way we are structure our codebase we had to enhance the prediction strategies.

We use Crystalball to predict which spec files should run and pass them into the Knapsack_Pro framework where the selected specs get to run in parallel on each feature branch. It is also worth mentioning that we also leverage the Gitlab Child Pipeline Feature to dynamically determine how many parallel runners we need based on the number of specs we need to run.

Impact

Faster feedback from CI jobs to developers since we are not running a full set of RSpec tests on every change.
Reduced impact of flaky tests. Because we are running only the selected specs on most CI pipelines, flaky tests will be encountered less often.
Huge CI cost reduction, since there are much fewer runners meaning much fewer working EC2 instances.

Implementing Crystalball

Prerequisite We installed the Crystalball gem in our project following the documentation in the Crystalball project

Implementation

First, we need to generate an execution map that records the relationship between Ruby source code and their spec files. Later this map will be used by the prediction strategies to predict specific specs to run.

We created an initializer in our Rails app to set up the Crystalball map generator that only runs in test to insulate all the configurations.

1if Rails.env.test? && ENV['CRYSTALBALL'] == 'true'
2  require 'crystalball'
3  require 'crystalball/rails'
4  require 'crystalball/map_generator/factory_bot_strategy'

1ALLOCATED_OBJECTS_MONITOR = [<the objects array goes here>]

1Crystalball::MapGenerator.start! do |config|
2    config.register Crystalball::MapGenerator::CoverageStrategy.new
3    config.register Crystalball::MapGenerator::AllocatedObjectsStrategy.build(only: ALLOCATED_OBJECTS_MONITOR, root: Dir.pwd)
4    config.register Crystalball::MapGenerator::DescribedClassStrategy.new
5    config.register Crystalball::Rails::MapGenerator::I18nStrategy.new
6    config.register Crystalball::Rails::MapGenerator::ActionViewStrategy.new
7    config.register Crystalball::MapGenerator::FactoryBotStrategy.new
8  end
9end

This instructs Crystalball which strategies to use to track Ruby files and specs relationships during a full RSpec test run. We created a scheduled CI pipeline job called Rspec Map Update to call the command CRYSTALBALL=true bundle exec rspec spec and generate the execution map. Since developers commit code all the time we have to run this job regularly to keep the execution map up to date.

After the map has been generated, we verify the map is valid and compare it with the previous version of the map to see if it has changed. If it is valid and changed, then we upload the map to an S3 bucket to store data between runs. Now when we run the RSpec tests on feature branches, we pull the latest map from the S3 bucket into spec/crystalball_data.yml for usage in the prediction builder below.

Since developers are always actively adding Ruby files and RSpec tests, we have to understand that there will be a moment after a developer merges new Ruby code and before the next Rspec Map UpdateRspec Map Update job finishes where the map is out-of-date. However, this problem can be accommodated by adopting multiple Crystalball prediction strategies to increase the breadth of tests selected.

You can find all the map generation strategies documented here.

2. Create a spec_prediction_builder.

This is the place we configure all the prediction strategies. One of the strategies is called ModifiedExecutionPaths which relies on the execution map generated above to predict specs.

1module Crystalball
2  module RSpec
3    class SpecPredictionBuilder < Crystalball::RSpec::PredictionBuilder
4      def predictor
5        super do |p|
6          # Build more links from ruby file to rspec examples
7          p.use Crystalball::Predictor::AssociatedSpecs.new(
8            from: %r{config/env_accessors/(.*).rb},
9              to: "./spec/initializers/%s_spec.rb"
10              )
11          p.use Crystalball::Predictor::RegexSpecs.new(
12            scope: "./spec/**/*_spec.rb",
13            from: %r{models/(.*).rb},
14              to: "./spec/graphql/mutations/%s/(.*).rb"
15              )

1# more strategies here ...
2
3          p.use Crystalball::Predictor::ModifiedSpecs.new
4          p.use Crystalball::Predictor::ModifiedExecutionPaths.new
5          p.use Crystalball::Predictor::ModifiedSupportSpecs.new
6        end
7      end
8    end
9  end
10end

To be more specific about the tests we run we also created our own strategy class to use regular expressions to select a group of specs based on our needs.

We call the class RegexSpecs, and it is similar to the existing AssociatedSpecs strategy from the Crystalball gem.

1require 'crystalball/predictor/strategy'
2
3module Crystalball
4
5 class Predictor
6   # This strategy is almost the same as associated_specs.rb, but the only difference is that the `to` parameter also accept regex.
7   # Used with `predictor.use Crystalball::Predictor::FilenamePatternSpecs.new(from: %r{models/(.*).rb}, to: "./spec/models/%s_spec.rb")`.
8   # When used will look for files matched to `from` regex and use captures to fill `to` regex to
9   # get paths of proper specs
10   class RegexSpecs
11     include Strategy
12
13     # @param [file glob] scope - to find all the spec files scope to work with
14     # @param [Regexp] from - regular expression to match specific files and get proper captures
15     # @param [Regexp] to - regex in sprintf format to get proper files using captures of regexp
16     def initialize(scope:, from:, to:)
17       @scope = scope
18       @from = from
19       @to = to
20     end
21
22     def call(diff, _map)
23       super do
24         regex_string = diff.map(&:relative_path).grep(from).map { |source_file_path| to % captures(source_file_path) }
25
26         regex_string.flat_map { |regex| Dir[scope].grep(Regexp.new regex)}
27       end
28     end
29
30     private
31
32     attr_reader :scope, :from, :to
33
34     def captures(file_path)
35       match = file_path.match(from)
36       if match.names.any?
37         match.names.map(&:to_sym).zip(match.captures).to_h
38       else
39         match.captures
40       end
41     end
42   end
43 end
44end

3. Next part is to configure the runner in the crystalball.yml file. Please check code comments for each line’s purpose.

1# This is to tell where to locate the map generated by running full rspec

1execution_map_path: "spec/crystalball_data.yml"

1# The builder lists all the prediction strategies
2requires:
3  - "./spec/spec_prediction_builder.rb"
4
5prediction_builder_class_name: "Crystalball::RSpec::SpecPredictionBuilder"

1# These two lines are to let Crystalball compare the HEAD of the branch to our staging branch to figure out the code changes. If you would like to test it on your local, you need to change the `diff_to` to nil
2diff_from: "origin/staging"
3
4diff_to: "HEAD"

4. Now let’s generate the prediction file by running CRYSTALBALL_CONFIG=spec/crystalball.yml bundle exec crystalball — dry-run > tmp/crystalball_prediction

The spec file paths should appear in the tmp/crystalball_prediction file given that we have made Ruby changes.

We also need to remove the redundant log lines from the file, so that we can pass them directly into the Knapsack_pro framework to run RSpec in parallel.

1PREDICTION_FILE = "tmp/crystalball_prediction"
2
3lines_of_tests = File.readlines(PREDICTION_FILE).select { |line| line[/^.\/.*\.rb/] } # remove unnecessary logs
4File.open(PREDICTION_FILE, "w") { |f| lines_of_tests.each { |line| f.puts line } }

5. Alright now mission accomplished if we only needed to get RTS RSpec going. We now have a prediction file that only contains mixed spec paths (./spec/services/rma/create_spec.rb./spec/services/rma/create_spec.rb) and example paths (./spec/models/brand_spec.rb[1:8:5]./spec/models/brand_spec.rb[1:8:5]). We could pass the file into RSpec directly.

Here is a snippet of what we generated:

1./spec/lib/customizable_spec.rb[1]
2./spec/models/patient_spec.rb[1:7]
3./spec/models/disbursal_spec.rb[1]
4./spec/services/rma/create_spec.rb
5./spec/models/brand_spec.rb[1:8:5]
6./spec/models/patient_spec.rb[1:3]
7./spec/models/patient_spec.rb[1:8]
8./spec/services/foi/rollup_spec.rb
9./spec/models/brand_spec.rb[1:8:3]

6. However we would like to have the selected specs running in parallel as well, so we jumped into the KnapsackPro world to tackle this.

KnapsackPro Integration

The Knapsack project has a free version but it doesn’t support the flag KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE which is used to accept the spec files we generated. So we use KnapsackPro for this goal. See the documentation here.

While implementing the integration, we experienced an issue that we couldn’t resolve and had to put a workaround for it. As we mentioned above, the prediction file is mixed with spec file paths and example paths. And we found that a random number of random specs were being ignored with a log message All examples were filtered out when we ran them with KnapsackPro. Here is an example:

1690D, [2021-09-08T19:11:44.479339 #375] DEBUG -- : [knapsack_pro] POST https://api.knapsackpro.com/v1/queues/queue

1691D, [2021-09-08T19:11:44.479398 #375] DEBUG -- : [knapsack_pro] API request UUID:

1692D, [2021-09-08T19:11:44.479415 #375] DEBUG -- : [knapsack_pro] API response:

1693D, [2021-09-08T19:11:44.479458 #375] DEBUG -- : [knapsack_pro] {"queue_name"=>"1664:c5ae646aac62c6432022b4a898230328", "build_subset_id"=>nil, "test_files"=>[{"path"=>"spec/graphql/mutations/wholesale_cart/update_spec.rb[1:5:1]", "time_execution"=>0.0}]}

1694I, [2021-09-08T19:11:44.479732 #375]  INFO -- : [knapsack_pro] To retry in development the subset of tests fetched from API queue please run below command on your machine. If you use --order random then remember to add proper --seed 123 that you will find at the end of rspec command.

1695I, [2021-09-08T19:11:44.479775 #375]  INFO -- : [knapsack_pro] bundle exec rspec --format documentation --format RspecJunitFormatter --out tmp/rspec.xml  --default-path spec "spec/graphql/mutations/wholesale_cart/update_spec.rb[1:5:1]"

1696Run options: include {:ids=>{"./spec/graphql/mutations/wholesale_cart/update_spec.rb"=>["1:5:1"]}}

1697All examples were filtered out

1698Randomized with seed 24869

We worked with KnapsackPro to debug this issue and it turned out the issue was around how we were using the RSpec filters. However, we didn’t manage to resolve the issue and our workaround is still in place. Thanks to KnapsackPro for helping us troubleshoot the issue and their support is really responsive and prompt.

Before we sent the prediction file into KnapsackPro, we trimmed the example paths and ran the whole spec file for each prediction instead. So the prediction file will end up with something like this:

1./spec/lib/customizable_spec.rb
2./spec/models/patient_spec.rb
3./spec/models/disbursal_spec.rb
4./spec/services/rma/create_spec.rb
5./spec/models/brand_spec.rb
6./spec/models/patient_spec.rb
7./spec/models/patient_spec.rb
8./spec/services/foi/rollup_spec.rb
9./spec/models/brand_spec.rb

We are running more than we need to but we avoid missing any potentially affected specs. We are still improving this process. Please leave your comments in this post if you have more insights about this.

Now we get back to how we implemented the Knapsack Pro.

We installed KnapsackPro by following the official documentation according to our CI/CD provider (which is self-managed Gitlab).
Created a bash script with a function triggering the RSpec tests.

1function rts_run {

1# pipe the prediction file into knapsack for parallel running rspec

1KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \

1KNAPSACK_PRO_PROJECT_DIR=. \

1KNAPSACK_PRO_CI_NODE_TOTAL=$CI_NODE_TOTAL \

1KNAPSACK_PRO_CI_NODE_INDEX=$(($CI_NODE_INDEX-1)) \

1KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=<KnapsackPro token> \

1KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=tmp/crystalball_prediction \

1BUNDLE_GEMFILE="${BUNDLE_GEMFILE:-Gemfile}" \

1bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

1}

Here, $CI_NODE_TOTAL and $(($CI_NODE_INDEX-1)) are the GitLab environment variables provided to tell KnapsackPro how many runners we have so that KnapsackPro can distribute the specs from tmp/crystalball_prediction to the runners, optimistically.

With this command, KnapsackPro can finally accept the prediction file and get our RTS-RSpec tests running in parallel mode.

Monitoring & Approving

We didn’t replace the original RSpec tests with the RTS-RSpec right away. Instead, to prove the validation and effectiveness of the new strategy, we kept these two jobs running side by side on our pipelines. We created Prometheus metrics to collect all the data on these jobs for us to analyze.

We did find some edge cases where Crystalball predictions missed covering certain code changes. Using the data from our metrics, we added coverage using our RegexSpecs prediction strategy.

One thing worth mentioning is that we still run the whole RSpec test suite in our main pipeline before any production deployment, so we can always catch missing cases and continue to improve our prediction strategy.

Providing Feedback

Given the complicated ecosystem of a Rails application and the massive list dependencies for each RSpec example, it is possible to miss running a relevant RSpec test with some code changes.

We want to keep improving our strategies, so please leave your questions and comments.

I didn’t cover how we make GitLab orchestrate the right amount of runners for dynamic prediction files — please also let me know if you are interested in the topic.

Do you like this content? Make sure to follow our publication! We are also hiring! Check out all open positions at Fullscript: https://fullscript.com/careers

Integrating Crystalball and Knapsack_Pro to achieve RTS with RSpec

Share this post