SlideShare a Scribd company logo

Rocket Fuelled Cucumbers

•
•
Joseph Wilk
Joseph Wilk

Rocket Fuelled Cucumbers discusses strategies for dealing with slow cucumber test suites, including: - Using Spork to preload support code to speed up test runs - Tagging tests to run focused subsets based on features, filenames, or tags - Distributing tests across multiple servers using Testjour to parallelize testing - Looking to cloud providers like EC2 to gain additional hardware resources - Dividing applications and tests along service boundaries to isolate components

1 of 99
Downloaded 88 times
Rocket Fuelled Cucumbers Joseph Wilk
Dealing with an Outbreak of Cucumbers Joseph Wilk
No Cucumbers where harmed in the making of this presentation. Just mildly shaken up
Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them
Cucumber Feature: Filling Cucumbers with • BDD framework rocket fuel • Plaintext Scenario: Rocket fuel Given I have trained monkeys • Promotes When I click the launch button communication Then the monkeys should not die Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end
Cucumber provides the venue Your code makes it slow Shame on you
What are Acceptance Tests? • Customer-facing • Cut through the whole stack • Examples of how the system is intended to be used
Rocket Fuelled Cucumbers
You made me Cuke
Rocket Fuelled Cucumbers
1 minute ~ build
Local builds 1 minute ~ build
Local builds Autotest 1 minute ~ build
Local builds Autotest Growl 1 minute ~ build
Rocket Fuelled Cucumbers
Rocket Fuelled Cucumbers
Don’t Panic
Slow testing problems Running all the Cukes takes too long! Development is slowed down by the time to run focused Cukes
Types of Feedback + Speed Development - Confidence - Speed Test Build + Confidence
Happiness is a 10 min build time
Slow builds are the enemy of continuous integration
Development Feedback Its too slow to run the cukes. Just push the code and run away
Reuse - Spork $ spork cucumber $ cucumber --drb require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git
Slow services • Search Before('@solr') do • Solr Solr.boot unless Solr.running? • Sphinx end • Databases • Mongo Feature: Searching • Redis @solr • Mysql Scenario: Indexed search • Message Queues Given ... When ... • RabbitMQ Then ...
Run Just Enough Tests Autotest # * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb Cucover @analyzer = Rcov::CodeCoverageAnalyzer.new
http://vimeo.com/12500864
Slicing Features @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed Tagging $ cucumber --tags @media Filenames $ cucumber visitor_* Subsystems $ cucumber features/admin/* Profiles $ cucumber --profile admin
Just enough Database
Just enough Database INSERT INTO `cities` (`name`) VALUES ('Test name') SELECT * FROM `cities` INSERT INTO WHERE (`cities`.`id` = 105838) `venues` ... SELECT * FROM Cache city data `venues` WHERE ... venue data Cache
Test Build Feedback Commit code Have a 1 hour nap
Options • Hardware • Intelligent test selection • Tighter focused tests • Divide and Conquer
Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours
120 sad minutes of my life...
$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help
$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:
Testjour Worker Slave Worker Redis Work Worker Master Slave Queue Worker Worker Slave Worker
Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
Looking to the Cloud EC2, Rackspace Cloud, Windows Azure
Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours
Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes
Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000
Ec2 Disc image AMI slim-sumo
Ec2 Disc image AMI Launch slim-sumo
Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch slim-sumo
Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} slim-sumo
http://vimeo.com/12467100
require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end
Rocket Fuelled Cucumbers
“Buy lots of hardware”
“Buy lots of hardware” “more hardware please”
“Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
“Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
Just run the tests that matter...
Failure probability matters European Computing manufacturer
Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours
Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours Tests that were most ~ 2 hours likely to fail
http://vimeo.com/12467122
Rocket Fuelled Cucumbers
Tests that never fail
Tests that never fail
Tests which Tests that regularly fail never fail
Tests which Tests that regularly fail never fail Daily Build
Tests which Tests that regularly fail never fail Daily Build Nightly Build
Tests which Tests that regularly fail never fail Daily Build Nightly Build $cucumber --tags ~@nightly $cucumber --tags @nightly
Flickering Confidence
Flickering Confidence FAIL
Flickering Confidence FAIL FAIL
Flickering Confidence FAIL FAIL PASS
Flickering Confidence FAIL FAIL PASS Main @flicker Flicker
Divide and conquer 7 7 7 1 hour 7 7 7 7 7 7 • Rails - Engines • Service Orientated Architecture
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer
Divide and Concuquer Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer related(artist) Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Divide and conquer related(artist) Related Rails HTML Artists Service unit test unit test Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!
Pairwise Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”
Pairwise http://www.screencast.com/t/NGI0NjVk
Just when you thought things were going well We should add some funky Ajax features
JavaScript Browser based In-memory Selenium, Watir Celerity
Need for speed Webrat 0.8 Celerity 12 Watir 12 Selenium 12.9 0 3.75 7.5 11.25 15 start-up time (seconds)
Capybara sudo gem install capybara Like Webrat but bigger envjs gem install envjs Johnson gem install johnson --prerelease spidermonkey
http://vimeo.com/12467133
Capybara-envjs
Rocket Fuelled Cucumbers
Find the right compromise for you
Find the right compromise for you Slow test builds produce slow release cycles
Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data
Thanks Its been emotional Joseph Wilk http://blog.josephwilk.net @josephwilk

More Related Content

Rocket Fuelled Cucumbers

  • 2. Dealing with an Outbreak of Cucumbers Joseph Wilk
  • 3. No Cucumbers where harmed in the making of this presentation. Just mildly shaken up
  • 4. Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them
  • 5. Cucumber Feature: Filling Cucumbers with • BDD framework rocket fuel • Plaintext Scenario: Rocket fuel Given I have trained monkeys • Promotes When I click the launch button communication Then the monkeys should not die Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end
  • 6. Cucumber provides the venue Your code makes it slow Shame on you
  • 7. What are Acceptance Tests? • Customer-facing • Cut through the whole stack • Examples of how the system is intended to be used
  • 11. 1 minute ~ build
  • 13. Local builds Autotest 1 minute ~ build
  • 14. Local builds Autotest Growl 1 minute ~ build
  • 18. Slow testing problems Running all the Cukes takes too long! Development is slowed down by the time to run focused Cukes
  • 19. Types of Feedback + Speed Development - Confidence - Speed Test Build + Confidence
  • 20. Happiness is a 10 min build time
  • 21. Slow builds are the enemy of continuous integration
  • 22. Development Feedback Its too slow to run the cukes. Just push the code and run away
  • 23. Reuse - Spork $ spork cucumber $ cucumber --drb require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end git://github.com/timcharper/spork.git
  • 24. Slow services • Search Before('@solr') do • Solr Solr.boot unless Solr.running? • Sphinx end • Databases • Mongo Feature: Searching • Redis @solr • Mysql Scenario: Indexed search • Message Queues Given ... When ... • RabbitMQ Then ...
  • 25. Run Just Enough Tests Autotest # * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb Cucover @analyzer = Rcov::CodeCoverageAnalyzer.new
  • 27. Slicing Features @media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed Tagging $ cucumber --tags @media Filenames $ cucumber visitor_* Subsystems $ cucumber features/admin/* Profiles $ cucumber --profile admin
  • 29. Just enough Database INSERT INTO `cities` (`name`) VALUES ('Test name') SELECT * FROM `cities` INSERT INTO WHERE (`cities`.`id` = 105838) `venues` ... SELECT * FROM Cache city data `venues` WHERE ... venue data Cache
  • 30. Test Build Feedback Commit code Have a 1 hour nap
  • 31. Options • Hardware • Intelligent test selection • Tighter focused tests • Divide and Conquer
  • 32. Testjour Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours
  • 33. 120 sad minutes of my life...
  • 34. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help
  • 35. $ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:
  • 36. Testjour Worker Slave Worker Redis Work Worker Master Slave Queue Worker Worker Slave Worker
  • 37. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  • 38. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  • 39. Testjour Worker rsync - code Slave Worker Redis Work Worker Master Slave Queue Worker Worker rsync - code Slave Worker
  • 40. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure
  • 41. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours
  • 42. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes
  • 43. Looking to the Cloud EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000
  • 44. Ec2 Disc image AMI slim-sumo
  • 45. Ec2 Disc image AMI Launch slim-sumo
  • 46. Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch slim-sumo
  • 47. Ec2 Disc image ec2 ec2 ec2 ec2 AMI node node node node Launch Build ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]} slim-sumo
  • 49. require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end
  • 51. “Buy lots of hardware”
  • 52. “Buy lots of hardware” “more hardware please”
  • 53. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
  • 54. “Buy lots of hardware” “more hardware please” “Just a little bit more hardware”
  • 55. Just run the tests that matter...
  • 56. Failure probability matters European Computing manufacturer
  • 57. Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours
  • 58. Failure probability matters European Computing manufacturer Automated test suite ~ 18 hours Tests that were most ~ 2 hours likely to fail
  • 63. Tests which Tests that regularly fail never fail
  • 64. Tests which Tests that regularly fail never fail Daily Build
  • 65. Tests which Tests that regularly fail never fail Daily Build Nightly Build
  • 66. Tests which Tests that regularly fail never fail Daily Build Nightly Build $cucumber --tags ~@nightly $cucumber --tags @nightly
  • 70. Flickering Confidence FAIL FAIL PASS
  • 71. Flickering Confidence FAIL FAIL PASS Main @flicker Flicker
  • 72. Divide and conquer 7 7 7 1 hour 7 7 7 7 7 7 • Rails - Engines • Service Orientated Architecture
  • 78. Divide and Concuquer Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 79. Divide and conquer Rails Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 80. Divide and conquer Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 81. Divide and conquer related(artist) Related Rails Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 82. Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 83. Divide and conquer related(artist) Related Rails HTML Artists Service Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 84. Divide and conquer related(artist) Related Rails HTML Artists Service unit test unit test Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”
  • 85. Don’t use Acceptance tests http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html Heresy!
  • 86. Pairwise Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....
  • 87. | Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | .... “most faults are caused by interactions of at most two factors”
  • 89. Just when you thought things were going well We should add some funky Ajax features
  • 90. JavaScript Browser based In-memory Selenium, Watir Celerity
  • 91. Need for speed Webrat 0.8 Celerity 12 Watir 12 Selenium 12.9 0 3.75 7.5 11.25 15 start-up time (seconds)
  • 92. Capybara sudo gem install capybara Like Webrat but bigger envjs gem install envjs Johnson gem install johnson --prerelease spidermonkey
  • 97. Find the right compromise for you Slow test builds produce slow release cycles
  • 98. Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data
  • 99. Thanks Its been emotional Joseph Wilk http://blog.josephwilk.net @josephwilk