The improvement process that I’ll walk you through is known commonly as DMAIC (Define, Measure, Analyze, Improve, Control).  For those not familiar with DMAIC, Google can tell you everything you need to know.  But, basically DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimise business processes.  We’ll use DMAIC in this post to improve the speed of your Cucumber tests.

Define

You’ve probably already defined the problem, your tests are too slow.  What you haven’t done is define success.  You will define success in terms of execution time.  For instance, if your Cukes are taking 30 mins you might define success as, “Make all Cukes run in 20 mins or less”, others might define success as, “Cukes running in 5 mins or less”.  This is up to you and your team to decide.  Just remember, the amount of time it takes to run your tests will be inversely proportional to number of times you will run them.  So the faster they are the more often you will run them.

Measure

You know how long it takes to run the suite.  You might even know how long it takes to run each scenario or sets of scenarios.  What you need to know now is how long it takes each step to run.  Wait! Before you go diving back into your steps to instrument them with timers and such, let’s see what Cucumber can do to help us.
Fortunately for us Cucumber offers a number of helpful options for formatting the output of our test runs.  One of those options is --format usage. Using the ‘usage’ format will show us which steps take the longest to run, how many times each step executed and if any steps are unused.  I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.

Cucumber Output using --format usage

Analyze

With our new data in hand we can begin to zero in on the trouble spots.  The usage report displays the steps ordered from slowest to fastest.  The report also shows the number of times each step executed.  Your slowest step might not be the biggest problem.  You should focus on the steps that run the slowest and the most often.  Also, consider why the step executed so often.  If you wrote the scenarios following a behavior driven development (BDD) cycle then the scenario was written to define behavior in the system. As a result you may find many scenarios exercising the same part of the system in only slightly different ways.  Viewing the scenarios in hindsight you should consider if you can consolidate or eliminate some of the scenarios.

Improve

Now that we know where to focus our efforts it’s time to make our improvements.

Consolidate

In the ‘Analyze’ phase you considered if you could remove or consolidate scenarios.  Now is the time to do that work.  Deleting scenarios is fun and easy.  Consolidating scenarios takes more effort and thought.  Look for recurring patterns such as screen navigation and data setup.  If you notice that you repeatedly setup the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation.  From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing while maintaining the ability to independently execute the scenario.

Learn to wait

One of the most common causes of slow downs in cukes is ‘sleep’ing.  People often use sleep to wait for AJAX activity to complete.  This is the most unpredictable and wasteful way to control test execution.  Imagine that an AJAX action normally takes between .5 and 2 seconds to complete.  If we sleep to wait for the action to complete how long should we sleep?  Let’s say we decide to sleep for 3 seconds.  If that action ever takes more than 3 seconds the test fails.  Also, every time the action takes less than 3 seconds the difference is wasted doing nothing and if this is one of your most executed steps those seconds quickly turn into minutes.

Instead of sleeping you must learn to ‘wait’.  The Selenium client includes a number of ‘wait_for’ methods that will improve the predictability of your steps and will remove the waste caused by sleeping.

• wait_for_text(value) — Simply waits for the text you specify to appear in the DOM.  Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.
• wait_for_element(xpathLocator) — Allows you to specify an exact xpath locator to wait for before continuing.
• wait_for_no_element(xpathLocator) and wait_for_no_text(value)– Allows you to wait for the removal of an xpath locator or text value from the DOM before continuing.

Good Software Engineering

Once you’ve ‘picked’ all the ‘low hanging fruit’, you’re left with your software engineering skills and training to profile, tune and tweak the last bits of improvement from your steps.

Scenario Tagging

Cucumber provides a simple yet powerful ability to add tags to scenarios using the @tagname. If you’ve made all the improvements that you can reasonably make but your tests still aren’t fast enough, then you can use tags to define a subset of scenarios to execute.  For instance, you could tag the scenarios with @smokeTest tag and run them by specifying ‘ –tag @smokeTest’ at the end of your cucumber command.  Don’t forget to run your full suite of Cukes nightly, or better yet schedule them to run more frequently via your continuous integration server.  Checkout the Tags page on the Cucumber wiki for more details on tags, https://github.com/cucumber/cucumber/wiki/Tags

Control

Congratulations.  Give yourself a pat on the back.  Hopefully you’ve met and surpassed your success criteria.  Wait a second though, we’re not done. You need to keep an eye on things.  Don’t let your test times creep up. That’s what the ‘Control’ phase is all about and it goes on for as long as you care about the amount of time required to run the scenarios. Make the ‘care and feeding’ of your cukes a regular part of your development cycle and they will provide your application with benefits for years to come.

LIVESTRONG.org blog: http://bit.ly/zwK2AQ

Danny’s Survivor Summit page: http://bit.ly/wiev8B

You can read all of my Agile Architect columns at http://adtmag.com/Articles/List/Agile-Architect.aspx

kanban |ˈkänˌbän| – noun

a Japanese manufacturing system in which the supply of components is regulated through the use of a card displaying a sequence of specifications and instructions, sent along the production line.

“Now wait.  We are not in Japan, and we are definitely not a production line.” Understood, however we can take this system and apply it to our practice.

Kanban in practical terms is best described with a picture.

Myles Bogner, Ph.D, VP Research and Development
with Special Asynchrony Guest Jason Carter, President of Aegis Strategies

As organizations embark on improving their Enterprise Architecture, Portfolio Management, and Enterprise Governance efforts, a frequent question tends to be, where are we now?  Below we offer a set of questions to guide the creation of an “as-is” assessment around these three areas.  We’ve categorized these questions into seven broad categories.

Governance

• What are your governance bodies, documents, and processes?
• How are IT investments prioritized and funded across your enterprise?
• What are the guiding principles that you would like to achieve through enterprise governance?
• Are your governance organizations formally chartered; where are they?
• Is there an IT governance process for routine requirements different from a strategic IT governance process?
• Does your community of lead architects come together regularly in a forum; is this forum chartered?
• Is there a formal tie between the enterprise architecture and engineering communities; if so, what is the concept of operations?
• Is there a formal tie between your organization and any external communities of interest; if so, what are the governance charters?
• Is there a formal enterprise architecture configuration and control board; if so, is it chartered and executing?
• Is there a prescribed methodology for software development across the organization to ensure secure, timely delivery?
• Are the organization’s architectural rules succinctly captured?
• Is there a configuration control board around a common computing environment; is this chartered?
• Specifically for government organizations, what are the governance procedures for ensuring Clinger-Cohen Act compliance?

Information

Like several Asynchrony projects, our team uses Cucumber to automate our acceptance tests. Since our project’s product is a website, our Cucumber steps use Selenium Webdriver (via the Capybara ruby gem) to automate browser interactions. As our project’s code base and feature set has grown, the amount of time it takes to run our entire acceptance test suite has likewise grown. By late spring of this year, this cost of time grew to a totally unacceptable 45 minutes per run, and we found ourselves no longer running all the tests before pushing code to our repository. Unsurprisingly, we started to encounter significant rework as stories inadvertently broke each others’ tests. We experimented with a number of different and creative approaches to try to ease the pain of the long test times, like running the tests in a “headless” browser, splitting the test run in parallel across multiple CPUs and cores, and marking some tests as “stable” and running those less frequently. All these approaches had their own benefits and drawbacks, and we eventually had to fall back to the tried and true technique of profiling our tests to find the performance bottlenecks.

7.3349736 /^I have completed registration as a proxy user$/
  6.7926899 And I have completed registration as a  proxy user
  7.8772572 And I have completed registration as a proxy user
6.6970380 /^I have completed registration as a basic user$/
  6.1421517 And I have completed registration as a basic user
  7.2519242 And I have completed registration as a basic user
5.4250364 /^I finish my enrollment$/
  5.4250364 And I finish my enrollment

Cucumber has some built-in support for this in the form of its “usage” output formatter. The data provided by this formatter is very valuable, but there are a few issues that are helpful to understand up front. Most importantly, it doesn’t recursively capture data when steps call other steps, so some steps may appear to be called very few times even though they might be called frequently by other steps. Also, while the output shows the average running time for each different step along with the actual running time for each call, it doesn’t show either the total number of calls or the total accumulated time for all calls to the same step. As a result, to get the real “big picture” of which steps have the most impact, one needs to count lines of output and multiply by the average time.

3.5672804 /^(?:|I )log in as "([^"]*)"$/
  3.4786973 And I log in as "john8"
  3.3762024 And I log in as "joe_the_great"
  3.2333490 And I log in as "MyUser"
  0.0000773 And I log in as "MyUser"
  3.7173720 And I log in as "MyUser"
  3.2421371 And I log in as "friendlyjoe"

We had 321 Cucumber scenarios which took a total of 39½ minutes to run on our fastest workstation. Even using the parallel_test gem to leverage the machine’s 8 cores, this test suite still took nearly 20 minutes to complete. After analyzing the output of profiling, we found that in any given run we were logging into our application 248 times. Since each login step took an average of 3.6 seconds to run, that meant our test run was spending almost 15 minutes just logging in, so this step was an obvious candidate to try to improve. Let’s take a look at what the original version of that step looked like:

Given /^(?:|I )log in as "([^"]*)"$/ do |nickname|
  visit logout_path
  visit login_path
  fill_in "username", :with => nickname
  fill_in "password", :with => "pass1234"
  click_button "Log in"
end

This step uses Capybara/Webdriver to tell the browser to go to the logout page (in case we were already logged in), go to the login page, fill in the username and password fields, and click the submit button. This approach is perfect for “When” steps – times when our scenario is testing the user’s login interaction itself. All other times, this is actually a “Given” – a precondition for a scenario, and not the actual functionality being tested directly. For comparison, here are two example scenarios, one for each case.

Scenario: Enrolled users should be able to log in.
  Given an enrolled user with nickname “joe_user”
  When I log in as “joe_user”
  Then I should see “Welcome, joe_user!”

Scenario: Logged-in users should be able to see their own profile.
  Given I am logged in as “joe_user”
  When I go to the “my profile” page
  Then I should see “edit your profile”

Most of the time, any “Given” step should not need to directly drive a browser to do its work. (The most obvious big exception to this rule is the case of “Given I am on ”.) In this case, we modified the step for “Given I log in as …” so that it could log in to the application without needing to use the browser. Our new version of this step looks like this:

Given /^(?:|I )log in as "([^"]*)"$/ do |nickname|
  browser = Capybara.current_session.driver.browser
  browser.manage.delete_all_cookies
  http_response = post_to(login_path, { 'user[nickname]' => nickname,
                                        'user[password]' => 'pass1234' })
  visit login_path unless current_path
  browser.manage.add_cookie parse_set_cookie_header(http_response['set-cookie'])
end

First, we “log out” of the application by deleting all cookies from the browser. Next we send an HTTP POST to the login controller directly from the cucumber code. Then we tell the browser to load a page if it hasn’t already, because setting cookies without having loaded a page is an error in all browsers. Finally we parse the session cookie from the response and set it in the browser directly.

0.1478878 /^(?:|I )log in as "([^"]*)"$/
  0.1518410 And I log in as "john8"
  0.1560398 And I log in as "joe_the_great"
  0.1462007 And I log in as "MyUser"
  0.0001179 And I log in as "MyUser"
  0.1591520 And I log in as "MyUser"
  0.1520781 And I log in as "friendlyjoe"

The resulting change in performance was dramatic, dropping the total time for the test suite to 12 minutes, 44 seconds. The average running time for this version of the “Given I log in as…” step is now 0.15 seconds. Combining this improvement with our use of the parallel_test gem means we can now run our entire acceptance test suite on an 8-core workstation in under 5 minutes, making it much less painful for us to run the entire test suite each time we are ready to push code to our repository.

There was another issue that we encountered on the way to this improvement – in the current version of Selenium Webdriver, the only browser for which this technique works out of the box is Chrome. There is a bug in the firefox driver that raises exceptions when calling addCookie while testing against localhost (the very scenario we most needed). We have created a Github fork of the Selenium2 code and fixed this bug, and will work with the Selenium development team in the near future to see this fix applied to the core Selenium project. There is also an open bug in the Selenium driver for Internet Explorer, where addCookie silently fails, forcing us to fall back to the slower browser-driven login approach on our IE testing.