Faster Cucumber Tests
by John Sextro on April 11th, 2012 at 6:58 am
Let’s face it you can never have enough money and your tests never run fast enough. I can’t help you with your money, but if you follow the steps in this post I can help you speed up your Cucumber tests.
The improvement process that I’ll walk you through is known commonly as DMAIC (Define, Measure, Analyze, Improve, Control). For those not familiar with DMAIC, Google can tell you everything you need to know. But, basically DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimise business processes. We’ll use DMAIC in this post to improve the speed of your Cucumber tests.
Define
You’ve probably already defined the problem, your tests are too slow. What you haven’t done is define success. You will define success in terms of execution time. For instance, if your Cukes are taking 30 mins you might define success as, “Make all Cukes run in 20 mins or less”, others might define success as, “Cukes running in 5 mins or less”. This is up to you and your team to decide. Just remember, the amount of time it takes to run your tests will be inversely proportional to number of times you will run them. So the faster they are the more often you will run them.
Measure
You know how long it takes to run the suite. You might even know how long it takes to run each scenario or sets of scenarios. What you need to know now is how long it takes each step to run. Wait! Before you go diving back into your steps to instrument them with timers and such, let’s see what Cucumber can do to help us.
Fortunately for us Cucumber offers a number of helpful options for formatting the output of our test runs. One of those options is --format usage. Using the ‘usage’ format will show us which steps take the longest to run, how many times each step executed and if any steps are unused. I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.
Cucumber Output using --format usage
Analyze
With our new data in hand we can begin to zero in on the trouble spots. The usage report displays the steps ordered from slowest to fastest. The report also shows the number of times each step executed. Your slowest step might not be the biggest problem. You should focus on the steps that run the slowest and the most often. Also, consider why the step executed so often. If you wrote the scenarios following a behavior driven development (BDD) cycle then the scenario was written to define behavior in the system. As a result you may find many scenarios exercising the same part of the system in only slightly different ways. Viewing the scenarios in hindsight you should consider if you can consolidate or eliminate some of the scenarios.
Improve
Now that we know where to focus our efforts it’s time to make our improvements.
Consolidate
In the ‘Analyze’ phase you considered if you could remove or consolidate scenarios. Now is the time to do that work. Deleting scenarios is fun and easy. Consolidating scenarios takes more effort and thought. Look for recurring patterns such as screen navigation and data setup. If you notice that you repeatedly setup the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation. From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing while maintaining the ability to independently execute the scenario.
Learn to wait
One of the most common causes of slow downs in cukes is ‘sleep’ing. People often use sleep to wait for AJAX activity to complete. This is the most unpredictable and wasteful way to control test execution. Imagine that an AJAX action normally takes between .5 and 2 seconds to complete. If we sleep to wait for the action to complete how long should we sleep? Let’s say we decide to sleep for 3 seconds. If that action ever takes more than 3 seconds the test fails. Also, every time the action takes less than 3 seconds the difference is wasted doing nothing and if this is one of your most executed steps those seconds quickly turn into minutes.
Instead of sleeping you must learn to ‘wait’. The Selenium client includes a number of ‘wait_for’ methods that will improve the predictability of your steps and will remove the waste caused by sleeping.
wait_for_text(value)— Simply waits for the text you specify to appear in the DOM. Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.wait_for_element(xpathLocator)— Allows you to specify an exact xpath locator to wait for before continuing.wait_for_no_element(xpathLocator)andwait_for_no_text(value)– Allows you to wait for the removal of an xpath locator or text value from the DOM before continuing.
Good Software Engineering
Once you’ve ‘picked’ all the ‘low hanging fruit’, you’re left with your software engineering skills and training to profile, tune and tweak the last bits of improvement from your steps.
Scenario Tagging
Cucumber provides a simple yet powerful ability to add tags to scenarios using the @tagname. If you’ve made all the improvements that you can reasonably make but your tests still aren’t fast enough, then you can use tags to define a subset of scenarios to execute. For instance, you could tag the scenarios with @smokeTest tag and run them by specifying ‘ –tag @smokeTest’ at the end of your cucumber command. Don’t forget to run your full suite of Cukes nightly, or better yet schedule them to run more frequently via your continuous integration server. Checkout the Tags page on the Cucumber wiki for more details on tags, https://github.com/cucumber/cucumber/wiki/Tags
Control
Congratulations. Give yourself a pat on the back. Hopefully you’ve met and surpassed your success criteria. Wait a second though, we’re not done. You need to keep an eye on things. Don’t let your test times creep up. That’s what the ‘Control’ phase is all about and it goes on for as long as you care about the amount of time required to run the scenarios. Make the ‘care and feeding’ of your cukes a regular part of your development cycle and they will provide your application with benefits for years to come.
Going Faster With a Back-Seat (Web) Driver
by Matt Campbell on June 17th, 2011 at 10:42 am
Slow, long running test suites can be almost as dangerous to a project’s health as not having tests at all. The presence of tests gives developers, managers and customers a sense of security about the stability of the code base, but that sense of security is only valid if those tests are running (and passing) frequently – preferably continuously. When test suites take too long to run, they cause a productivity drain to the developers, costing us and our customers money. When this productivity drain gets bad enough, developers tend to react to it by avoiding running the test suites, eliminating much of their value. That’s why it’s very important for project teams to keep their test suites performing quickly.
Building Eclipse RCP Applications the Agile Way
by mhainline on June 4th, 2010 at 2:41 am
For a long time now, the Eclipse Rich Client Platform has suffered from a severe lack of support for one of the core elements of Agile programming–Continuous Integration. The support for building and packaging an RCP application through the IDE itself was very different from the headless support for those same tasks through PDE/Build. The testing support for JUnit 4 available through the IDE was not even present in the Eclipse Test Framework. The situation has markedly improved in Eclipse 3.6 with advances in the Eclipse Test Framework and the addition of Buckminster and p2 as a possible replacement for PDE/Build, but setting up a continuous integration system for Eclipse RCP remains a time-consuming and difficult task. An Agile project really needs to have continuous integration up and running hopefully from day one. To make this easier, I’ve created a quick-start template project called Lightning, which demonstrates how to effectively combine Buckminster, the Eclipse Test Framework, and SWTBot for integration tests. It has been tested on Linux and Windows, though Buckminster on the Mac seems to still have a few kinks to work out.
Buckminster touts itself as a system that works the same headlessly as it does in the IDE, and while not fully living up to that promise yet, it comes close. Nevertheless, for our build we chose to simply kick off the headless version of Buckminster through ant–a solution that works equally well anywhere. In fact, since the headless version of Buckminster is a little hard to get installed, we put that into the build process as well. This uses the p2 director application available from the Buckminster download site.
The first step is to download Lightning and unzip it. Because we are using Buckminster, we can actually materialize our target platform for the project from the Helios p2 repository. To do so, simply navigate to the buckminster.build.feature and run the command “ant materialize.target.platform”, which will materialize the target platform to a directory <unzip location>/lightning/workspace-target-platform. That will serve as the target platform for the IDE workspace, and includes plugins the actual build doesn’t need, such as source for Eclipse. Import the lightning plugins into a new Eclipse 3.6 workspace, switch the target platform to the new platform you just materialized, and you should be all set to follow the code for the rest of this blog.
We’ll start with a quick rundown of the tools we use for building and testing our new RCP product.
ant
At the top level ant is used to manage all the other tools. It provides the top-level targets, and gives us everything we need to actually run the build from Eclipse, the command line, or Hudson. For more information on ant, see the Ant Manual.
p2
p2 is the provisioning system in Eclipse, replacing the old update manager. It is available both within the IDE as well as in a stand-alone utility Eclipse distribution that is intended to be run from the command line. p2 can be used to both download and install Eclipse plugins and RCP products. For more information on p2, see the p2 Wiki.
Buckminster
Buckminster is a build system for Eclipse products and plugins, and can be run from within the IDE (with the proper tooling installed) as well as headlessly from the command line. We use the command-line version in our build system, so no additional tooling is required in the IDE. Buckminster can inspect artifacts such as plugins and features in order to determine all of their dependencies, and can pull those dependencies from p2 repositories. Buckminster is somewhat similar to Maven, but with a focus on Eclipse artifacts. For more information on Buckminster, try to read through the Buckminster Manual. No really. Go on. I dare you.
Eclipse Test Framework
The Eclipse Test Framework allows us to run JUnit4 tests as plugin tests from the command line. This means that the tests run within the OSGi environment provided by Eclipse, and have access to things such as the Extension Registry.
SWTBot
SWTBot is a framework for running tests that interact with the user interface directly. These can be used to click buttons and check results from the user interface, and we use this for User Acceptance Testing. SWTBot comes both as a set of tools for use in the Eclipse IDE as well as in a headless distribution. We use the headless distribution in the build process, but include the other tools in the workspace target platform. They do not play very well together.
Hudson
We really haven’t created a continuous integration system until we’re actually running it continuously. Hudson is my favorite CI system, and is quite easy to set up. Hudson includes a Buckminster plugin, but we since we can launch everything through ant, we don’t actually need it. By relying on ant we ensure that the build can be run both from the developer box as well as the continuous integration system without risking differences in configuration.
Now that the tools are out of the way, let’s take a look at the build process itself. You can kick off the full build process by just running the command “ant” from the buckminster.build.feature directory.
The first step in the build process installs the p2 utility, called the “director”. This in turn is used to install the headless version of Buckminster. The next step uses Buckminster to build the product.
There are two products we will build in this process. The first is the test product, which will include everything in the production product along with all tests and test dependencies. The second product is the actual production product that we can then ship to our customers. If it seems a little strange that we need two products, just remember that we’re actually going to run our tests inside an OSGi environment, and so we need everything packed into plugins, including our test code.
When Buckminster runs, it needs to run in an Eclipse workspace, in just the same way it would run if we were running it from within the IDE. The first step of the Buckminster script we run sets up a new workspace and target platform in order to allow Buckminster to build. The target platform is imported using a target definition file, which simply tells Buckminster where to place any plugins it later downloads for the target platform. Next, the import command actually executes an import of those plugins as defined in a “cquery” file. The .cquery file points to an Eclipse feature that was created especially for Buckminster, along with a list of places to find all the plugins that are included in that feature, which is stored in an “rmap” file.
The rmap points to a few different places to find plugins. Some of these are from p2 repositories on the internet, some are from our local source project checkouts, and some are from local p2 repositories. The local p2 repositories were actually created earlier in the build process using the p2 director application and a list of pre-built plugins. Once the cquery is imported, Buckminster has a fully materialized target platform along with all of the checked out projects imported into its own workspace area, and is now ready to build the projects. Note that we could have set up our build to use the workspace-target-platform we’ve already created and the build would probably be a bit faster, but we wanted to show how Buckminster can pick out all of the plugins it needs and assemble a target platform on the fly. Notice that we had to create a local p2 repository for the Eclipse Test Framework as well. In the future we hope that Eclipse will put the ETF into one of their public p2 repositories, and in fact there is a bug out there for that right now.
Once we build all the plugins in the workspace with Buckminster, we execute a Buckminster target. The normal target we would execute would be one that is automatically created for us by Buckminster by virtue of the fact that we are inside an Eclipse feature. That target is called site.p2, and tells Buckminster to install all of the plugins into a new temporary p2 repository. From there we could do whatever we like with them. In our case, however, we have added a different target in a “cspex” file. This target will call site.p2, but also use p2 to install the product into the file system. In this case, we install our test product with the command “create.product.test”. Notice that this command actually runs an embedded version of ant that comes with Buckminster to perform the work.
After our test product is installed in the file system, we need to run the tests using the Eclipse Test Framework. The Eclipse Test Framework includes an ant file to run it, but this ant file was a little limited, so we replace that functionality in our own ant file. The test framework is launched as an Eclipse application, and can only run a single test or test suite at a time. Because of that we have created an AllTests suite for each plugin fragment.
The test target in ant launches the test framework from within our test product that we installed earlier. It passes in the plugin that contains the tests for our test fragment, and then saves the resulting JUnit test reports. These test reports can be displayed by Hudson or the Eclipse IDE. It also runs the SWTBot tests in a very similar manner, and saves the results as well.
Assuming all tests pass, we package the final product. The build system can package the product for multiple platforms.
I hope this project is useful to some of you Agile RCP developers out there. Let me know how it works out for you, or if you have any questions.
Asynchrony Twitter Feed
- asynchrony: Debuting on Twitter: Our new logo! We'll be rolling the new look out to the website soon! Kudos to our designers for their hard work. April 27, 2012
- asynchrony: Tonight is @STLMobileDev! @MarkSawyier on his startup, @BonfyreApp; @markshiz on cross-platform dev with Android NDK. Talks start at 6:30p! March 22, 2012
- asynchrony: RT @railsgrammer: Kanban: It changed my life. A new post at @asynchrony 's blog http://t.co/cHTryvNi February 9, 2012
- asynchrony: RT @StlMobileDev: Thu Jan 26th: Doors: 6p Talks: 6:30p. @jessitron on Android storage; Srdjan Grubor on a factory dashboard for Android. ... January 24, 2012
- asynchrony: RT @SandiStJohn: Asynchrony is looking to hire java/andriod developers for perm opportunities. Call me today for more details 314-678- ... January 11, 2012
