Agile
Faster Cucumber Tests
by John Sextro on April 11th, 2012 at 6:58 am
Let’s face it you can never have enough money and your tests never run fast enough. I can’t help you with your money, but if you follow the steps in this post I can help you speed up your Cucumber tests.
The improvement process that I’ll walk you through is known commonly as DMAIC (Define, Measure, Analyze, Improve, Control). For those not familiar with DMAIC, Google can tell you everything you need to know. But, basically DMAIC is an improvement cycle commonly used in Six Sigma to improve and optimise business processes. We’ll use DMAIC in this post to improve the speed of your Cucumber tests.
Define
You’ve probably already defined the problem, your tests are too slow. What you haven’t done is define success. You will define success in terms of execution time. For instance, if your Cukes are taking 30 mins you might define success as, “Make all Cukes run in 20 mins or less”, others might define success as, “Cukes running in 5 mins or less”. This is up to you and your team to decide. Just remember, the amount of time it takes to run your tests will be inversely proportional to number of times you will run them. So the faster they are the more often you will run them.
Measure
You know how long it takes to run the suite. You might even know how long it takes to run each scenario or sets of scenarios. What you need to know now is how long it takes each step to run. Wait! Before you go diving back into your steps to instrument them with timers and such, let’s see what Cucumber can do to help us.
Fortunately for us Cucumber offers a number of helpful options for formatting the output of our test runs. One of those options is --format usage. Using the ‘usage’ format will show us which steps take the longest to run, how many times each step executed and if any steps are unused. I’ve slightly modified some of the example feature files provided with the Cucumber source to generate the following output.
Cucumber Output using --format usage
Analyze
With our new data in hand we can begin to zero in on the trouble spots. The usage report displays the steps ordered from slowest to fastest. The report also shows the number of times each step executed. Your slowest step might not be the biggest problem. You should focus on the steps that run the slowest and the most often. Also, consider why the step executed so often. If you wrote the scenarios following a behavior driven development (BDD) cycle then the scenario was written to define behavior in the system. As a result you may find many scenarios exercising the same part of the system in only slightly different ways. Viewing the scenarios in hindsight you should consider if you can consolidate or eliminate some of the scenarios.
Improve
Now that we know where to focus our efforts it’s time to make our improvements.
Consolidate
In the ‘Analyze’ phase you considered if you could remove or consolidate scenarios. Now is the time to do that work. Deleting scenarios is fun and easy. Consolidating scenarios takes more effort and thought. Look for recurring patterns such as screen navigation and data setup. If you notice that you repeatedly setup the same data to run different scenarios, it’s a good bet that those scenarios are ripe for consolidation. From the regression testing perspective, you want to maximize the amount of testing performed while minimizing the amount of work you have to do in the system to prepare for the testing while maintaining the ability to independently execute the scenario.
Learn to wait
One of the most common causes of slow downs in cukes is ‘sleep’ing. People often use sleep to wait for AJAX activity to complete. This is the most unpredictable and wasteful way to control test execution. Imagine that an AJAX action normally takes between .5 and 2 seconds to complete. If we sleep to wait for the action to complete how long should we sleep? Let’s say we decide to sleep for 3 seconds. If that action ever takes more than 3 seconds the test fails. Also, every time the action takes less than 3 seconds the difference is wasted doing nothing and if this is one of your most executed steps those seconds quickly turn into minutes.
Instead of sleeping you must learn to ‘wait’. The Selenium client includes a number of ‘wait_for’ methods that will improve the predictability of your steps and will remove the waste caused by sleeping.
wait_for_text(value)— Simply waits for the text you specify to appear in the DOM. Should this value already exist, execution will continue without waiting, so carefully select a value that you know won’t exist until the AJAX action completes.wait_for_element(xpathLocator)— Allows you to specify an exact xpath locator to wait for before continuing.wait_for_no_element(xpathLocator)andwait_for_no_text(value)– Allows you to wait for the removal of an xpath locator or text value from the DOM before continuing.
Good Software Engineering
Once you’ve ‘picked’ all the ‘low hanging fruit’, you’re left with your software engineering skills and training to profile, tune and tweak the last bits of improvement from your steps.
Scenario Tagging
Cucumber provides a simple yet powerful ability to add tags to scenarios using the @tagname. If you’ve made all the improvements that you can reasonably make but your tests still aren’t fast enough, then you can use tags to define a subset of scenarios to execute. For instance, you could tag the scenarios with @smokeTest tag and run them by specifying ‘ –tag @smokeTest’ at the end of your cucumber command. Don’t forget to run your full suite of Cukes nightly, or better yet schedule them to run more frequently via your continuous integration server. Checkout the Tags page on the Cucumber wiki for more details on tags, https://github.com/cucumber/cucumber/wiki/Tags
Control
Congratulations. Give yourself a pat on the back. Hopefully you’ve met and surpassed your success criteria. Wait a second though, we’re not done. You need to keep an eye on things. Don’t let your test times creep up. That’s what the ‘Control’ phase is all about and it goes on for as long as you care about the amount of time required to run the scenarios. Make the ‘care and feeding’ of your cukes a regular part of your development cycle and they will provide your application with benefits for years to come.
Tags: BDD, cucumber, Cukes, Ruby, selenium, Test-driven development, testing
Kanban: It Changed My Life
by railsgrammer on February 8th, 2012 at 5:52 pm
Well, maybe not my life, but my project’s life. A bit reading and patience will allow me to explain. But first, a definition of the word: Kanban.
kanban |ˈkänˌbän| – noun
a Japanese manufacturing system in which the supply of components is regulated through the use of a card displaying a sequence of specifications and instructions, sent along the production line.
“Now wait. We are not in Japan, and we are definitely not a production line.” Understood, however we can take this system and apply it to our practice.
Kanban in practical terms is best described with a picture.
We have multiple queues drawn by string, or marker on a whiteboard, or imaginary (it depends on the project). Each queue represents a phase of our development process. A new project, for example, may have the following queues: Waiting, Code In Process, Code Complete, Acceptance Testing In process, Acceptance Testing Complete, Ready for Customer Review, Ready for Production.
As requirements, from our client or customer, are recorded, they are written onto story cards and placed into the Waiting queue. When someone begins working, that person picks up the first card and moves it over to the next queue to begin Coding. When the story has been completely coded, it goes to Code Complete so that the person who provides acceptance testing can pick it up and begin working on that story. This process continues to happen until it reaches the end.
Stories get worked on from the right side first, so that it can reach completion as fast as possible. This provides the most efficient and focused flow of progress.
- Recording, prioritizing, and tracking customer requirements.
- Showing progress of work in progress.
- Communicating when something is complete and ready for the next phase of the development process (Acceptance Testing).
- Providing visibility for when work is running out and new requirements are needed from the customer or client.
Kanban provides a ton of invaluable assets, as a system, to your project. Such as: efficiency, focus, communication, limit work in process, prioritization, visibility on status.
We also limit work in process (WIP), because work in process is considered waste. It is considered waste because it, at the state of being work in process, is unusable code. If at any point in time the customer says “Let’s change priorities completely” or if for some reason the project finished prematurely, the incomplete code is wasted.
We apply WIP limits by restricting the number of stories that can be in a particular queue. If you can’t move a queue forward because the WIP limit is already full, you are forced to either clear the queue ahead or find someone who can. This is very important because it prevents the project from consuming a block in flow of progress.
Our primary Kanban consists of the following queues. Feel free to use the exact same queues for your project but beware that every project is different and requires a different kanban. Pay attention to the pain of your project, start simple and adjust based on that pain.
- Backlog
- Waiting (limit 3 features)
- Code In Process (limit 3 stories)
- Code Review Ready (optional, limit 1 story)
- Code Review In Process (optional, limit 1 story)
- Quality Acceptance Ready
- Quality Acceptance In Process
- Quality Acceptance Complete
- Feature Waiting for Stories (limit 3 features)
- Feature Ready for Customer Review
- Release Waiting for Features
- Release Ready to be tested on Staging
- Release Ready for Production
- Released to Production
a Japanese manufacturing system in which the supply of components is regulated through the use of a card displaying a sequence of specifications and instructions, sent along the production line.
Your project’s development cycle is a production line. In order to get the most quality and the most work completed think about work in process, notice where stories have the longest cycle time (time a story spends in a given queue), which queue fills up too fast (perhaps you need more resources to satisfy that queue, or you need a greater WIP limit).
If you are developing cars, you want as many completed, high quality cars as you can build. You don’t want 500 windows, 200 doors, 6 handles, 3 mirrors and 1 car. Same goes for developing an application. You don’t want 10 features in process and 1 feature completed; you aren’t guaranteed time to finish the work in process.
I hope this post was useful to you. I would love to get some feedback.
This post is the first in a series I plan on publishing about how Kanban changed my life. Check in to see the additional posts. I will also update this post with links.
Tags: agile, change, dev, Kanban, metrics, project management, risk management
New Agile Design Article Posted at UX Magazine
by janderson on August 1st, 2011 at 12:18 pm
Curious about how UX design works in an agile world? Check out our latest article at UX Magazine to find out: Change on a Dime: Agile Design
Tags: agile, design, ui, User Experience, ux
Asynchrony Twitter Feed
- asynchrony: Debuting on Twitter: Our new logo! We'll be rolling the new look out to the website soon! Kudos to our designers for their hard work. April 27, 2012
- asynchrony: Tonight is @STLMobileDev! @MarkSawyier on his startup, @BonfyreApp; @markshiz on cross-platform dev with Android NDK. Talks start at 6:30p! March 22, 2012
- asynchrony: RT @railsgrammer: Kanban: It changed my life. A new post at @asynchrony 's blog http://t.co/cHTryvNi February 9, 2012
- asynchrony: RT @StlMobileDev: Thu Jan 26th: Doors: 6p Talks: 6:30p. @jessitron on Android storage; Srdjan Grubor on a factory dashboard for Android. ... January 24, 2012
- asynchrony: RT @SandiStJohn: Asynchrony is looking to hire java/andriod developers for perm opportunities. Call me today for more details 314-678- ... January 11, 2012
