When you have a web app that is under constant development Continuous Integration and Continuous Deployment are incredibly useful practices. Planned releases may seem like a less risky approach, but they are not. When you do something a lot you keep getting good at it. We release our product ten times a day every day and we are loving it!

Here is the shocking truth: In our company, any developer can commit new code to our source control system and the changes become live on our servers in less than 5 minutes. This means I can write a bug fix or new feature right now and it takes several minutes for it to be live for 300 thousand users.

When I tell this to other developers their eyes get wide and they look at me like I am crazy. Then I explain the secret but they still don’t believe it until they see it work.

Our secret is basically not a big secret if you are following the continuous integration/deployment gurus: Automated Tests. We have hundreds of tests running on each and every release. Even a single, tiny, little failure on one of these tests shuts down the release. That’s how we keep it sane.

We have three kinds of automated tests and use Hudson to execute them:
1. Unit Tests
2. Integration Tests
3. Browser Tests

Today, I will be focusing on the third one: Browser Tests. Since our web application heavily depends on JavaScript, we have to test it on a browser. In fact, we have to test it on many browsers. We test it on different versions of Internet Explorer, Firefox, Google Chrome and Safari. The tests are executed using the Selenium framework.

Problems with In-house Selenium Testing

The biggest problem with running Selenium tests is creating and managing many testing environments. Usually if you are running Selenium tests constantly on a machine it cannot be used by anyone. So in practice, they have to be on dedicated machines. And if your releases depend on the Selenium tests, you have make sure they are always on and running smoothly. It is a lot of maintenance headache.

The other problem with Selenium is that it is slow. It takes selenium long time launch up the browser do the tests. Unless you have many dedicated machines the tests have to run sequentially.

To speed up our Selenium tests I looked for ways to run them in parallel. My first approach was to use Amazon EC2, since I am already pretty familiar with it. That turned out to be not a great solution. We still had to maintain many selenium environments. Keeping many EC2 instances always on can be quite expensive. Launching new instances on demand turned out to be too slow for our case.

Final Solution: Sauce OnDemand

Then, I found out about Sauce Labs’s Sauce OnDemand service. This is basically an on demand service that let’s you run Selenium tests instantly on the cloud. Although documentation is limited, it has many good code samples. So, setting it up was pretty straightforward. The great thing about it is that you can choose from many browser setups and run your tests on them instantly. No need to maintaining dedicated machines.

Unfortunately, there was no documentation about running tests in parallel in PHP. So, we first looked into using PHPUnit’s new parallel processing features: @runTestsInSeparateProcesses and @runInSeparateProcess. They turned out to be too buggy to be usable at this time. So, we hacked together a simple script that runs the tests on the shell as background processes.

A Simple Example

Since there are no sample code available for doing this in PHP, I’d like to share a simplified version of the script we hacked together yesterday. It is kind of rough since we first focused on making this work. It is not clean or documented but it should give you a good jump start.

1. BrowserTests.php: Includes actual selenium tests. I simplified it for this example and only left one test. On our production version we have all the tests here. Since we prefer to save time and run all tests at once on a specific browser. You can create these tests using Selenium IDE or Sauce IDE.

2. allBrowserTests.php: You can run this on the shell then add it to Hudson or whatever you use for your Continuous Integration. Basically, this script runs selenium tests on the background in parallel. It then checks their log files to see if they finished successfully or failed.

3. SeleniumTestHelper.php: This is the integration script with SauceLabs. You should change the domain name, Sauce username and access key to your own.

Download all files files here. You must first have phpunit installed to use them.

Happy testing!

This entry was posted on Wednesday, September 22nd, 2010.