|
|
|
Subscribe to Advertising & Marketing Review!
Contact Ken Custer at 303-277-9840.
|
|
|
|
|
|
|
Beta Testing Your Website
by Glen Emerson Morris
Beta testing is a critical phase of developing a successful Website because it provides you with your last chance to find and fix problems with it before it goes live. Beta testing should not be confused with user, or usability, testing as usability testing just covers whether the Website is designed well enough for customers to use it conveniently and without problems.
Beta testing includes considerably more issues, including load testing, failure recovery, security, and platform and browser compatibility, among others. Of these, load testing can be the most expensive. (A few years ago I conducted load testing on a Website for the Center for Medicare/Medicaid Services, and the bill was $10,000 to rent LoadRunner for one month to test with a load 250 simultaneous users.) However, it's usually browser/platform testing that's the most problematic.
One of the annoying issues of developing Websites is that the same code will look differently on different browsers running on different operating systems (aka platforms). At one point in 2000 WebMD had five testers just to cover different variations of MS Internet Explorer. Sometimes there would be major screen draw differences between minor variations of versions. Things are much better than those days, but we're not out of the woods by any means. If you're going to launch or update a Website, you need to test it for OS browser compatibility issues before it goes live, and this can be a lot of work.
The first step, taken as early as the design phase, is to determine what platform/browser combinations will be supported. It's a good idea to study the current statistics of your Website and see what people are actually using to log on with. Most server logs can provide this information very easily. A good approach is to import the data into an Excel spreadsheet and sort the list by number of users for each platform/browser, and decide where the cutoff point should be on the basis of diminishing returns.
For nearly any Website the matrix should include Windows, Mac and possibly Linux platforms, and browser combinations including Internet Explorer, Firefox and Safari, and possibly some other browsers, too. The total matrix is the number of operating systems, including sub variants of each, multiplied by the number of browsers, and the sub variants of each.
For instance, a matrix might include Windows Vista, XP, with browsers including IE 6, 7 and 8, Firefox 2 and 3, and possibly Safari and Opera. A basic matrix then would be 2 operating systems times 5 browsers (just counting IE and Firefox versions) or 10 different combinations to be tested, just for Windows compatibility. Even this simplifies things, because this matrix doesn't include service packs.
When you consider that there are over a dozen browsers available for Windows, the matrix starts getting large, and this is before you add in Macs, or cell phones, which are increasingly used to surf the net.
If your company is developing a Website that will include cell phones as a supported platform, it's highly likely that you won't have the budget to test all the possible cell phone models your customers might be using. One solution is to farm the work out to a QA service that specializes in cell phone QA testing. If that proves too expensive, the other option is to make the Website available to the general public on a limited basis, and use the public as an external beta site.
If you decide to use your customers as beta testers it is important that they be made aware that as it's a beta version and they should expect to find some problems with it, and it's also very important that your new Website is really at the beta stage of development.
The first criteria for being at the beta phase of development is that the software's feature set is complete. The second criteria is that the software is free of all severity one and two defects, meaning the software doesn't crash and all features actually work reliably. For practical purposes, you won't really know the software is ready for customer beta testing until after it's gone through a full round of internal testing and passed. It doesn't pay to make it available earlier, and can even be counter productive if you do. Only a small set of the general public is willing to try a beta version of any software, and prematurely releasing a Website with major problems can drive many of them away from testing later versions (or builds, to use the technical term).
There are two major issues to consider when using the general population as testers. The general population does not have the right skills to write up a good definition of a software problem, including steps to reproduce, and you have to set up some reliable method of getting the problems customers report to the engineers so they can fix it.
One approach is to have a standard form customers can fill out and email to you. Another approach, more costly but more effective, is to have an online database available for them to enter the information into. Either way, the critical information you need from them is what operating system and browser versions they had problems with, and as best they can describe it, how to reproduce the problem. It's a good idea to provide your customer beta testers with information about how to find the browser and operating system info. I know this is basic stuff, but a lot of your customers won't know how to get this information on their own.
With either approach, it's a good idea to have a person assigned to screen the defects customers report, weed out the useless ones (and some will be useless), contact the customer for more information if needed, and finally, make sure the defects get assigned to the right engineer.
If you follow these steps, your Website will probably launch successfully, but the work doesn't end there. Any time a new or updated version of the major operating systems or browsers is released, you should run your Website through a full set of tests. So, assign someone the job of monitoring new OS and browser releases and have a process in place to trigger testing when upgrades happen. It's work, but quality comes at a price.
Glen Emerson Morris was recently a senior QA Consultant for SAP working on a new product to help automate compliance with the Sarbanes-Oxley law, an attempt to make large corporations at least somewhat accountable to stockholders and the law.
He has worked as a technology consultant for Yahoo!, Ariba, WebMD, Inktomi, Adobe, Apple and Radius.
Copyright © 1994 - 2008 by Glen Emerson Morris
All Rights Reserved
'
keywords: Internet advertising, Internet marketing, business, advertising, Internet, marketing.
For more advertising and marketing help, news, resources and information visit our Home Page.
Back to top
|
|
|
