Testing 1, 2, 3

February 1, 2012 12:00 AM

The Container Store Inc. aims to make it easy for customers to shop across its multiple channels, in its physical stores, on the web and through mobile commerce. A consumer can configure a closet or home office space using the online elfa Design Center at, then walk into a store and share that configuration through her iPhone with a designer expert. For the finishing touch, she can complete the configuration project on an in-store elfa Design Center screen.

"We try to make it seamless for customers to start online and finish in the store, or plan something in the store and finish online at home," says Brad Schneider, who is head of The Container Store's web development team and oversees the retailer's quality assurance department. "This is not just an afterthought for us. We make sure from the launch of design tools that they can be multichannel. We want to make sure customers can start in one channel and finish in another, or do it any way they want."

In order to be sure its web site is playing its part, The Container Store employs several testing services so that it can verify that promotions are leading to sales, that customers can figure out how to use the elfa Design Center and that all the features of the site are not slowing down page load times, a problem that can not only drive away shoppers but lead Google to penalize a retailer's site in search rankings.

That kind of comprehensive testing program can make a big difference in a retailer's financial performance, particularly if a retailer is on a strong growth curve and experiencing sharp growth in traffic volume, experts say. "Running performance testing on a regular basis will uncover bottlenecks and give time to improve performance before the event occurs and your business and/or infrastructure is unable to scale to demand," IDC Retail Insights says in a report it issued last year.

The good news for e-retailers is that testing does not have to be expensive, and that they can sometimes get results in a matter of days, or even hours.

More shoes, please

At The Container Store, comprehensive testing showed the retailer that its elfa design tool wasn't showing all the options customers had available to design a customized closet space—a void that wasn't uncovered until pointed out by experienced testers participating in a uTest testing program.

The retailer works with uTest, a company that provides access to more than 50,000 testers in 180 countries who, typically in groups of 20 to 50 for a single project, will access a retailer's web content from different web or mobile browsers, then post comments about any bugs to a uTest-hosted portal. A retailer client can click a comment to send questions to a tester to better understand any problems he reported. UTest can cost as low as about $500 for a single test, but large retailers can pay $3,000 to $5,000 per month for an ongoing program of extensive tests, a uTest spokeswoman says.

Now in its second year of working with uTest to check the functionality of web site features, The Container Store learned from its testers that its online elfa Design Center closet configuration tool wasn't showing users all of the available options for including shoe racks of various sizes, leaving out what can be a crucial selling point for some shoppers, Schneider says.

In addition to pointing out such missing features, uTest testers typically will also provide a detailed review of how they navigated through a retailer's site and where problems cropped up. "UTesters usually provide more value to their bug findings than just 'This doesn't work,' they provide some explanation of how they got there," says Jerry Welch, manager of the software quality assurance group at The Container Store. "It provides an explanation of what our customers might have been doing."

Where's that store?

The retailer also found that a new store locator wasn't working properly on some mobile browsers. "UTest found out that particular ZIP codes weren't returning results, so we caught it before going live with it," Schneider says. He adds that The Container Store can get a team of uTest testers on a project quickly.

That came in handy when the retailer was about to launch a new mobile site last spring but its marketing team raised questions about whether it was ready to release. The Container Store arranged for 20 testers within moments, Welch says, and within 36 hours they had identified 60 issues with the new site. The retailer was able to work with its mobile developer, Usablenet, to ensure a fully functional mobile site within a week, including product research, social media links and checkout.

The Container Store also works with several other performance testing providers, including Compuware Gomez for monitoring web server uptime and downtime, Neustar Inc.'s BrowserMob for testing the ability of web pages to handle heavy traffic loads, Usability Sciences for checking site usability by consumers, and Tealeaf Technology Inc. for viewing site visitors' online sessions and analyzing, for instance, what might have led to site abandonment. The retailer also gathers feedback from site visitors through OpinionLab, which enables consumers to enter ratings and comments on a retail site.

Godiva's gifts

Even with business unusually brisk during the holiday shopping season, Godiva Chocolatier Inc. makes time to test the best mix of product images in its web site's merchandising displays, says Mahender Nathan, vice president of Godiva's direct business.

A recent test of multiple displays of a new 18-piece Parfait Chocolates Gift Box, for example, produced a winning combination that resulted in a 9.4% conversion rate lift, compared to an alternate display.

The winning gift box display continues to run today for all site visitors. The display shows an image of the partially uncovered gift box, revealing the chocolate pieces inside. Directly beneath the main image are three smaller images revealing alternate arrangements of the chocolates; each of the smaller images can be moused over to make them appear as the main image. The three smaller images are hosted by Monetate, which provides technology that Godiva uses to test multiple merchandising displays with various segments of site visitors. Godiva uses the Monetate TestLab, part of Monetate's Agility Suite of web testing software.

And testing is not just a matter of finding the right merchandise display to convert the most customers, Nathan says. The tests also help to determine, for instance, how much Godiva should invest in product images to get the best return on investment. "The ROI of our photo budget has increased as well," by showing which type of images, and how many of them, produce the best conversion rates, Nathan says.

Simpler is better

Another e-retailer found through testing that simplifying its site navigation would lead to more sales. After noticing too many visitors leaving without placing an order on the e-retailer ran a test and found shoppers would respond better to a simpler navigation bar with fewer choices to click.

Running a test through of how site visitors responded to two navigation bars—one busy with several click options, the other simple with only four options—Headsets found that the simpler one resulted in a 10% higher visitor-to-purchase conversion rate, says Matt Burnaford, vice president of marketing.

The retailer also learned that many of its customers, even ones who had purchased from Headsets before, didn't know enough about the many types of headsets to steer their way through site search and navigation.

The previous navigation bar offered shoppers five buttons to click to select among several types of headsets, including Wireless Office Headsets, Corded Office Headsets and Best Sellers. The nav bar also offered several other links, including Headset Wizard, Benefits of Headsets and Staff Recommendations.

But the retailer learned that many customers responded better to a nav bar that let them simply choose the type of device they planned to use a headset on, rather than having to first to choose among headset categories like wireless, corded or best sellers.

After Headsets ran the test through, its new winning nav bar now appears with only four link buttons: Office Headsets, Home Headsets, PC Headsets and Cell Headsets. The only other information in the navigation display is the retailer's 800 number, its hours of customer service, and a shopping cart icon with the number of carted items.

"The other navigation clearly wasn't working, but the leaner, simpler look produced the increase in conversion rates," Burnaford says.

One-hour service

Running the test was a simple matter of filling out a form on to get test results within about one hour, he adds. To run a test, a retailer gives the URL of the page to be tested, describes a task for site visitors (such as "Buy a headset"), and indicates the type of consumer by demographics (including gender, age and income group) from's available panel of consumer testers from throughout the U.S.

The participating testers then video-record themselves using the client retailer's site; once the test is complete, the retailer receives a copy of each tester's video so the merchant can view and listen to the recorded session. Retailers also receive a written summary from testers on any problems they encountered in using the site, and retailers can follow up to ask testers questions after each test. UserTesting charges $39 per tester.

Burnaford didn't say how many testers Headsets used in the navigation test, but notes that Headsets will use anywhere from one to 10 testers for any one test. A spokesman for says clients are free to choose any number of testers, but that most clients tend to run tests with either about five testers or 10-12 testers.

Keep the customer smiling

In some cases, retailers run tests with tiny in-house staffs., a gourmet cookie and gifts retailer that launched on the Magento e-commerce platform in 2010, has been generating triple-digit sales increases each year since then, says Adam Golomb, the retailer's director of e-commerce. A potential problem it faces is from traffic spikes driven by online marketing programs, primarily through the LivingSocial local deals network, Golomb says.

With just a two-person e-commerce team, the retailer constantly monitors its site performance to ensure that customers driven to its site by marketing can have a good shopping experience, traffic spikes and all, he adds. The site offers a feature for designing one's own batch of cookies, including shape, color of icing, and an illustration that can be created with icing, such as a smiley face.

Any one of these steps could lead to a glitch with the underlying software that leaves the consumer waiting for the site to respond, Golomb says. Working with technology from Yottaa Inc., the retailer gets reports that show, for instance, the site bounce rates associated with the load times for each interactive feature within a specified time period. "We look at every technology integration on our site and identify if a certain integration is slowing down the site, then we review software code to identify any issues," he says.

SmileyCookie uses Yottaa to monitor site performance as experienced from multiple web and mobile browsers. Boston-based Yottaa, which was launched in 2009, provides "cloud" or Internet-based monitoring services, which it currently offers for no charge, CEO Coach Wei says. It also provides web site optimization services that can cost from about $129 to several hundred dollars per month, based on the number of page views, Wei says.

With all the options retailers have for testing how well their sites provide a good online shopping experience, it should be easier than ever for them to keep their customers smiling and coming back.




Top Solution Providers