Smarter testing

March 1, 2016 03:06 PM

Website testing is getting increasingly sophisticated. Rather than rely on simple A/B tests that measure whether a website change will produce significant results, a number of retailers are turning to testing protocols that enable them to feed customer data in before testing or layer that information on for analysis afterward.

“The technology is evolving and the statistical models are becoming more refined and useful,” says Chris Goward, founder of WiderFunnel, a conversion optimization agency that helps retailers use testing software.

For example, when Matt Greene joined Gardener’s Supply Co. two years ago as a marketing coordinator, the e-retailer of gardening tools and home and yard décor was not actively testing its site. In earlier attempts the retailer used the Content Experiments tool built into Google Analytics to split-test its website content. Content Experiments allowed the Gardener’s Supply to create tests such as changing text or button colors, but at the time, testing itself wasn’t seen as a big, high opportunity idea, Greene says. It wasn’t easy to use, either. It wasn’t until vendors came to the market with tools that could deliver deeper insights that could improve sales that Gardener’s Supply decided to make testing part of its regular operations, Greene says.

“In the past couple years, as the number of tests we’ve run has increased, so too has the company’s adoption of testing as an ongoing, innovative process,” says Greene, now in the newly created position of e-commerce analyst and testing strategist. “The tools we now use support that strategy.”

That means Gardener’s Supply is running tests that integrate more data, such as tests triggered by actions consumers take on the site. Those tests are producing significant results. A test that expanded the product selection consumers saw after they conducted a search, for example, led to a 7% increase in the add-to-cart rate and an 8% jump in conversions, Greene says.

“If you’re a shopper visiting a product page, would you rather see unrelated products that we’ve subjectively chosen to show you, or relevant alternative products that suit your intentions and are driven by machine learning?” Greene says. “We believed users want to see the latter, and it turns out we were right.”

Greene now intends to work with even more data-driven hypotheses, such as measuring the preferences and actions of new versus returning visitors, customers versus non-customers and consumers visiting the site from certain devices or browsers. “Having that kind of information and being able to act on it quickly provides online retailers with a major advantage,” Greene says.

Many other e-retailers too are finding that more rigorous online testing and analysis models are delivering insights that can drive sales. Further, such tests can now be managed by non-technical staff, as opposed to requiring the help of information technology staff. E-retailers say testing tools are becoming more user-friendly, are more easily integrated into the software systems that run their e-commerce sites and are more capable of feeding in other data sources to let e-retailers gather more valuable responses. For example, the tools are letting e-retailers track online shoppers throughout their shopping trips, and then segment the shoppers to learn details like their churn rates, lifetime value and the total revenue per segment of shoppers.

At startup e-retailer Native Cos., the natural deodorant maker grouped consumers into two buckets—returning customers and new or lapsed customers to examine whether consumers prefer a free travel-size deodorant sample or a $6 discount off the purchase of three full-size deodorants (from $36 to $30).

Within a day of launching the test, Native had a clear answer:  By a 22% margin customers preferred the free deodorant sample. The result led Native to show travel-size deodorant to website visitors who haven’t bought anything and those who haven’t made a purchase for longer than two months.

Previously, e-retailers have needed IT’s help to set up testing software and run tests like Native’s effort. But that’s no longer the case. Some content management systems include testing software, says Scott Smigler, CEO of Exclusive Concepts Inc., a marketing agency that works with small and mid-market retailers, including and That enables marketers to brainstorm an idea and launch a test in minutes, rather than days or weeks, he says.

Not only are more testing tools available, they’re built for non-technical workers, Greene says.

“The advances are in both the sophistication of the tools and the fact that more people than ever can now get involved in the testing process,” he says. At Gardener’s Supply, e-commerce and web-development teams set up tests, often after consulting with creative, marketing and merchandising teams.

Some tools have direct integrations with e-commerce platforms, which typically benefit more advanced testers as they can hook into product feeds and content, Smigler says. Other tools require testers to paste a line of JavaScript into the retailer’s website code, he says.

The testing services Gardener’s Supply uses, Optimizely and Evergage, require a line of JavaScript on the retailer’s website. Evergage also stores the retailer’s entire product catalog and content library in its system, Greene says.

“This gives us the ability to test personalized experiences that dynamically serve products and/or content to our site visitors,” he says. “For example, Evergage is able to look at our whole product catalog and make product recommendations based on advanced algorithm ingredients, such as trending products from the last 24 hours.” Evergage hooks into Gardener’s Supply’s e-commerce platform software from Demandware Inc., expanding the testing and personalization capabilities that aren’t available using Demandware alone, Greene says.

A hurdle for retailers is to find the right testing vendor and set up the tests properly without getting lulled into thinking that installing the software will solve their problems, WiderFunnel’s Goward says.

Greene says Gardener’s Supply finds Optimizely’s technology works best for testing big changes to the user experience. For example, the e-retailer used to show shoppers who added a product to the cart a drop-down cart to confirm their action. It used Optimizely, where pricing starts at $50 a month, to create a pop-up confirmation instead, which allows it to more aggressively cross-sell other products. Greene says Evergage’s strength is in behavioral targeting and product recommendations and it provides simple ways to test campaigns. Fees for Evergage’s services are based on traffic levels and selected modules; for example, a retail site with fewer than 100,000 visitors a month can expect to pay about $20,000-$30,000 a year for the tools., an e-retailer that sells apparel and merchandise to college fraternities and sororities, has used SiteSpect Inc. technology to better organize the buttons on’s mobile checkout page and improve conversion rates, says Robb Haas,’s chief operating officer.

SiteSpect lets retailers present different variations of features to customer groups based on customer lifetime value, geography, device and other characteristics. E-retailers can also use the technology to test different website images, form fields, and static and dynamic content. Pricing starts at $75,000 for an annual subscription.

SiteSpect integrates with e-retail sites at the Domain Name System level, which means web traffic passes through SiteSpect servers before reaching the merchant’s website, says Smigler, who advised on the SiteSpect implementation.

During that pass-through, SiteSpect collects data on visitors, such as the device they are using and the traffic source they’re coming from. Then, based on the rules of the test that have been established, it changes the website experience for a shopper to test different elements using a “find/replace” concept. It also lets retailers perform segmentation retroactively, meaning it shows a retail client how a test performed across dozens or more segments and browsing scenarios to discover insights the retailer may not have expected, Smigler says.

Only smartphone users to saw the checkout page test. The test showed that both new and return shoppers visited product pages immediately before they placed an order, suggesting that even returning visitors were browsing before they completed the purchase they had started on their last visit, Haas says.

While the new mobile checkout page increased the rate at which first-time visitors completed a purchase by 46%, the average amount each customer spent remained flat when looked at across all shoppers. But could also see repeat visitors were 34% more likely to buy than new visitors, and they spent 21% more, Haas says.

The example shows how e-retailers can do deeper post-test analysis to reveal insights that can make an impact. SiteSpect’s post-test technology drills down across scenarios, even to the point, for, of figuring out the patterns of mobile repeat visitors who entered through a search engine click who were on the site for more than five minutes and used filters on the category page. This differed from GreekGear’s previous testing method, which required the retailer to select two or three scenarios to focus on in advance of a test.

The results of some tests can be revealing, or even counterintuitive. Oberweis Dairy has 42,500 online customers whose home deliveries of milk, ice cream, eggs, bacon, yogurt, cinnamon rolls and other foods account for about one-third of Oberweis’ undisclosed total yearly revenues, says Bruce Bedford, vice president of marketing analytics and consumer insights. The dairy also sells wholesale to supermarkets and operates ice cream shops.

For close to a decade, the dairy made the same introductory offer online and through direct mail to consumers to get them to try its home delivery service, offering free home delivery for six months. Oberweis knew the offer had worked to bring new customers in, but many customers quit the service after they had to start paying the regular $2.99 per order delivery fee.

Oberweis used business analytics software vendor SAS Institute Inc. to measure the response of new customers to a different offer. Instead of offering free delivery for six months, Oberweis offered delivery for 99-cents per order for a year. After 12 months, the retailer’s standard delivery fee applied.

The lengthy test delivered results that challenge the belief that consumers like free more than paid. Oberweis found that 30% more of the consumers who became Oberweis customers through the 99-cent promotion versus the free promotion stayed on when the regular rate kicked in, resulting in a significant boost in yearly sales. “We discovered that, for many people, certain things only have value if they paid for them,” Bedford says.

The test for the online ordering service is one example of how the dairy implements more sophisticated testing methods across the organization to improve its business. For its ice cream shops, for example, it feeds weather forecast data into SAS to help set staffing levels—the idea being that when it gets warm, more customers will visit Oberweis shops for a scoop. The dairy found that the dew point level, which is a measure of the amount of moisture in the air, is a better predictor of higher foot traffic than the temperature, and now aligns staffing levels to follow it.

Testing programs today can do more than show whether consumers are more likely to press a blue button over a green one, having developed to the point where the answers they tease out can drive meaningful bottom-line results.




AlertBot (
Website Monitoring Industry Leader. Test web pages, login areas, checkout processes and more. Track performance and be alerted of errors. Take a Free Trial.




Top Solution Providers