Perils and pitfalls of ad testing

Over the next 113 days, lots of campaigns will be looking at lots of
ads — nearly $10 billion worth, according to some estimates — trying to
decide which are effective and which aren’t, which are worth airing and
which shouldn’t be broadcast. The evidence suggests they will often be
wrong, especially if they rely on instinct or focus-group testing.

A recent experiment demonstrates again that neither professionals nor the public (who constitute focus groups) can predict which ads will work and which won’t. Researchers from the Universities of Michigan and Oregon, as well as UCLA, asked members of the target population to rate the effectiveness of three anti-smoking ad campaigns. The semi-focus group consistently ranked ad campaign “B” as far and away the most effective, “C” as least effective, with A in between. Industry experts (akin to campaign consultants) ranked B and C as more effective than A.

{mosads}The ads had a measurable goal — they asked people to call an anti-smoking hotline. So when the ads were actually broadcast, it was possible to measure their real-life effectiveness in increased calls.

How did the focus groups and the experts perform? Poorly.

Campaign B, forecast to be the most effective, did increase call volume 11.5 times compared to no advertising at all. But campaign C, which had been rated the least effective, actually increased call volume by 32 times, making the ads judged least potent, in fact, nearly three times more effective than those the public and the experts judged most convincing.

The folks at Go Daddy learned a similar lesson from their experience. Experts panned their first Super Bowl ad. MBA students and faculty at Northwestern’s prestigious Kellogg School of Management ranked the raunchy spoof as one of the six worst Super Bowl ads that year. An Ad Age headline labeled it a “loser.”

Wrong again.

A random-sample survey, before and after the game, showed Go Daddy enjoyed the biggest increase in both awareness and favorability. The real proof was in sales. The day after the ad aired the company’s market share skyrocketed by over 50 percent.

The most important limitation of focus groups is actually a limitation of the human mind. People are very poor reporters of their own decisionmaking processes.

We can ask them how and why they make decisions or which ad will motivate them, and they will answer those questions, but their answers often bear little resemblance to reality.

One classic experiment illustrates the point. Under the guise of consumer study, women shoppers were asked to rate the quality of five pairs of nylon stockings (OK, it was a while ago). What they didn’t know was that all five were identical — they were simply five pairs of the same stockings. But by 4-to-1, shoppers picked the pair displayed on the far right — the last one most eyes naturally encountered. Asked why, subjects responded with all kinds of explanations about colors and quality, explanations that could not have been accurate because the stockings were identical. Told that it might have been the position of the stocking (the only possible explanation), the shoppers insisted that had been totally irrelevant to their choice. They expressed preferences and were happy to explain them, but those explanations had nothing to do with reality. So it is often with participants in focus groups.

Moreover, newer research introduces us to a cousin of Heisenberg’s Uncertainty Principle — the very act of putting thoughts into words, as focus group participants do, can change their thoughts.

And the experts? Well, perhaps we are not quite as expert as we like to believe.

So test ads before you spend millions airing them. But as another ad used to warn — “don’t try this at home” — well-crafted techniques mean the difference between effective strategy and nonsense.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leader of the Senate and the Democratic whip in the House.