Mythbusting for landing pages
Scott Brinker on Wednesday, May 27, 2009 at 08:53AM 
Looking forward to SMX Advanced in Seattle next week, where I’ll be on a panel for Mythbusting PPC Urban Legends, along with Reid Spice of iCrossing and Frederick Vallaeys of Google, moderated by Jeff Rohrs of ExactTarget.
I love the mythbusting theme, as superstitions abound in search marketing — nowhere more so than with landing pages and post-click marketing.
I’m sure we’ll cover a wide range of myths, especially in what is certain to be a lively Q&A, but there are 3 landing page myths that I’m going to debunk head on in my presentation:
1. A landing page is a single page.
If you’ve been reading this blog, you already know we’re big advocates of thinking of paths more than pages — that very often the post-click experience needs to include two or three or four steps. Still, within the search industry, the myth of the landing page persists; there are gobs of best practices around optimizing a single page, but very little insight about leveraging multi-step paths.
Personally, I blame the terminology: “landing page” sure sounds like a single page. It boxes in your thinking right from the start, just by talking about it. That’s why we’ve been on a crusade to shift the discussion to post-click marketing.
To help new page-to-post-click converts make the shift, I’ll cover three scenarios where a multi-step path is better than a single page:
- When you want to segment respondents.
- When you want to ease someone into a process.
- When multiple steps actually builds engagement.
In the meantime, for a nice introduction to creative path thinking, see Justin’s recent post Landing Pages are Packaging.
2. Flash on landing pages is evil.
Pages built entirely in Flash, that begin with a looooooong “Loading…” introduction are, indeed, a really bad idea for a landing page. But that’s a far extreme of a very broad continuum of possibilities.
A much better way to harness Flash is to embed fast-loading Flash objects as part of an HTML page. By fast-loading, I mean resources that load in less than a second — no clunky animations or painfully slow loading phases — as fast or faster than most flat graphics.
And the upside of incorporating Flash can be significant to your brand and conversion rate:
- Beautiful rendering of fonts and layered text over graphics.
- AJAX-like effects as people hover over buttons and call-to-action links.
- Features such as image thumbnail viewers that gently engage respondents without page reloads. • Tab-like views to remove clutter from the page while still providing deep content.
This can really help your post-click experience stand out from the crowd. And by building these objects so that they take parameters — for all text, images, and call-to-action links — a Flash designer can hand-off a Flash that can be reused again and again by a front-line search marketer with context-specific dynamic content. Immensely cost-effective.
For more details, read how Graphics are the Enemy! (but cool graphical experiences are not).
3. Multivariate testing (MVT) is better than A/B testing.
Ah, this is one of my favorite sacred cows to turn into hamburger. My post last year Playing Russian roulette with your landing pages generated a fair amount of stir (primarily from vendors of MVT software who did not appreciate me pointing out that the emperor had no clothes).
In all fairness, I’ll admit that there are scenarios where MVT is the right tool for the right job. But when it comes to testing landing pages and post-click marketing experiences — especially multi-step paths — A/B testing is probably the better choice 9-times-out-of-10.
Why? Here are some of the key reasons to choose A/B testing:
- Requires less traffic to achieve statistical significance.
- Tests fewer variations, but variations can be apples and oranges (motto: “test big ideas”).
- Elements within a test can have dependences (e.g., matched headlines and images).
- It’s easy for marketers to visualize each test variation for approval.
- There’s zero risk of an accidentally bad combination being tested.
- Multi-page paths of varying length can be tested against each other.
- Often easier and faster to set up an impromptu test.
That last point shouldn’t be underestimated. The power of testing in online marketing is only good if you’re able to test lots of ideas, whenever you want, quickly, without a lot of overhead. If it takes you a long time to configure a test, and a relatively long time to get and analyze the results, you’re missing out on the competitive advantage of speed.
Isn’t mythbusting fun?
Reader Comments (10)
I'm really looking forward to this session next week! I'm hoping that I don't subscribe to many of the myths being busted, but you never know.
Thanks, Kristin! I think this should be a lively session exactly because most of us have some PPC myths in the back of our heads -- I'm looking forward to hearing from the other panelists and the audience Q&A to learn about a few new ones myself.
Please do come over and introduce yourself! Always great to meet fellow post-click marketing aficionados.
Good article, Scott. I totally agree on your first two myths and am glad you're reminding people of this. I am not persuaded so much about myth #3. Some MVT experiments can be quite easy to set up and reveal answers that no A/B test would ever uncover, because they are counter-intuitive to test. I don't want to hawk any vendors in your space, but at least one allows testing to occur with no changes to the Web site (just some magic with a UI configurator), so that kind of test might even be easier to set up than an A/B test, especially in a big Web site environment with a slow-moving IT team. If your point that MVT isn't always better than A/B testing, i agree, but it's rare than any technique is always better than another in every situation. I think MVT is something that oughtto be strongly considered any time A/B testing is used, because it often (usually?) helps produce better results. (And. no I am not an MVT vendor.)
Thanks for the thought-provoking post.
Hi, Mike. Thanks for the comment! I'm a fan of yours, so it's great to have you weigh in on this.
I agree with you completely that neither A/B testing nor MVT is absolutely better than the other. It's very much a function of circumstances and objectives.
The main point of my "mythbusting" is that MVT seems to be more hyped, and so even in situations where an A/B test would be better, I think many marketers are lured (or shamed) into MVT instead. I'd like to counterbalance that because -- just as there are some tests that are better done via MVT -- there are many tests that can only be done via A/B testing.
I listed a few of the circumstances in which A/B is a better choice -- primarily for apples vs. oranges tests and situations where a smaller amount of traffic will be tested in a certain timeframe. I think the apples vs. oranges capability (different layouts, different flows, different interdependent elements on the page) is probably the most important because it encourages a very different kind of testing -- that is particularly useful in the context of landing pages.
Since I'm a vendor of post-click marketing software myself, maybe it would lend more weight to my case to quote a completely different vendor, Google's Website Optimizer team, that in sheer volume has probably seen more A/B tests and MVT experiments than anyone else. Here's a quote re:A/B testing from their site (http://www.google.com/intl/en/websiteoptimizer/articles/testtypes.html):
(Emphasis added.)
Aside from net impact to conversion rates, there's another dimension that I find interesting in considering A/B vs. MVT. Often, MVT is promoted because it enables hundreds or thousands of combinations to be tested without the marketer having to explicitly create them. Although that can certainly be a feature, I'd argue that there's a dark side to that too: because we humans usually think linearly, not combinatorially, it's hard for the marketer to predict in advance if there are any accidentally "unacceptable" combinations in a large MVT test. I'm not talking about combinations that simply underperform, but combinations, that when viewed by a specific respondent who is unwittingly part of an experiment, actually damage the brand with that individual.
A/B testing, in contrast, is an incredibly straightforward what-you-see-is-what-you-get approach.
But, my cheerleading for A/B testing aside, the right thing for any marketer to do is to consider the testing options for a particular situation, appreciate the pros and cons of each method, and choose the one that's best for them.
Hi Scott,
thank you for this interesting post and a constructive discussion with Mike. Being a big enthusiast of multivariate testing I nevertheless agree with you that sometimes it is not worthwhile to overcomplicate things. A/B Split Tests have the advantages you mention, but as Mike has pointed out in some situations MVT can reveal some interesting results. What I have been trying to explore in my blog is the application of MVT for optimising international websites for different cultures - this I believe is a rather novel approach to MVT altogether.
In any case, thanks for your interesting contribution - I will continue to read your blog.
Best wishes - Jeremy
Hi, Jeremy -- was just checking out your blog on multicultural MVT, very cool stuff.
Really appreciate having experienced MVT practitioners such as yourself weigh in to confirm that this case for A/B testing is true and valid. There are scenarios where each offers the right capabilities for the job, and it's important to match the right tool to the right mission.
The scenario in which you seem to advocate using MVT the most -- optimizing an organization's primary website -- is actually one I agree with wholeheartedly. I think MVT is usually better for optimizing within a website because:
* you generally have more traffic to the website as a whole;
* you're usually constrained in the layout according to the UI framework;
* multi-pages flows are harder to predict/control due to universal navigation;
* more items in the UI are "independent" and can be varied accordingly;
* websites are usually designed for a "common denominator" experience;
The sort of big, bold tests that one can do with landing pages -- which you really should do with landing pages -- would often be difficult and/or inappropriate in the context of the main website.
So, just as I made the assertion that the features of A/B testing makes it better for landing pages 9-times-out-of-10 -- and that's an argument not directly for the technical properties of the testing mechanism, but rather for the philosophy of what to test and how to test in landing pages, which simply happens to align with the features of A/B testing -- I would also readily concede that testing inside a primary website is often better served by MVT.
The real debate then, isn't so much A/B testing vs. MVT, as much as it is: are landing pages part of your website or something else (part of your advertising)? I firmly believe that the scales weigh towards the latter. That even if landing pages eventually connect respondents into your primary website, the landing page itself -- or even a broader multi-step landing path -- has a more immediate mission of connecting with the respondent in the context of the ad or email they just clicked on and winning that critical first impression moment relative to all click-throughs to the competition. In that scenario, big, bold testing -- unencumbered by most of the constraints of the primary web site -- makes a tremendous impact.
Thanks for contributing to the discussion!
Hi Scott,
Great topic. I agree with your point that people shouldn't restrict themselves to single page landing pages and that there are opportunities to improve conversion rates with multi-step paths. In fact, most landing pages are part of a multi-step path anyway and even if the path is linear, each page should be considered during optimization.
To your second point, at Widemile we have pioneered testing Flash. Several years ago, in a test for The Weather Channel (case study: http://widemile.com/resources/case-studies/the-weather-channel-notify), we discovered that a Flash header outperformed several static images, and was in fact a major contributor towards increasing conversions. Widespread Flash adoption and broadband have since lowered the reasons for avoiding Flash on landing pages.
While I agree with much of your post, I think you give too much credit to A/B Testing versus MV Testing. Both tools have specific uses, and A/B Testing’s weakness is the lack of data you get with your results. You know which page wins, but have no data about how any of your content affected the conversion event. With MV Testing you can test lots of different messaging and discover which specific piece caused an increase, and by how much. The value of these insights shouldn’t be overlooked because it tell you a lot about your audience, and what to test next.
I think to improve your point, you could say that A/B testing used in conjunction with MVT leads to the greatest conversion increases, and by limiting yourself to only one method, you can expect suboptimal results. Here is a simple, solid methodology that incorporates both kinds of testing:
1) Start by measuring basic analytics (set a baseline for traffic patterns and conversions)
2) Perform an A/B test (test big ideas to find the best performing template)
3) Perform an MV Test on the winner (find the best performing content)
4) Do a follow-on MV Test on discovered segments (isolate and improve
traffic segments that perform significantly different from the mean)
Lastly, all the "Russian Roulette" weaknesses you point out about MVT can be remedied with proper test design. I agree that A/B is easier conceptually, but as marketers become more sophisticated and testing-focused, MVT will be the logical tool of choice for testing offers, pricing, hero shots, buttons, descriptions, and a lot more. Even in terms of simplicity, designing and implementing a completely new page (or funnel) for an A/B test is a lot of work compared to creating new copy and images for an existing page for an MV Test.
Regards,
Frans Keylard
Widemile
Hi, Frans -- thanks for joining in the discussion.
So, we agree completely on multi-page landing experiences and using Flash to boost post-click marketing performance. That's good. And we're both in favor of testing and optimization. That's good too.
As for A/B testing versus MVT, well, you're the director of optimization for a company that sells an MVT platform. You've spent many years doing MVT experiments, and I'd imagine that by now you must be one of the most experienced MVT people in the industry. While I respect you on both counts, it's also fair to point out that your perspective is colored by selling a product/service that is tied to your proposed methodology and that "proper test design" is a lot easier when you're a professional test designer. (Okay, you thoroughly check the revolver to make sure it's not loaded before playing Russian roulette.)
I'd also be careful about proclaiming the explanatory power of MVT too much. There are definitely caveats in there -- about the amount of traffic you need, the trade-offs with interaction effects, and the sticky reality that factor variations in a marketing context are rarely linearly-related "levels". To be sure, you can get more data, and run more analyses on it, but it's most certainly not a magic answer box. I've talked to a number of marketers, who after running extensive and expensive MVT tests found themselves scratching their heads with the exact question: what did we actually learn?
However, I do generally agree with you for using MVT in more traditional website optimization, when you're stuck with a particular page layout, and the best you can do is swap in alternate headlines and hero shots. But the issue I take with the glorification of MVT in the context of landing pages is that "implementing a completely new page (or funnel)" doesn't have to be a lot of work. (This is where I will reciprocally point out that I'm biased, having created a software platform to rapidly produce and test highly creative landing pages and conversion paths.) And it is creative post-click marketing, not cookie-cutter templates, that has the greatest impact.
A/B testing enables highly imaginative landing pages tests; MVT constrains them, both in their structure and in the quantity of traffic required to try a new idea. I think your equating "big ideas" with "templates" in step #2 of your methodology is where you've gotten stuck in limiting the possibilities according to an MVT-centric worldview. Break free from thinking about templates to envisioning experiences. Don't let the math muffle the marketing before you even get started.
My point is that embracing A/B testing in post-click marketing -- the way we're advocating -- is more about a philosophical leap from plain old landing page stencils and the stifling way in which they were previously produced and managed than it is an academic comparison of testing methodologies.
I guess I really don't expect to convince you -- we're probably both happier thinking that our companies each have competitive differentiation over the other -- but it is fun to debate it with you. Next time, let's do this over beer.
Hello Scott,
I am a firm believer in A/B testing and have been doing it for nearly a decade, and A/B testing is a reliable tool in my testing arsenal. You mentioned that our Optimization platform is a MVT platform, and yes, MVT is certainly a distinguishing feature, but it also supports all possible flavors of A/B testing: A/B (proportional & disproportional) in addition to the “big” Fractional and Full Factorial Multivariate testing features. It is hard to dispute that tools with more features will give users a wider range of testing methodologies.
Earlier you said: “In all fairness, I’ll admit that there are scenarios where MVT is the right tool for the right job. But when it comes to testing landing pages and post-click marketing experiences — especially multi-step paths — A/B testing is probably the better choice 9-times-out-of-10.” This is akin to saying my car has 4 gears, but I prefer to never leave second gear. Here are your gears: 1) Basic analytics, 2) A/B testing. 3) Multivariate testing, 4) Advanced segmentation and targeting. The truth is that 9-times-out-of-10, you can guarantee a LOWER conversion rate by ONLY using A/B testing instead of following it up with MVT. I being conservative because in my experience the follow-on MVT has improved upon the A/B winner 10-times-out-of-10 and the big lifts have performed as advertised.
It’s a bit unfair to characterize me by saying “your perspective is colored by selling a product/service that is tied to your proposed methodology and that ‘proper test design’ is a lot easier when you're a professional test designer.“ To the first part, I would counter that when you really understand both MV and A/B testing, it greatly increases your capabilities and options, but if you only have A/B testing at your disposal you are naturally limited in what you can test effectively. To the second part, I will discuss the merits of proper test design and how incompetent design will also hurt A/B testing!
I would like to focus on methodology and stay platform agnostic, but to further set the record straight, the platform I use was designed from scratch around tried and tested methodology based on years of direct full-service client engagements, not the other way around. I have been A/B testing as a marketer for over 10 years, with or without custom built tools, or a team of web developers and creative artists at my disposal. I strongly support great creative and a strong UX, and it has been a critical component of my testing and tool design strategy since day one – which was to design tests using imaginative landing pages, microsites, and funnels. In full-service mode almost all test pages are custom-built using creative and marketing best practices and these ideas are always tested against the control page(s). The goal has always been to allow the marketer to safely test marketing ideas.
I am enjoying your mythbusting concept, so I’d like to add some based on some issues you raised:
Myth 1) "A/B testing enables highly imaginative landing pages tests; MVT constrains them"
You mentioned that “When you're stuck with a particular page layout, the best you can do is swap in alternate headlines and hero shots (with MVT)” – This could not be farther from the truth. A/B Testing still requires good test design methodology (which is what you cite as a weakness of MVT in the hands of marketers). As a matter of fact, A/B testing imposes more restrictions because you must avoid throwing the baby out with the bath water if you omit or add a critical factor or level in the challenger. The content and messaging of the page is highly important to the marketer and to conversions. Aside from how pages look – you can get huge lifts if the customer better understands the offer, and this requires too much patience and time with only A/B testing at your disposal. Marketers want results fast.
Let’s view some other options:
- Use the content streaming technology to display multiple versions of the whole page or major parts of the page (aka Univariate testing - 1 factor, n level test)
- Devote the levels of one factor to testing the swapping out of CSS to alter the entire appearance of the page or site. This allows you to test the whole page layout while simultaneously testing other content. It would take several successive A/B tests to accomplish what can be done in a single MV test run. Now you know the best layout AND content.
- Use the levels of a factor to move one or multiple elements around the page (i.e. test button placement)
- “A vs. Multivariate B testing” - the original page gets some of the traffic while the MVT test gets the rest during the same timeframe. Doesn’t take that much longer than a straight MV Test, and doesn’t take that much longer to run, gives you the best of both worlds.
Myth 2) "You need a lot of traffic to do MV Tests"
Typically you will see lifts that are equal or greater to the initial A/B test lift is to continue with MV Testing. It’s simply too inefficient to use traditional multipage A/B to test such important things as offers and pricing and the number of successive test runs you would need with A/B is completely inefficient.
- It's easy to prove that Fractional Factorial MVT requires the same or less traffic to reach the same significance levels compared to an equivalent A/B Test and scales much better too.
- Most smaller fractional tests need just 300 conversions to stabilize.
Myth 3) "Interaction Effects are a trade-off"
- I am not sure what you mean here, MVT handles interaction effects very well and full factorial testing is based on testing all possible combinations – and certainly better than A/B testing which does not handle interaction effects at all.
- Most interactions are random and do not have a significant impact on conversion behavior – it’s critical to have a tool that can distinguish between high and low Factor significances - why bother with interactions that cannot guarantee a lift?
- In most cases, to truly prove the existence of meaningful interactions you need a lot of traffic. Most academics don’t like the blunt truth; and that is that main effects are almost always just as good or better than interactions in terms of significance. You must work really hard to create a test where a powerful interactions can conclusively beat main effects with real traffic – most examples are not scenarios you would want to test because they lack common sense. Believe me, I have tried for years to make a case for powerful interactions. In the end this is more of a fractional vs. full factorial argument than anything to do with A/B testing. Even fractional factorial with main effects-only can detect the presence of interactions due to abnormally high error levels when the design is not saturated and you have sufficient traffic.
- Lastly, interactions are more important when you are testing funnels, because there is a very measurable relationship between levels, click outs, and conversions that must be understood by marketers. For instance understanding the relationship between a) show or hide pricing vs. b) click-outs or full conversions.
Myth 4) "Factor variations in a marketing context are rarely linearly-related 'levels'"
First let’s fix the terminology and call “non-linearly related” levels “nominal (or discrete) variables,” and I am glad you raised this, because this is actually one of the most insightful areas of MV testing. You cite “(A/B) Tests fewer variations, but variations can be apples and oranges (motto: ‘test big ideas’)” as a strength of A/B testing, but it is really one of the major advantages of MV testing and quite frankly a debilitating limitation of a strict adherence to A/B testing.
Of course, it is always good to use “linearly-related” variables (or interval or ordinal) because a marketer can infer a direction of improvement from the results of a test. However, even for nominal variables, an MV test can provide valuable information about the significance of effects, which levels drive a higher conversion rate, and a more accurate prediction of an optimal conversion rate.
It’s a good practice to test nominal or interval variables because of the wealth of insight it delivers about your audience and their thresholds. Here are some of the key examples that make sense to a marketer from a testing perspective:
- Interval variables: Discount Offer 0% - 25% - 50% - 75% (numerical distance between levels)
- Nominal variables: Lifestyle vs. product hero shot (levels with no natural ordering based on preference)
- Ordinal variables: Subcompact, compact, mid-size, large automobile (levels with an inherent natural ordering)These kinds of variables also lend themselves well to segmentation and targeting because they may tie to personal preferences which can be derived and optimized.
Myth 5) MVT is not a magic box of answers for marketers
I am not going to argue that there is such a thing as a magic box, but anything that delivers more data to the marketers, and allows marketers to get answers to specific marketing questions is arguably a major improvement.
- Marketers will learn little about their testing if they don’t have hypotheses about their audience and test to see whether their assumptions are correct. Garbage in-Garbage out also applies to A/B testing.
- A/B testing and MVT differ because MVT allows the ability to ask questions and measure the results, e.g.:
--- How far can you push fear-based messaging before it yields negative returns?
--- How much discount should you offer before improvements taper off?
--- How many form fields can we show before lead quality or business requirements suffer?
I truly don’t think we are that different in our methodology approach outside of me having both A/B and MVT at my disposal, and I have always enjoyed your perspectives on this matter. Many of the testing weaknesses you identify come from a lack of test methodology, and this is one reason many people quickly abandon A/B and MVT testing on Google Website Optimizer.
I’ll take you up on that beer the next time we are on a panel together, but it may be a while - I have my hands full! :-)
Thank you for the opportunity to exchange ideas, and keep up the good work!
Regards,
Frans Keylard
Crikey, Frans -- that's one long, complex comment.
To a certain degree, this makes one of my points: the complexity of MVT is non-trivial. In many agile landing page test opportunities, these issues are far more than most marketers want -- or need -- to bite off.
Despite your rants against A/B testing, the reality is that they're easy to conceptualize, set up, and quickly run with a small amount of traffic. You can explain an A/B test to your team and your boss in 2 minutes. Diving into MVT experimental design is clearly a much more advanced task -- one that it doesn't surprise me often requires outside consultants and service providers such as yourself.
I still think that we have a fundamentally different philosophy of what's being tested, not just how it's being tested. Your analogy about the car with 4 gears -- putting segmentation as the last possible tweak, long after exhausting significant A/B and MVT experiments, is diametrically on the other end of the spectrum of our approach. Most of the time, we believe that segmentation is the primary axis by which to test in the first place.
And rather than go down an endless cycle of optimizing every last ounce out of a single experiment -- what I would call a classic case of diminishing returns -- we believe that marketers are often better served by making big wins, quickly and agilely, on an opportunity and then moving on to the next.
Your claim that MVT typically gives equal or greater lifts than A/B doesn't jive with what Google (a neutral third-party in our debate here) has seen over thousands and thousands of tests and posted on their web site (and what Tom Leung, their product manager, said in our post-click marketing session at SES). Certainly in our experience, A/B testing -- done with the right philosophy -- often achieves double and triple digit increases in conversion rates. To claim that you would yet double that again either strains credibility or questions why your A/B tests aren't that effective.
Alas, while there's more I disagree with in your comment -- pretty much all of your "myths" -- you're wearing me down. I will, however, concede three points:
1. There are scenarios where MVT is exactly what's required -- your example of testing a variety of prices (which has a lot of other dangers in the age of social media, but we'll save that debate for another time) is a good one. That is not mutually exclusive with my claim that there's plenty of scenarios where A/B testing better fits the bill.
2. The way in which you talk about using A/B testing doesn't sound particularly attractive to me either.
3. Software becomes the embodiment of philosophy and strategy. We've each designed products around our respective strategies, and they're very different. What matters is the results our respective customers achieve by adopting our philosophy/strategy or yours. I accept that I'm not going to change yours.