In the Direct Marketing world, we have many self-proclaimed experts who claim to have a "secret" that will deliver wonders. As a marketer, you go through "cycles of disbelief" as I call them. You listen to someone whose proposition appears to make sense. You try it out and see no results. You withdraw for a while. Then you run across another "guru", your need to believe kicks in and the cycle repeats. Obviously something is missing. If you could only articulate it. It's on the tip of your tongue. Here it is:
If we are armed with the knowledge to TEST the market and interpret the results of those tests, then we don't need expensive books and costly methods written by "gurus". The market itself is telling us what works by the response it provides back to us.
So testing is the discipline that helps you uncover what works. Once you master the techniques of running tests, your market tells you what works, and you no longer have to listen on faith to any "experts" and their advice. It is always delectable to read articles and be informed as to what others have to say. But now you are also an expert, with enough knowledge to decide whether you agree or not with the claims of any other expert. At that point you will discover that some of the people enjoying a big spotlight are at times flat-out wrong.
Let us start by identifying the two major schools of thought dealing with market testing:
A/B Split method: A/B testing, or split testing, is a method of advertising testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates. A classic direct mail tactic, this method has been recently adopted within the interactive space to test tactics such as banner ads, emails and landing pages.
Significant improvements can be seen through testing elements like copy text, layouts, images and colors. However, not all elements produce the same improvements, and by looking at the results from different tests, it is possible to identify those elements that consistently tend to produce the greatest improvements.
Employers of this A/B testing method will distribute multiple samples of a test, including the control, to see which single variable is most effective in increasing a response rate or other desired outcome. The test, in order to be effective, must reach an audience of a sufficient size that there is a reasonable chance of detecting a meaningful difference between the control and other tactics.
Multivariate method: Multivariate statistics or multivariate analysis in statistics describes a collection of procedures which involve observation and analysis of more than one statistical variable at a time. The word multivariate is defined as: "having or involving a number of independent mathematical or statistical variables". Sometimes a distinction is made between bivariate and multivariate statistics.
There are many different models of proposed multivariate solutions, each with its own type of analysis:
Clustering systems
Hotelling's T-square
Multivariate analysis of variance (MANOVA) methods
Discriminant function
Regression analysis
Principal components analysis
Redundancy analysis
Correspondence analysis
Linear discriminant analysis
Logistic regression
Artificial neural networks
Multidimensional scaling
Canonical correlation analysis
Recursive partitioning
The Multivariate methods, while interesting, are much like stock market performance predictor software. Their advocates show a religious zeal, and it is admirable. The purpose of most methods is to find patterns and extrapolate what might happen based on these results. Unfortunately, guessing is much like prayer. It requires a dose of faith.
Neuroscience tells us that we have a primal human need to see patterns in the unknown. Patterns make us feel comfortable, because now the unknown makes a little more sense to us. Things that don't conform to our need to "create a definition, to catalogue", will puzzle and unnerve us. We need to label things, we are simply wired that way in our mind. To cater to that all-important human urge to label the world, there are dozens of pseudo-scientists willing to sell you a magic cure that will make you feel safe again in a little world where things make sense again. We invite you to use a true scientific method. Then the data you use for your decisions comes straight from the marketplace, not from extrapolation.
Multivariate testing is a very interesting discipline for you to study on your own, since it may eventually become quite useful as both technology and man's insight of his own understanding take leaps forward in the near future. In my opinion, until then, it adresses our need for comfort quite well; however I have yet to see results backed-up by numbers (with a few exceptions, in which the methods can't really be called multivariate). I am sure there may be applications where certain complex forecasting or extrapolating models are useful indeed. I am not questioning the ability of a multivariate method to show results eventually; I am simply doubting whether you can afford the technology and the sheer astronomical size of the lists required to derive multivariate conclusions. In the end, smart marketing is all about profitability and scalability.
We have also once built a neural web offer system that presented multiple versions and combinations of a page and an offer to entice people in. Based on which combinations turned into actual orders, the system eventually started displaying the "producing" combination of elements more often. A very interesting concept, and at first look the "multivariate" approach was sound indeed. However, such an approach starts from a flawed assumption. It dares to assume there is a certain "holy grail" combination of the right title, the ideal copy, images, discount, order form, etc - that when combined just right will actually hypnotize many people into buying your product. That is a very flawed assumption, and it takes incredibly high numbers to prove for or against. If explaining it takes so long, imagine tracking it. It is your choice if the change in your bottom line is worth playing with these concepts.
There are companies out there selling 5 and 6 figure systems that will track and analyze every step of an order or campaign, abd companies that go bankrupt under the weight of these leechers. There are cheaper ways to do the same thing. Just ask.
Assuming for a moment that the way to arrive at the best solutions is by rolling the dice - that is not really a science, it is only "wishful thinking". Furthermore, the minimum sampling size required would be exponentially larger than in linear testing. In other words, even if the "throw of the dice" random trial model did work, a test to prove that it does so beyond doubt would be outside the scope or budget of any small business. It is only the lazy people looking for a shortcut (or to avoid learning) that readily believe in the logic of a "fluke", just like the old lady ahead of you in line at the grocery who religiously buys her lottery tickets, turning your coffee buying experience into a 1/2 hour nightmare, because she doesn't understand how statistics are stacked-up against her.
What makes a campaign good has less to do with "magic combinations" and more to do with (a) being well informed, (b) knowing how to apply that knowledge economically and (c) deriving conclusions through a correct interpretation of the results. With that in mind, A/B Split is our preferred discipline of testing, and the accepted norm used by science in general to test and to prove the validity of any hypotesis. Double-blind medical studies use a variable and a control group (split test).
A/B SPLIT TESTING METHOD
In this chapter, we will concern ourselves with this particular methodology proven to work by millions of dollars in testing, the A/B Split method of testing. Some of the techniques you will learn here will become an integral part of all your future marketing efforts. Knowing A/B Split will also pave the way for you to be able to play with some multivariate concepts in a smart fashion. Everything you learn here will help you make better and lean marketing decisions based on facts, information and proper interpretation.
Preliminary rule: "don't be an expert"
Don't presume to know your market. Test instead.
In order to know what to test for, you must start with an issue you would like to resolve.
All split-tests start with the question WHICH.
There is a way to get to that question, as you can see below.
The progression of the questions leading to "which" is as follows:
(a) Why
(b) What
(c) Which
Example:
(a) Why is my shopping cart losing customers in the middle of the checkout process?
After you find a likely answer to your question you move on to WHAT.
Answer: It appears we ask them to sign-up and create a membership after they decide to pay for items in the cart, and SIGNUP is a long and cumbersome multi-form process. Our tracking system indicates that all lost orders happen during the completion of that sign-up.
(b) What can we do to make this an easier and more pleasant interaction?
Answer: We will try what we think are some improved versions of this form versus the original form. We will send an equal number of orders through each of the improved versions. One form will reduce the data collected. One will improve esthetics. One will have pop-up help for each field. One will feature LIVE CHAT. We will then analyze the results side by side against the control version (our original). We expect the results to show which element has the most impact in retaining orders. Then we can combine the best elements.
AND OUR ALL IMPORTANT TEST QUESTION:
(c) Which of the elements tested helped the shopping cart to retain the most customers?
VARIABLES
The VARIABLE was the SIGNUP FORM.
The values of our test variable were the different and separately measured improvements we tested it for.
Your variable may be a smaller page element: the headline, the image displayed online, the call to action on the outside of an envelope. An element which you test is called a variable. The changes you make to that variable (and track) become the values of the variable tested.
In the example above, a better test for those of us on a limited budget would be to incorporate ALL the form improvements in one test (make signup more pleasing, reduce the amount of data collected, add pop-up help buttons, add live chat), and in another test eliminate the signup form all together to see if that signup form is indeed your bottleneck. These two versions of our control variable will provide sufficient insight to draw meaningful conclusions.
Control Group: this is the group of people receiving your "unchanged" offer at the same time with a similar group (or groups) who receives your improved offer(s). Using a control group at the same time to a similar group of people eliminates possible distorsion effects. For example it eliminates the possibility that a mailing may have failed because it was sent at a time when people were gloomy about the economy. Or it prevents the assumption that the success of another test had to do more with external factors than with the improvement you applied to it.
Best test channels:
If you have an email list, some of the best testing is done by sending your control message and altered versions to smaller batches of the same email group. The success of an email campaign is greatly dependant on the Landing Page.
Pay Per Click
In lack of an existing email list, the next best venue for a great market test with quick results is placing side by side Pay Per Click ads to test a call to action, a landing page, a message, an incentive offere versus another and so on. The success of a PPC ad is greatly dependant on the Landing Page as well, even more so - since in an email you know who they are, whereas in PPC you are trying to determine who they are.
TEST VALIDITY
Sample size sufficiency:
You're watching CNN and the results of a public poll are announced:
"60% of the American public believe in God".
Then in small letters, it says: we have interviewed 300 people, the poll results are based on their answers. As a marketer, some of the questions that come to mind may be:
- who are these 300 people, what profession, what gender, do they really represent the rest of us accurately?
- what happens if we move the test to the Bible belt, and then back to a large city?
and a more important question:
- is a test of 300 large enough to accurately represent what the nation as a whole feels on the subject?
Fortunately, there is a way to determine sample size sufficiency. Determining sample size is a very important issue because samples that are too large may waste time, resources and money, while samples that are too small will provide inaccurate results.
The minimum number of observations needed for a test to be accurate is called "n".
It is said that when that number is higher, even if the experiment is repeated, the results should be fairly similar (higher accuracy).
When the number is smaller, the results will have lower accuracy.
Another question may be:
How many times shall I toss a coin to reach a point where it fell to the ground of both sides about equally?
Is 100 tosses enough? What about 1000? When is the number sufficient, so I can stop?
Statistical Inference - combining probability and statistics
Result interpretation:
This free publication ends here, and our free consultation can show you examples of how others use our services.
For help in building SMART and USEFUL campaigns and web-based applications on a sensible budget, contact us:
Option Quest, LLC
888-590-0666 x1
option@optionquest.com
Free consultation!
You owe it to your bottom line to graduate from plain web sites and fruitless mailings.
There is much more to learn on this subject. It is becoming apparent as I write, that this chapter alone can be a book in itself. So I will end this exciting introduction to market testing here, and wish your work in Direct Marketing is as successful as it can be. Add our infrastructure to your great campaign ideas, and your rates off response will change dramatically.
