How Multivariate Testing Can Boost Your Email Campaign Results by Denise Keller An A/B split test is essentially an either/or procedure. Sending out a thousand emails with subject line A is compared to another thousand sent with subject line B to see which one drew the best metrics. Given that the average email newsletter can have dozens of specific aspects that can have a measurable impact on the campaign’s results, the process of testing them all though A/B split testing can become an infinite process. Multivariate testing allows you to transcend the limits of simple A/B split testing by being able to test multiple variable factors at once, or more than two variations for each individual variable. The art & science of testing multiple variables & values Not only can you test subject line A vs. B vs. C and so on, but you can also test those subject lines in conjunction with various preheader texts, image positionings, and calls to action. For example, you might have: Subject line: Check out this week’s sale; Blowout sale this week; 50% off everything this week only Call to action: Click here; Download the sale brochure now; Get an additional 10% off voucher today only Image position: Upper right; upper left; middle left Therefore you are testing three variables with three values each. Increasing the number of factors requires a mushrooming number of emails Each multivariate test is different and the addition of variables and values will increase the number of composite emails required. While a multivariate testing of three variables with three values each will require 27 different emails, one with eight variables with four values each will need 65,536 separate emails to be composed and tested. Multivariate testing of numerically high variables and values must be reserved for the largest email marketers as fully accurate results can only be had in those cases by sending out emails in the millions. The N*R*C=S Formula If you are testing six variables with three values each, you will require a total of 729 separate emails. The formula to determine the number of subscribers required to achieve reliable statistical accuracy is: Number of Emails Composed [N] x Historical Number of Readers For 1 Conversion [R] x Number of Conversions Required For Accuracy [C] = Number of Subscribers Required Or N*R*C=S If we assume that you will establish that the minimum number of conversions to these pages to generate the most reliable results is 100, and your conversion rate averages at 5% (one out of 20), you will plug in the numbers as follows: 729 [N] x 20 [R] x 100 [C] = 1,458,000 This test would only be suitable to a major national email marketer who has 1.5 million email addresses on hand to carry out the multivariate testing. You don’t need millions of subscribers for reliable multivariate testing These massive types of multivariate testing are not necessarily obligatory for most email marketers. Smaller subscription lists can be catered to simply by running a multiple number of multivariate testing. If the six variables with three values each test is split into two tests of three variables with three values each, the numbers per test become: 27 [N] x 20 [R] x 100 [C] = 54,000 Multiplied by two tests, the total number of subscribers required drops to barely over 100,000 which places it within the reach of a far greater number of email marketing list sizes. Even smaller list sizes may be accommodated by simply lowering the [C] factor to a smaller number, such as the statistically legitimate minimum of 25: 27 [N] x 20 [R] x 25 [C] = 13,500 This lower [C] allows for just 27,000 subscribers for a dual 3×3 test. The results of these smaller tests will not provide the breadth of statistical insight and precise accuracy in every possible permutation that the 1.5 million email test would, but can certainly be of significant validity in the process of assisting any email marketer to make critical determinations about their campaign content. Takeaway: Multivariate testing can help every email marketer in refining their subscriber approach to maximize results, and thus warrants application to all campaigns, large or small. Once you master it, this type of testing can give you micro-accurate info on what works for your email campaigns. Source: http://www.theemailguide.com/email-...aign-results-by-denise-keller-benchmarkemail/ .. Roundy's Thoughts: I would only have time to do this if I were the actual advertiser with the product, so finding the absolutely best subject and body would be worth all this time... I'm amazed that some people go to such lengths to split test, but I don't see a typical email publisher doing this.
I agree Round. MV in split testing is really cool if you are an advertiser (or the marketing nerd team for said advertiser). For a pub - A/B split testing works best. And my advice is: test only ONE element at a time - diff subject or from name or body variation. To go to the lengths she is describing is near ludicrous and probably not worth the efforts. A good A/B has many times added immediate extra digits to the bottom line when we discover the winning variant. Jedi
