Multivariate Testing - FAQ

What is Multivariate testing? Is it the same as AB testing?
Multivariate Testing allows you to test up to 4 variations of an offer, whereas AB testing compares only two versions. This BLOX Audience+ feature allows you to experiment with variations of price, messaging, presentation of offers and even links to see which performs the best. BLOX Audience+ automatically calculates the conversions and conversion rate for you, then automatically makes the winning variant the default offer group after the testing period concludes.

For additional information check out the How to setup Multivariate Testing.

Why should I run AB tests?
With Multivariate Testing, you can make informed decisions based on empirical evidence, rather than guesswork. By understanding which variations resonate most with your audience, you can refine your BLOX Audience+ personalized paywalls to maximize subscription revenue.

How do I decide what to test?
Start by identifying an element that you’d like to test, and a hypothesis. You can create variants of paywall headline, price/rate, how you describe the price (“$1 for a week for 1 year” or “$52 for a year”), call-to-action buttons, use of Featured, or even the presence of an image. Prioritize elements that are critical to achieving your objectives and have the potential to influence user behavior.  For best results, keep it simple. 

Can I test multiple things at once?
For optimal results, it's best to keep tests straightforward by making only one alteration per variant Offer Group. Introducing multiple changes per variant can complicate the process of pinpointing which change resonated most with your users.

How long should I run a Multivariate test?
Generally, larger sample sizes provide more reliable results. The duration of a test should be sufficient to capture an adequate volume of data while accounting for factors like traffic fluctuations and seasonal variations. As a rule of thumb, run tests for at least 2 weeks to account for variations in user behavior. Additionally, consider factors such as the size of the audience of the rule you’re testing. Running a Multivariate test on Anonymous will yield more results faster, since it is the largest group of users in the funnel. If you run a test on Former subscribers, the audience group is smaller to begin with, and may need a month or more to provide results of any significance.

What kind of data is provided for Multivariate tests?
BLOX Audience+ automatically calculates impressions, conversions, and conversion rate for each Access Offer Group in the test. This data is emailed when the test concludes and is available in the BLOX Access Control admin any time. Overall conversions and wall events are calculated in Google Analytics and Data Insights on the Subscription dashboard.

How often is the Multivariate test data updated in BLOX?
BLOX Audience+ automatically updates the data on the Results tab hourly. When you open the Results tab, you'll see a timestamp indicating how long ago it was updated. On larger sites with a high amount of traffic, the updates may take longer than a few hours.
At the conclusion of the test, the results will be emailed to the address you specified.

Does this work for 3rd-party circulation systems?
Yes, Multivariate tests can be run on sites that use circulation systems for purchase, such as Newzware. However, results for conversions will not be available in BLOX since the purchases are in another system.

How much does it cost?
If you already have BLOX Audience+, this new feature is FREE.  If you don’t have BLOX Audience+, you will need to upgrade from Subscriptions Pro or BLOX Audience. Contact your sales rep or submit a support ticket and our customer service team will be happy to help you.

Do I need BLOX Audience+ to take advantage of this feature?
Yes. If you don’t have BLOX Audience+, you will need to upgrade from Subscriptions Pro or BLOX Audience. Contact your sales rep or submit a support ticket and our customer service team will be happy to help you.