AB testing FAQ
What does A/B testing help you do?
It allows you to compare different versions of your product data structure or content to determine which one performs better.
E.g. Test a new custom label, or see what impact adding information to the product title has on the performance. If it is positive, you can edit the title accordingly for all applicable products in the future.
Is this feature available for all sites?
Yes.
New sites will only have the "Exports" tab
Old sites will have "Exports" and "Exports"
You can continue using the legacy "Exports" tab without problem but if you wish to do A/B testing you need to select "Exports".
How does it work?
For each export channel (let's say Google Merchant Center) you can activate a certain number of Dataflow variants. Based on the number of Dataflows that you set up, our algorithm will split your products in the same amount of groups. With each Dataflow variant you can create different versions of the same feed.
E.g. If you activate 2 variants, ~50% of your products will run through one, and the other ~50% through the other. Activate 3 variants and your products will be split ~33% / ~33% / ~33%, and so on.
Are all Dataflow variants exported separately?
No. Once you have edited and prepared your different Dataflow variants and want to export your products for testing, all Dataflows will merge, meaning all products will be placed in ONE feed again so that only one final feed is exported for your selected channel. Within this same feed some products will have version A, and others version B of which ever comparison you are doing.
So can I have two totally different Dataflows?
You will have a different Dataflow for each variant. The pre-defined export template will be the same for every variant within an export. But you can choose to add boxes for one Dataflow only, if this is the element you want to A/B test.
Can I remove connection for one Dataflow only?
Similarly, you can choose to delete an attribute (or remove a connection) in a Dataflow variant.
Please note: Any data edits made between Import and Intermediate feeds will still impact ALL the export channels (as before).
Selecting one Dataflow variant as main feed for export
When you determine the number of Dataflow variants, one of them must be activated as your "main" feed. This is just so that you can select the export destination and does not affect your final feed template. The final export feed will be a combination of all the existing attributes or connections. So if Dataflow A is your "main" export and Dataflow B has an attribute added to it, the export will still include that attribute. Products in Dataflow A will simply be blank for this attribute.
Are there any requirements to use the A/B testing feature?
There are no technical requirements. You can even use your own tracking if you want by adding a different parameter for each Dataflow variant.
Don't do these common mistakes when using A/B testing:
Don't test more than one thing at a time. It will become difficult to assess what factor accounted for the decrease or increase in performance.
Don't stop too early. Your goal is to find out whether "Dataflow A" or "Dataflow B" is the better one. Give your feed enough time to get a significant result before drawing a conclusion.
Don't give up after the first test. The first test may not be the best one. But use this experience for the next one.
Be careful when A/B testing for a small feed. The bigger your feed and the longer you test, the more accurate your results will be.
Don't forget about other influencing factors such as seasonality or not receiving enough traffic from the publisher to begin with.
Don't test nonsense: Naturally a Dataflow with product images will perform better than a Dataflow without.