Productsup AB testing explained

What does A/B testing help you do?

It allows you to compare different versions of your product data structure or content to determine which one performs better.

E.g. Test a new custom label, or see what impact adding information to the product title has on the performance. If it is positive, you can edit the title accordingly for all applicable products in the future.

AB testing FAQ

What does A/B testing help you do?

It allows you to compare different versions of your product data structure or content to determine which one performs better.

E.g. Test a new custom label, or see what impact adding information to the product title has on the performance. If it is positive, you can edit the title accordingly for all applicable products in the future.

Is this feature available for all sites?

Yes.

  • New sites will only have the "Exports A/B" tab

  • Old sites will have "Exports A/B" and "Exports"

You can continue using the legacy "Exports" tab without problem but if you wish to do A/B testing you need to select "Exports A/B".

How does it work?

For each export channel (let's say Google Merchant Center) you can activate a certain number of Dataflow variants. Based on the number of Dataflows that you set up, our algorithm will split your products in the same amount of groups. With each Dataflow variant you can create different versions of the same feed.

E.g. If you activate 2 variants, ~50% of your products will run through one, and the other ~50% through the other. Activate 3 variants and your products will be split ~33% / ~33% / ~33%, and so on.

Are all Dataflow variants exported separately?

No. Once you have edited and prepared your different Dataflow variants and want to export your products for testing, all Dataflows will merge, meaning all products will be placed in ONE feed again so that only one final feed is exported for your selected channel. Within this same feed some products will have version A, and others version B of which ever comparison you are doing.

So can I have two totally different Dataflows?

You will have a different Dataflow for each variant. The pre-defined export template will be the same for every variant within an export. But you can choose to add boxes for one Dataflow only, if this is the element you want to A/B test.

Can I remove connection for one Dataflow only?

Similarly, you can choose to delete an attribute (or remove a connection) in a Dataflow variant.

Please note: Any data edits made between Import and Intermediate feeds will still impact ALL the export channels (as before).

Selecting one Dataflow variant as main feed for export

When you determine the number of Dataflow variants, one of them must be activated as your "main" feed. This is just so that you can select the export destination and does not affect your final feed template. The final export feed will be a combination of all the existing attributes or connections. So if Dataflow A is your "main" export and Dataflow B has an attribute added to it, the export will still include that attribute. Products in Dataflow A will simply be blank for this attribute.

Are there any requirements to use the A/B testing feature?

There are no technical requirements. You can even use your own tracking if you want by adding a different parameter for each Dataflow variant.

Don't do these common mistakes when using A/B testing:

  • Don't test more than one thing at a time. It will become difficult to assess what factor accounted for the decrease or increase in performance.

  • Don't stop too early. Your goal is to find out whether "Dataflow A" or "Dataflow B" is the better one. Give your feed enough time to get a significant result before drawing a conclusion.

  • Don't give up after the first test. The first test may not be the best one. But use this experience for the next one.

  • Be careful when A/B testing for a small feed. The bigger your feed and the longer you test, the more accurate your results will be.

  • Don't forget about other influencing factors such as seasonality or not receiving enough traffic from the publisher to begin with.

  • Don't test nonsense: Naturally a Dataflow with product images will perform better than a Dataflow without.

How to set up AB testing

1. Add multiple variants

To use the A/B testing feature you need to work with more than one Dataflow so you can compare the different variants. To create multiple Dataflow variants you need to add the same export channel more than once at least. In Exports A/B click on Add Export again:

6005182f88587066bf2751d81c47032d.png

Try to give it a different name. If you use the same name, the platform will automatically add a number after the name to differentiate the Dataflows.

Example:

22b03281ea5689b2e1ec7625303e1fcb.png

2. Select the main variant / activate deactivate a variant

2.1. Select main variant

You can set one main variant per channel. Change the main variant by clicking on On / Off. Note: If all variants are Off then the feed won't be exported - so make sure you select one, You can edit the export settings (destination, pricing, filename...) for only the one selected main variant (Don't worry, all variants will be merged into one feed at the end before the export).

Each time you change main variant, you will need to add the destination, set the filename etc. again.

b8ca04da143587fdaa2dbb1b46423807.png

2.2. Activate / deactivate a variant

In A/B, click on On / Off to activate / deactivate a variant.

Only activated variants (in the A/B column) will be used for A/B testing.

c16d0ebd04a136fb99fee65749e343ce.gif

2.3. Activate / deactivate an Export

If all variants are set on Off for upload, then the channel is completely deactivated. To export to it again, activate at least one variant.

3. Manage Dataflow variants

Activated variants are available in Dataflow.

efa0e2b7d9855152306c0f2c6c5548d1.png

You can add, delete and edit connections. Remember that every change between Import and Intermediate affects ALL the export channels.

The products will go through one of the Dataflow variants, and then later be merged into one feed again but keeping only their unique Dataflow optimizations.

Example: Dataflow variant 1Attributes are connected, no box

ae2f0b613e7fda6c49551120d2f90702.png

Example: Dataflow variant 2

Difference with Dataflow variant 1:

  • Test example 1: one extra connection: additional_image_link

  • Test example 2: one connection less: gtin

  • Test example 3: one extra box: Static Value in Title ("TEST")

Even though all 3 test changes are shown in the same screenshot below, remember to only test one change at a time so that you can accurately judge which edit caused a performance change.

0c4e0583be646374d8b1b124b48030e7.png

Result: the product feed contains 8 products.

After exporting the feed, it will look like this:

  • The first 4 products go through Dataflow variant 1. additional_image_link remains empty because there is no connection in the Dataflow for this attribute.

  • The last 4 products go through Dataflow variant 2. gtin remains empty because there is no connection in the Dataflow for this attribute.

  • Title contains only "TEST" for the last 4 products because of the box in Dataflow variant 2. This does not affect products going through Dataflow variant 1.

id

title

gtin

image_link

additional_image_link

price

123

Jeans 402

725184712258

URL1

 

34.95

234

Microwave Grand Prix

88698603659

URL2

 

21.52

345

Laptop Powerstark

725184712135

URL3

 

35.47

456

Sneakers ConAir

884420588689

URL4

 

50.00

567

TEST

 

URL5

URL2

150.23

678

TEST

 

URL6

URL3

69.95

789

TEST

 

URL7

URL4

79.95

891

TEST

 

URL8

URL5

18354

All attributes connected at least in one active Dataflow variant will be exported.