A/B testing
Last edited: 2023/09/27
A/B testing (also known as split testing or bucket testing) is a method of comparing two versions of the wrapper against each other to determine which one performs better.
For example:
Objective of the test | Version 1 | Version 2 |
Bidder performance | 5 bidders A, B, C, D, E | 5 bidders A, B, C, D, F |
Lazy load | ON | OFF |
Ad unit | Ad unit 1a(with associated formats and sizes) | Ad unit 1b
(with associated formats and sizes) |
When you have more than 2 versions, the test is called a multivariate test. In Prebid Stack you can plug as many versions as you want.
VersionsHow to set up an A/B test?
In the left tab menu, go to A/B testing. Then click on <Add new>.
You will be directed to the A/B test screen.
You need to :
- Select the domain for which you want to run an A/B test
- Provide the test a name
- Select the versions you want to test. You need to select min. 2 versions. If you want to use more than 2, simply click on <Add next>
- For each selected version, define the traffic share.
- Make sure your total traffic share equals 100% (= the total traffic bar is green. Otherwise, it is red and you cannot <Save>
How to start and end an A/B test?
Start a test
For any domain, you can simply start a test by either:
- Editing the test you want to run and clicking <Start test> or
- In your A/B test table, by switching the status to <ON>
End a test
For any test running (Status is <ON> in the A/B test table), you can end the test either:
- By editing the test you want to run and clicking <End test> or
- In your A/B test table, by switching the status to <OFF>
Afterwards, you need to choose which version you want to plug from the test. (you must have at least 1 version plugged in the wrapper).
Can I have several A/B tests for 1 domain?
You cannot have many A/B tests running simultaneously on 1 domain. However, you can have 1 test with as many versions as you require so you can just split your traffic in multiple versions: e.g. v1 and v2 to test a timeout and v3 and v4 to test a bidder configuration.
Monitoring the test results of A/B test
In our Prebid Analytics tool, under your A/B testing tab, you can see the table of all ongoing tests.
Furthermore, you can create a GAM report
Create a report:
- Reporting > Reports > <New report>
- Then in General settings, <Add filter> select <key-values> using the key values of each of the versions used in your test. (you can filter key-values which contain “qw_”)
- In Dimensions, you can select <key-values>, then select proper metrics and click <Run> (or <Save>)