Skip to content

A/B Testing

The MAX A/B Testing feature allows you to use real-time data to test, and to measure the impact of, any change that you make to your waterfall configuration.

You can experiment with different ad monetization strategies and identify opportunities that will maximize LTV. Such strategies might include adding or removing a network to see incrementality, adding a new price point, or experimenting with regional waterfall optimizations.

Creating an A/B Test

Take the following steps to create an A/B test:

  1. Select MAX > Mediation > Manage > Ad Units in your MAX Dashboard.

  2. Click the ad unit for which you would like to set up an A/B test. The Edit Ad Unit window appears.

  3. From the menu in the Default Waterfall tab, select Create AB Test. Edit Ad Unit. Name, Platform, Ad Type, Default Waterfall. Preview Waterfall, Create AB Test.

  4. You then see a small pop-up dialog that describes the process. Check the “Copy existing ad unit configuration…” checkbox and click Create AB Test to copy your existing waterfall to your new test.

  5. When you create a new A/B test, by default MAX applies your Test configuration to 50% of your users. You can modify this value, for example if you want to run tests longer at a lower volume of traffic. To do so, change the value of Test Group Allocation when you create your A/B test. Create AB Test: AB Test allows you to test a different ad unit configuration on a specific % of your users. Once you have analyzed the results of your test, you can either “Promote” these settings to all users or “Deprecate” the test. Test Group Allocation: 50%. ☑ Copy existing ad unit configuration for Test Ad Unit (can be edited later). Create AB Test.

  6. You see a new toggle called Test in the Default Waterfall tab. In this toggle setting, you see your test configuration (which, at first, is the same as the waterfall configuration you just copied). Type in a new AB Test Name for your test, and start changing the waterfall according to what you want to test.

    AppLovin recommends that you name your A/B tests with a taxonomy that reflects the start time for the A/B test (for example, “0812_01UTC_FBBiddingAdded”). This makes it easier for you to keep track of changes and to monitor the results.

    Ad Type. Default Waterfall. AB Test Name. Participating Networks.

  7. After you make the changes that distinguish your test waterfall, click Save. The MAX ad unit page updates with your new A/B test.

  8. When you return to your ad unit page, your default view will still be your original control waterfall. Click the Control toggle to switch to Test and compare your two waterfalls.

    You can visualize each waterfall setting by selecting Preview Waterfall from the menu in the Default Waterfall tab. Then, in the Waterfall Preview dialog, choose the country for which you want to visualize your waterfall. You then see a visual representation of the non-bidder ad networks in your waterfall for that country (and also a list of any bidder networks, which supplement your waterfall).

    Preview Waterfall, End AB Test

Monitoring Results

Monitor the impact of your test in terms of requests, impressions, and revenue, for 24–48 hours after the test begins. Do this in MAX > Mediation > Analyze > Advanced Reporting or MAX > Mediation > Analyze > A/B Tests. You then can either promote your test waterfall to all of your users, or you can deprecate it and revert to the original control configuration. Your A/B test should appear in production and be present in your reports within 20 minutes after you save it.

Advanced Reporting

Any change that you notice reflected in requests, impressions, and revenue indicates a change in the IMP/DAU, REQ/DAU, and ARPDAU. This is probably a result of the changes you made in your test waterfall.

The example screenshot above shows a 28% drop in IMP/DAU, a 16% decrease in REQ/DAU, and a 27% drop in ARPDAU. If these results continue after a few hours (to generate a larger sample), this test would be a good one to deprecate, due to the potential revenue loss.

To deprecate or promote a test:

  1. Go to the Default Waterfall tab of your ad unit page.
  2. Set the Control/Test toggle to Test.
  3. Select End AB Test from the menu. Preview Waterfall, End AB Test.
  4. In the End AB Test dialog that appears, click either Deprecate Test or Promote Test. If the result of this test exceeded the “Control” in performance, select “Promote Test”, if not, you can select “Deprecate Test” to end the test and revert to the “Control” group. Make sure to Save your changes at the end to ensure that the AB Test changes are correctly applied.
  5. Click Save. Your changes will be live in production traffic in a few minutes.

Understanding Results

MAX provides an A/B test result page for each test that generates over 10,000 impressions in the test group. This page shows results the day after these impressions were generated. The A/B Tests page (MAX > Mediation > Analyze > A/B Tests) lists and links to these pages.

A/B Tests. Application, Test Name, Test Start Date, Ad Unit, Ad Type, Waterfall, Result

You can read this A/B test result page to understand the Network Revenue share-of-voice changes, the ARPDAU lift anticipated with the test, and many other data points that will help you make the right decision to promote or deprecate the test.

A/B Test Results