AB Experiments (Satori)

Introduction

AB Experiments are set up on the Satori Dashboard. Ensure your Satori instance is set up and you can access your game dashboard.

This document provides instructions for setting up the Example AB Test use case described below.

Satori’s terminology differs from other tools, such as Firebase. Below you can find how the corresponding pairs of Firebase terms in Satori:

Firebase vs. Satori:

——————————

Remote Config → Feature Flag

Experiment Cohort → Experiment Variant

Note: Looker also uses Experiment Cohort to refer to Experiment Variant

Requirements

1. Make sure that you have set up Remote Configuration as experiments depend on remote configurations following the guide here: Remote Configuration

2. Configure your Lion - Remote Config settings to use Satori as the remote config adapter. More info here.

3. Design your experiment - what are your hypotheses, goals, and game levers? Please see our Experiment Best Practices.

4. LionSDK will automatically fire the ab_cohort event to LionAnalytics, allowing us to analyze the experiment results.

   1. Every experiment needs a unique name, and we need a special (unique for the experiment) feature flag that tracks which variant the player has been assigned for that experiment. LionSDK will automatically check for a feature flag with the following naming convention and use that to fire the required ab_cohort event for you.

      • “exp_[EXPERIMENT NAME]" (replace [EXPERIMENT NAME] with your unique name)

   2. Ensure you have all necessary events firing in your game that you will use for analysis. See our Analytics: 1. Planning guide for more information.

5. Follow Satori’s conceptual documentation here to understand the concept of experiments and experiment phases. If you need to change the current experiment configurations, stop the current phase, fix and start a new one.

   1. See our example below to understand better how to set up your Experiment correctly.

6. Analyze Results - Once your experiment is complete, please see the Dashboards & Analysis section here: Dashboard & Analysis

Example Experiment (AB Test) Use Case

1. The Product team wants to run an experiment using Satori Remote Config to optimize the level at which interstitial ads start showing in the game. The Product team wants to run an experiment using Satori Remote Config on the game King Wing (3 letter code: KWI) to evaluate the impact of different interstitial ad timings. The experiment is named kwi_and_new.inter.timer, with the following variants and configurations:

   Variants

   inter_between_time

   inter_start_time

   control

   90

   90

   aggressive

   60

   60

   passive

   120

   120

2. Each variant/cohort of the experiment will receive a different value for the inter_between_time and the inter_start_time parameter, allowing the team to treat users differently by assigning them different values.

3. There will be 3 experiment variants: “control” , “aggressive”, and passive.

4. Users will be equally distributed among the experiment variants.

5. The experiment will be concluded, and existing participants will be grandfathered. This means that they will continue to receive their existing treatments, while no new users will be added to the experiment.

Example Dashboard & Client Setup

1. On the Satori dashboard go to the Feature Flags tab and click the "Create Feature Flag" button and create 3 new feature flags called inter_between_time ,inter_start_time and exp_kwi_and_NewInterTimer. These 3 parameters will be used in our experiment setup.

   1. The exp_kwi_and_NewInterTimer parameter serves as an internal identifier to track which cohort variant a player belongs to. This parameter will be set to one of three values: control, aggressive, or passive.

      

      

      
2. Create an Experiment called kwi_and_new.inter.timer following the instructions provided here.

1. Setup the Metrics and Target Audience

2. Setup your Variants.

   • Looking back at our initial guideline here, we want to have 3 groups. The control, aggressive and passive.

     
3. Select the Feature Flags we want to override. We will use the inter_between_time, inter_start_time , and exp_kwi_and_NewInterTimer we had created earlier. Make sure to name the exp_kwi_and_NewInterTimer parameter we had created to match the group it belongs to.

   
4. On the Schedule a Phase step, select your Start and End time for this experiment and click "Create"