Experiment Events
Overview
The goal of incorporating Experimental events is to help drive analyses of A/B tests. This is required for teams that run A/B tests and would like to use Looker or have the team’s Data member analyze the experiment’s performance.
Experiment Naming Convention
To keep experiment information consistent across games for easier analysis, please do your best to use the following naming convention:
{game_code}_{platform}_{test.name}
- game_code - Three-letter game code, e.g. FNI, SBP, FMT, etc.
- If game doesn’t have a Three-letter code, just put name:
game.that.doest.have.code
- If game doesn’t have a Three-letter code, just put name:
- platform
- ios - ios
- and - android
- all - in case you run this test on both platforms at the same time (start and end dates should be equaled)
- test.name - whatever test name you like, but if you need include multiple words, use dot symbol
.
instead of spaces.
Available Events
ab_cohort
: Indicates individual user’s membership in an experiment. The event should be fired every session users log in
Critical Parameters
- Required
experiment_name
: the name of the experiment, e.g., “midLevelInter”experiment_cohort
: the name of the variant or cohort within the experiment, e.g., “inter_no,” “inter_yes.”
Example
- Cake Sort Puzzle 3D
-
The game ran an experiment called
new_inter_timer
to evaluate the impact of different interstitial timings in the game. -
The setup of the experiment is as follows:
Variants interBetweenTime interStartTime Aggressive (33.3%)
60 60 Baseline (33.3%)
90 90 Passive (33.3%)
120 120 -
Each variant received a different value for the experiment’s
interBetweenTime
andinterStartTime
remote configs. It is crucial to note that each variant used both remote configurations. -
✅ The team should have fired the ab_cohort event based on the user’s allocation to the variant as either of the following:
LionAnalytics.AbCohort("new_inter_timer", "Aggressive")
LionAnalytics.AbCohort("new_inter_timer", "Passive")
LionAnalytics.AbCohort("new_inter_timer", "Baseline")
-
❌ Instead, the team fired the ab_cohort event incorrectly as either of the following fired the event:
LionAnalytics.AbCohort("interBetweenTime", "60")
LionAnalytics.AbCohort("interBetweenTime", "90")
LionAnalytics.AbCohort("interBetweenTime", "120")
LionAnalytics.AbCohort("interStartTime", "60")
LionAnalytics.AbCohort("interStartTime", "90")
LionAnalytics.AbCohort("interStartTime", "120")
-
Implementation
- The event should be fired every session users log in. If a user is in multiple tests, the event should be fired multiple times
- This event will be attached as a global parameter and will be included in the payload of all proceeding events after it is fired once
- Multiple experiments can be attached at once
- Only fire this event for players included in the experiment. This produces unnecessary data costs for pre-processing the data that will not be used in evaluating experiment results.
Event Validation
-
If you do not have access to Looker, please use the LionAnalytics QA Tool following the instructions here: LionAnalytics QA Tool
-
If you have access to Looker, ensure that critical parameter values should be present [ref]
-
The value of the parameter should follow the following format:
- {”experiment_name”: [”cohort_name”]}
-
Example:
{"mid_level_inters":["midlevel_no"]}
-
Dashboard & Analysis
Analyses
General A/B test performance
- whether the test result validates the product hypothesis
- How much uplift/decrease was there in the KPIs that the experiment aimed at moving?
- Are there any significant differences in the changes in the target KPIs?
Metrics
DAU
Installs
Ad Impressions and Revenue
Ad and IAP revenue (by Cohort/Tenure day), which can be used for calculating RPI
Avg. Sessions
Total Usage
Retention
Explore
AB Test Daily KPI: this Explore provides all the essential Daily KPIs and supports grouping by Experiment and Experiment Cohorts
Dashboards
Experiment Dashboard: this dashboard provides statistical significance calculation for the selected KPI
AB - Studio Management: this dashboard focuses on showing multiple KPIs and their differences at once