> For the complete documentation index, see [llms.txt](https://lionstudios.gitbook.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://lionstudios.gitbook.io/readme/analytics/experiment-events.md). # Experiment Events ### Overview The goal of incorporating Experimental events is to help drive analyses of A/B tests. This is required for teams that run A/B tests and would like to use Looker or have the team’s Data member analyze the experiment's performance. ### Experiment Naming Convention To keep experiment information consistent across games for easier analysis, please do your best to use the following naming convention: `{game_code}_{platform}_{test.name}` * **game\_code** - Three-letter game code, e.g. FNI, SBP, FMT, etc. * If game doesn’t have a Three-letter code, just put name: `game.that.doest.have.code` * **platform** * *ios* - ios * *and* - android * *all* - in case you run this test on both platforms at the same time (start and end dates should be equaled) * **test.name** - whatever test name you like, but if you need include multiple words, use dot symbol `.` instead of spaces. ### Available Events * `ab_cohort`: Indicates individual user’s membership in an experiment. The event should be fired every session users log in ### Critical Parameters * Required * `experiment_name`: the name of the experiment, e.g., “midLevelInter” * `experiment_cohort`: the name of the variant or cohort within the experiment, e.g., “inter\_no,” “inter\_yes.” ### Example * [Cake Sort Puzzle 3D](https://play.google.com/store/apps/details?id=com.gamebrain.piesort\&hl=en\&gl=US) * The game ran an experiment called `new_inter_timer` to evaluate the impact of different interstitial timings in the game. * The setup of the experiment is as follows: | Variants | interBetweenTime | interStartTime | | ------------------ | ---------------- | -------------- | | Aggressive (33.3%) | 60 | 60 | | Baseline (33.3%) | 90 | 90 | | Passive (33.3%) | 120 | 120 | * Each variant received a different value for the experiment's `interBetweenTime` and `interStartTime` remote configs. It is crucial to note that each variant used both remote configurations. * ✅ The team should have fired the ab\_cohort event based on the user’s allocation to the variant as either of the following: * `LionAnalytics.AbCohort("new_inter_timer", "Aggressive")` * `LionAnalytics.AbCohort("new_inter_timer", "Passive")` * `LionAnalytics.AbCohort("new_inter_timer", "Baseline")` * ❌ Instead, the team fired the ab\_cohort event incorrectly as either of the following fired the event: * `LionAnalytics.AbCohort("interBetweenTime", "60")` * `LionAnalytics.AbCohort("interBetweenTime", "90")` * `LionAnalytics.AbCohort("interBetweenTime", "120")` * `LionAnalytics.AbCohort("interStartTime", "60")` * `LionAnalytics.AbCohort("interStartTime", "90")` * `LionAnalytics.AbCohort("interStartTime", "120")` ### Implementation * The event should be fired every session users log in. If a user is in multiple tests, the event should be fired multiple times * This event will be attached as a global parameter and will be included in the payload of all proceeding events after it is fired once * Multiple experiments can be attached at once * Only fire this event for players included in the experiment. This produces unnecessary data costs for pre-processing the data that will not be used in evaluating experiment results. ### Event Validation * If you do not have access to Looker, please use the LionAnalytics QA Tool following the instructions here: [**LionAnalytics QA Tool**](/readme/analytics/validation/lion-analytics-qa-tool.md) * If you have access to Looker, ensure that critical parameter values should be present \[[ref](https://lionstudios.cloud.looker.com/explore/lionstudios_analytics/adjust_events_qa_explore?qid=gLk3ekF8KMPkmlRdJIEmIl\&toggle=fil)] * The value of the parameter should follow the following format: * {”experiment\_name”: \[”cohort\_name”]} * Example: ```json {"mid_level_inters":["midlevel_no"]} ``` ### Dashboard & Analysis #### Analyses General A/B test performance * whether the test result validates the product hypothesis * How much uplift/decrease was there in the KPIs that the experiment aimed at moving? * Are there any significant differences in the changes in the target KPIs? #### Metrics DAU Installs Ad Impressions and Revenue Ad and IAP revenue (by Cohort/Tenure day), which can be used for calculating RPI Avg. Sessions Total Usage Retention #### Explore [AB Test Daily KPI](https://lionstudios.cloud.looker.com/explore/lionstudios_analytics/adjust_experiment_daily_kpis): this Explore provides all the essential Daily KPIs and supports grouping by Experiment and Experiment Cohorts #### Dashboards [Experiment Dashboard](https://lionstudios.cloud.looker.com/dashboards/142?Session+Date+Filter=\&KPI+Selector=d7%5E_retention\&Install+Date+Filter=2023%2F04%2F11+to+2023%2F04%2F25\&App+Version+%28Session%29+Filter=\&Network+Type+Filter=\&Campaign+Filter=\&Country+Type+Filter=\&Is+Customer+%28Yes+%2F+No%29=\&Is+New+User+%28Yes+%2F+No%29=\&AB+Test+Filter=%22mid_level_inters%22\&Game+Filter=Wordle%21\&Control+Variant=%22midlevel_no%22\&Player+Tenure+Filter=%3E%3D0\&Variants=): this dashboard provides statistical significance calculation for the selected KPI [AB - Studio Management](https://lionstudios.cloud.looker.com/dashboards/393?Game+Filter=Wordle%21\&AB+Test+Filter=%22mid_level_inters%22\&Control+Variant=%22midlevel_no%22\&Campaign+Filter=\&Country+Type+Filter=\&Install+Date+Filter=2023%2F04%2F11+to+2023%2F04%2F25\&Network+Type+Filter=\&Platform=\&Kpi+Selector=dac%5E_over%5E_dau\&Country+Filter=\&Player+Tenure+Filter=%5B0%2C14%5D\&App+Version+%28Session%29+Filter=\&Session+Date+Filter=\&App+Version+%28Install%29+Filter=\&Variant=\&DAU=%3E0\&Payer+Status+%28Yes+%2F+No%29=\&Is+Customer+%28Yes+%2F+No%29=): this dashboard focuses on showing multiple KPIs and their differences at once --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://lionstudios.gitbook.io/readme/analytics/experiment-events.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.