The Data Science of Experimental Design
3h 36mIntermediate2020-05-04
Authors

Monika Wahi
Data Science and Biotech Expert
Course details
Interested in learning how to create an online experiment that helps you better understand your business? This course can help you get up to speed. Instructor Monika Wahi shows learners without a background in experimental design how to build an A/B test for a web page, run the test, analyze the data, and make decisions based on the results of the test. Monika begins by explaining exactly what A/B testing is and under what circumstances it is useful. She then covers potential strategies for increasing conversion rates, as well as how to choose both A and B conditions for testing. Next, she explains how to define conversion rates and develop and document case definitions, conduct a baseline analysis in Excel and, based on the results of the analysis, design an A/B test. Plus, she demonstrates how to conduct a chi-square test in Excel and get a sample size estimate using G*Power.
Skills covered
Data AnalysisData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOne-Off
Concepts
Introduction
- Conducting an experiment online
- What you should know
Introduction to Experimental Testing
- Experiment is a type of study
- Features of an experiment
- Circumstances for experimental testing
- When not to do an experiment
- Systems ready for experimental testing
- Comparability of experimental conditions
Defining Conversions
- Trying to increase conversions
- Different types of conversions
- Case definition of conversion
- Measuring a conversion
- Considering time period for conversions
- Rates versus frequencies of conversions
Defining Conversion Rates
- Identify and prioritize conversions
- Operationalize counting conversions
- Document conversion case definitions
- Brainstorm denominators
- False positives and negatives
- Document denominators
- Determine time frames
Baseline Descriptive Analyses
- Baseline time-series analyses
- Data handling
- Baseline results as a guide
- Thinking about increasing conversions
- Strategies to increase conversions
- Planning a campaign
Designing the Experiment
- Designing the test
- Testing the implementation
- Choosing a test statistic
- Choosing the chi-squared test
- Chi-squared test in Excel
Sample Size and Statistics
- Installing G_Power
- Using G_Power
- Sample size simulation
- Planning the timeline
- Stratified analysis
- Conditional tests
Analyzing and Interpreting the Data
- Overall analysis approach
- Time-series analysis
- Chi-squared analysis
- Interpretation
Conclusion and Next Steps
- What actions can we take
- Report writing