Block 98: A/B Testing Fundamentals
Apply statistics to real-world A/B testing scenarios.
Concepts
- Setting up a controlled experiment
- Determining minimum sample size
- Two-proportion z-test with statsmodels
- Multiple testing correction (Bonferroni)
Code Examples
See exercise below.
Exercise
Simulate an A/B test: variant A converts at 10%, B at 12%. How large a sample do you need to detect this at α=0.05? Run the test with simulated data and report whether the result is significant.
Homework
What is p-hacking? How can you design an experiment to avoid it? Friday