I recently took the course A/B Testing on Udacity. It provides a final project to help me understand the concepts and know the statistical concepts. A/B testing, at its most basic, is a way to compare two versions of something to figure out which performs better. -- Amy Gallo, A Refresher on A/B Testing, HBR It is a very common method for web analytics and product improvement. From this Udacity project, I had an overview of this method... Project Overview: Udacity tested a change on its homepage. If the student clicked "start free trial", they were asked how much time they had available to devote to the course. If the student indicated 5 or more hours per week, they would be taken through the checkout process as usual. If they indicated fewer than 5 hours per week, a message would appear indicating that Udacity courses usually require a greater time commitment for successful completion, and suggesting that the student might like to access the course materials for free. At this point, the student would have the option to continue enrolling in the free trial, or access the course materials for free instead. This screenshot shows what the experiment looks like. Step 1: Choosing Invariant Metrics (for Sanity Checks later, these metrics shouldn't be changed during the test, or the experiment setup is incorrect)
Step 3: Calculating Standard Deviation *The formula: sqrt(p*(1-p)/N)
Step 4: Select a evaluation metrics and calculate the page views the experiment needs
Control Group and Experiment Group Page View Needed: 1. According to Gross Conversion: 2*[25835/0.08] = 645875 2. According to Retention:2*[39115/(0.08 * 0.20625)] = 47412121 3. According to Net Conversion: 2*(27413/0.08) = 685325 Choose the bigger number, however, 47 million is too much, then choose the second large number - according to Net Conversion, 685325 Step 5: Choosing Duration and Exposure (how long the experiment lasts)
Step 6: Sanity Checks (use invariant metrics -- step 1) With 95% Confidence Interval
Step 7: Effect Size Tests
Step 8: Sign Tests
Number of Failure - when Experiment Group less than Control Group: 19 Number of Days: 23 Probability: 0.5 Two tailed p-value: 0.0026
Number of Failure - when Experiment Group less than Control Group: 13 Number of Days: 23 Probability: 0.5 Two tailed p-value: 0.6776 Summary: Because in the Effect Size Tests, the net conversion doesn't have statistically significance nor the Practically significance, which means the experiment the payment number during the 14 days doesn't increase. Therefore, the experiment shouldn't launch to a bigger group.
If you would like to take the course and have a try, here is the link: https://www.udacity.com/course/ab-testing--ud257 And it's FREE!
0 Comments
Leave a Reply. |
AuthorLan Jiang is a data analyst with a media industry origin. She is enthusiastically learning about the latest machine learning and data tools to know the audience and customers thoroughly. Archives
January 2019
Categories
All
|