Week 9 of my minidegree in conversion rate optimization from CXL was overwhelming.
This week I completed the Statistics For A/B Testing course.
Let’s take a look at the lessons:
– Basics of Causal Inference
– Statistical Significance & Other Estimates
– Statistical Power & Sample Size Calculations
– Multivariate Tests
– Running Concurrent A/B Tests
– Tests With Multiple Outcomes
– Non-binomial Data
– Statistics For Percentage Change
– Asking The Right Questions
– Communicating Statistical Results
Wow! What a snooze-fest.
It took all of about 15-minutes into the first lesson for me to experience flashbacks to first year university stats 101. I hated it then, and I hate it now.
The approach taken in this course was kind of like throwing you into deep, freezing cold water to teach you how to swim.
Let’s start with just the title of the first lesson … the FIRST lesson!!!
“Lesson 1: Basics of Causal Inference”
What the f***?! Already I feel sleepy.
One of the lesson objectives is to “understand the difficulties involved in making sense of data in a noisy world.”
That’s funny considering that’s what I think of this course – noise that didn’t make sense.
There was one interesting thing I plucked from this first lesson. And that was, when you run A/A tests you will actually see a variability in the results. In fact, if you didn’t know that it was an A/A test, you might actually choose one “variation” to be better than the other.
Lesson #2: Statistical Significance & Other Estimates
The first objective of this lesson is to understand the need for the p-value and what it means.
What is the p-value, you may be asking?
Well, according to Georgi…
“The p-value is the probability, under the specified statistical model for the null hypothesis, of observing a statistic as extreme or more extreme than the observed.”
And here is the screenshot of the accompanying graphic…
Just looking at this definition and graphic makes me panic.
To get a better grasp of an apparently important concept, I turned to YouTube. In less than 30-seconds I found a video that does an excellent job of describing what the p-value is.
“If the p-value is low, the null hypothesis must go”
The same thing is true for the concept of standard deviation.
According to Georgi, the standard deviation is…
“A measurement of central tendency which quantifies the amount of dispersion in a given set of data.”
Can you say, WTF?
Now compare that definition to the one I found on MathIsFun.com…
“The Standard Deviation is a measure of how spread out numbers are.”
Okay, one final poke at this lesson…
Another important concept introduced in this video was the z-score.
According to Georgi…
“The z-score is a measurement of the difference between the mean of a distribution and a given observation expressed in number of standard deviations.”
Or put more simply by statisticshowto.com…
A z-score (also called a standard score) gives you an idea of how far from the mean a data point is.”
Let’s look at another lesson…
“Lesson 3: Statistical Power & Sample Size Calculations
It appears to me that this should be one of the most important lessons. It’s objectives include:
- Learn how to determine the required sample size for a simple A/B test.
- Be able to plan a fixed-sample A/B test so it achieves a target power at a specified minimum effect of interest.
Early in the video, Georgi talks about the two types of errors in A/B testing:
- Type I (false positive)
- Type II (false negative)
Here’s a screenshot of the graphic used in the video to demonstrate the two types of errors…
Not bad but still filled with a bunch of jargon. Compare that to this graphic I found on abtasty.com…
I mean, come on! It’s got smiley faces. Already that makes it more engaging than Georgi’s.
Not only is the graphic on ABTasty.com better, but their use of real-life examples is far more effective than what’s found in the CXL course video.
One last example of why this course needs to be scrapped and re-done…
“Lesson 8: Statistics for percentage change”
According to Georgi Georgiev, the course instructor, this is an interesting topic because most people are getting it wrong – he says with a sanctimonious little smirk.
The lesson objectives are listed as follows:
- Explore the difference between absolute and relative or percentage difference
- Define a proper statistical model for percent change
- Calculate statistical estimates for working with percent solutions.
What I don’t see on this list is…
“Statistics for percentage change: What is it and why does it matter?”
My point with all of this is this…
This information could have been presented in a FAR, FAR, BETTER way. I understand that CXL wants to have the top 1% of people in their fields be the experts to appear in their courses.
But they need to weigh the value of presentation too.
A perfect example is the course on Google Analytics presented by Chris Mercer. It’s a technical topic delivered by a dynamic presenter who strives to make the content understandable to most people.
I can’t, for the life of me, understand how they could include this course and only give us a basic level course on Google Tag Manager.
If you’re going to drag me into the deep end of anything, I’d rather learn more about how to use GA and GTM than be given this level of info on stats.
I don’t foresee any marketer getting into CRO and thinking they could pull of the statistical analysis stuff on their own. Give me the basics so I understand the importance and role of the stats analysis, and teach me how to hire an expert if needed.
There’s no way I feel confident enough after this course to go in and apply anything taught here. If I can’t buy software to do it for me, I am hiring someone to do it.
So in conclusion…
Stats play an important role in A/B testing to determine that the decisions made are made based on valid data.
Georgi is obviously a smart dude but is not the right person to be presenting this topic.
This course needs to be scrapped and redone with another presenter, using content that is less filled with jargon, and more relatable.