As a math/stats/data person who doesn't dabble much in web optimization -- can someone explain to me what's awesome about this?
Not to belittle this nice package -- it looks like a basic stats calculator for calculating sample size confidence levels with friendly visualization, and I'm just trying to understand what's being valued on the market/industry right now. Is it because current A/B testing software doesn't provide these basic calculations? Or is it that it's well presented and visualized to a lay crowd?
Like all of A/B testing, it's applying a _very_ old statistical method (Chi-Square was one of the first modern statistical techniques--by that I mean it's 113 years old) to an area where statistics has not commonly been used. This makes it seem wonderful and novel as countless people suddenly realize that statistics can be applied to fields that were previously untouched by quantitative analyses.
The statistics being used in the A/B testing world is stuff you would've learned in your very first statistics class, it seems. From the success of Optimizely and VWO that the focus is definitely more on the viz and presentation than it is on using any cutting-edge techniques.
Gotcha, and thanks -- it just seemed trivial and I was under the (false) assumption that confidence levels and selecting appropriate sample size should be common knowledge, given how much polls are used in day-to-day life.
Good to know there's plenty of opportunity to bring better stats to high tech. Of course, I understand a lot of the value comes from making those things applicable and meaningful to the users...
Power analysis and CI's should be elementary, but I would assert that they are actually not commonplace. Most people have a very surface-level understanding of the latter, and little understanding of the former. In my opinion, A/B Testing has actually done a great service to power analysis. I have seen many experiments in the academic world (social sciences are somewhat notorious for this) forgoing the power analysis for various reasons (fear: they would not be able to get the sample size needed for 80% power, inability to control sample size: you take whatever you can get with a convenience sample). As a statistician, I breathe a sigh of relief with the amount of emphasis power analysis receives in the A/B world. It's a step in the right direction (if you're an acolyte to the dark world of Neyman-Pearson).
As for bringing better stats to high tech, I've thought of this as a wonderful challenge. I'd especially like to see more focus on not violating modeling assumptions (more non and semi-parametrics), and using some more modern techniques from the ML and Bayes literature.
Hypothesis testing is so last century :). Would love to discuss it further with some similarly-inclined HN folks.
Sorry for all the parentheticals. You'd think I was a lisp programmer with the amount of parenthesis I used.
Not to belittle this nice package -- it looks like a basic stats calculator for calculating sample size confidence levels with friendly visualization, and I'm just trying to understand what's being valued on the market/industry right now. Is it because current A/B testing software doesn't provide these basic calculations? Or is it that it's well presented and visualized to a lay crowd?