Simplifying P-Values and Hypothesis Testing [Uber Statistics Interview Question]

Describe hypothesis testing and p-values in layman's terms.

This is the same question as problem #6 in the Statistics Chapter of Ace the Data Science Interview!

The process of testing whether data supports a particular hypotheses is called hypothesis testing and involves measuring parameters of a population’s probability distribution. This process typically employs at least two groups, one a control that receives no treatment and the other(s), which do receive the treatment(s) of interest.

Examples could be: the impact of a marketing campaign on booked rides, the conversion rates for particular user flows in a product, etc. Testing also involves two hypotheses, the null hypothesis, which assumes no significant difference between the groups, and the alternative hypothesis, which assumes a significant difference in the measured parameter(s) as a consequence of the treatment.

A p-value is the probability of observing the given test results given the assumptions made by the null hypothesis. The lower this probability, the higher the chance that the null hypothesis should be rejected. If the p-value is lower than the pre-determined significance level α, generally set at 0.05, then it indicates that the null hypothesis should be rejected in favor of the alternative hypothesis. Otherwise, the null hypothesis cannot be rejected, and it cannot be concluded that the treatment has any significant effect.

P-values in the context of a normal distribution: Image Source