Describe what Type I and Type II errors are, and the tradeoffs between them.

This is the same question as problem #7 in the Statistics Chapter of Ace the Data Science Interview!

Both errors are relevant in the context of hypothesis testing. A type I error is when one rejects the null hypothesis when it is correct, and is known as a false positive. A type II error is when the null hypothesis is not rejected when the alternative hypothesis is correct; this is known as a false negative.

In layman's terms, a type I error is when we detect a difference, when in reality there is no significant difference in an experiment. Similarly, a type II error occurs when we fail to detect a difference, when in reality there is a significant difference in an experiment.

The level of a type I error is given by the level of significance α, whereas the type II error level is given by β. Usually, 1-α is referred to as the confidence level, whereas 1-β is referred to as the statistical power of the test being conducted. Note that, in any well-conducted statistical procedure, we both α and β to be small.

However, based on the definition of the two errors, it is impossible to make both errors small simultaneously for a fixed experiment setup; the larger α is, the smaller β is. Based on the experiment and the relative importance of false positives and false negatives, a Data Scientist must decide which thresholds to adopt for any given experiment.

Note that experiments are set up so that both 1-α and 1-β are relatively high (.95, and .8 respectively). That said, if the experiment setup is flexible, it always helps to have a larger sample size and multiple experiments.