Confidence Intervals

Assignment

The purpose of this programming assignment is for you to get a hands-on experience with confidence intervals. This will be useful for you to understand how reliable is the predicting of the performance of a classifier.

Program the following experiment in a programming language of your choice.

1. Simulate a biased coin toss with probability of success p (use p as a parameter). In mathematical terms this is a Bernoulli trial with probability p.
2. Now simultate tossing N such biased coins and count the proportion of successes that you had.
3. Repeat the above experiment 10000 times and plot the distribution of successes by using a histogram with 100 intervals. Do the experiments for p=0.75, 0.80, 0.90, and for each case try N=100 and N=1000.
4. For each case, use your statistical knowledge to compute 80%, 90%, and 99% confidence intervals for p (as if it was unknown to you). Do the experimental results match the theory?