Why do we choose the Bayesian approach over the Frequentist approach? What are the differences between these approaches?
The first thing to note is that these two views are complementary and helpful for building intuition. But they are different perspectives.
The core idea underlying the Frequentist view of probability is the frequency of an outcome in a repeated experiment. As an example, here’s a Frequentist argument for the probability of a fair coin coming up heads
The probability that a fair coin comes up heads is 0.5 because if we flip a fair coin a large number of times, the fraction of the number of heads over the number of flips gets very close to 0.5.
There are rigorous techniques for saying what I did above but that’s the idea: repeat an experiment many times and examine a numerical result.
On the other hand, the core idea underlying the Bayesian view is our degree of belief about an event. A helpful consequence of this view is that we can assign probabilities to our hunches/hypotheses, even in the case where we’re unable to repeat the experiment. For example, “The patient will respond poorly to this medication” or “This candidate will win the election.” The ability to assign probabilities to these kind of circumstances which will happen only once is an important reason for choosing a Bayesian view of probability.