Stochastic Independence versus Stochastic Dependence
In order to fully understand the Bayesian rule it is important to be familiar with some concepts of standard probability theory. Assume we have two events, let’s call them A and B. The probability that event A occurs is P(A) and the probability that event B occurs is P(B). If event A and event B are independent from each other, the probability that both events are occurring at the same time, also known as the joint probability .
In order for two events to be independent the occurrence of the event A cannot have any influence on the occurrence of the event B and vice versa. As an example let’s assume event A is defined as earning more than 10.000 per month and event B is defined as getting six when throwing a die. Both events are not, or at least not as far as I know, interfering. The probability of having six when throwing a die is not changing if it is a person who earns more than 10.000 per month, nor is the probability for a person earn more than 10.000 per month changing if this person has a six when throwing a die.
In Bayesian statistics such cases are less relevant and it becomes much more interesting when events are not independent. To create an example where to event are linearly dependent we simply have to redefine the event B from the example above. Event B is now not having a six, when throwing a die anymore, but event B is defined as being the CEO of a company with more than 10.000 employees. It should be intuitive that the probability of event A, earning more than 10.000 per month, is changing with the occurrence of event B. The probability of earning more than 10.000 should be higher when a person is a CEO of a company with more than 10.000 employees. On the other hand, the probability of event B, a person being a CEO of a company with more than 10.000 employees, should be higher when this person earns more than 10.000 per month.
The following two equations represent the cases where events are stochastically independent and stochastically dependent
Where is the probability for event A under the condition the event B has occurred, we therefore refer to it as the conditional probability.
Maybe I misread/misunderstood but you had previously defined the independence of errors from the coefficients in OLS as/by P(e_i | X)=0, now you define independence of A,B as P(A|B)=P(A)P(B). Are you assuming P(e_i)=0? Moreover, I know we assume E[e_i]=0, but I am not sure what you mean by P(e_i)=0. Is P the distribution/density of e_i? Cool blog, though, keep it up.
Hi there. Thanks for you comment. And sorry for getting back to you so late. We assume that P(e_i | X)=0. This means that P(e_i)P(X)=0. What this means in the end is that we assume that residuals cannot be explained by the explanatory variables. Hope this makes things clearer. Cheers, ad