Bayes theorem

In the previous post we learned conditional probability: how the chance of an event changes once we know that another event has happened. Bayes’ theorem takes the very next step. It lets us reverse a conditional probability, so that from an effect we can work backwards to its most likely cause. In this post we will build the idea from scratch, understand the intuition, and then solve nine practice questions from easy to advanced, each with a complete step-by-step solution.

In this article

  1. What is Bayes’ theorem?
  2. Conditions necessary to use it
  3. The formula
  4. Law of total probability
  5. Intuition for beginners
  6. A 5-step recipe
  7. Question 1: the lost parcel (3 methods)
  8. Question 2: the examination
  9. More practice questions (3 to 9)
  10. Common mistakes
  11. Quick revision

What is Bayes’ Theorem?

Definition

Bayes’ theorem gives the probability that a particular cause was responsible for an observed effect. If an event A is known to have happened, and it could have happened through several different routes E1, E2, …, En, Bayes’ theorem tells us the probability that it happened through a specific route Ei.

Conditional probability usually answers a forward question: “If this cause is present, how likely is the effect?” Bayes’ theorem answers the backward question: “The effect has been observed. How likely is it that this particular cause produced it?”

Here are a few everyday forms of this backward question:

  • A product is found defective. What is the chance it came from Machine 2?
  • A medical test is positive. What is the chance the person really has the disease?
  • A student answered correctly. What is the chance the student copied the answer?
  • A parcel is lost. What is the chance the second post office lost it?

Conditions Necessary for Bayes’ Theorem

Bayes’ theorem in its general form works only when the “causes” are set up properly. Check these conditions before you use it:

  1. Mutually exclusive causes. The events E1, E2, …, En cannot happen together. Exactly one of them is responsible.
  2. Exhaustive causes. Together they cover every possibility: E1 ∪ E2 ∪ … ∪ En = whole sample space. This means P(E1) + P(E2) + … + P(En) = 1.
  3. Non-zero probabilities. P(Ei) > 0 for every cause, and the observed event has P(A) > 0 (we cannot divide by zero).
  4. Known inputs. We know the prior probabilities P(Ei) and the likelihoods P(A | Ei).
  5. The event A has already happened. The question is always of the type “given that A occurred, find the probability of Ei.”
How to spot a Bayes question

Look for phrases such as “given that”, “it is found that”, “if it is observed that”, “what is the probability that it came from”. The result is known; the cause is the unknown.

The Formula

Simple (two-event) form

Start from the multiplication rule of conditional probability. The same event “A and B both happen” can be written in two ways:

P(A ∩ B) = P(A) × P(B | A)    and    P(A ∩ B) = P(B) × P(A | B)

Both are equal, so P(B) × P(A | B) = P(A) × P(B | A). Dividing by P(B) gives Bayes’ theorem:

P(A | B) = P(B | A) × P(A)P(B)

General form (several causes)

When the causes E1, E2, …, En satisfy the conditions above, and A is the observed event:

P(Ei | A) = P(Ei) × P(A | Ei)P(E1)P(A | E1) + P(E2)P(A | E2) + … + P(En)P(A | En)
TermNameMeaning in plain words
P(Ei)PriorHow likely the cause was before we saw the evidence.
P(A | Ei)LikelihoodIf this cause is true, how likely is the evidence A?
P(A)Evidence (total probability)Overall chance of seeing A, through any cause.
P(Ei | A)PosteriorHow likely the cause is after we saw the evidence. This is what we want.

Law of Total Probability (the denominator)

The denominator of Bayes’ theorem is not magic. It is the law of total probability. If E1, …, En are mutually exclusive and exhaustive, then the event A can happen only along one of these routes, so we add up the probability of every route:

P(A) = P(E1) P(A | E1) + P(E2) P(A | E2) + … + P(En) P(A | En)

Each term is “reach the cause, and then produce A”. Adding all such routes gives the total chance of A. So Bayes’ theorem is simply:

P(Ei | A) = probability of the route through Eiprobability of all routes that give A

Intuition for Beginners

Imagine 100 emails. Suppose 20 are spam and 80 are genuine. The word “winner” appears in 15 of the 20 spam emails and in 8 of the 80 genuine emails.

You open an email and see the word “winner”. How likely is it spam?

  • Emails containing “winner”: 15 (spam) + 8 (genuine) = 23.
  • Of these 23, spam emails are 15.
  • So P(spam | “winner”) = 15/23, which is about 65.2%.

That is Bayes’ theorem in action, without any formula. We shrank the world to only those emails that contain the evidence, and then asked what share of that shrunken world belongs to spam. The formula does exactly the same with probabilities instead of counts.

Two ideas to remember

1. Reverse is not equal. P(wet road | rain) is high, but P(rain | wet road) can be low, because a sprinkler or a water tanker can also wet the road. P(A | B) and P(B | A) are different numbers.

2. Start belief, update belief. The prior is your starting belief. The evidence pushes it up or down. The posterior is the updated belief.

A 5-Step Recipe to Solve Any Bayes Question

  1. Name the causes. Define E1, E2, … and check they are mutually exclusive and add up to 1.
  2. Name the evidence. Define A, the event that is “given” in the question.
  3. List the inputs. Write the priors P(Ei) and the likelihoods P(A | Ei).
  4. Find P(A). Use the law of total probability (the denominator).
  5. Divide. P(Ei | A) = P(Ei) P(A | Ei) / P(A). Simplify and sanity-check: all posteriors must add up to 1.

Question 1: The Lost Parcel (Three Methods)

Question 1Easy to Medium

A parcel is sent from sender A to receiver B, passing one after the other through two sorting centres. Each centre independently loses an incoming parcel with probability 1/4, regardless of what happens to other parcels. Given that the parcel is lost, find the probability that it was lost at the second centre.

Solution

First of all, notice that “given that the parcel is lost” is a condition. Our whole world shrinks to the situation in which the parcel is lost.

Method 1: Reduced sample space
  1. Original probability = 1. After the condition, the reduced world has probability P(parcel is lost).
  2. The parcel is lost if it is lost either by the first centre or by the second centre (but only if the first did not lose it):
    P(lost) = P(lost by first) + P(not lost by first and lost by second)
    = 1/4 + (3/4)(1/4) = 1/4 + 3/16 = 4/16 + 3/16 = 7/16
  3. Favourable probability (lost by the second centre) = (3/4)(1/4) = 3/16.
  4. Required probability = favourable ÷ reduced world:
    3/167/16 = 3/7
Method 2: Bayes’ theorem
  1. Define the events.
    E1 = parcel is lost by the first centre
    E2 = parcel is not lost by the first centre, but is lost by the second
    E3 = parcel is lost by neither centre
    A = the parcel is lost
    These three events are mutually exclusive and exhaustive.
  2. Priors.
    P(E1) = 1/4
    P(E2) = (3/4)(1/4) = 3/16
    P(E3) = (3/4)(3/4) = 9/16
    Check: 4/16 + 3/16 + 9/16 = 16/16 = 1
  3. Likelihoods. If E1 or E2 happened, the parcel is certainly lost; if E3 happened, it is certainly not lost.
    P(A | E1) = 1,   P(A | E2) = 1,   P(A | E3) = 0
  4. Write the formula. We want P(E2 | A):
    P(E2 | A) = P(E2) P(A | E2)P(E1) P(A | E1) + P(E2) P(A | E2) + P(E3) P(A | E3)
  5. Calculate the denominator (this is P(A), by the law of total probability):
    (1/4)(1) + (3/16)(1) + (9/16)(0) = 4/16 + 3/16 + 0 = 7/16
  6. Calculate the numerator:
    P(E2) P(A | E2) = (3/16)(1) = 3/16
  7. Divide and simplify:
    P(E2 | A) = 3/167/16 = 3/7
Method 3: Visual probability tree

Start with Parcel at the top. Three branches leave it, one for each of the mutually exclusive and exhaustive events E1, E2 and E3. From each event, one arrow shows whether the parcel is lost: 1 means lost and 0 means not lost.

Parcel
1/4 E1Lost by first centre 1 Lost
(3/4)(1/4) = 3/16 E2Not lost by first, lost by second 1 Lost
(3/4)(3/4) = 9/16 E3Lost by neither 0 Not lost

The green box (E2) is the favourable path.

  1. The branches cover everything.
    P(E1) + P(E2) + P(E3) = 1/4 + 3/16 + 9/16 = 4/16 + 3/16 + 9/16 = 16/16 = 1
  2. What the final labels mean. They are the indicator of the event “parcel is lost”: 1 if the parcel is lost, 0 if it is not lost.
  3. Probability that the parcel is lost. Multiply each branch probability by its indicator and add:
    P(A) = (1/4)(1) + (3/16)(1) + (9/16)(0) = 4/16 + 3/16 + 0 = 7/16
  4. Favourable path. The parcel is lost by the second centre only along the middle branch E2, with probability 3/16.
  5. Divide. Favourable path ÷ total probability of the condition:
    3/167/16 = 3/7

All three methods give the same answer: P(lost by second centre | parcel is lost) = 3/7

Sanity check

The chance that the first centre lost it is (1/4) ÷ (7/16) = 4/7. Then 4/7 + 3/7 = 1, as it must be. The first centre gets the larger share simply because it sees every parcel, while the second centre sees only those that survived the first.

Question 2: The Examination

Question 2Medium

In an examination, an examinee either guesses, copies, or knows the answer to a multiple-choice question with four choices, of which exactly one is correct. The probability that the examinee guesses is 1/3 and the probability that the examinee copies is 1/6. The probability that the answer is correct, given that the examinee copied it, is 1/8. Find the probability that the examinee copied the answer, given that the answer was correct.

Solution
  1. Define the events.
    G = examinee guesses,   C = examinee copies,   K = examinee knows the answer
    A = the answer is correct
    G, C and K are mutually exclusive and exhaustive (the examinee does exactly one of the three).
  2. Find the missing prior. Since the three add up to 1:
    P(K) = 1 − 1/3 − 1/6 = 6/6 − 2/6 − 1/6 = 1/2
  3. Find the likelihoods.
    Guessing among 4 choices: P(A | G) = 1/4
    Copying (given in the question): P(A | C) = 1/8
    Knowing the answer: P(A | K) = 1
  4. Write the formula. We want P(C | A):
    P(C | A) = P(C) P(A | C)P(G) P(A | G) + P(C) P(A | C) + P(K) P(A | K)
  5. Denominator (law of total probability):
    (1/3)(1/4) + (1/6)(1/8) + (1/2)(1)
    = 1/12 + 1/48 + 1/2
    = 4/48 + 1/48 + 24/48 = 29/48
  6. Numerator:
    P(C) P(A | C) = (1/6)(1/8) = 1/48
  7. Divide:
    P(C | A) = 1/4829/48 = 1/29

Required probability = 1/29 (about 0.0345)

More Practice Questions (Easy to Advanced)

Try each question on your own first. Then compare your steps with the solution.

Question 3Beginner

Bag I contains 3 red and 2 blue balls. Bag II contains 1 red and 4 blue balls. A fair coin is tossed: heads means Bag I is chosen, tails means Bag II. One ball is drawn from the chosen bag and it is red. Find the probability that it came from Bag I.

Solution
  1. Events. B1 = Bag I chosen, B2 = Bag II chosen, R = red ball drawn.
  2. Priors. P(B1) = P(B2) = 1/2.
  3. Likelihoods. P(R | B1) = 3/5,   P(R | B2) = 1/5.
  4. Denominator. P(R) = (1/2)(3/5) + (1/2)(1/5) = 3/10 + 1/10 = 4/10 = 2/5.
  5. Bayes. P(B1 | R) = (1/2)(3/5) ÷ (2/5) = (3/10) ÷ (4/10) = 3/4.

Answer: 3/4 (so Bag II has the remaining 1/4)

Question 4Easy

In a factory, Machine M1 makes 60% of the items and Machine M2 makes the other 40%. Two percent of M1‘s items are defective and 5% of M2‘s items are defective. An item is picked at random and found to be defective. What is the probability that it was made by M2?

Solution
  1. Events. M1, M2 = made by that machine; D = item is defective.
  2. Priors. P(M1) = 0.60,   P(M2) = 0.40.
  3. Likelihoods. P(D | M1) = 0.02,   P(D | M2) = 0.05.
  4. Denominator. P(D) = 0.60 × 0.02 + 0.40 × 0.05 = 0.012 + 0.020 = 0.032.
  5. Bayes. P(M2 | D) = 0.020 ÷ 0.032 = 5/8 = 0.625.

Answer: 5/8

Notice

M2 makes fewer items (40%) but is more error-prone, so it is responsible for the majority (62.5%) of defective items.

Question 5Medium

A disease affects 1% of a population. A test detects the disease in 95% of people who have it, but also gives a false positive for 5% of healthy people. A randomly chosen person tests positive. What is the probability that this person actually has the disease?

Solution
  1. Events. S = person is sick, H = person is healthy, T = test is positive.
  2. Priors. P(S) = 0.01,   P(H) = 0.99.
  3. Likelihoods. P(T | S) = 0.95,   P(T | H) = 0.05.
  4. Denominator. P(T) = 0.01 × 0.95 + 0.99 × 0.05 = 0.0095 + 0.0495 = 0.059.
  5. Bayes. P(S | T) = 0.0095 ÷ 0.059 = 19/118 ≈ 0.161, i.e. about 16.1%.

Answer: about 16.1%

Check with counts

Take 10,000 people. 100 are sick and 95 of them test positive. 9,900 are healthy and 495 of them test positive. Total positives = 590, of which only 95 are truly sick: 95/590 ≈ 16.1%. The surprising answer comes from the very low prior (1%): healthy people are so many that their false positives outnumber the true positives.

Question 6Medium

Three factories A, B and C supply 50%, 30% and 20% of the bulbs in a shop. The percentages of defective bulbs from A, B and C are 1%, 2% and 3% respectively. A bulb bought from the shop is defective. Find the probability that it came from factory C.

Solution
  1. Events. FA, FB, FC = bulb from that factory; D = bulb is defective.
  2. Priors. 0.5, 0.3, 0.2 (they add up to 1).
  3. Likelihoods. P(D | FA) = 0.01, P(D | FB) = 0.02, P(D | FC) = 0.03.
  4. Route probabilities.
    Through A: 0.5 × 0.01 = 0.005
    Through B: 0.3 × 0.02 = 0.006
    Through C: 0.2 × 0.03 = 0.006
  5. Denominator. P(D) = 0.005 + 0.006 + 0.006 = 0.017.
  6. Bayes. P(FC | D) = 0.006 ÷ 0.017 = 6/17 ≈ 0.353.

Answer: 6/17

Sanity check

The three posteriors are 5/17 (A), 6/17 (B) and 6/17 (C). They add up to 17/17 = 1, as they should.

Question 7Advanced

A box contains three coins: one is two-headed, one is fair, and one is biased with P(head) = 1/3. A coin is chosen at random and tossed twice. Both tosses show heads. Find the probability that the chosen coin is the two-headed coin.

Solution
  1. Events. C1 = two-headed coin, C2 = fair coin, C3 = biased coin; HH = both tosses are heads.
  2. Priors. P(C1) = P(C2) = P(C3) = 1/3.
  3. Likelihoods. The tosses are independent, so we multiply:
    P(HH | C1) = 1 × 1 = 1
    P(HH | C2) = (1/2)(1/2) = 1/4
    P(HH | C3) = (1/3)(1/3) = 1/9
  4. Denominator.
    P(HH) = (1/3)(1) + (1/3)(1/4) + (1/3)(1/9) = (1/3)(1 + 1/4 + 1/9) = (1/3)(49/36) = 49/108
  5. Bayes.
    P(C1 | HH) = 1/349/108 = 36/10849/108 = 36/49

Answer: 36/49 (about 73.5%)

Intuition

Before tossing, the two-headed coin had a prior of only 1/3 (about 33%). Seeing two heads in a row is a lot more likely under the two-headed coin than under the others, so the belief jumps to about 73.5%. More heads would push it even higher.

Question 8Advanced

30% of incoming emails are spam. The word “offer” appears in 60% of spam emails and in 10% of genuine emails. The word “free” appears in 50% of spam emails and in 5% of genuine emails. Assume that, within spam and within genuine emails separately, the two words appear independently. An email contains both words. Find the probability that it is spam.

Solution
  1. Events. S = spam, N = genuine, W = email contains both words.
  2. Priors. P(S) = 0.30,   P(N) = 0.70.
  3. Likelihoods (independence lets us multiply the two word probabilities):
    P(W | S) = 0.6 × 0.5 = 0.30
    P(W | N) = 0.1 × 0.05 = 0.005
  4. Route probabilities.
    Through spam: 0.30 × 0.30 = 0.09
    Through genuine: 0.70 × 0.005 = 0.0035
  5. Denominator. P(W) = 0.09 + 0.0035 = 0.0935.
  6. Bayes. P(S | W) = 0.09 ÷ 0.0935 = 180/187 ≈ 0.963.

Answer: about 96.3%

Why this matters

Each extra piece of evidence multiplies the odds in favour of spam. This is the idea behind the classic “naive Bayes” spam filter.

Question 9Advanced

A parcel travels through three post offices in sequence. Each post office independently loses an incoming parcel with probability 1/5. Given that the parcel is lost, find the probability that it was lost by the third post office.

Solution
  1. Events. E1, E2, E3 = parcel is lost by the 1st, 2nd, 3rd office respectively; E4 = not lost by any office. A = parcel is lost.
  2. Priors.
    P(E1) = 1/5 = 25/125
    P(E2) = (4/5)(1/5) = 20/125
    P(E3) = (4/5)(4/5)(1/5) = 16/125
    P(E4) = (4/5)3 = 64/125
    Check: 25 + 20 + 16 + 64 = 125, so the total is 1
  3. Likelihoods. P(A | E1) = P(A | E2) = P(A | E3) = 1 and P(A | E4) = 0.
  4. Denominator. P(A) = 25/125 + 20/125 + 16/125 + 0 = 61/125. (Quick check: 1 − 64/125 = 61/125.)
  5. Bayes. P(E3 | A) = (16/125) ÷ (61/125) = 16/61.

Answer: 16/61 (about 26.2%)

Common Mistakes to Avoid

  • Swapping the condition. Using P(A | Ei) when the question wants P(Ei | A). Always ask: “What has already happened?”
  • Forgetting the denominator. The numerator alone is only one route. You must divide by the total of all routes.
  • Priors that do not add to 1. If they do not, a cause is missing (like “knows the answer” in Question 2).
  • Ignoring the base rate. A very rare cause stays unlikely even with a good test (Question 5).
  • Overlapping causes. If two causes can happen together, Bayes’ theorem in this form cannot be applied directly.

Quick Revision

  • Bayes’ theorem finds the probability of a cause given an observed effect.
  • Causes must be mutually exclusive and exhaustive, with P(Ei) > 0 and P(A) > 0.
  • Formula: P(Ei | A) = P(Ei) P(A | Ei) ÷ Σ P(Ej) P(A | Ej).
  • The denominator is the law of total probability: P(A).
  • Think of it as favourable path ÷ all paths that give the evidence.
  • Sanity check: all posterior probabilities must add up to 1.

Missed the basics? Read the previous post on Conditional Probability first, then come back and redo Questions 1 and 2 without looking at the solutions.

Leave a Comment

Your email address will not be published.

← Previous: Conditional Probability Tutorial