B-17 Survivorship Bias: What the Bomber Story Actually Teaches About Missing Data

Psychology / Statistics

The bomber survivorship story is useful because it reveals a core statistical danger: the cases you can observe may be systematically different from the cases that disappeared from the dataset. Abraham Wald’s wartime aircraft work was deeper than the popular red-dot meme.

The popular bomber story captures a real statistical problem

Damage on returning aircraft reflects places a plane could often be hit and still survive. The missing aircraft contain information absent from the visible sample.

Test section 1 with one selection question about “the popular bomber story captures a real statistical problem”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 1 on “the popular bomber story captures a real statistical problem”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Wald’s work was more technical than the meme

The Statistical Research Group memoranda developed methods for estimating aircraft vulnerability using damage observed among survivors.

Test section 2 with one selection question about “wald’s work was more technical than the meme”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 2 on “wald’s work was more technical than the meme”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

The red-dot diagram is not a wartime artifact

The widely shared silhouette with bullet holes is a modern explanatory illustration and should not be treated as a document produced by Wald.

Test section 3 with one selection question about “the red-dot diagram is not a wartime artifact”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 3 on “the red-dot diagram is not a wartime artifact”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Selection on survival changes the distribution

If damage in one region is more lethal, aircraft hit there are less likely to return, so returning planes underrepresent those hits.

Test section 4 with one selection question about “selection on survival changes the distribution”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 4 on “selection on survival changes the distribution”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Missing data can be informative

Absence is not always random. When appearing in the dataset depends on survival, the missingness mechanism carries information about the outcome.

Test section 5 with one selection question about “missing data can be informative”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 5 on “missing data can be informative”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Survivorship bias is a selection mechanism

Observed data are not necessarily false; they are conditional on having survived long enough to become observable.

Test section 6 with one selection question about “survivorship bias is a selection mechanism”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 6 on “survivorship bias is a selection mechanism”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

The slogan oversimplifies vulnerability analysis

Actual analysis must consider exposure, hit rates, damage severity, aircraft structure, mission conditions, and return probability.

Test section 7 with one selection question about “the slogan oversimplifies vulnerability analysis”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 7 on “the slogan oversimplifies vulnerability analysis”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Success stories reproduce the same trap

Studying only successful companies can overstate behaviors shared by winners while excluding failed firms that used the same practices.

Test section 8 with one selection question about “success stories reproduce the same trap”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 8 on “success stories reproduce the same trap”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Investment databases can drop failure

Funds, products, managers, and strategies that disappear may be absent from retrospective samples, making survivor performance look stronger.

Test section 9 with one selection question about “investment databases can drop failure”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 9 on “investment databases can drop failure”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Career advice can select on winners

High performers may emphasize visible habits while underweighting timing, networks, selection, industry conditions, and people who followed similar paths without the same result.

Test section 10 with one selection question about “career advice can select on winners”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 10 on “career advice can select on winners”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Product analytics can hide churned users

Research based only on active users can optimize for survivors while missing the reasons others abandoned the product.

Test section 11 with one selection question about “product analytics can hide churned users”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 11 on “product analytics can hide churned users”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Case studies select for success

A case study can show possibility but does not estimate average causal effect when failures and comparison groups are missing.

Test section 12 with one selection question about “case studies select for success”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 12 on “case studies select for success”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Correction starts by mapping who disappeared

Ask which cases failed, exited, were rejected, never reported, or became impossible to measure, then ask why.

Test section 13 with one selection question about “correction starts by mapping who disappeared”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 13 on “correction starts by mapping who disappeared”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

Counterfactual thinking improves interpretation

The important question is what the dataset would look like if the missing cases were visible and whether that would change the conclusion.

Test section 14 with one selection question about “counterfactual thinking improves interpretation”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 14 on “counterfactual thinking improves interpretation”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

The durable lesson is the data-generating process

Before trusting a sample, understand the mechanism that allowed each case to enter it and the mechanism that removed the rest.

Test section 15 with one selection question about “the durable lesson is the data-generating process”: what determines whether a case becomes observable? If observation depends on survival, retention, reporting, approval, or success, the sample is selected.

For survivorship section 15 on “the durable lesson is the data-generating process”, trace the data-generating process explicitly: population, selection event, observed cases, missing cases, and inference. This preserves Wald’s deeper lesson that missingness can contain information about vulnerability.

B 17 Survivorship Bias field lab 1

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 1, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 1 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 1 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 2

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 2, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 2 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 2 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 3

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 3, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 3 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 3 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 4

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 4, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 4 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 4 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 5

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 5, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 5 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 5 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 6

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 6, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 6 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 6 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 7

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 7, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 7 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 7 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 8

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 8, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 8 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 8 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 9

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 9, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 9 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 9 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 10

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 10, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 10 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 10 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 11

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 11, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 11 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 11 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 12

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 12, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 12 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 12 by documenting the selection mechanism beside every reported estimate that depends on survivors.

B 17 Survivorship Bias field lab 13

Pick a dataset containing only active, successful, retained, approved, or reported cases. Write down which cases vanished before measurement and why. For survivorship audit 13, record which population existed before selection so later analysis cannot confuse survivors with the full set.

Compare the observed average with plausible scenarios for missing cases. The goal is not to invent precise values but to test how sensitive the conclusion is to non-random disappearance. State the missing-case scenario in survivorship audit 13 that would materially reverse the visible conclusion.

Add one sentence to the final analysis explaining the selection mechanism. If the conclusion changes once that sentence is visible, the original story depended too heavily on survivors. Close survivorship audit 13 by documenting the selection mechanism beside every reported estimate that depends on survivors.

Sources and further reading

Millionaire Lesson editorial analysis on b 17 survivorship bias. Source-backed claims are separated from managerial interpretation.