Detecting and Handling Missing Data

Why Data Goes Missing

Before you fix a gap, understand why it exists. A missing value can mean a sensor failed, a user skipped an optional form field, a join key didn't match, or a value was legitimately not applicable (a "spouse's name" field for an unmarried respondent). Each of these has a different fix, and guessing wrong can quietly bias your results.

Statisticians group missingness into three patterns. Data is Missing Completely at Random (MCAR) when the fact that a value is missing has nothing to do with any variable in the dataset — a random subset of survey forms got lost in transit. Data is Missing at Random (MAR) when the missingness relates to other observed variables — older respondents are more likely to skip an income question, but within an age group the missingness is random. Data is Missing Not at Random (MNAR) when the missingness relates to the missing value itself — people with very high or very low incomes are the ones most likely to skip the income question. MNAR is the dangerous case, because the rows you're missing are systematically different from the rows you have.

Diagnosing the Pattern

You rarely know the true mechanism, but you can gather evidence. Cross-tabulate missingness against other columns: if a "response time" field is missing far more often for mobile users than desktop users, that's a strong hint the missingness is related to platform, not random chance. Plot the count of missing values by category, by date, or by segment. A pattern that clusters — by region, by date range, by data source — points to a systematic cause (a broken integration, a form redesign) rather than random noise, and systematic causes usually need a different fix than random ones.

Strategies for Handling Missing Values

Deletion is the simplest option. Dropping a row is reasonable when missingness is rare, appears random, and the analysis has enough remaining rows to stay reliable. Dropping a column is reasonable when a field is missing so often that no fix would make it trustworthy. Deletion is dangerous under MNAR, since you'd be removing exactly the rows that behave differently.

Imputation fills gaps with an estimated value. Simple imputation — using the column mean, median, or mode — is fast and fine for small amounts of random missingness, but it shrinks variance and can mute real relationships if overused. Median is generally safer than mean for skewed data, since a handful of extreme values won't drag it around. Forward or backward fill works well for ordered data like time series, where the most recent known value is a reasonable stand-in for a temporary gap. Model-based imputation — predicting a missing value from other columns using regression or a nearest-neighbors approach — is more accurate but adds complexity and can overstate your confidence if you forget the value was estimated.

A middle-ground option is to flag and fill: impute a placeholder value, but also add a boolean column marking which rows were imputed. This lets any downstream model or reviewer account for the uncertainty instead of treating estimated values as if they were observed.

Choosing the Right Strategy for the Situation

Match the fix to the stakes and the mechanism. For a one-off exploratory chart, dropping a handful of random rows is fine. For a metric that will drive a business decision, prefer a documented imputation method over silent deletion, and always report how much data was missing and how you handled it. When missingness looks systematic (MNAR), consider whether the honest answer is "we can't reliably measure this for this subgroup" rather than papering over the gap with an imputed number.

Practical Review Checklist

Before moving on, confirm that you can:

• Explain the difference between MCAR, MAR, and MNAR in your own words
• Cross-tabulate missingness against other columns to check for a pattern
• Choose between deletion, simple imputation, and model-based imputation with a reason
• Flag imputed values so they're distinguishable from observed ones
• State, in your report, how much data was missing and how it was handled

Conclusion

Missing data isn't a single problem with a single fix — it's a diagnostic question first and a technique second. Spend the time understanding the pattern before you pick a strategy, and always be transparent about what you filled in versus what was actually observed.