Correlation, Segmentation, and Outlier Detection in EDA

Measuring Correlation Between Variables

Correlation quantifies how strongly two numeric variables move together, on a scale from -1 to 1. A value near 1 means they rise together, near -1 means one rises as the other falls, and near 0 means little linear relationship. A correlation matrix — every numeric column compared against every other — is a fast way to scan a wide dataset for relationships worth investigating further, especially useful in the early stages of EDA before you know which pairs of variables matter.

Correlation has real limits. The standard (Pearson) correlation coefficient only captures linear relationships — two variables can have a strong curved relationship and still show a correlation near zero. It's also highly sensitive to outliers, where a single extreme point can create or hide an apparent relationship. Always pair a correlation number with a scatter plot before trusting it; the number alone can hide a shape that a five-second glance at a chart would catch immediately.

Segmenting Data to Find Hidden Patterns

Aggregate numbers can hide the story. A flat overall trend can conceal two segments moving in opposite directions and canceling each other out — a classic case of Simpson's paradox, where a trend present in several groups of data disappears or reverses when the groups are combined. Segmenting by obvious dimensions — region, customer tier, acquisition channel, time period — is often the fastest way to find where a pattern actually lives.

Good segmentation starts from a hypothesis, not a random slice-and-dice. Ask "would I expect this metric to behave differently for new versus returning customers?" and then check, rather than mechanically splitting by every available column and hoping something looks interesting. That said, a broad first pass across several plausible segments is a reasonable way to generate hypotheses worth digging into further.

Spotting Outliers During Exploration

During EDA, an unusual value is information, not just noise to be cleaned away. A single day with revenue ten times the normal average might be a data error — or it might be the day a major promotion ran, which is exactly the kind of insight EDA exists to surface. Before deciding an outlier is a problem, ask what event or segment might explain it.

Plotting a metric over time is often the fastest way to catch outliers with an obvious explanation: a spike that lines up with a known campaign, holiday, or outage tells its own story. Outliers with no obvious explanation are the ones worth flagging for the data cleaning pass covered earlier in this course, since they're more likely to be genuine errors.

Correlation Is Not Causation

Two variables moving together doesn't mean one causes the other. They might share a common cause (ice cream sales and drowning incidents both rise with summer heat), the causal direction might be reversed from what seems intuitive, or the relationship might simply be a coincidence in a small sample. Treat every correlation surfaced during EDA as a hypothesis to investigate — through a controlled experiment, a natural experiment, or at minimum a plausible causal mechanism — not as a conclusion ready to present.

Practical Review Checklist

Before moving on, confirm that you can:

• Read a correlation matrix and identify pairs worth a closer look
• Explain why correlation should always be checked against a scatter plot
• Describe what Simpson's paradox is and why segmentation can reveal it
• Explain why an outlier during EDA is information, not just an error to remove
• Give an example of correlation without causation

Conclusion

The relationships between variables — not the variables in isolation — are usually where the real insight in a dataset lives. Correlate broadly, segment with a hypothesis in mind, and stay skeptical of any relationship until you understand why it exists.