Correlation, Skew & Data Preprocessing

Correlation and Skew with Python

Correlation (Pearson) shows how two numeric columns move together — from -1 to +1.

df.corr() builds a correlation matrix for all numeric columns.

Real-life example: Ice cream sales and temperature correlate — both rise in summer, not because one eats the other.

import pandas as pd

df = pd.DataFrame({"hours": [1, 2, 3, 4], "score": [55, 65, 75, 85]})
print(df.corr())

Skew Distributions

Skew measures asymmetry. Positive skew has a long right tail.

df.skew() on numeric columns guides preprocessing choices.

Real-life example: Income data is often right-skewed — a few very high earners pull the tail right.

import pandas as pd

df = pd.DataFrame({"values": [1, 2, 2, 3, 100]})
print(df.skew())

Data Preprocessing

Preprocessing cleans and transforms raw data before modeling — handle missing values, encode categories, scale numbers.

Good preprocessing often beats fancy algorithms on messy data.

Real-life example: Preprocessing is washing vegetables before cooking — same recipe fails on dirty inputs.

Data Rescale

MinMaxScaler maps features to a range (often 0–1). Helps algorithms sensitive to magnitude.

Real-life example: Rescaling is converting all measurements to the same unit before comparing height and weight charts.

from sklearn.preprocessing import MinMaxScaler

X = [[10], [20], [30]]
scaler = MinMaxScaler()
print(scaler.fit_transform(X))

Normalize Data with Python

Normalizer scales each row to unit length (L2 norm). Common for text features and some distance models.

Real-life example: Normalizing is adjusting each student's mix of subjects to compare fairly when totals differ.

from sklearn.preprocessing import Normalizer

X = [[1, 2, 3], [4, 5, 6]]
print(Normalizer().fit_transform(X))

Standardized Data with Python

StandardScaler sets mean 0 and std 1 — z-score normalization. Works well when data is roughly Gaussian.

Real-life example: Standardizing is grading on a curve — everyone's score relative to class average.

from sklearn.preprocessing import StandardScaler

X = [[0], [10], [20]]
print(StandardScaler().fit_transform(X))

Binarized Data with Python

Binarizer converts values above a threshold to 1, else 0. Used in image masks and simple flags.

Real-life example: Binarizing is a pass/fail stamp — scores above 40 become pass (1), else fail (0).

from sklearn.preprocessing import Binarizer

X = [[0.5], [1.2], [2.8]]
print(Binarizer(threshold=1.0).transform(X))