Data Preprocessing and Feature Engineering

Data preparation essentials

Handle missing values with drop/impute strategy.
Encode categorical variables.
Scale numeric features for distance-based models.
Split train/validation/test before peeking at outcomes.

Pipeline example

Scikit-learn preprocessing pipeline

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer

numeric_features = ["age", "salary"]
categorical_features = ["city", "education"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler())
    ]), numeric_features),
    ("cat", Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore"))
    ]), categorical_features),
])