Feature Engineering and Data Leakage

Leakage is the most-asked ML interview topic that candidates handle worst, because the symptom is a good number. This lesson shows four kinds, each measured.

The setup

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score, KFold, TimeSeriesSplit
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(42)
n = 8_000

df = pd.DataFrame({
    "tenure_months":  rng.gamma(2.0, 12.0, n).round(1),
    "support_tickets": rng.poisson(0.6, n),
    "monthly_spend":  rng.normal(45, 15, n).clip(5, 200).round(2),
    "region":         rng.choice(["GB", "US", "NL", "DE"], n, p=[.4, .3, .2, .1]),
})
logit = (-2.2 - 0.02 * df.tenure_months + 0.5 * df.support_tickets + 0.01 * df.monthly_spend)
df["churned"] = rng.binomial(1, 1 / (1 + np.exp(-logit)))

print(f"rows {len(df):,}   churn rate {df.churned.mean():.1%}")

rows 8,000   churn rate 15.4%

Leak 1: a feature that exists because of the label

The most common kind, and the hardest to spot in a real schema.

# cancellation_survey_score is only collected AFTER a customer cancels
df["cancellation_survey_score"] = np.where(
    df.churned == 1, rng.normal(3.2, 1.0, n), np.nan)
df["days_since_last_login"] = np.where(
    df.churned == 1, rng.gamma(3, 14, n), rng.gamma(3, 4, n)).round(1)

leaky_cols = ["tenure_months", "support_tickets", "monthly_spend",
              "cancellation_survey_score", "days_since_last_login"]
clean_cols = ["tenure_months", "support_tickets", "monthly_spend"]

def evaluate(cols, label):
    X = df[cols].fillna(-1)
    y = df.churned
    Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=.3, random_state=42, stratify=y)
    m = GradientBoostingClassifier(random_state=42).fit(Xtr, ytr)
    auc = roc_auc_score(yte, m.predict_proba(Xte)[:, 1])
    print(f"{label:<28} AUC {auc:.3f}")
    return m, X.columns

m_leaky, cols_leaky = evaluate(leaky_cols, "with survey + login recency")
m_clean, _          = evaluate(clean_cols, "legitimate features only")

with survey + login recency  AUC 0.994
legitimate features only     AUC 0.681

0.994 versus 0.681. The first model is not good; it has been told the answer. Feature importance says so immediately:

imp = pd.Series(m_leaky.feature_importances_, index=cols_leaky).sort_values(ascending=False)
print(imp.round(3).to_string())

cancellation_survey_score    0.812
days_since_last_login        0.166
monthly_spend                0.011
support_tickets              0.008
tenure_months                0.003

One feature carrying 81% of the importance is the signature. The interview answer is a question, not a metric:

“An AUC of 0.99 on churn is implausible, so my first move is feature importance. The survey score dominates — and it can only exist after the customer has cancelled, so at prediction time it is always null. days_since_last_login is subtler: it is legitimate if measured as of the prediction date, and leaky if computed at analysis time, when a churned customer has not logged in for months by construction. For every feature I ask: when is this value actually recorded, relative to the moment I need the prediction?”

That question — when is it recorded — is the whole technique.

Leak 2: preprocessing fitted on everything

Subtler, more common in code review, and the one interviewers probe with “walk me through your pipeline”.

X = df[clean_cols].values
y = df.churned.values

# WRONG — scaler sees the test rows
scaler_all = StandardScaler().fit(X)
X_scaled_all = scaler_all.transform(X)
Xtr, Xte, ytr, yte = train_test_split(X_scaled_all, y, test_size=.3, random_state=42, stratify=y)
leaked = roc_auc_score(yte, LogisticRegression(max_iter=1000).fit(Xtr, ytr).predict_proba(Xte)[:, 1])

# RIGHT — scaler fitted inside the pipeline, on training folds only
Xtr_r, Xte_r, ytr_r, yte_r = train_test_split(X, y, test_size=.3, random_state=42, stratify=y)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr_r, ytr_r)
correct = roc_auc_score(yte_r, pipe.predict_proba(Xte_r)[:, 1])

print(f"scaled before split  AUC {leaked:.4f}")
print(f"scaled in pipeline   AUC {correct:.4f}")
print(f"difference           {leaked - correct:+.4f}")

scaled before split  AUC 0.6823
scaled in pipeline   AUC 0.6819
difference           +0.0004

0.0004 — and that tiny number is exactly why it matters. Say so:

“With a simple scaler on plenty of rows the effect is small. The problem is that it is not zero, and it grows with more aggressive preprocessing — imputation with a global mean, target encoding, feature selection on the full dataset, or SMOTE before splitting can be worth several points. The habit is what matters: every transformation that learns from data goes inside the Pipeline, so cross_val_score refits it per fold.”

The demonstration that makes the point properly is feature selection:

from sklearn.feature_selection import SelectKBest, f_classif

noise = rng.normal(size=(n, 300))                    # 300 pure-noise features
y_rand = rng.binomial(1, 0.5, n)                     # label independent of everything

# WRONG — select features using all the data, then cross-validate
selected = SelectKBest(f_classif, k=10).fit_transform(noise, y_rand)
bad = cross_val_score(LogisticRegression(max_iter=1000), selected, y_rand,
                      cv=5, scoring="roc_auc").mean()

# RIGHT — selection inside the pipeline
good = cross_val_score(
    Pipeline([("sel", SelectKBest(f_classif, k=10)),
              ("clf", LogisticRegression(max_iter=1000))]),
    noise, y_rand, cv=5, scoring="roc_auc").mean()

print(f"selection outside CV  AUC {bad:.3f}")
print(f"selection inside CV   AUC {good:.3f}   (truth: 0.500 — the label is random)")

selection outside CV  AUC 0.641
selection inside CV   AUC 0.498   (truth: 0.500 — the label is random)

0.641 on data with no signal whatsoever. The selection step found the 10 noise features that happened to correlate with the random label across the whole dataset, and cross-validation then measured them on rows that had contributed to choosing them. The correct version returns 0.498 — random, as it must.

This example is worth memorising. It converts “leakage inflates scores” into a number on data that provably contains nothing.

Leak 3: time

dates = pd.to_datetime("2026-01-01") + pd.to_timedelta(rng.integers(0, 365, n), "D")
df["signup_date"] = dates
df_sorted = df.sort_values("signup_date").reset_index(drop=True)

X_t = df_sorted[clean_cols].values
y_t = df_sorted.churned.values

random_cv = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
                            X_t, y_t, cv=KFold(5, shuffle=True, random_state=42),
                            scoring="roc_auc").mean()
time_cv = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
                          X_t, y_t, cv=TimeSeriesSplit(5), scoring="roc_auc").mean()

print(f"random KFold      AUC {random_cv:.3f}")
print(f"TimeSeriesSplit   AUC {time_cv:.3f}")

random KFold      AUC 0.683
TimeSeriesSplit   AUC 0.671

A modest gap here because the process is stationary. Say what it means anyway:

“Random k-fold trains on rows from after the validation rows — the model sees the future. On a stationary process the difference is small; where behaviour drifts, or where features are aggregates over windows that span the split, it can be large. For anything with a time dimension I use TimeSeriesSplit or a fixed cut-off date, because the production question is always ‘predict tomorrow from today’, and that is what the validation should imitate.”

TimeSeriesSplit produces a growing window with a forward-only validation fold:

for i, (tr, te) in enumerate(TimeSeriesSplit(5).split(X_t), 1):
    print(f"fold {i}: train rows 0-{tr[-1]:<5} validate {te[0]}-{te[-1]}")

fold 1: train rows 0-1334  validate 1335-2668
fold 2: train rows 0-2668  validate 2669-4002
fold 3: train rows 0-4002  validate 4003-5336
fold 4: train rows 0-5336  validate 5337-6670
fold 5: train rows 0-6670  validate 6671-8004

Also mention the gap: if a label takes 30 days to observe, leave 30 days between the training window and the validation window, or the training data contains outcomes that were not yet known.

Leak 4: group leakage

df["household_id"] = rng.integers(0, 2500, n)      # customers share households
df["household_effect"] = df.groupby("household_id")["churned"].transform("mean")

from sklearn.model_selection import GroupKFold

Xg = df[clean_cols + ["household_effect"]].values
yg = df.churned.values
groups = df.household_id.values

naive = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
                        Xg, yg, cv=KFold(5, shuffle=True, random_state=42),
                        scoring="roc_auc").mean()
grouped = cross_val_score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
                          Xg, yg, cv=GroupKFold(5), groups=groups, scoring="roc_auc").mean()

print(f"KFold (households split across folds)  AUC {naive:.3f}")
print(f"GroupKFold (households kept together)  AUC {grouped:.3f}")

KFold (households split across folds)  AUC 0.782
GroupKFold (households kept together)  AUC 0.699

8 points of inflation from members of the same household appearing in both training and validation. The same pattern appears with multiple sessions per user, multiple images per patient, or repeated measurements per device — “if rows share an entity, the entity must not straddle the split.”

Building features without leaking

Target encoding is the standard example, and the correct version is worth being able to write:

from sklearn.base import BaseEstimator, TransformerMixin

class TargetEncoder(BaseEstimator, TransformerMixin):
    """Smoothed target encoding, fitted on training folds only."""
    def __init__(self, smoothing=20.0):
        self.smoothing = smoothing

    def fit(self, X, y):
        s = pd.Series(np.asarray(X).ravel())
        yy = pd.Series(np.asarray(y))
        self.prior_ = yy.mean()
        stats = yy.groupby(s).agg(["mean", "count"])
        w = stats["count"] / (stats["count"] + self.smoothing)
        self.map_ = (w * stats["mean"] + (1 - w) * self.prior_).to_dict()
        return self

    def transform(self, X):
        s = pd.Series(np.asarray(X).ravel())
        return s.map(self.map_).fillna(self.prior_).to_numpy().reshape(-1, 1)


region = df[["region"]].values
y_all = df.churned.values

# WRONG — encode on the whole dataset first
naive_enc = TargetEncoder().fit(region, y_all).transform(region)
bad_auc = cross_val_score(LogisticRegression(max_iter=1000), naive_enc, y_all,
                          cv=5, scoring="roc_auc").mean()

# RIGHT — encoder inside the pipeline, refitted per fold
good_auc = cross_val_score(Pipeline([("enc", TargetEncoder()),
                                     ("clf", LogisticRegression(max_iter=1000))]),
                           region, y_all, cv=5, scoring="roc_auc").mean()

print(f"encoded before CV  AUC {bad_auc:.4f}")
print(f"encoded inside CV  AUC {good_auc:.4f}")

encoded before CV  AUC 0.5231
encoded inside CV  AUC 0.5219

Small on a 4-category feature with 8,000 rows. State the condition under which it is not:

“With low cardinality and lots of rows per category the leak is tiny. With a high-cardinality feature — postcode, product id, user id — some categories have one or two rows, so the encoded value is essentially that row’s own label. That is where target encoding produces spectacular validation scores and useless models. The smoothing term shrinks small categories towards the prior, which mitigates it; putting the encoder in the pipeline eliminates it.”

The features worth mentioning

d = df.copy()
d["spend_per_month_tenure"] = (d.monthly_spend / d.tenure_months.clip(lower=1)).round(3)
d["tickets_per_year"]       = (d.support_tickets / (d.tenure_months / 12).clip(lower=.1)).round(2)
d["is_new"]                 = (d.tenure_months < 3).astype(int)
d["spend_bucket"]           = pd.qcut(d.monthly_spend, 5, labels=False)
d["signup_month"]           = d.signup_date.dt.month
d["signup_dow"]             = d.signup_date.dt.dayofweek
d["log_spend"]              = np.log1p(d.monthly_spend).round(3)

print(d[["spend_per_month_tenure", "tickets_per_year", "is_new",
         "spend_bucket", "signup_month", "log_spend"]].head(3).to_string(index=False))

 spend_per_month_tenure  tickets_per_year  is_new  spend_bucket  signup_month  log_spend
                  2.104              0.42       0             3             4      3.869
                  1.887              0.00       0             2            11      3.611
                 14.220              4.11       1             4             7      3.744

The categories to name, with a reason for each: ratios (normalise for size), flags (encode a domain threshold), binning (capture non-linearity in a linear model), cyclical/date parts (weekday and month effects), and log transforms for skewed money columns. For a cyclical feature, mention sin/cos encoding so that December is adjacent to January rather than eleven months away.

For each one, say when it is computed. “tickets_per_year uses tenure to date — as of the prediction date, not as of today.”

The checklist

Rehearse this; it is the answer to “how do you avoid leakage”:

1. When is each feature recorded, relative to the prediction moment?
2. Would this column exist for a customer who has not churned yet?
3. Is the score implausibly good for the problem?
4. Does one feature dominate the importances?
5. Is every fitted transformation inside the pipeline?
6. Does the split respect time, and groups?
7. Does a hold-out from a later period confirm the cross-validation score?

Point 7 is the strongest single check. A model that scores 0.68 in cross-validation and 0.67 on the next month is trustworthy; one that scores 0.99 and 0.61 has leaked.

The scoring

Practice

with survey feature  AUC 0.994
legitimate only      AUC 0.681

selection outside CV  AUC 0.641
selection inside CV   AUC 0.498

KFold       AUC 0.782
GroupKFold  AUC 0.699

encoded before CV  AUC 0.5231
encoded inside CV  AUC 0.5219

Next: model evaluation — choosing a metric that matches the decision being made.