I5 · Machine Learning Foundations

Feature Engineering and Selection

Feature engineering is creating new input features from raw data using domain knowledge, so that patterns become easier for a model to learn. Common techniques include extracting parts of dates, computing ratios and differences, binning numbers into ranges, combining features into interactions, aggregating history (such as a customer's average spend) and transforming skewed values with logarithms. Feature selection keeps only the features that help, using filter methods (statistical scores), wrapper methods (trying subsets with a model, such as recursive feature elimination) or embedded methods (models that select as they learn, such as lasso or tree importance).

12 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

A food-delivery company records the order time as "2024-03-15 19:42". A model cannot do much with that string. Turn it into "hour = 19, weekday = Friday, dinner rush = yes" and the model suddenly predicts delivery times far better. Good features often matter more than a fancier algorithm.

Feature engineering is creating new input features from raw data using domain knowledge, so that patterns become easier for a model to learn. Common techniques include extracting parts of dates, computing ratios and differences, binning numbers into ranges, combining features into interactions, aggregating history (such as a customer's average spend) and transforming skewed values with logarithms. Feature selection keeps only the features that help, using filter methods (statistical scores), wrapper methods (trying subsets with a model, such as recursive feature elimination) or embedded methods (models that select as they learn, such as lasso or tree importance).

In simple words

A doctor does not just read your height and weight separately; they compute your BMI, because that single number says more about health. Feature engineering is creating such helpful numbers. Feature selection is like packing for a trip: take what you need and leave the rest, so the bag stays light.

Overview

  • Feature engineering creates informative inputs from raw data using domain knowledge.
  • Feature selection keeps the inputs that help and removes the rest.
  • Good features often improve a model more than a fancier algorithm.
  • Every feature must be available at prediction time, or it leaks the answer.
  • Fewer, better features reduce overfitting, cost and privacy risk.
  • Deep learning learns many features itself, but tabular problems still benefit from engineering.

Why it matters

On tabular data, careful features usually give bigger gains than switching algorithms. Removing useless or leaky features reduces overfitting, speeds up training, makes models easier to explain and cheaper to run, and avoids collecting data that is not needed, which also helps privacy.

Key terms

Feature engineering
Creating new, more useful inputs from raw data.
Interaction feature
A feature combining two others, such as distance × rain.
Binning
Grouping a number into ranges, such as age bands.
Aggregation feature
A summary of related history, such as a user's average order value.
Filter, wrapper and embedded selection
Scoring features alone, testing subsets with a model, or letting the model select.
Curse of dimensionality
As features grow, data becomes sparse and models need far more examples.

Types

Feature engineering and selection each come in several types. These are the main techniques.

1. Transformation features

New versions of one feature: logs, powers, bins and flags.

  • How it works: Apply a function to a column, such as log(1 + x), x², or a bin like age band 18–25, to expose the pattern a model needs.
  • Example: is_peak_hour from the order hour; log of income.
  • When to use it: Use them when a relationship is curved or has thresholds that a linear model cannot learn directly.

2. Combination (interaction and ratio) features

Features built from two or more others: products, ratios and differences.

  • How it works: Multiply, divide or subtract columns that interact in reality.
  • Example: Price per square foot; distance × rain for delivery delays; BMI from weight and height.
  • When to use it: Use them when domain knowledge says two factors work together.

3. Aggregation and time features

Summaries of related history and parts of dates.

  • How it works: Group by an entity and compute counts, means or recency from past rows only; extract hour, weekday and month from timestamps.
  • Example: A customer's number of orders in the last 30 days; days since last purchase.
  • When to use it: Use them in customer, sensor and transaction data; compute them carefully to avoid leakage.

4. Filter selection methods

Scoring each feature on its own against the target.

  • How it works: Correlation, F-tests, chi-square and mutual information rank features quickly; SelectKBest keeps the top k.
  • Example: Mutual information picking distance, preparation time and rain while ignoring ten noise columns.
  • When to use it: Use them as a fast first pass, remembering they miss features that only help in combination.

5. Wrapper and embedded selection methods

Selecting features by training models: recursive elimination (wrapper), and lasso or tree importance (embedded).

  • How it works: RFE repeatedly drops the weakest feature; lasso sets weak weights to zero; permutation importance measures how much shuffling a feature hurts validation accuracy.
  • Example: RFE reducing 60 electricity-usage features to 12 with no loss in accuracy.
  • When to use it: Use them when features interact and you can afford the extra training time.

Explained in detail

Common feature-engineering techniques

TechniqueExampleWhy it helps
Date and time partsHour, weekday, month, is_holiday from a timestampExposes daily, weekly and seasonal patterns
Ratios and differencesPrice per square foot; days since last purchaseCaptures relationships the model would struggle to learn
Binning and flagsis_peak_hour, age band 18–25Lets linear models use non-linear effects
Log transformlog(income), log(views)Tames long tails and outliers
AggregationsCustomer's average spend over 90 daysAdds history to a single row
Text and category handlingWord counts; one-hot or target encodingTurns non-numbers into numbers

Domain knowledge first, and beware of leakage

The best features come from understanding the problem. A delivery manager knows that rain slows riders, that lunch and dinner rushes create queues, and that a restaurant's average preparation time matters. Each idea becomes a feature to test. But every feature must be available at prediction time and must not contain the answer. "Actual delivery time" or "customer complaint filed" would make a delivery model look brilliant in testing and useless in real use. Aggregations need special care: a customer's average spend must be computed from past orders only, never including the order being predicted. Building features inside a scikit-learn pipeline, or computing them only from training folds, prevents such leakage.

Selecting features

Filter methods score each feature on its own, for example by correlation, an F-test (SelectKBest with f_regression) or mutual information, which also detects non-linear links; they are fast but ignore interactions. Wrapper methods such as recursive feature elimination train a model, drop the weakest feature and repeat, which is slower but considers features together. Embedded methods select during training: lasso sets weak weights to zero, and tree ensembles provide importances. Always choose features using training data only, ideally inside cross-validation, and confirm on a test set that the smaller model is at least as good. Highly correlated pairs, such as height in centimetres and in inches, add nothing new; keep one. In the code below, a linear model using the raw timestamp predicts delivery time with an average error of 5.1 minutes, while the engineered features cut it to 3.2 minutes. When ten columns of random noise are added, mutual information picks distance, preparation time, rain, hour and distance × rain as the top five, and none of the noise.

How it works

  1. Talk to domain experts and list ideas for features.
  2. Create features with code that uses only information available at prediction time.
  3. Compare a baseline model with and without the new features using cross-validation.
  4. Select features with filter, wrapper or embedded methods on training data.
  5. Confirm on the test set and document each feature's meaning.

The maths

Interaction feature
Log transform for skewed values

An order placed at 19:42 on a Friday, 6 km from the restaurant, in the rain becomes hour = 19, weekday = 4 (counting Monday as 0), is_peak = 1 because 19:00 to 21:00 is the dinner rush, distance_km = 6, rain = 1 and distance × rain = 6. A linear model can now learn "peak hours add about 10 minutes" and "each rainy kilometre adds extra time", effects it could not learn from the raw timestamp. Suppose the model also receives 10 random columns of noise: a filter method that scores each column's link to delivery time ranks the real features at the top and the noise near zero, so the noise can be dropped.

In code

delivery_features.py
import numpy as np
import pandas as pd
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
n = 2000
orders = pd.DataFrame({
    "time": pd.Timestamp("2024-03-01") + pd.to_timedelta(rng.integers(0, 60 * 24 * 60, n), unit="min"),
    "distance_km": rng.uniform(0.5, 10, n).round(1),
    "prep_min": rng.integers(5, 30, n),
    "rain": (rng.random(n) < 0.2).astype(int),
})
hour = orders.time.dt.hour
peak = hour.between(12, 13) | hour.between(19, 20)
orders["minutes"] = 10 + 3 * orders.distance_km + orders.prep_min + 10 * peak + 2.5 * orders.distance_km * orders.rain + rng.normal(0, 4, n)

raw = pd.DataFrame({"timestamp": orders.time.astype("int64") // 10**9, "distance_km": orders.distance_km, "prep_min": orders.prep_min, "rain": orders.rain})
eng = raw.drop(columns="timestamp").assign(hour=hour, is_peak=peak.astype(int), weekday=orders.time.dt.weekday, dist_x_rain=orders.distance_km * orders.rain)

for name, X in [("raw features", raw), ("engineered features", eng)]:
    X_tr, X_te, y_tr, y_te = train_test_split(X, orders.minutes, random_state=0)
    mae = mean_absolute_error(y_te, LinearRegression().fit(X_tr, y_tr).predict(X_te))
    print(f"{name:<20} test MAE = {mae:.1f} minutes")

noisy = eng.join(pd.DataFrame(rng.normal(size=(n, 10)), columns=[f"noise{i}" for i in range(10)]))
X_tr, X_te, y_tr, y_te = train_test_split(noisy, orders.minutes, random_state=0)
selector = SelectKBest(mutual_info_regression, k=5).fit(X_tr, y_tr)
print("Top 5 features by mutual information:", list(X_tr.columns[selector.get_support()]))

Output

raw features         test MAE = 5.1 minutes
engineered features  test MAE = 3.2 minutes
Top 5 features by mutual information: ['distance_km', 'prep_min', 'rain', 'hour', 'dist_x_rain']

Try it yourself

Predicting electricity bills

A power company wants to warn customers whose next bill will be unusually high.

Useful feature or leakage?

A model predicts whether a loan will default, at the time of application.

  • Years at current job

  • Monthly income divided by loan instalment

  • Debt-collection call made

  • Number of missed payments on this loan

Common mistakes

  • Creating features from information that is not available at prediction time.
  • Selecting features on the full data set before splitting.
  • Adding hundreds of features without checking that they help.

Check yourself

Question 1

1. Which is an engineered feature?

Question 2

2. Recursive feature elimination is an example of…

Question 3

3. Why must aggregation features use only past data?

Summary

  • Feature engineering turns raw data into informative inputs using domain knowledge.
  • Dates, ratios, flags, logs and aggregations are common, but every feature must be available at prediction time.
  • Select features with filter, wrapper or embedded methods on training data, and confirm on a test set.

Finished reading?

Sources: Kuhn and Johnson, Feature Engineering and Selection (free online book) · scikit-learn: feature selection · Zheng and Casari, Feature Engineering for Machine Learning (O'Reilly)