Definition
A food-delivery company records the order time as "2024-03-15 19:42". A model cannot do much with that string. Turn it into "hour = 19, weekday = Friday, dinner rush = yes" and the model suddenly predicts delivery times far better. Good features often matter more than a fancier algorithm.
Feature engineering is creating new input features from raw data using domain knowledge, so that patterns become easier for a model to learn. Common techniques include extracting parts of dates, computing ratios and differences, binning numbers into ranges, combining features into interactions, aggregating history (such as a customer's average spend) and transforming skewed values with logarithms. Feature selection keeps only the features that help, using filter methods (statistical scores), wrapper methods (trying subsets with a model, such as recursive feature elimination) or embedded methods (models that select as they learn, such as lasso or tree importance).
In simple words
A doctor does not just read your height and weight separately; they compute your BMI, because that single number says more about health. Feature engineering is creating such helpful numbers. Feature selection is like packing for a trip: take what you need and leave the rest, so the bag stays light.
Overview
- Feature engineering creates informative inputs from raw data using domain knowledge.
- Feature selection keeps the inputs that help and removes the rest.
- Good features often improve a model more than a fancier algorithm.
- Every feature must be available at prediction time, or it leaks the answer.
- Fewer, better features reduce overfitting, cost and privacy risk.
- Deep learning learns many features itself, but tabular problems still benefit from engineering.
Why it matters
On tabular data, careful features usually give bigger gains than switching algorithms. Removing useless or leaky features reduces overfitting, speeds up training, makes models easier to explain and cheaper to run, and avoids collecting data that is not needed, which also helps privacy.
Key terms
- Feature engineering
- Creating new, more useful inputs from raw data.
- Interaction feature
- A feature combining two others, such as distance × rain.
- Binning
- Grouping a number into ranges, such as age bands.
- Aggregation feature
- A summary of related history, such as a user's average order value.
- Filter, wrapper and embedded selection
- Scoring features alone, testing subsets with a model, or letting the model select.
- Curse of dimensionality
- As features grow, data becomes sparse and models need far more examples.
Types
Feature engineering and selection each come in several types. These are the main techniques.
1. Transformation features
New versions of one feature: logs, powers, bins and flags.
- How it works: Apply a function to a column, such as log(1 + x), x², or a bin like age band 18–25, to expose the pattern a model needs.
- Example: is_peak_hour from the order hour; log of income.
- When to use it: Use them when a relationship is curved or has thresholds that a linear model cannot learn directly.
2. Combination (interaction and ratio) features
Features built from two or more others: products, ratios and differences.
- How it works: Multiply, divide or subtract columns that interact in reality.
- Example: Price per square foot; distance × rain for delivery delays; BMI from weight and height.
- When to use it: Use them when domain knowledge says two factors work together.
3. Aggregation and time features
Summaries of related history and parts of dates.
- How it works: Group by an entity and compute counts, means or recency from past rows only; extract hour, weekday and month from timestamps.
- Example: A customer's number of orders in the last 30 days; days since last purchase.
- When to use it: Use them in customer, sensor and transaction data; compute them carefully to avoid leakage.
4. Filter selection methods
Scoring each feature on its own against the target.
- How it works: Correlation, F-tests, chi-square and mutual information rank features quickly; SelectKBest keeps the top k.
- Example: Mutual information picking distance, preparation time and rain while ignoring ten noise columns.
- When to use it: Use them as a fast first pass, remembering they miss features that only help in combination.
5. Wrapper and embedded selection methods
Selecting features by training models: recursive elimination (wrapper), and lasso or tree importance (embedded).
- How it works: RFE repeatedly drops the weakest feature; lasso sets weak weights to zero; permutation importance measures how much shuffling a feature hurts validation accuracy.
- Example: RFE reducing 60 electricity-usage features to 12 with no loss in accuracy.
- When to use it: Use them when features interact and you can afford the extra training time.
Explained in detail
Common feature-engineering techniques
| Technique | Example | Why it helps |
|---|---|---|
| Date and time parts | Hour, weekday, month, is_holiday from a timestamp | Exposes daily, weekly and seasonal patterns |
| Ratios and differences | Price per square foot; days since last purchase | Captures relationships the model would struggle to learn |
| Binning and flags | is_peak_hour, age band 18–25 | Lets linear models use non-linear effects |
| Log transform | log(income), log(views) | Tames long tails and outliers |
| Aggregations | Customer's average spend over 90 days | Adds history to a single row |
| Text and category handling | Word counts; one-hot or target encoding | Turns non-numbers into numbers |
Domain knowledge first, and beware of leakage
The best features come from understanding the problem. A delivery manager knows that rain slows riders, that lunch and dinner rushes create queues, and that a restaurant's average preparation time matters. Each idea becomes a feature to test. But every feature must be available at prediction time and must not contain the answer. "Actual delivery time" or "customer complaint filed" would make a delivery model look brilliant in testing and useless in real use. Aggregations need special care: a customer's average spend must be computed from past orders only, never including the order being predicted. Building features inside a scikit-learn pipeline, or computing them only from training folds, prevents such leakage.
Selecting features
Filter methods score each feature on its own, for example by correlation, an F-test (SelectKBest with f_regression) or mutual information, which also detects non-linear links; they are fast but ignore interactions. Wrapper methods such as recursive feature elimination train a model, drop the weakest feature and repeat, which is slower but considers features together. Embedded methods select during training: lasso sets weak weights to zero, and tree ensembles provide importances. Always choose features using training data only, ideally inside cross-validation, and confirm on a test set that the smaller model is at least as good. Highly correlated pairs, such as height in centimetres and in inches, add nothing new; keep one. In the code below, a linear model using the raw timestamp predicts delivery time with an average error of 5.1 minutes, while the engineered features cut it to 3.2 minutes. When ten columns of random noise are added, mutual information picks distance, preparation time, rain, hour and distance × rain as the top five, and none of the noise.
How it works
- Talk to domain experts and list ideas for features.
- Create features with code that uses only information available at prediction time.
- Compare a baseline model with and without the new features using cross-validation.
- Select features with filter, wrapper or embedded methods on training data.
- Confirm on the test set and document each feature's meaning.
The maths
An order placed at 19:42 on a Friday, 6 km from the restaurant, in the rain becomes hour = 19, weekday = 4 (counting Monday as 0), is_peak = 1 because 19:00 to 21:00 is the dinner rush, distance_km = 6, rain = 1 and distance × rain = 6. A linear model can now learn "peak hours add about 10 minutes" and "each rainy kilometre adds extra time", effects it could not learn from the raw timestamp. Suppose the model also receives 10 random columns of noise: a filter method that scores each column's link to delivery time ranks the real features at the top and the noise near zero, so the noise can be dropped.
In code
import numpy as np
import pandas as pd
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(0)
n = 2000
orders = pd.DataFrame({
"time": pd.Timestamp("2024-03-01") + pd.to_timedelta(rng.integers(0, 60 * 24 * 60, n), unit="min"),
"distance_km": rng.uniform(0.5, 10, n).round(1),
"prep_min": rng.integers(5, 30, n),
"rain": (rng.random(n) < 0.2).astype(int),
})
hour = orders.time.dt.hour
peak = hour.between(12, 13) | hour.between(19, 20)
orders["minutes"] = 10 + 3 * orders.distance_km + orders.prep_min + 10 * peak + 2.5 * orders.distance_km * orders.rain + rng.normal(0, 4, n)
raw = pd.DataFrame({"timestamp": orders.time.astype("int64") // 10**9, "distance_km": orders.distance_km, "prep_min": orders.prep_min, "rain": orders.rain})
eng = raw.drop(columns="timestamp").assign(hour=hour, is_peak=peak.astype(int), weekday=orders.time.dt.weekday, dist_x_rain=orders.distance_km * orders.rain)
for name, X in [("raw features", raw), ("engineered features", eng)]:
X_tr, X_te, y_tr, y_te = train_test_split(X, orders.minutes, random_state=0)
mae = mean_absolute_error(y_te, LinearRegression().fit(X_tr, y_tr).predict(X_te))
print(f"{name:<20} test MAE = {mae:.1f} minutes")
noisy = eng.join(pd.DataFrame(rng.normal(size=(n, 10)), columns=[f"noise{i}" for i in range(10)]))
X_tr, X_te, y_tr, y_te = train_test_split(noisy, orders.minutes, random_state=0)
selector = SelectKBest(mutual_info_regression, k=5).fit(X_tr, y_tr)
print("Top 5 features by mutual information:", list(X_tr.columns[selector.get_support()]))Output
raw features test MAE = 5.1 minutes engineered features test MAE = 3.2 minutes Top 5 features by mutual information: ['distance_km', 'prep_min', 'rain', 'hour', 'dist_x_rain']
Try it yourself
Predicting electricity bills
A power company wants to warn customers whose next bill will be unusually high.
Useful feature or leakage?
A model predicts whether a loan will default, at the time of application.
Years at current job
Monthly income divided by loan instalment
Debt-collection call made
Number of missed payments on this loan
Common mistakes
- Creating features from information that is not available at prediction time.
- Selecting features on the full data set before splitting.
- Adding hundreds of features without checking that they help.
Check yourself
Summary
- Feature engineering turns raw data into informative inputs using domain knowledge.
- Dates, ratios, flags, logs and aggregations are common, but every feature must be available at prediction time.
- Select features with filter, wrapper or embedded methods on training data, and confirm on a test set.
Finished reading?
Sources: Kuhn and Johnson, Feature Engineering and Selection (free online book) · scikit-learn: feature selection · Zheng and Casari, Feature Engineering for Machine Learning (O'Reilly)