Feature Engineering for Production ML: Feature Stores, Point-in-Time Correctness, and the Kaggle Gap

Feature engineering is where most of the value in machine learning comes from. I’ve seen teams spend months tuning model architectures while neglecting the features that feed those models. After nine years of building production ML systems, I can say with confidence that a well-featured logistic regression will outperform a poorly-featured deep neural network in most real-world scenarios. This post covers the practical aspects of feature engineering for production systems — the stuff that separates Kaggle notebooks from systems that serve millions of predictions daily.

The Kaggle vs. Production Gap

What Kaggle Gets Right

Kaggle competitions teach excellent feature engineering habits. Competitors spend 80% of their time on features and 20% on model tuning, which is roughly the right ratio. Kaggle also teaches you to think creatively about features — combining columns, extracting date parts, creating interaction terms, and engineering domain-specific signals.

What Kaggle Gets Wrong

Kaggle’s feature engineering is fundamentally different from production in three ways:

Static data: Kaggle gives you a fixed dataset. You compute features once and train on them. In production, features must be computed continuously — for training (from historical data) and for serving (from real-time data). The computation must be identical in both cases, or you get training-serving skew.
No time dimension: Kaggle datasets are typically snapshots. Production data has a temporal dimension — features must be computed as they would have been at the time of prediction, not with future information. This is point-in-time correctness, and violating it is the most common source of data leakage in production ML.
Unlimited compute: Kaggle notebooks can spend hours computing features for a static dataset. Production feature pipelines must compute features within latency budgets (milliseconds for real-time serving) and cost budgets (terabytes of data processed daily).

Feature Stores: The Missing Infrastructure

What Is a Feature Store?

A feature store is a centralized platform for storing, managing, and serving ML features. It bridges the gap between feature engineering (typically done by data scientists in notebooks) and feature serving (needed by production models in real-time).

The core capabilities of a feature store:

Feature registration: A central catalog of all features, with metadata (owner, description, data type, freshness SLA).
Feature computation: A way to compute features from raw data, both in batch (Spark jobs) and real-time (stream processing).
Feature storage: Efficient storage of computed features, optimized for both batch training (Parquet files) and real-time serving (Redis/DynamoDB).
Point-in-time joins: The ability to retrieve feature values as they were at a specific point in time, preventing data leakage.
Feature serving: Low-latency retrieval of feature values for online prediction.

My Feature Store Journey

I’ve used three feature stores in production: Feast, Tecton, and a custom-built solution. Here’s my honest assessment:

Feast (open-source):

Pros: Simple, lightweight, good for teams starting with feature stores. Supports both batch and online serving.
Cons: Limited real-time feature support. The online store is basic (Redis/DynamoDB). You’ll outgrow it if you need sophisticated streaming features.
My recommendation: Start here if you’re building your first feature store.

Tecton (managed):

Pros: Excellent real-time feature support, managed infrastructure, sophisticated point-in-time joins.
Cons: Expensive, vendor lock-in, steep learning curve.
My recommendation: Use if you have the budget and need sophisticated real-time features at scale.

Custom-built:

Pros: Exactly fits your needs, no vendor lock-in.
Cons: Significant engineering investment (3-6 months for a usable system), ongoing maintenance burden.
My recommendation: Build custom only if your requirements are truly unique and no existing solution fits.

Feature Store Architecture

Raw Data (S3/Kafka)
    ↓
Feature Computation (Spark batch / Flink streaming)
    ↓
Offline Store (Parquet on S3) ← for training
Online Store (Redis)          ← for serving
    ↓
Feature API (gRPC/REST)       ← for model inference

Point-in-Time Correctness: The Silent Killer

What Is Point-in-Time Correctness?

Point-in-time correctness means that when you compute features for a training example, you only use data that was available at the time the prediction would have been made. This sounds obvious, but it’s violated constantly in practice.

A Concrete Example

Suppose you’re building a churn prediction model. You want to predict whether a user will churn in the next 30 days. Your features include “number of support tickets in the last 90 days.”

Wrong (data leakage): Compute the feature using the current number of support tickets. If you’re training on historical data from January 2025, you’d be using support ticket data from October 2024 to January 2025 — but some of those tickets were filed after the prediction date.

Right (point-in-time correct): Compute the feature using only support tickets that existed at the prediction date. For a January 1, 2025 prediction, use tickets from October 2, 2024 to January 1, 2025.

How to Implement Point-in-Time Correctness

The key is to always join features using a timestamp:

# WRONG: Current feature values
features = user_features_table.join(user_id)

# RIGHT: Point-in-time correct feature values
features = point_in_time_join(
    entity_df=predictions_df,  # columns: user_id, prediction_timestamp
    feature_df=user_features_table,
    entity_col="user_id",
    event_timestamp_col="feature_timestamp",
    prediction_timestamp_col="prediction_timestamp",
)

In Feast, this is built in:

training_df = store.get_historical_features(
    entity_df=training_entities,  # DataFrame with entity columns + event_timestamp
    features=[
        "user_features:total_purchases",
        "user_features:days_since_last_visit",
        "user_features:avg_session_duration",
    ],
).to_df()

The entity_df must include an event_timestamp column that specifies when each prediction would have been made. Feast retrieves the feature values that were most recently computed before that timestamp.

Common Point-in-Time Violations

Using future data: The most obvious violation. Features computed after the prediction timestamp.
Using aggregated features without temporal bounds: “Total purchases” should be “total purchases as of the prediction date,” not “total purchases to date.”
Using target-derived features: Features that are computed from the target variable or its derivatives. For example, “average order value in the 30 days after prediction” — this is literally using the future.
Leaking through joins: Joining a table that contains future information. For example, joining user data with product data where the product data includes future pricing.

Data Leakage: Detection and Prevention

Types of Data Leakage

Target leakage: Features that are computed using information from the target variable. Example: Using “returned the product” as a feature to predict “will return the product.”

Train-test contamination: Information from the test set leaking into the training set. Example: Normalizing features using statistics computed across the entire dataset (including test data).

Temporal leakage: Using future information to make predictions about the past. This is the point-in-time correctness violation described above.

Detection Strategies

Feature importance analysis: If a feature has unexpectedly high importance, investigate why. A feature that perfectly predicts the target is suspicious — it might be leaking.
Single-feature AUC: Compute the AUC of each feature individually against the target. Features with AUC > 0.95 are suspicious — they’re too good to be true.
Time-based validation: Train on data before a cutoff date and evaluate on data after. If performance drops dramatically compared to random train-test split, you likely have temporal leakage.
Feature correlation with target over time: Plot the correlation between each feature and the target over time. If the correlation drops sharply when you add a temporal gap between feature computation and target observation, the feature is likely leaked.

Prevention Strategies

Use a feature store with point-in-time joins. This is the most reliable prevention mechanism.
Implement strict train-test splits based on time. Never use random splits for time-series data.
Code review feature engineering logic. Every feature should have a clear definition of what data it uses and from what time window.
Automated leakage detection in CI/CD. I run a leakage detection script as part of the model training CI/CD pipeline. It checks for suspicious feature-target correlations and flags them for review.

Automated Feature Generation

Why Automate?

Manual feature engineering is powerful but slow. For a new problem, it can take weeks to engineer a good feature set. Automated feature generation can produce a baseline feature set in hours, which you then refine manually.

Featuretools and Deep Feature Synthesis

Featuretools is the most mature automated feature generation library. It uses Deep Feature Synthesis (DFS) to automatically generate features by applying aggregation and transformation primitives to a relational dataset.

import featuretools as ft

# Define the entity set
es = ft.EntitySet("customer_data")
es = es.add_dataframe(
    dataframe_name="customers",
    dataframe=customers_df,
    index="customer_id",
)
es = es.add_dataframe(
    dataframe_name="transactions",
    dataframe=transactions_df,
    index="transaction_id",
    time_index="transaction_date",
)
es = es.add_relationship("customers", "customer_id", "transactions", "customer_id")

# Generate features
feature_matrix, feature_defs = ft.dfs(
    entityset=es,
    target_dataframe_name="customers",
    max_depth=2,
    agg_primitives=["count", "sum", "mean", "std", "min", "max"],
    trans_primitives=["month", "weekday", "is_weekend"],
)

DFS generates features like:

COUNT(transactions) — number of transactions per customer
SUM(transactions.amount) — total spend per customer
MEAN(transactions.amount) — average transaction amount
STD(transactions.amount) — transaction amount variability
MONTH(transactions.transaction_date) — month of most recent transaction

Production Considerations for Automated Features

Automatically generated features are great for exploration but require careful vetting for production:

Interpretability: Can you explain what this feature means? A feature like MEAN(SUM(SUM(transactions.amount))) is meaningless. I filter out features with nesting depth > 2.
Computability: Can this feature be computed in real-time? Features that require scanning a user’s entire transaction history are expensive to compute at serving time. I filter for features that can be computed from pre-aggregated statistics.
Leakage risk: Automated features can inadvertently create leaked features. I run the leakage detection pipeline on all auto-generated features before including them.
Feature count: DFS can generate thousands of features. Most are useless. I use feature selection (mutual information, recursive feature elimination) to keep only the top 100-200 features.

Feature Engineering Patterns for Production

Pattern 1: Multi-Window Aggregates

Compute aggregates over multiple time windows (1h, 24h, 7d, 30d, 90d). This captures both short-term and long-term behavior:

def compute_multi_window_features(events_df, windows=['1h', '24h', '7d', '30d', '90d']):
    features = {}
    for window in windows:
        window_events = filter_by_window(events_df, window)
        features[f'count_{window}'] = len(window_events)
        features[f'sum_{window}'] = window_events['amount'].sum()
        features[f'mean_{window}'] = window_events['amount'].mean()
    return features

Production tip: Pre-compute the 90d aggregates and derive shorter windows from them. This reduces computation cost — you only need to aggregate once, not five times.

Pattern 2: Ratio and Rate Features

Ratios are often more predictive than raw counts:

Conversion rate (purchases / visits)
Return rate (returns / purchases)
Engagement rate (actions / sessions)
Growth rate (current period / previous period)

Pattern 3: Recency Features

“Days since last X” is consistently one of the most predictive feature types:

Days since last visit
Days since last purchase
Days since last support ticket
Days since last email open

These features capture the decay of customer engagement over time.

Pattern 4: Embedding Features

For high-cardinality categorical variables (product IDs, user IDs, content IDs), embeddings are more effective than one-hot encoding:

# Train embeddings on user-product interaction data
user_embeddings = train_embeddings(interactions_df, dim=64)
product_embeddings = train_embeddings(interactions_df, dim=64)

# Use embeddings as features
features['user_embedding'] = user_embeddings[user_id]
features['product_embedding'] = product_embeddings[product_id]
features['user_product_similarity'] = cosine_similarity(
    user_embeddings[user_id], product_embeddings[product_id]
)

Pattern 5: Temporal Features

Time-based features capture cyclical patterns:

Hour of day (sine/cosine encoding)
Day of week
Month of year
Is holiday
Days until next holiday
Season

Important: Use sine/cosine encoding for cyclical features, not raw numbers. Hour 23 and hour 0 are adjacent, but raw encoding makes them seem far apart.

Feature Engineering for Different Model Types

Tree-Based Models (LightGBM, XGBoost)

Tree-based models are robust to feature scaling and can handle missing values natively. Focus on:

Interaction features (feature A × feature B)
Ratio features
Bucketed continuous features (though trees handle continuous features well)
High-cardinality categoricals (use target encoding or catboost encoding)

Neural Networks

Neural networks are sensitive to feature scaling and require careful preprocessing:

Normalize continuous features (standard scaling or min-max scaling)
Use embeddings for categoricals
Provide interaction features explicitly (NNs can learn interactions, but explicit features help)
Handle missing values with a mask feature (1 = present, 0 = missing)

Linear Models (Logistic Regression, Linear SVM)

Linear models require the most feature engineering:

Polynomial features for non-linear relationships
Bucketed continuous features (decile bins)
One-hot encoding for categoricals (limit to top-N to avoid dimensionality explosion)
Feature scaling is critical

Lessons Learned

Lesson 1: Feature Engineering Is Iterative

Your first feature set will be mediocre. That’s fine. The process is: engineer features → train model → analyze errors → engineer features to address those errors → repeat. Each iteration should focus on the model’s biggest weaknesses.

Lesson 2: Document Every Feature

Every feature should have a documented definition: what it means, how it’s computed, what data it uses, and what time window it covers. Without documentation, features become tribal knowledge and eventually get recomputed incorrectly.

Lesson 3: Monitor Feature Distributions

Feature distributions shift over time. A feature that was highly predictive six months ago might be useless today. I monitor feature distributions weekly using PSI and investigate any feature with PSI > 0.2.

Lesson 4: The Feature Store Pays for Itself

The initial investment in a feature store is significant. But the payoff — reduced training-serving skew, feature reuse across models, point-in-time correctness, and centralized feature documentation — pays for itself within 6 months for any team with more than 3 models in production.

Lesson 5: Don’t Over-Engineer

Start simple. Use basic aggregates (count, sum, mean) and temporal features (recency, frequency). Add complexity only when the simple features aren’t sufficient. I’ve seen teams engineer hundreds of exotic features when 20 well-chosen simple features would have performed just as well.

Conclusion

Feature engineering for production ML is a discipline that requires understanding both the data science (what features are predictive?) and the engineering (how do we compute and serve features reliably?). The feature store is the central piece of infrastructure that bridges these two worlds. Invest in point-in-time correctness to prevent data leakage, automate feature generation to accelerate iteration, and document everything to maintain institutional knowledge. The model matters, but the features matter more.