Data Science in Banking and Finance: Production Lessons from the Trenches

Banking was one of the earliest adopters of data science, and it remains one of the most demanding. After nine years of building models for financial institutions — credit scoring systems processing millions of applications, fraud detection pipelines catching transactions in real time, and customer segmentation frameworks driving marketing strategy — I’ve accumulated hard-won lessons that the textbooks don’t cover.

Finance is different from other domains in ways that matter deeply for data science practitioners. The data is temporal by nature, the stakes are measured in millions of dollars, the regulatory environment constrains what you can do, and the class imbalance problems are extreme. This post distills the patterns that hold up across these challenges.

Credit Scoring: The Foundation

Credit scoring is the backbone of retail banking data science. Every loan application, every credit card approval, every line of credit increase — these decisions are informed by predictive models. The core problem is simple: predict whether a borrower will default. The execution is anything but.

Feature Engineering for Credit

The features that predict credit default are well-established, but getting them right requires careful engineering:

import pandas as pd
import numpy as np

def engineer_credit_features(applications: pd.DataFrame, bureau_data: pd.DataFrame) -> pd.DataFrame:
    """Engineer credit scoring features from application and bureau data."""
    features = applications[["application_id"]].copy()

    # === Debt-to-Income Ratio ===
    features["dti_ratio"] = (
        applications["total_monthly_debt_payments"]
        / applications["monthly_income"].clip(lower=1)
    )

    # === Bureau Utilization ===
    # Percentage of available credit being used
    features["credit_utilization"] = (
        bureau_data.groupby("customer_id")["current_balance"].sum()
        / bureau_data.groupby("customer_id")["credit_limit"].sum().clip(lower=1)
    )

    # === Payment History ===
    # Number of delinquent accounts in last 12 months
    features["delinquent_accounts_12m"] = bureau_data[
        bureau_data["days_past_due"] > 30
    ].groupby("customer_id").size()

    # === Credit Age ===
    # Months since oldest account opened
    features["credit_age_months"] = (
        pd.Timestamp.now() - bureau_data.groupby("customer_id")["open_date"].min()
    ).dt.days / 30

    # === Inquiry Velocity ===
    # Number of credit inquiries in last 6 months (high velocity = risky)
    features["inquiries_6m"] = bureau_data[
        (bureau_data["inquiry_date"] >= pd.Timestamp.now() - pd.Timedelta(days=180))
    ].groupby("customer_id").size()

    # === Stability Features ===
    features["employment_tenure_months"] = applications["employment_tenure_years"] * 12
    features["address_tenure_months"] = applications["address_tenure_years"] * 12

    return features

The most underappreciated feature in credit scoring is velocity. Not just “how much debt do you have” but “how fast are you accumulating it?” A customer who opened 3 new credit accounts in the last 6 months is fundamentally different from one who has had the same 3 accounts for 5 years, even if their current balances are identical.

Temporal Leakage: The Silent Killer

Temporal leakage is the most dangerous pitfall in credit modeling, and it’s shockingly easy to introduce:

# WRONG — temporal leakage
# Using features calculated AFTER the application date
def bad_feature_engineering(applications, transactions):
    """This will give you great backtest results and terrible production performance."""
    # These transactions happen AFTER the application decision
    future_transactions = transactions[
        transactions["date"] > applications["application_date"]
    ]
    features["future_spending"] = future_transactions.groupby("customer_id")["amount"].sum()
    return features  # Looks great in validation! Fails in production!

# RIGHT — point-in-time features
def good_feature_engineering(applications, transactions):
    """Only use information available at the time of application."""
    features = []
    for _, app in applications.iterrows():
        customer_txns = transactions[
            (transactions["customer_id"] == app["customer_id"])
            & (transactions["date"] <= app["application_date"])
        ]
        feature_row = {
            "application_id": app["application_id"],
            "txn_count_30d": len(customer_txns[customer_txns["date"] >= app["application_date"] - pd.Timedelta(days=30)]),
            "avg_balance_90d": customer_txns[
                customer_txns["date"] >= app["application_date"] - pd.Timedelta(days=90)
            ]["balance"].mean(),
        }
        features.append(feature_row)
    return pd.DataFrame(features)

The rule is absolute: every feature must be calculable using only information available at the moment the prediction would be made in production. This means no future transactions, no bureau data updated after the application date, no outcome-informed features.

I’ve seen teams achieve 0.95 AUC in validation that dropped to 0.65 in production because of subtle temporal leakage. The most insidious form: using a feature derived from a dataset that’s updated nightly, where the nightly batch includes data from the same day as the application but after the decision point.

Reject Inference

A unique challenge in credit scoring: you only observe outcomes for applicants who were approved. Rejected applicants — who might have been good or bad borrowers — are invisible. This creates a biased training dataset.

def reject_inference_fuzzy_augmentation(
    approved_model,
    rejected_applications: pd.DataFrame,
    approved_labels: pd.Series,
    n_iterations: int = 5,
) -> tuple:
    """Soft-label reject inference using model predictions."""
    # Score rejected applications
    rejected_probs = approved_model.predict_proba(rejected_applications)[:, 1]

    # Assign fuzzy labels based on model probability
    # Don't hard-assign — use probabilistic weighting
    rejected_weight_good = 1 - rejected_probs
    rejected_weight_bad = rejected_probs

    # Iteratively retrain with weighted rejected samples
    for iteration in range(n_iterations):
        # Create augmented training set
        X_augmented = pd.concat([
            approved_features,
            rejected_applications,
        ])
        y_augmented = pd.concat([
            approved_labels,
            pd.Series(np.where(rejected_probs > 0.5, 1, 0)),  # Soft labels
        ])
        sample_weights = np.concatenate([
            np.ones(len(approved_labels)),
            np.where(rejected_probs > 0.5, rejected_probs, 1 - rejected_probs),
        ])

        # Retrain
        model = lgb.LGBMClassifier()
        model.fit(X_augmented, y_augmented, sample_weight=sample_weights)

        # Update rejected predictions
        rejected_probs = model.predict_proba(rejected_applications)[:, 1]

    return model, rejected_probs

Reject inference is controversial. Purists argue you shouldn’t train on synthetic labels. Pragmatists argue that ignoring the rejected population creates a model that only works for the population it’s seen. I fall in the pragmatist camp, but with guardrails: always compare models with and without reject inference, and monitor for instability in the rejected population scoring.

Fraud Detection: Real-Time ML Under Extreme Imbalance

The Scale of Imbalance

In banking fraud detection, you’re typically looking at 0.01-0.1% fraud rate. That’s not just imbalanced — it’s extreme. A model that predicts “not fraud” for every transaction achieves 99.9% accuracy.

from sklearn.metrics import precision_recall_curve, average_precision_score
import lightgbm as lgb

def train_fraud_detector(transactions: pd.DataFrame, labels: pd.Series):
    """Train fraud detection model with proper handling of extreme imbalance."""
    # Calculate class weights
    n_negative = (labels == 0).sum()
    n_positive = (labels == 1).sum()
    scale_pos_weight = n_negative / n_positive

    params = {
        "objective": "binary",
        "metric": "average_precision",  # PR-AUC, not ROC-AUC
        "learning_rate": 0.05,
        "num_leaves": 63,
        "min_child_samples": 100,
        "scale_pos_weight": scale_pos_weight,
        "subsample": 0.8,
        "colsample_bytree": 0.8,
        "verbose": -1,
    }

    # Use stratified k-fold
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

    models = []
    for train_idx, val_idx in cv.split(transactions, labels):
        train_data = lgb.Dataset(
            transactions.iloc[train_idx],
            label=labels.iloc[train_idx],
        )
        val_data = lgb.Dataset(
            transactions.iloc[val_idx],
            label=labels.iloc[val_idx],
        )

        model = lgb.train(
            params,
            train_data,
            num_boost_round=2000,
            valid_sets=[val_data],
            callbacks=[lgb.early_stopping(100)],
        )
        models.append(model)

    return models

Use PR-AUC (Average Precision), not ROC-AUC, for fraud detection evaluation. ROC-AUC is misleading with extreme class imbalance — a model can achieve high ROC-AUC while having terrible precision at the operating threshold. PR-AUC focuses on the minority class performance that actually matters.

Feature Engineering for Fraud

Fraud features need to capture behavioral anomalies — deviations from a customer’s normal pattern:

def engineer_fraud_features(
    transaction: dict,
    customer_history: pd.DataFrame,
    merchant_stats: pd.DataFrame,
) -> dict:
    """Engineer real-time fraud features for a single transaction."""
    features = {}

    # === Velocity Features ===
    # Transactions in last N minutes/hours
    for window in [5, 30, 60, 1440]:  # minutes
        window_txns = customer_history[
            customer_history["timestamp"] >= transaction["timestamp"] - pd.Timedelta(minutes=window)
        ]
        features[f"txn_count_{window}m"] = len(window_txns)
        features[f"txn_amount_sum_{window}m"] = window_txns["amount"].sum()

    # === Behavioral Deviation ===
    # Z-score of current transaction amount vs. customer's history
    customer_mean = customer_history["amount"].mean()
    customer_std = customer_history["amount"].std()
    features["amount_zscore"] = (
        (transaction["amount"] - customer_mean) / max(customer_std, 1.0)
    )

    # === Geographic Anomaly ===
    # Distance from customer's typical location
    typical_lat = customer_history["latitude"].median()
    typical_lon = customer_history["longitude"].median()
    features["distance_from_typical_km"] = haversine_distance(
        transaction["latitude"], transaction["longitude"],
        typical_lat, typical_lon,
    )

    # === Time Anomaly ===
    # Is this transaction outside the customer's typical hours?
    typical_hours = customer_history["timestamp"].dt.hour
    features["hour"] = transaction["timestamp"].hour
    features["is_unusual_hour"] = int(
        features["hour"] < typical_hours.quantile(0.05)
        or features["hour"] > typical_hours.quantile(0.95)
    )

    # === Merchant Risk ===
    features["merchant_fraud_rate"] = merchant_stats.loc[
        transaction["merchant_id"], "fraud_rate"
    ]

    return features

The most predictive fraud features are behavioral deviations, not absolute values. A $5,000 transaction isn't inherently suspicious — it is if the customer's average transaction is$50. A transaction at 3 AM isn’t suspicious — it is if the customer has never transacted outside 8 AM-10 PM.

Real-Time Scoring Architecture

Fraud detection must happen in real time — before the transaction is authorized. This means sub-100ms scoring latency:

import redis
import numpy as np

class RealTimeFraudScorer:
    """Production fraud scoring with Redis-backed feature store."""

    def __init__(self, model, redis_client: redis.Redis):
        self.model = model
        self.redis = redis_client
        self.feature_ttl = 86400 * 30  # 30 days

    def score(self, transaction: dict) -> dict:
        """Score a transaction in real time."""
        customer_id = transaction["customer_id"]

        # Get customer history from feature store
        history_key = f"customer:{customer_id}:transactions"
        history = self._get_customer_history(history_key)

        # Calculate features (must be fast — <10ms)
        features = self._calculate_features(transaction, history)

        # Score
        fraud_probability = self.model.predict_proba(
            np.array(features).reshape(1, -1)
        )[0, 1]

        # Decision logic
        decision = "approve"
        if fraud_probability > 0.8:
            decision = "decline"
        elif fraud_probability > 0.3:
            decision = "review"  # Queue for human review

        # Update history (after decision, before returning)
        self._update_history(history_key, transaction)

        return {
            "fraud_probability": fraud_probability,
            "decision": decision,
            "features": features,
        }

The Redis-backed feature store is critical. You can’t query a database for each transaction — you need features available in milliseconds. The trade-off is that your feature store needs to be updated after each transaction, which introduces eventual consistency. In practice, a few seconds of latency in feature updates is acceptable for fraud detection.

Customer Segmentation

Beyond RFM

Traditional RFM (Recency, Frequency, Monetary) segmentation is a starting point, but modern banking demands richer segmentation:

from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

def advanced_segmentation(customer_data: pd.DataFrame, n_segments: int = 6):
    """Multi-dimensional customer segmentation."""
    features = customer_data[[
        # Transaction behavior
        "avg_monthly_txn_count",
        "avg_monthly_spend",
        "spend_volatility",
        # Product penetration
        "num_products",
        "has_mortgage",
        "has_investment",
        # Digital engagement
        "mobile_logins_per_month",
        "digital_txn_pct",
        # Value metrics
        "total_relationship_value",
        "revenue_12m",
    ]].copy()

    # Standardize
    scaler = StandardScaler()
    features_scaled = scaler.fit_transform(features.fillna(0))

    # Dimensionality reduction for visualization
    pca = PCA(n_components=2)
    features_2d = pca.fit_transform(features_scaled)

    # Clustering
    kmeans = KMeans(n_clusters=n_segments, random_state=42, n_init=10)
    segments = kmeans.fit_predict(features_scaled)

    # Profile segments
    customer_data["segment"] = segments
    segment_profiles = customer_data.groupby("segment").agg({
        "avg_monthly_spend": "mean",
        "num_products": "mean",
        "revenue_12m": "mean",
        "digital_txn_pct": "mean",
    })

    return customer_data, segment_profiles

The key insight from banking segmentation: the most valuable segments are not always the highest spenders. Customers with moderate spending but high digital engagement and multiple products often have the highest lifetime value because they’re cheaper to serve and more loyal.

Explainability: Regulatory Requirement, Not Nice-to-Have

In banking, model explainability isn’t optional. Regulators require it. Customers have a right to know why they were denied credit. And your risk management team needs to understand what the model is doing.

import shap

def explain_credit_decision(model, customer_features: pd.DataFrame, customer_idx: int):
    """Generate explanation for a single credit decision."""
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(customer_features.iloc[[customer_idx]])

    # Get feature contributions
    feature_importance = pd.DataFrame({
        "feature": customer_features.columns,
        "value": customer_features.iloc[customer_idx].values,
        "shap_value": shap_values[1][0],  # For fraud/not-fraud class
    }).sort_values("shap_value", ascending=False)

    # Generate human-readable explanation
    top_positive = feature_importance.head(3)
    top_negative = feature_importance.tail(3)

    explanation = {
        "decision": "denied" if model.predict(customer_features.iloc[[customer_idx]])[0] == 1 else "approved",
        "confidence": float(model.predict_proba(customer_features.iloc[[customer_idx]])[0, 1]),
        "top_risk_factors": [
            {"feature": row["feature"], "impact": row["shap_value"], "value": row["value"]}
            for _, row in top_positive.iterrows()
        ],
        "top_mitigating_factors": [
            {"feature": row["feature"], "impact": row["shap_value"], "value": row["value"]}
            for _, row in top_negative.iterrows()
        ],
    }

    return explanation

SHAP values are the industry standard for model explainability in finance. They provide consistent, theoretically grounded feature attributions. But be careful: SHAP explains the model, not the world. A feature with high SHAP importance doesn’t necessarily cause the outcome — it’s correlated with the model’s prediction.

Model Monitoring in Production

Finance models degrade faster than most because the underlying data distribution shifts with economic conditions:

def monitor_model_drift(
    reference_predictions: np.ndarray,
    production_predictions: np.ndarray,
    reference_features: pd.DataFrame,
    production_features: pd.DataFrame,
) -> dict:
    """Monitor for model drift in production."""
    from scipy.stats import ks_2samp

    drift_metrics = {}

    # Prediction drift
    ks_stat, p_value = ks_2samp(reference_predictions, production_predictions)
    drift_metrics["prediction_drift_ks"] = ks_stat
    drift_metrics["prediction_drift_p"] = p_value

    # Feature drift
    feature_drift = {}
    for col in reference_features.columns:
        ks_stat, p_value = ks_2samp(
            reference_features[col].dropna(),
            production_features[col].dropna(),
        )
        feature_drift[col] = {"ks_stat": ks_stat, "p_value": p_value}

    drift_metrics["feature_drift"] = feature_drift

    # Alert conditions
    drift_metrics["alerts"] = []
    if ks_stat > 0.1:
        drift_metrics["alerts"].append("PREDICTION_DRIFT")
    for col, metrics in feature_drift.items():
        if metrics["ks_stat"] > 0.15:
            drift_metrics["alerts"].append(f"FEATURE_DRIFT_{col}")

    return drift_metrics

Set drift thresholds conservatively. In credit scoring, a model that was trained during an economic expansion will systematically underpredict defaults during a recession. You need to detect this shift and trigger a model review before it causes material losses.

Conclusion

Data science in banking and finance demands a unique combination of technical skill, regulatory awareness, and business understanding. The temporal nature of financial data, the extreme class imbalance in fraud detection, the regulatory requirements for explainability, and the real-time performance constraints make this one of the most challenging domains for ML practitioners.

The patterns I’ve described — point-in-time feature engineering, PR-AUC-driven fraud model evaluation, behavioral deviation features, SHAP-based explainability, and continuous drift monitoring — are the foundation of production-grade financial ML systems. But the most important lesson is this: in finance, the cost of a wrong prediction is directly measurable in dollars. This clarity is both a blessing and a curse. A blessing because it forces rigorous evaluation. A curse because it means every mistake has a price tag.

Build your models with the assumption that they will be audited, challenged, and required to explain themselves. If you can do that, you’re building something that will survive in production.