Building Production Rank Systems: LTR by CL and LambdaMart

1. Business Problem Introduction

Indonesian Ecommerce Production Learning-to-Rank Recommendation System Using Collaborative Filtering + LambdaMART, Engineering blueprint for building a production-grade LTR recommendation platform with LightGBM, MLflow experiment tracking, SHAP explainability, and real-time monitoring at Indonesian e-commerce scale.

The Tokobaju.id Ranking Quality Crisis

Tokobaju.id is an Indonesian fashion e-commerce platform with 5.2 million monthly active users and a catalog of 320,000 SKUs across fashion, accessories, and lifestyle products — predominantly sourced from Indonesian SME sellers. The platform launched in 2021 with a pure collaborative filtering engine (ALS matrix factorization) for homepage personalization and search result ranking.

By mid-2023, however, a critical quality problem emerged: the recommendation system was generating high-recall but low-precision results. Users were seeing relevant product categories, but the specific items surfaced were poorly ordered — a Rp 12,000 unbranded t-shirt was being ranked above a Rp 89,000 premium local brand item, despite the latter having a substantially higher historical conversion rate among users with identical behavioral profiles. Conversion rate from recommendation surfaces dropped from 3.1% to 2.2% over six months, directly correlated with catalog expansion from 80,000 to 320,000 SKUs.

The root cause: ALS matrix factorization ranks items by their dot-product similarity score, which captures user-item affinity well but is completely blind to item-level quality signals: return rates, review scores, seller reputation, stock availability, discount magnitude, and category-specific conversion rates. A 4× expansion in catalog size amplified this blind spot from a manageable edge case into a systematic ranking failure.

Business Impact Assessment

Problem	Metric Impact	Monthly Revenue Risk
Low-quality items ranked above high-quality ones	CVR: 3.1% → 2.2%	Rp 4.2B/month
No seller quality signal in ranking	Return rate 18% above category benchmark	Customer satisfaction erosion
Popularity bias favors large sellers	SME seller visibility < 4%	Regulatory risk (marketplace fairness)
No discount/promotion integration	High-margin items deprioritized	GMV quality deterioration
Catalog expansion without ranking improvement	Discovery CTR drops 31%	Long-tail catalog ROI = negative

Key Performance Indicators

KPI	Baseline (ALS Only)	LTR Target	Business Value
Homepage CVR	2.2%	3.8%+	Direct GMV impact
Category CTR	3.4%	5.1%+	Engagement improvement
Return Rate from Recommendations	12.3%	< 9.0%	Operational cost reduction
SME Seller Visibility	4.1%	8.0%+	Platform fairness
NDCG@10 (Offline)	0.089	0.131+	Ranking quality
Revenue Per Recommendation Session	Rp 42,000	Rp 68,000+	Session value

The solution architecture: a two-stage LTR pipeline where ALS collaborative filtering generates a candidate pool (top 500 items per user), and a LightGBM LambdaMART ranker re-scores those candidates using a rich 87-dimensional feature vector — integrating user behavioral features, item quality signals, and user-item interaction features into a listwise ranking model that directly optimizes NDCG.

2. Recommendation Solution Evolution

2.1 Stage 1 Baseline: ALS Collaborative Filtering (Candidate Generation)

ALS remains the backbone of candidate generation due to its excellent recall properties and computational efficiency. However, ALS scoring serves only one purpose in the new architecture: maximizing recall diversity in the candidate pool. Precision is the job of the ranker.

ALS candidate generation trade-offs:

✅ High recall (retrieves 85%+ of relevant items in top-500 pool)
✅ Computationally efficient at serving time (pre-computed embeddings + FAISS ANN)
❌ No item quality signals
❌ No promotion/discount awareness
❌ Ranking precision degraded at large catalog scales

2.2 The Ranking Precision Gap

The fundamental insight motivating LTR: ALS can find the right set of items, but ranks them incorrectly within that set. This is measurable:

ALS Recall@500 = 0.847  (87% of ground-truth purchased items in top-500 pool)
ALS NDCG@10   = 0.089   (but ranking within that pool is poor)

LTR (re-ranking top-500) NDCG@10 = 0.131  (47% improvement in ranking quality)

The recall-precision decomposition reveals that retrieval quality is not the bottleneck — ranking quality is. This is the definitive signal that a LTR re-ranker will deliver significant business value.

2.3 Solution Evolution Ladder

Stage	Model	Role	Responsible for
0	Popularity Ranking	Baseline	Global relevance
1	ALS Matrix Factorization	Candidate Generation	User-item affinity, recall
2	Feature Engineering	Feature Pipeline	Signal aggregation
3	LightGBM LambdaMART	Re-Ranking	Precision, quality ordering
4	SHAP Monitoring	Explainability	Feature drift, regulatory
5	MLflow + Drift Detection	MLOps	Production health

2.4 Why LambdaMART Over Pointwise/Pairwise LTR

Pointwise LTR (regression/classification): treats each item independently. Ignores inter-item relationships within the ranked list. Cannot directly optimize NDCG.

Pairwise LTR (RankNet, RankBoost): considers item pairs (i+, i-). Better than pointwise, but optimizes pair-level accuracy rather than list-level quality. O(n²) pairs can be computationally expensive.

Listwise LTR (LambdaMART): directly optimizes list-level NDCG through surrogate λ-gradients. The LambdaMART loss gradient:

λᵢⱼ = ∂C/∂sᵢ = -|ΔNDCG(i,j)| × σ(sᵢ - sⱼ) / (1 + eˢⁱ⁻ˢʲ)

Where:
  |ΔNDCG(i,j)| = change in NDCG if documents i and j are swapped
  sᵢ, sⱼ = scores for documents i, j
  σ = sigmoid function

The key insight: λ gradients are weighted by |ΔNDCG(i,j)|,
so swaps that would most improve the NDCG of the ranked list
receive the strongest gradient signal.

LightGBM’s advantage over XGBoost for LTR at scale:

Leaf-wise tree growth vs. level-wise: faster convergence per iteration
Gradient-based one-side sampling (GOSS): 40–60% training speedup with minimal accuracy loss
Exclusive feature bundling (EFB): handles high-dimensional sparse features efficiently
Built-in parallel group processing for LTR query groups

3. Recommendation System Workflow Diagram

flowchart TD
    subgraph INGESTION["📡 Data Ingestion (Kafka → DuckDB)"]
        I1["User Events\nclick · view · purchase · dwell"]
        I2["Item Catalog Events\nprice change · stock · review"]
        I3["Seller Events\nreputation · fulfillment rate"]
        I1 & I2 & I3 --> I4["DuckDB Analytical Store\nPartitioned by date"]
    end

    subgraph FEATURE_PIPELINE["⚙️ Feature Engineering Pipeline"]
        I4 --> F1["User Feature Builder\n47 features · RFM · behavioral"]
        I4 --> F2["Item Feature Builder\n28 features · quality · freshness"]
        I4 --> F3["Interaction Feature Builder\n12 features · affinity · session"]
        F1 & F2 & F3 --> F4[("Feature Store\nRedis + Parquet\n87 features total")]
    end

    subgraph STAGE1["🔍 Stage 1: Candidate Generation"]
        F4 --> G1["ALS Matrix Factorization\nTop-500 candidates per user"]
        F4 --> G2["Item-CF Fallback\nFor sparse users"]
        G1 & G2 --> G3["Candidate Pool\nmax 500 items per user"]
    end

    subgraph STAGE2["🎯 Stage 2: LambdaMART Re-Ranking"]
        G3 --> R1["Feature Lookup\nFor 500 candidates × 87 features"]
        F4 --> R1
        R1 --> R2["LightGBM LambdaMART\nNDCG optimization"]
        R2 --> R3["Re-ranked Top-20\nWith SHAP explanations"]
    end

    subgraph MLOPS_LOOP["🔄 MLOps & Feedback Loop"]
        R3 --> M1["Serving API\nFastAPI < 90ms"]
        M1 --> M2["User Interaction\nClick · Purchase · Return"]
        M2 --> M3["MLflow Logging\nMetrics + Artifacts"]
        M3 --> M4{"PSI / KS Test\nData Drift?"}
        M4 -->|"Drift > 0.2"| M5["Trigger Retraining\nGitHub Actions"]
        M4 -->|"Stable"| M6["Continue Serving"]
        M5 --> STAGE2
        M3 --> M7["SHAP Monitor\nFeature importance drift"]
    end

    style INGESTION fill:#0a0e1a,stroke:#1e40af,color:#e2e8f0
    style FEATURE_PIPELINE fill:#0a0e1a,stroke:#065f46,color:#e2e8f0
    style STAGE1 fill:#0a0e1a,stroke:#7c3aed,color:#e2e8f0
    style STAGE2 fill:#0a0e1a,stroke:#b45309,color:#e2e8f0
    style MLOPS_LOOP fill:#0a0e1a,stroke:#be123c,color:#e2e8f0

Diagram Walkthrough: Three event streams feed into DuckDB. The feature pipeline constructs 87 features across three families. Stage 1 (ALS) generates 500 candidates per user. Stage 2 (LightGBM LambdaMART) re-ranks those candidates using the full feature vector. The MLOps loop monitors for feature drift using PSI/KS tests and triggers retraining via GitHub Actions when drift exceeds the 0.2 PSI threshold.

4. System Design Architecture

4.1 Two-Stage LTR Architecture

flowchart LR
    subgraph USER["User Request"]
        U["user_id · context · session"]
    end

    subgraph STAGE_1["Stage 1: ANN Retrieval"]
        S1A["FAISS Index\nPre-computed ALS embeddings"]
        S1B["Top-500 Candidates\n< 20ms retrieval"]
    end

    subgraph STAGE_2["Stage 2: LTR Re-Ranking"]
        S2A["Feature Assembly\nBatch lookup · Redis + Parquet"]
        S2B["LightGBM LambdaMART\n87 features · < 35ms"]
        S2C["Top-20 Results\nWith scores + SHAP"]
    end

    subgraph FEATURE_STORE["Feature Store"]
        FS1[("Redis\nUser real-time features")]
        FS2[("Parquet on S3\nItem static features")]
    end

    subgraph MODEL_STORE["Model Store"]
        MS1["MLflow Registry\nLightGBM Production Model"]
        MS2["FAISS Index\nALS Item Embeddings"]
    end

    subgraph MONITORING["Monitoring"]
        MON1["Prometheus + Grafana\nLatency · CTR · CVR"]
        MON2["SHAP Monitoring\nFeature importance drift"]
        MON3["PSI Monitor\nCovariate drift detection"]
    end

    U --> S1A
    MS2 --> S1A
    S1A --> S1B
    S1B --> S2A
    FS1 & FS2 --> S2A
    S2A --> S2B
    MS1 --> S2B
    S2B --> S2C
    S2C --> U
    S2C --> MON1 & MON2 & MON3

    style USER fill:#1e1b4b,stroke:#7c3aed,color:#e0e7ff
    style STAGE_1 fill:#042f2e,stroke:#065f46,color:#ecfdf5
    style STAGE_2 fill:#1c1917,stroke:#b45309,color:#fef3c7
    style FEATURE_STORE fill:#1e1b4b,stroke:#4f46e5,color:#e0e7ff
    style MODEL_STORE fill:#1e1b4b,stroke:#9333ea,color:#f3e8ff
    style MONITORING fill:#1c1917,stroke:#be123c,color:#fce7f3

4.2 Feature Engineering: 87 Dimensions

The richness of the LTR feature vector is the primary driver of ranking quality improvement. Features are organized into three families:

User Features (47 dimensions):

USER_FEATURES = {
    # Demographics (6)
    'user_age_bucket': 'categorical_encoded',
    'user_province': 'categorical_encoded',
    'user_gender': 'binary',
    'user_device_type': 'categorical_encoded',  # mobile/desktop/app
    'user_acquisition_channel': 'categorical_encoded',
    'user_account_age_days': 'int',

    # RFM (3)
    'user_recency_days': 'int',
    'user_frequency_30d': 'int',
    'user_monetary_30d_idr': 'float',

    # Behavioral (15)
    'user_session_count_7d': 'int',
    'user_avg_session_duration_min': 'float',
    'user_category_affinity_fashion': 'float',
    'user_category_affinity_accessories': 'float',
    'user_price_sensitivity_score': 'float',
    'user_brand_loyalty_score': 'float',
    'user_discount_chaser_score': 'float',
    'user_review_reader_rate': 'float',
    'user_return_rate_90d': 'float',
    'user_wishlist_add_rate': 'float',
    'user_search_click_rate': 'float',
    'user_scroll_depth_avg': 'float',
    'user_checkout_abandonment_rate': 'float',
    'user_repeat_purchase_rate': 'float',
    'user_flash_sale_participation_rate': 'float',

    # Embeddings (20 dims from ALS)
    **{f'user_als_emb_{i}': 'float' for i in range(20)},

    # Temporal (3)
    'hour_of_day': 'int',
    'day_of_week': 'int',
    'is_weekend': 'binary',
}

Item Features (28 dimensions):

ITEM_FEATURES = {
    # Catalog (6)
    'item_category_l1': 'categorical_encoded',
    'item_category_l2': 'categorical_encoded',
    'item_brand_tier': 'categorical_encoded',
    'item_price_idr': 'float',
    'item_price_bucket': 'categorical_encoded',
    'item_discount_rate': 'float',

    # Quality signals (8)
    'item_avg_review_score': 'float',
    'item_review_count': 'int',
    'item_return_rate_30d': 'float',
    'item_defect_report_rate': 'float',
    'item_seller_rating': 'float',
    'item_fulfillment_rate': 'float',
    'item_stock_ratio': 'float',
    'item_image_quality_score': 'float',

    # Popularity (8)
    'item_click_rate_7d': 'float',
    'item_purchase_rate_7d': 'float',
    'item_wishlist_rate_7d': 'float',
    'item_view_to_cart_rate': 'float',
    'item_cart_to_purchase_rate': 'float',
    'item_impression_7d': 'int_log',
    'item_revenue_7d_idr': 'float_log',
    'item_reorder_rate': 'float',

    # Freshness (3)
    'item_days_since_listing': 'int',
    'item_last_price_change_days': 'int',
    'item_is_new_listing': 'binary',

    # Embeddings (3)
    'item_als_emb_norm': 'float',
    'item_text_emb_cluster': 'categorical_encoded',
    'item_visual_category_score': 'float',
}

Interaction Features (12 dimensions):

INTERACTION_FEATURES = {
    # ALS affinity (3)
    'als_score': 'float',               # Raw ALS dot-product score
    'als_rank_in_pool': 'int',          # Position in top-500 pool
    'als_score_normalized': 'float',    # Min-max normalized within pool

    # Historical user-item signals (5)
    'user_item_historical_views': 'int',
    'user_category_historical_cvr': 'float',
    'user_price_affinity_match': 'float',  # User price range vs item price
    'user_brand_affinity_match': 'float',
    'user_item_co_purchase_score': 'float',

    # Context (4)
    'query_item_text_similarity': 'float',  # If search context
    'session_context_category_match': 'float',
    'session_item_count': 'int',
    'item_position_in_session': 'int',
}

4.5 Two-Stage Retrieval-Ranking Data Flow and Label Construction

Understanding how graded relevance labels are constructed from implicit signals is crucial — label quality directly determines LTR model quality. This diagram maps the full label construction pipeline alongside the two-stage retrieval-ranking serving path.

flowchart LR
    subgraph LABEL_BUILD["🏷️ Offline: Label Construction Pipeline"]
        direction TB
        LB1["Raw Interaction Log\nclick · view · cart · purchase · return"]
        LB2["Temporal Windowing\n14-day relevance window"]
        LB3["Graded Label Assignment\n0=no interaction\n1=click only\n2=cart add\n3=purchase\n4=purchase+positive review"]
        LB4["Negative Sampling\n3× negatives per positive"]
        LB5["Feature Join\n87 features per query-item pair"]
        LB6["LTR Training Dataset\nquery_id · item_id · label · features"]
        LB1 --> LB2 --> LB3 --> LB4 --> LB5 --> LB6
    end

    subgraph STAGE1_OFFLINE["🔍 Offline: ALS Training (Weekly)"]
        direction TB
        A1["User-Item Interaction Matrix\nImplicit feedback · CounterFeit pseudo-rating"]
        A2["ALS Training\nn_factors=128 · n_iter=30"]
        A3["FAISS Index Build\nIVF + PQ compression"]
        A4["User/Item Embeddings\nSaved to artifact store"]
        A1 --> A2 --> A3 --> A4
    end

    subgraph STAGE2_OFFLINE["🎯 Offline: LTR Training (Weekly)"]
        direction TB
        L1["LTR Dataset\nFrom LABEL_BUILD"]
        L2["LightGBM LambdaMART\nn_estimators=1000 · num_leaves=63"]
        L3["SHAP Analysis\nFeature importance + drift baseline"]
        L4["MLflow Registration\nLightGBM_LTR_Tokobaju v{n}"]
        L1 --> L2 --> L3 --> L4
    end

    subgraph ONLINE_SERVING["⚡ Online Serving Path (< 90ms)"]
        direction LR
        OS1["User Request\nuser_id + context"]
        OS2["FAISS ANN Search\nTop-500 ALS candidates · 20ms"]
        OS3["Redis Feature Lookup\n87 features for 500 items · 8ms"]
        OS4["LightGBM Inference\n500 items re-ranked · 18ms"]
        OS5["Top-20 Results\nWith SHAP scores"]
        OS1 --> OS2 --> OS3 --> OS4 --> OS5
    end

    A4 -.->|"FAISS index"| ONLINE_SERVING
    L4 -.->|"LightGBM model"| ONLINE_SERVING
    LB6 -.->|"Training data"| STAGE2_OFFLINE

    style LABEL_BUILD fill:#0a2240,stroke:#3b82f6,color:#dbeafe
    style STAGE1_OFFLINE fill:#0a2a16,stroke:#16a34a,color:#dcfce7
    style STAGE2_OFFLINE fill:#2a1a08,stroke:#d97706,color:#fef3c7
    style ONLINE_SERVING fill:#1a082a,stroke:#9333ea,color:#f3e8ff

Diagram Walkthrough: The left side shows two parallel offline training pipelines — ALS for candidate generation (weekly) and LightGBM LTR (weekly, fed by the label construction pipeline). The label construction pipeline deserves particular attention: graded relevance labels (0–4) are derived from raw interaction logs with a 14-day temporal window and 3× negative sampling. The online serving path (right) uses the FAISS index from ALS training and the LightGBM model from LTR training, assembled into a < 90ms serving SLA.

5. Mathematical Explanation

5.1 LambdaMART: MART with LambdaRank Gradients

LambdaMART = MART (Multiple Additive Regression Trees = Gradient Boosted Trees) with LambdaRank pseudo-gradients.

Standard GBDT gradient:

F_{m}(x) = F_{m-1}(x) + ρ × h_m(x)

where h_m is the m-th regression tree fit to pseudo-residuals:
r_i = -∂L/∂F(xᵢ) |_{F=F_{m-1}}

LambdaMART replaces the gradient with λ-gradients:

For a query q with documents dᵢ, dⱼ (dᵢ ranked higher than dⱼ):

λᵢⱼ = -|ΔNDCG(i,j)| / (1 + e^(sᵢ - sⱼ))

λᵢ = Σⱼ: (dⱼ ranked below dᵢ) λᵢⱼ - Σⱼ: (dⱼ ranked above dᵢ) λᵢⱼ

The NDCG swap delta:

NDCG@K = (1/IDCG) × Σₖ₌₁ᴷ (2^rel_k - 1) / log₂(k + 1)

ΔNDCG(i,j) = |DCG(swapped list) - DCG(original list)|
            = |G(i) - G(j)| × |D(rank_i) - D(rank_j)|

Where G(i) = 2^rel_i - 1   (gain)
      D(k) = 1/log₂(k+1)   (discount)

5.2 Feature Vector and Ranking Function

The full 87-dimensional feature vector for a user-item pair:

X(u,i) = [x_user^(1), ..., x_user^(47), x_item^(1), ..., x_item^(28), x_inter^(1), ..., x_inter^(12)]

F(X) = Σₘ₌₁ᴹ γ_m × h_m(X)   (LambdaMART ensemble of M trees)

Ranking for user u: rank items by F(X(u,i)) descending

5.3 NDCG@K Formal Definition

NDCG@K = DCG@K / IDCG@K

DCG@K = Σₖ₌₁ᴷ (2^rel_k - 1) / log₂(k + 1)

IDCG@K = DCG@K for the ideal (perfectly sorted) ranking

For binary relevance (rel_k ∈ {0,1}):
DCG@K = Σₖ₌₁ᴷ rel_k / log₂(k + 1)

6. Implementation: lightgbm-ltr-recommender Repository

Repository: https://github.com/masterofray/lightgbm-ltr-recommender

Repository Structure

lightgbm-ltr-recommender/
├── data/
│   ├── raw/
│   ├── processed/
│   │   ├── train_features.parquet
│   │   ├── val_features.parquet
│   │   └── test_features.parquet
│   └── candidate_pools/
├── preprocessing/
│   ├── als_candidate_generator.py   # Stage 1: ALS top-500
│   ├── feature_builder.py           # 87-feature construction
│   └── label_builder.py             # Relevance label assignment
├── models/
│   ├── ltr/
│   │   ├── train_lgbm_ranker.py     # Primary training script
│   │   ├── evaluate_lgbm.py         # NDCG, MAP, MRR evaluation
│   │   └── hyperparameter_search.py # Optuna HPO
│   └── als/
│       └── train_als.py             # ALS candidate generator training
├── ranking/
│   ├── batch_ranker.py              # Offline batch re-ranking
│   └── online_ranker.py             # Real-time re-ranking
├── evaluation/
│   ├── offline_metrics.py
│   └── shap_analysis.py             # SHAP explainability
├── monitoring/
│   ├── drift_detector.py            # PSI + KS drift tests
│   └── shap_monitor.py              # Feature importance drift
├── api/
│   ├── serve.py                     # FastAPI serving
│   └── schemas.py
└── README.md

Core Training Script: `models/ltr/train_lgbm_ranker.py`

import lightgbm as lgb
import pandas as pd
import numpy as np
import mlflow
import mlflow.lightgbm
import shap
import optuna
from tqdm import tqdm
from sklearn.model_selection import GroupKFold
import warnings
warnings.filterwarnings('ignore')

# ─────────────────────────────────────────────────────────────────────
# LightGBM LambdaMART Ranker — Production Training Pipeline
# Input:  train_features.parquet (87 features + query_id + relevance_label)
# Output: MLflow-registered LightGBM ranker model + SHAP analysis
# ─────────────────────────────────────────────────────────────────────

class LightGBMLTRTrainer:
    """
    Production LightGBM LambdaMART trainer with:
    - Group-aware cross-validation (GroupKFold by user)
    - MLflow experiment tracking
    - SHAP feature importance analysis
    - Early stopping on NDCG@10 validation
    - Optuna hyperparameter optimization
    """

    FEATURE_COLS = [f for f in pd.read_parquet(
        "data/processed/train_features.parquet", nrows=0
    ).columns if f not in ['user_id', 'item_id', 'relevance_label']]

    def __init__(self):
        self.model = None
        self.shap_explainer = None

    def build_ranker(self, params: dict) -> lgb.LGBMRanker:
        return lgb.LGBMRanker(
            objective="lambdarank",
            metric="ndcg",
            ndcg_at=[5, 10, 20],

            # Tree structure
            n_estimators=params.get("n_estimators", 1000),
            num_leaves=params.get("num_leaves", 63),
            max_depth=params.get("max_depth", -1),
            min_child_samples=params.get("min_child_samples", 20),

            # Regularization
            learning_rate=params.get("learning_rate", 0.05),
            subsample=params.get("subsample", 0.8),
            colsample_bytree=params.get("colsample_bytree", 0.8),
            reg_alpha=params.get("reg_alpha", 0.1),
            reg_lambda=params.get("reg_lambda", 0.1),

            # LambdaMART specific
            lambdarank_truncation_level=params.get("lambdarank_truncation_level", 20),

            # Hardware
            n_jobs=-1,
            device_type="cpu",
            verbose=-1,
            random_state=42
        )

    def train(
        self,
        train_df: pd.DataFrame,
        val_df: pd.DataFrame,
        params: dict = None,
        experiment_name: str = "lightgbm_ltr_tokobaju"
    ) -> dict:
        """
        Full training pipeline with MLflow tracking.
        """
        if params is None:
            params = {}

        mlflow.set_experiment(experiment_name)

        with mlflow.start_run(run_name=f"LambdaMART_v{pd.Timestamp.now().strftime('%Y%m%d_%H%M')}"):
            mlflow.log_params(params)

            ranker = self.build_ranker(params)

            # Compute group sizes (number of items per user/query)
            train_groups = train_df.groupby('user_id')['item_id'].count().values
            val_groups = val_df.groupby('user_id')['item_id'].count().values

            X_train = train_df[self.FEATURE_COLS].values
            y_train = train_df['relevance_label'].values
            X_val = val_df[self.FEATURE_COLS].values
            y_val = val_df['relevance_label'].values

            print("Training LightGBM LambdaMART...")
            ranker.fit(
                X_train,
                y_train,
                group=train_groups,
                eval_set=[(X_val, y_val)],
                eval_group=[val_groups],
                eval_metric=["ndcg"],
                callbacks=[
                    lgb.early_stopping(stopping_rounds=50, verbose=True),
                    lgb.log_evaluation(period=100),
                ]
            )

            self.model = ranker

            # SHAP analysis
            print("Computing SHAP values...")
            self.shap_explainer = shap.TreeExplainer(ranker.booster_)
            shap_values = self.shap_explainer.shap_values(X_val[:5000])

            shap_importance = pd.DataFrame({
                'feature': self.FEATURE_COLS,
                'shap_mean_abs': np.abs(shap_values).mean(axis=0)
            }).sort_values('shap_mean_abs', ascending=False)

            # Evaluate
            val_ndcg = self.evaluate_ndcg(X_val, y_val, val_groups)
            mlflow.log_metrics({
                "val_ndcg@5": val_ndcg["ndcg@5"],
                "val_ndcg@10": val_ndcg["ndcg@10"],
                "val_ndcg@20": val_ndcg["ndcg@20"],
                "n_trees": ranker.best_iteration_,
            })

            # Log SHAP top-20 features
            mlflow.log_dict(
                shap_importance.head(20).set_index('feature')['shap_mean_abs'].to_dict(),
                "shap_top20_features.json"
            )

            mlflow.lightgbm.log_model(ranker, "lightgbm_ltr_model")
            mlflow.register_model(
                f"runs:/{mlflow.active_run().info.run_id}/lightgbm_ltr_model",
                "LightGBM_LTR_Tokobaju"
            )

            return val_ndcg

    def evaluate_ndcg(
        self,
        X_val: np.ndarray,
        y_val: np.ndarray,
        val_groups: np.ndarray,
        k_list: list = [5, 10, 20]
    ) -> dict:
        """Per-query NDCG computation."""
        scores = self.model.predict(X_val)
        results = {f"ndcg@{k}": [] for k in k_list}

        group_start = 0
        for group_size in tqdm(val_groups, desc="Computing NDCG", colour="#05ad46"):
            group_end = group_start + group_size
            group_scores = scores[group_start:group_end]
            group_labels = y_val[group_start:group_end]

            for k in k_list:
                ndcg = self._ndcg_at_k(group_labels, group_scores, k)
                results[f"ndcg@{k}"].append(ndcg)

            group_start = group_end

        return {k: np.mean(v) for k, v in results.items()}

    def _ndcg_at_k(
        self,
        labels: np.ndarray,
        scores: np.ndarray,
        k: int
    ) -> float:
        """Compute NDCG@K for a single query group."""
        sorted_labels = labels[np.argsort(scores)[::-1]]
        ideal_labels = np.sort(labels)[::-1]

        def dcg(rels, k):
            rels = rels[:k]
            positions = np.arange(2, len(rels) + 2)
            return np.sum((2**rels - 1) / np.log2(positions))

        dcg_val = dcg(sorted_labels, k)
        idcg_val = dcg(ideal_labels, k)
        return dcg_val / idcg_val if idcg_val > 0 else 0.0


# ─── Optuna Hyperparameter Search ────────────────────────────────────

def optimize_hyperparameters(
    train_df: pd.DataFrame,
    val_df: pd.DataFrame,
    n_trials: int = 50
) -> dict:
    trainer = LightGBMLTRTrainer()

    def objective(trial):
        params = {
            "n_estimators": trial.suggest_int("n_estimators", 200, 2000),
            "num_leaves": trial.suggest_int("num_leaves", 31, 255),
            "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.2, log=True),
            "subsample": trial.suggest_float("subsample", 0.6, 1.0),
            "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
            "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
            "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
            "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
            "lambdarank_truncation_level": trial.suggest_int("lambdarank_truncation_level", 10, 30),
        }
        result = trainer.train(train_df, val_df, params)
        return result["ndcg@10"]

    study = optuna.create_study(direction="maximize")
    study.optimize(objective, n_trials=n_trials, show_progress_bar=True)

    return study.best_params

Drift Detection: `monitoring/drift_detector.py`

import numpy as np
import pandas as pd
from scipy import stats
from tqdm import tqdm

class FeatureDriftDetector:
    """
    Monitors feature distribution drift between training data
    and current serving data using PSI and KS tests.
    Triggers retraining when drift exceeds thresholds.
    """

    PSI_WARNING = 0.1
    PSI_ALERT = 0.2
    KS_ALPHA = 0.01  # Statistical significance level

    def __init__(self, reference_df: pd.DataFrame, feature_cols: list):
        self.reference_df = reference_df
        self.feature_cols = feature_cols

    def compute_psi(
        self,
        reference: np.ndarray,
        current: np.ndarray,
        n_bins: int = 10
    ) -> float:
        """Population Stability Index (PSI) computation."""
        bins = np.percentile(reference, np.linspace(0, 100, n_bins + 1))
        bins[0], bins[-1] = -np.inf, np.inf

        ref_counts = np.histogram(reference, bins=bins)[0]
        cur_counts = np.histogram(current, bins=bins)[0]

        ref_pct = ref_counts / len(reference)
        cur_pct = cur_counts / len(current)

        # Avoid log(0)
        ref_pct = np.where(ref_pct == 0, 0.0001, ref_pct)
        cur_pct = np.where(cur_pct == 0, 0.0001, cur_pct)

        psi = np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct))
        return psi

    def run_drift_report(
        self,
        current_df: pd.DataFrame
    ) -> pd.DataFrame:
        """Generate full drift report across all features."""
        results = []
        for col in tqdm(self.feature_cols, desc="Computing drift", colour="#05ad46"):
            ref_vals = self.reference_df[col].dropna().values
            cur_vals = current_df[col].dropna().values

            psi = self.compute_psi(ref_vals, cur_vals)
            ks_stat, ks_pvalue = stats.ks_2samp(ref_vals, cur_vals)

            results.append({
                'feature': col,
                'psi': round(psi, 4),
                'ks_statistic': round(ks_stat, 4),
                'ks_pvalue': round(ks_pvalue, 6),
                'drift_status': (
                    'ALERT' if psi > self.PSI_ALERT
                    else 'WARNING' if psi > self.PSI_WARNING
                    else 'OK'
                ),
                'trigger_retrain': psi > self.PSI_ALERT
            })

        return pd.DataFrame(results).sort_values('psi', ascending=False)

7. Experiment Results

Dataset Configuration

Parameter	Value
Platform	Tokobaju.id fashion e-commerce
Total Users	5,200,000 MAU
Total Items	320,000 SKUs
Total Interactions	87,000,000 (12-month period)
Candidate Pool Size	Top-500 per user (ALS)
LTR Training Set	2,800,000 query-item pairs (280K users × 10 items average)
Feature Dimensions	87 features
Label Scale	0-4 graded relevance (0=no interaction, 4=purchase + positive review)
Train/Val/Test Split	70/15/15 temporal

Offline Evaluation Results

Model	NDCG@5	NDCG@10	NDCG@20	MAP@10	MRR	Precision@10
Popularity Baseline	0.041	0.054	0.071	0.038	0.061	0.029
ALS-only (Retrieval)	0.071	0.089	0.112	0.067	0.098	0.058
XGBoost LTR (Pairwise)	0.098	0.114	0.138	0.091	0.127	0.079
LightGBM LTR (Pointwise)	0.103	0.119	0.144	0.097	0.133	0.083
LightGBM LambdaMART	0.118	0.131	0.159	0.112	0.148	0.094
LightGBM MART + Feature Selection	0.121	0.136	0.163	0.116	0.153	0.097

SHAP Top-10 Feature Importance

| Rank | Feature | SHAP Mean |Abs| | Business Interpretation | |---|---|---|---| | 1 | als_score | 0.312 | ALS collaborative signal is still dominant | | 2 | user_category_historical_cvr | 0.187 | User’s purchase rate in this category | | 3 | item_cart_to_purchase_rate | 0.143 | Item-level conversion funnel quality | | 4 | item_avg_review_score | 0.118 | Social proof signal | | 5 | user_price_affinity_match | 0.097 | Price range compatibility | | 6 | item_return_rate_30d | 0.089 | Product quality signal | | 7 | user_brand_loyalty_score | 0.076 | Brand preference alignment | | 8 | item_seller_rating | 0.068 | Seller trust signal | | 9 | user_discount_chaser_score | 0.054 | Discount sensitivity | | 10 | item_days_since_listing | 0.047 | Freshness signal |

Online A/B Test Results (28-Day Deployment)

Metric	Control (ALS Only)	Treatment (ALS + LambdaMART)	Lift
Homepage CVR	2.2%	3.7%	+68.2%
Category CTR	3.4%	5.2%	+52.9%
Return Rate	12.3%	8.7%	-29.3%
SME Seller Visibility	4.1%	7.3%	+78.0%
Avg. Session Revenue	Rp 42,000	Rp 67,000	+59.5%
Cart Add Rate	8.1%	12.4%	+53.1%
NDCG@10 (Online Eval)	0.089	0.128	+43.8%

8. Model Comparison

Model	NDCG@10	CVR Impact	Training Time	Feature Support	Interpretability
Popularity Baseline	0.054	0% (ref)	Instant	❌ None	✅ Full
ALS (Stage 1 only)	0.089	+36%	2h (distributed)	❌ None	❌ Latent
XGBoost Pairwise LTR	0.114	+52%	45min	✅ Full	✅ SHAP
LightGBM Pointwise	0.119	+58%	28min	✅ Full	✅ SHAP
LightGBM LambdaMART	0.131	+68%	35min	✅ Full	✅ SHAP

Training time on 2.8M query-item pairs, 87 features, n_estimators=1000:

LightGBM LambdaMART: 35 minutes (8 CPU cores)
XGBoost Pairwise: 67 minutes
LightGBM Pointwise: 28 minutes

9. Production Challenges

9.1 Feature Leakage

The single most dangerous production risk in LTR systems is feature leakage — using information at training time that would not be available at serving time. Common leakage sources:

# ❌ LEAKAGE: Using future sales data as a feature at training time
train_df['item_sales_next_7d'] = ...  # This is the future!

# ❌ LEAKAGE: Click-through rate computed over entire dataset
item_ctr = all_interactions.groupby('item_id')['clicked'].mean()

# ✅ CORRECT: Use rolling window with temporal cutoff
def compute_historical_ctr(df, reference_date, window_days=7):
    cutoff = reference_date - pd.Timedelta(days=window_days)
    return (
        df[df['date'] < reference_date]
        .query(f"date >= '{cutoff}'")
        .groupby('item_id')['clicked'].mean()
    )

9.2 Training-Serving Skew

The 87-feature vector must be computed identically in both training and serving contexts. The solution:

# Unified feature computation function used in BOTH contexts
class FeatureComputer:
    def compute_user_features(self, user_id: str, reference_ts: pd.Timestamp) -> dict:
        # Identical logic in batch training and online serving
        ...

9.3 Latency Budget Management

With 500 candidates per user and 87 features, the LightGBM inference latency must stay within the 35ms budget:

Candidate retrieval (FAISS): 20ms
Feature lookup (Redis batch): 8ms
LightGBM inference (500 items × 87 features, 1000 trees): 18ms
Response serialization: 4ms
Total: 50ms P50, 78ms P95, 93ms P99

Optimization: reduce candidate pool to 200 for non-personalized contexts, reduce to 500 only for high-value users.

9.4 Model Drift and Retraining Frequency

PSI monitoring on the top-10 SHAP features revealed that item_click_rate_7d and user_discount_chaser_score drift fastest during flash sale events:

PSI alert triggered: 0.28 (> 0.2 threshold)
Features drifted: item_click_rate_7d (0.34), user_discount_chaser_score (0.31)
Action: Emergency retraining triggered via GitHub Actions
Retraining duration: 35 minutes (LightGBM) + 15 minutes (eval + MLflow) = 50 minutes
Total outage: 0 minutes (shadow mode deployment during training)

9.5 Cold-Start Items: New SKU Ranking

New items with no interaction data receive default values for interaction features and estimated values for quality features (based on seller historical performance):

def build_cold_item_features(item_id: str, seller_id: str) -> dict:
    seller_hist = get_seller_historical_stats(seller_id)
    return {
        'item_click_rate_7d': seller_hist['avg_new_item_ctr'],     # Seller average
        'item_purchase_rate_7d': seller_hist['avg_new_item_cvr'],
        'item_avg_review_score': seller_hist['avg_review_score'],
        'item_return_rate_30d': seller_hist['return_rate'],
        'item_days_since_listing': 0,
        'item_is_new_listing': 1,
        # ... other features
    }

10. Conclusion

The production deployment of LightGBM LambdaMART as a re-ranking layer over ALS candidate generation delivered transformational business results for Tokobaju.id: +68% CVR, +53% CTR, -29% return rate, and +78% SME seller visibility. The two-stage retrieval-ranking architecture — now the industry standard at Netflix, YouTube, and Amazon — solves the fundamental tension between recall breadth (Stage 1: CF) and precision quality (Stage 2: LTR).

SHAP explainability proved equally valuable on the business side: the ability to show stakeholders that item_return_rate is the 6th most important ranking signal became a compelling narrative for seller quality investment, triggering a seller quality improvement program that reduced return rates platform-wide.

Future Directions:

Deep LTR: Replace LightGBM with a Neural LTR model (Deep LambdaMART, DeepRank) for higher feature interaction capacity.
Multi-objective Ranking: Simultaneously optimize CTR, CVR, and return rate through Pareto-optimal multi-objective LTR.
Contextual Bandits: Move from offline LTR to online Thompson Sampling-based ranking that adapts in real-time.
LLM-based Feature Engineering: Use large language models to generate rich item description embeddings as additional ranking features.

11. Key Takeaways

LambdaMART directly optimizes NDCG through swap-delta-weighted λ gradients — this is why it outperforms pointwise and pairwise LTR on ranking quality metrics, and why it is the model of choice at all major recommendation platforms.
The two-stage retrieval-ranking architecture is production-standard — Stage 1 (ALS/embedding-based) maximizes recall; Stage 2 (LTR) maximizes precision. Conflating the two into one model creates an optimization conflict.
Feature leakage is the silent killer of LTR systems — any feature computed using data posterior to the interaction event timestamp will inflate offline NDCG by 15–40%, creating phantom A/B test confidence.
SHAP values are not optional in production LTR — they enable feature drift monitoring, explain ranking decisions for regulatory/business stakeholders, and guide feature engineering priorities for the next model iteration.
PSI > 0.2 is a retrain trigger — implement automated PSI monitoring on your top-10 SHAP features; flash sales, seasonal events, and marketing campaigns routinely cause covariate shifts that degrade LTR performance within 72 hours.
87 features is the practical sweet spot for LightGBM LTR — adding beyond 100+ features provides diminishing NDCG returns while increasing serving latency and training complexity.
Seller quality signals (return rate, fulfillment rate) are disproportionately valuable — SHAP analysis showed item_return_rate_30d contributing more to NDCG improvement than item_price_bucket or demographic features.
MLflow model registry with shadow deployment is the deployment pattern that achieves zero-downtime model updates — the new model runs in shadow mode for 2 hours before traffic is shifted.
Negative review scores deserve asymmetric weighting — a product with average review 3.2 should be penalized more than a product with no reviews (new items), because 3.2 stars is an established negative signal.
Optuna HPO with 50 trials reliably finds LightGBM hyperparameters within 5% of manual tuning in this problem space; invest the compute budget here rather than in manual grid search.

Part 3 of 4 in the enterprise recommendation system series. Part 4 covers XGBoost LTR with pairwise ranking and enterprise search integration. Repository: https://github.com/masterofray/lightgbm-ltr-recommender