Skip to content
Aryanto
Back to Blog

Building a Customer Data Platform: Unifying Siloed Data at Scale

A production-focused guide to building a Customer Data Platform (CDP) that unifies siloed customer data, handles identity resolution, engineers 200+ features in real-time, and maintains data quality at scale. Based on nine years of building data platforms.

• Aryanto, M.Si
data-engineering customer-data architecture

Building a Customer Data Platform: Unifying Siloed Data at Scale

Every company I’ve worked with has the same problem: customer data is scattered across dozens of systems — CRM, e-commerce platform, mobile app, email marketing, customer support, payment processor, loyalty program — and none of them agree on who the customer is. The CRM has an email address, the mobile app has a device ID, the loyalty program has a phone number, and the e-commerce platform has a browser cookie. Connecting these identities and building a unified customer profile is the core challenge of a Customer Data Platform (CDP), and it’s one of the most technically demanding data engineering problems I’ve encountered.

The Architecture: Lambda for Customer Data

I’ve settled on a lambda architecture for CDPs — a batch layer for historical accuracy and a speed layer for real-time freshness. Here’s the high-level architecture:

graph TD
    %% Nodes Definition
    DS[Data Sources]
    IL[Ingestion Layer]
    RES[Raw Events Store]
    
    BL[Batch Layer<br>List: Spark / Databricks]
    SL[Speed Layer<br>List: Flink / Kafka]
    
    UP[Unified Profile]
    RF[Real-time Features]
    
    SV[Serving Layer<br>Feature Store + API]

    %% Pipeline Flow
    DS --> IL
    IL --> RES
    
    %% Lambda Split
    RES --> BL
    RES --> SL
    
    %% Processing Flow
    BL --> UP
    SL --> RF
    
    %% Lambda Merge
    UP --> SV
    RF --> SV

    %% Custom Styling for Clean Tech Look
    style DS fill:#f9f9f9,stroke:#333,stroke-width:2px
    style RES fill:#f4f4f5,stroke:#2806a1,stroke-width:2px,stroke-dasharray: 5 5
    style BL fill:#eff6ff,stroke:#1e40af,stroke-width:2px
    style SL fill:#fef2f2,stroke:#991b1b,stroke-width:2px
    style SV fill:#f0fdf4,stroke:#166534,stroke-width:2px

Ingestion Layer

The ingestion layer must handle heterogeneous data sources with different formats, schemas, and reliability guarantees. I use a combination of:

  1. Event streaming (Kafka): For real-time event data from web, mobile, and IoT sources. Events are published to Kafka topics with a standardized schema (I use Avro with a schema registry).
  2. Batch ingestion (Airbyte/Fivetran): For periodic data from SaaS tools (Salesforce, HubSpot, Zendesk). These tools handle API rate limits, incremental sync, and schema detection.
  3. File-based ingestion (SFTP/S3): For legacy systems that can only export flat files. I use a custom parser that handles CSV, fixed-width, and Excel formats.

Critical design decision: All events land in a raw events store (S3 data lake in Parquet format) before any processing. This provides a complete audit trail and allows reprocessing when business logic changes. Never transform data in the ingestion layer — keep it as close to the source format as possible.

Schema Evolution

Customer data schemas change constantly. A new field gets added to the CRM, the mobile app changes its event structure, the e-commerce platform renames a column. Your CDP must handle schema evolution gracefully.

I use a schema registry (Confluent Schema Registry for Kafka, Apache Iceberg for the data lake) that enforces backward compatibility. New fields can be added, but existing fields cannot be renamed or removed without a migration plan. When an upstream system makes a breaking change, the ingestion layer detects it and alerts the data engineering team.

Production tip: Maintain a schema version for every data source. When a source changes its schema, increment the version and update the downstream processing code. This lets you handle the transition period where both old and new schemas are in flight.

Identity Resolution: The Heart of the CDP

The Problem

A single customer might appear as:

  • user_12345 (CRM, email-based)
  • device_abc123 (mobile app, device ID-based)
  • cookie_xyz789 (web, cookie-based)
  • loyalty_99887 (loyalty program, phone-based)
  • anon_55555 (web analytics, anonymous)

Identity resolution is the process of determining that all these identifiers belong to the same person.

Deterministic Matching

The first layer is deterministic matching — exact matches on known identifiers. If two records share the same email address, phone number, or loyalty ID, they’re the same person. This sounds simple, but in practice:

  • Email normalization: John.Doe@Gmail.com and johndoe@gmail.com are the same person. I normalize emails by lowercasing, removing dots in the local part (for Gmail), and removing plus-addressing.
  • Phone normalization: +1 (555) 123-4567 and 5551234567 are the same. I normalize to E.164 format.
  • Handling typos: john.doe@gmail.com and john.doe@gmial.com are probably the same person. I use edit distance matching for high-confidence fuzzy matches on email.

Probabilistic Matching

Deterministic matching handles the easy cases. For anonymous users (cookie-based, device-based), you need probabilistic matching — statistical techniques that estimate the probability that two identifiers belong to the same person.

I use a combination of:

  1. Device fingerprinting: Browser fingerprint (screen resolution, installed fonts, timezone, user agent) combined with IP address. Not perfect, but useful for linking anonymous sessions.
  2. Behavioral similarity: If two anonymous users have very similar browsing patterns (same products viewed, same time of day, same geographic area), they might be the same person.
  3. Supervised learning: Train a classifier on known identity matches (from deterministic matching) to predict matches for ambiguous pairs. Features include shared attributes, behavioral similarity, and temporal proximity.

Ethical consideration: Probabilistic matching is powerful but raises privacy concerns. I implement strict consent management — only match identities for users who have opted in to data collection. I also maintain a “forget me” mechanism that deletes all identity links when a user requests data deletion.

The Identity Graph

The output of identity resolution is an identity graph — a graph where nodes are identifiers and edges are identity links. I store this in a graph database (Neo4j) for flexible querying:

// Find all identifiers for a customer
MATCH (c:Customer {id: 'customer_12345'})-[:HAS_IDENTIFIER]->(i:Identifier)
RETURN i.type, i.value

// Find all events for a customer across all identifiers
MATCH (c:Customer {id: 'customer_12345'})-[:HAS_IDENTIFIER]->(i:Identifier)-[:HAS_EVENT]->(e:Event)
RETURN e.type, e.timestamp, e.properties
ORDER BY e.timestamp DESC

The identity graph is updated incrementally — new events can trigger new identity links, and existing links can be strengthened or weakened based on new evidence.

Feature Engineering: 200+ Features in Real-Time

The Feature Taxonomy

After building CDPs for multiple companies, I’ve developed a taxonomy of customer features that consistently provide predictive value:

Behavioral features (highest value):

  • Recency: Days since last visit, last purchase, last email open
  • Frequency: Visits per week, purchases per month, sessions per day
  • Monetary: Total spend, average order value, max single purchase
  • Engagement: Pages per session, time on site, email click rate

Temporal features:

  • Time-of-day patterns: When does this customer typically shop?
  • Day-of-week patterns: Weekday vs weekend preference
  • Seasonal patterns: Holiday shopping, birthday-related purchases
  • Trend features: Is this customer’s engagement increasing or decreasing?

Product affinity features:

  • Category affinity: Vector of purchase probabilities per category
  • Brand affinity: Preferred brands based on purchase history
  • Price sensitivity: Response to discounts, price elasticity estimate
  • Product lifecycle: Where is this customer in the product discovery → purchase → repurchase cycle?

Channel features:

  • Channel preference: Email vs push vs SMS vs in-app
  • Cross-channel behavior: How does this customer move between channels?
  • Attribution: Which marketing channels drive this customer’s conversions?

Social features:

  • Referral behavior: Has this customer referred others?
  • Review activity: Number and sentiment of reviews written
  • Community engagement: Forum posts, social shares

Real-Time Feature Computation

Some features must be computed in real-time. When a customer is browsing the website right now, you can’t wait for a nightly batch job to update their features. I use a stream processing layer (Apache Flink) to compute real-time features:

# Simplified Flink job for real-time feature computation
class CustomerFeatureProcessor(ProcessFunction):
    def process_element(self, event, ctx: Context):
        # Update session features
        session_state = ctx.get_state(self.session_state)
        session_state.update(event)

        # Compute real-time features
        features = {
            'events_in_session': session_state.event_count,
            'pages_viewed_in_session': session_state.unique_pages,
            'time_on_site_seconds': session_state.duration_seconds,
            'products_added_to_cart': session_state.cart_adds,
            'session_start_hour': session_state.start_hour,
            'is_returning_session': session_state.is_returning,
        }

        # Emit to feature store
        yield CustomerFeature(
            customer_id=event.customer_id,
            features=features,
            timestamp=event.timestamp,
        )

Batch Feature Computation

Complex features that require historical aggregation (lifetime value, RFM scores, cohort analysis) are computed in batch using Spark:

def compute_rfm_features(events_df, reference_date):
    """Compute RFM (Recency, Frequency, Monetary) features."""
    purchase_events = events_df.filter(col('event_type') == 'purchase')

    rfm = purchase_events.groupBy('customer_id').agg(
        datediff(lit(reference_date), max('event_date')).alias('recency_days'),
        countDistinct('order_id').alias('frequency'),
        sum('order_value').alias('monetary'),
        avg('order_value').alias('avg_order_value'),
        stddev('order_value').alias('order_value_stddev'),
        countDistinct('product_category').alias('category_breadth'),
    )

    # Add percentile ranks
    for metric in ['recency_days', 'frequency', 'monetary']:
        rfm = rfm.withColumn(
            f'{metric}_percentile',
            percent_rank().over(Window.orderBy(col(metric)))
        )

    return rfm

Feature Store Integration

All features — real-time and batch — are stored in a feature store (I use Feast). The feature store provides:

  1. Point-in-time correctness: When computing features for model training, you must use the feature values that were available at the time of prediction, not the current values. The feature store handles this automatically via point-in-time joins.
  2. Feature reuse: Multiple models can share the same features without duplicating computation.
  3. Feature freshness: The feature store tracks when each feature was last updated and can serve stale features with a freshness warning.

Data Quality: The Unsexy Foundation

Data Quality Framework

I implement data quality checks at three levels:

Ingestion-time checks:

  • Schema validation: Does the event match the expected schema?
  • Completeness: Are required fields present?
  • Format validation: Are dates valid? Are emails well-formed?

Processing-time checks:

  • Statistical validation: Are feature distributions within expected ranges?
  • Consistency: Do related features agree? (e.g., total_orders >= orders_last_30_days)
  • Freshness: Are features being updated on schedule?

Serving-time checks:

  • Staleness: Is the feature too old to be useful?
  • Coverage: What percentage of customers have this feature?
  • Distribution: Is the feature distribution in production matching the training distribution?

Data Quality Monitoring Dashboard

I maintain a Grafana dashboard that tracks:

  • Completeness: Percentage of customers with each feature populated.
  • Freshness: Distribution of feature update timestamps.
  • Distribution drift: PSI for each feature, compared against a reference window.
  • Anomaly rate: Percentage of records failing quality checks.

When any metric exceeds its threshold, an alert fires and the data engineering team investigates. This proactive approach catches data quality issues before they affect downstream models.

Lessons from Building CDPs

Lesson 1: Identity Resolution Is Never “Done”

New data sources constantly introduce new identifier types. A new partnership adds a new loyalty ID. A new marketing channel introduces a new tracking mechanism. Each new identifier type requires updates to the identity resolution logic. Plan for ongoing investment in this area.

Lesson 2: Real-Time Is More Expensive Than You Think

Computing features in real-time requires maintaining state (session data, running aggregates, windowed counts) in memory. At scale (millions of concurrent users), this requires significant infrastructure investment. Be selective about which features truly need real-time computation — most features can tolerate a 15-minute to 1-hour delay.

Lesson 3: Schema Changes Will Break Your Pipeline

The most common cause of CDP pipeline failures is upstream schema changes. A field gets renamed, a data type changes, a new enum value appears. Invest in schema validation and versioning from day one. Make schema changes visible and alert the team before they cause failures.

Lesson 4: Privacy Is Not Optional

Customer data platforms handle sensitive personal information. GDPR, CCPA, and other privacy regulations require specific capabilities: consent management, data deletion, purpose limitation, and data minimization. Build these capabilities into your CDP from the start — retrofitting privacy controls is much harder than designing them in.

Lesson 5: Start with the Use Cases

Don’t build a CDP in a vacuum. Start with specific use cases (personalized recommendations, churn prediction, marketing attribution) and build the data infrastructure to support them. A CDP that serves 5 concrete use cases is more valuable than a theoretical CDP that could serve any use case but serves none well.

Conclusion

Building a Customer Data Platform is one of the most technically challenging data engineering projects you can undertake. It spans real-time stream processing, batch computation, graph databases, feature stores, identity resolution, and data quality — all while handling privacy regulations and constantly evolving source schemas. The payoff is enormous: a unified view of the customer that powers personalization, analytics, and machine learning across the entire organization. But be prepared for a long journey — a production-grade CDP takes 12-18 months to build and requires ongoing investment to maintain.