Building a Churn Model That Actually Gets Used

The graveyard of accurate models

There’s a specific type of ML project that happens in almost every large organisation: the model that performs brilliantly in a Jupyter notebook and then quietly disappears into a PowerPoint deck never to be used again.

I’ve seen this happen with churn models more than any other ML use case. The reason is almost never technical. It’s because the people who built the model and the people who would act on its outputs are in different worlds, with different vocabularies, different incentives, and different definitions of “good enough.”

This is the story of a churn model that didn’t die that death — and what it took to keep it alive.

Starting with the business problem, not the algorithm

The first thing I did was not build a model. I spent two weeks with the retention team understanding how they actually worked. What I learned:

• They ran targeted retention campaigns every Monday morning — about 50,000 subscribers per campaign
• They were currently segmenting manually based on days since last usage and ARPU — a two-variable heuristic built in Excel
• The campaign team could act on a ranked list of subscribers. They could not act on a probability distribution
• They needed the list by Sunday evening. Not Sunday at 11:59pm. Sunday at 6pm, so there was review time
• They did not trust any output they couldn’t explain to their manager

That last point killed more ML projects than any technical failure. Explainability wasn’t a nice-to-have. It was table stakes.

Feature engineering from real data

The features that mattered most were not complicated. They were:

• Days since last voice call, last data session, last SMS
• Change in monthly data consumption over the last 30/60/90 days
• Number of calls to customer support in the last 60 days
• Contract end date proximity
• Recent roaming activity (proxy for travel behaviour change)
• Plan type and whether they’d previously rejected an upsell offer

def build_churn_features(subscriber_df, cdr_df, support_df):
    """Build churn prediction features from raw telecom data."""
    
    # Recency features
    last_activity = cdr_df.groupBy("subscriber_id").agg(
        F.max("event_date").alias("last_event_date"),
        F.max(F.when(F.col("event_type") == "VOICE", F.col("event_date"))).alias("last_voice_date"),
        F.max(F.when(F.col("event_type") == "DATA", F.col("event_date"))).alias("last_data_date")
    )
    
    # Consumption trend (30-day vs 90-day average)
    consumption = cdr_df.groupBy("subscriber_id", 
                                  F.date_trunc("month", F.col("event_date")).alias("month"))         .agg(F.sum("data_mb").alias("monthly_data_mb"))
    
    return subscriber_df         .join(last_activity, "subscriber_id", "left")         .join(consumption_trend, "subscriber_id", "left")         .withColumn("days_since_last_activity", 
                   F.datediff(F.current_date(), F.col("last_event_date")))

The model itself

We used a gradient boosted tree model (LightGBM via Python, then translated to PySpark MLlib for production scale). Deliberately not a deep learning model — the retention team needed to understand why a subscriber was flagged, and tree-based models offer feature importance that can be translated into plain English.

Model performance: AUC 0.81 on holdout. But more importantly: when we showed the top 10 churn risk factors for a sample subscriber to the retention team lead, she said “yes, that makes sense” without hesitation. That moment mattered more than the AUC.

The production architecture

The model runs on a PySpark job that:

1. Pulls yesterday’s CDR data from the Delta Lake silver layer
2. Joins with subscriber, contract, and support data
3. Scores all active subscribers (approximately 4 million)
4. Ranks by churn probability, filters to the top 60,000 (with a hard threshold of p > 0.35)
5. Writes the ranked list to a database table consumed by the CRM system
6. Sends a summary report to the retention team by 5:30pm Sunday

The job runs every Saturday night with a backup trigger on Sunday morning if the Saturday run fails or produces anomalous output (count deviation > 15% from the prior week triggers an alert).

What made it stick

Six months in, the model is still running. The retention team uses it every week. A few things that made the difference:

• The output format matched their existing workflow. They received a CSV-equivalent ranked list, exactly how they’d been consuming the Excel-based heuristic. No new tools to learn.
• We ran a 90-day A/B test against the old method. The ML-targeted group had 23% higher retention rate. Showing that number converted sceptics.
• We explained each subscriber in plain English. The CRM system shows “Flagged because: data usage dropped 67% in the last 30 days, contract expires in 45 days.” Not a probability. A reason.
• We were honest about errors. When the model got something obviously wrong, we documented it and updated the feature set. Trust is built over time, not announced.