Why I Rebuilt a CDR Pipeline from Scratch — and What I Learned

The system that “mostly worked”

Every telecom has one. The pipeline that’s been running since before anyone on the current team joined. The one that’s documented in a 300-page Word doc that was last updated in 2019. The one where the on-call runbook is basically “restart the service and pray.”

Our CDR (Call Detail Record) pipeline fell into this category. It processed tens of millions of records daily — call events, data sessions, SMS activity — feeding both billing and analytics systems. And it mostly worked.

Mostly is a dangerous word in data engineering. Mostly means your billing reconciliation has a 0.3% error rate that compounds weekly. Mostly means your churn model is trained on data where 1 in 200 records has the wrong subscriber ID. Mostly means every time a new record type gets introduced upstream, you’re firefighting for two days before realising the ingestion job silently dropped it.

What was actually wrong

The pipeline was built in three distinct phases by three different teams over seven years. Each team inherited the previous team’s assumptions and added their own workarounds on top. By the time I started working with it, the architecture looked like this:

• Raw files arrived on SFTP from multiple network elements in formats ranging from ASN.1 to CSV to custom fixed-width
• A collection of shell scripts and Java jobs parsed and normalised these into Oracle staging tables
• A nightly batch job ran transformations and loaded to the analytics warehouse
• Three separate reconciliation jobs ran post-load to catch errors — and regularly found them

The rebuild decision

The temptation with systems like this is incremental improvement. Add validation here, fix the schema handling there, replace the worst offending jobs one at a time. We’d been doing that for two years. It was like painting a ship while it was rusting from the inside.

The rebuild decision came from a specific incident: a upstream vendor changed their CDR format — a minor schema update, adding two new fields — and our pipeline silently dropped those records for eleven days before anyone noticed. Eleven days of missing call records. In billing. That’s when the conversation shifted from “can we afford to rebuild?” to “can we afford not to?”

The new architecture

We moved to PySpark on a Hadoop cluster with Delta Lake as the storage layer. The core design principles:

Schema-on-read with evolution

Instead of enforcing a rigid schema at ingestion, we land raw files as-is in a bronze layer and apply schema validation and transformation in the silver layer. Schema changes upstream trigger alerts, not silent drops. Delta Lake’s schema evolution capabilities mean adding new fields doesn’t break existing jobs.

# Schema enforcement with drift detection
def validate_and_ingest(df, expected_schema, source_name):
    actual_cols = set(df.columns)
    expected_cols = set(expected_schema.fieldNames())
    
    new_cols = actual_cols - expected_cols
    missing_cols = expected_cols - actual_cols
    
    if new_cols:
        alert_schema_drift(source_name, new_fields=list(new_cols))
    if missing_cols:
        raise SchemaMismatchError(f"Missing required fields: {missing_cols}")
    
    return df.select([col(c) for c in expected_schema.fieldNames()])

Idempotent processing

Every job can be re-run without producing duplicates. Delta Lake’s MERGE operations handle this gracefully — if a record already exists with the same key, it updates rather than inserts.

Observable by default

Record counts, processing times, and error rates are emitted as metrics at every stage. Alerts fire when counts deviate more than 5% from the rolling 7-day average.

What actually took the most time

Not the PySpark code. The hardest parts were:

• Historical data migration — 4 years of CDR data in various formats needed to be backfilled into the new structure without disrupting live operations
• Parallel run validation — running old and new pipelines in parallel for 6 weeks and reconciling every discrepancy found 23 legitimate bugs in the old system
• Stakeholder trust — downstream teams had built compensating workarounds for the old system’s quirks. Teaching them the new system was reliable enough to remove those workarounds took longer than the build itself

The results, six months later

Processing reliability: 99.7% (up from ~94%). The remaining 0.3% is source file delivery failures — genuinely upstream issues now, not pipeline bugs. Schema-related incidents: zero. Time to ingest a new CDR source type: down from “2-day firefight” to “half a day of configuration.” Reconciliation job count: reduced from 3 to 0. The data is now trusted enough that the billing team stopped running their own manual spot checks — the biggest compliment a data pipeline can receive.