The Problem

The existing Call Detail Record (CDR) pipeline was built on legacy batch processing that struggled to keep pace with growing data volumes. Processing windows kept slipping, failures were hard to diagnose, and the system had become fragile — small schema changes upstream could break the entire pipeline with little warning.

The Approach

Rather than patch the existing system further, the pipeline was re-architected around PySpark and Delta Lake, moving from brittle batch scripts to a proper lakehouse structure. This meant:

• Restructuring ingestion to handle schema evolution gracefully instead of failing on it
• Moving to distributed processing that scales with data volume rather than degrading under it
• Building in fault tolerance so partial failures don’t take down the whole run

The Outcome

The modernized pipeline handles significantly higher throughput with far greater reliability, and is structured so that future schema or volume changes are absorbed rather than requiring emergency fixes.

Specific throughput and reliability metrics are being prepared for publication and will be added here.

← Back to all work