Building Anomaly Detection on Streaming Telecom Data

The problem with finding out the next morning

Telecom networks generate events continuously — call attempts, data sessions, SMS, charging events. Our batch pipeline processed these overnight and analytics were ready by 6am. Which meant: if something went wrong at 2pm, we found out at 6am the next day. For a billing anomaly, that’s potentially 16 hours of incorrect charges accumulating. For a network element failure, that’s 16 hours of degraded service with no alert.

The ask was clear: catch anomalies within minutes, not the next morning.

Why we didn’t go straight to ML

The instinct with “anomaly detection” is to reach for machine learning — autoencoders, isolation forests, LSTM-based sequence models. We evaluated all of them. We chose statistical baselines instead. The reasons:

• Interpretability — when an alert fires, the operations team needs to understand why immediately. “The isolation forest score exceeded the threshold” is not actionable. “Call success rate dropped 23% below the 7-day average” is.
• Operational overhead — ML models need retraining, drift monitoring, feature pipelines. Statistical baselines need threshold tuning once and occasional review.
• False positive rate — our initial ML experiments had a false positive rate 3× higher than rolling statistics. Alert fatigue kills anomaly detection programmes faster than missed alerts.

The architecture

Events flow from network elements into Kafka topics. PySpark Structured Streaming consumes from Kafka, applies aggregations over sliding windows, computes deviations from rolling baselines, and writes alerts to a downstream table consumed by our ops dashboard.

from pyspark.sql import functions as F

# Structured Streaming from Kafka
stream = spark.readStream     .format("kafka")     .option("kafka.bootstrap.servers", KAFKA_BROKERS)     .option("subscribe", "cdr-events")     .load()     .select(F.from_json(F.col("value").cast("string"), CDR_SCHEMA).alias("data"))     .select("data.*")

# 5-minute tumbling window aggregation
windowed = stream     .withWatermark("event_time", "2 minutes")     .groupBy(
        F.window("event_time", "5 minutes"),
        "network_element",
        "event_type"
    )     .agg(
        F.count("*").alias("event_count"),
        F.avg("duration_sec").alias("avg_duration"),
        F.sum(F.when(F.col("result_code") != "SUCCESS", 1).otherwise(0)).alias("failure_count")
    )     .withColumn("failure_rate", F.col("failure_count") / F.col("event_count"))

Rolling baseline computation

For each metric, we maintain a rolling 7-day baseline (mean and standard deviation) using Delta Lake as the state store. Each new window value is compared against the baseline; deviations beyond 3σ trigger alerts.

def check_anomaly(current_value, baseline_mean, baseline_std, sigma_threshold=3.0):
    if baseline_std == 0:
        return False
    z_score = abs(current_value - baseline_mean) / baseline_std
    return z_score > sigma_threshold

Tuning for alert quality

The first week in production, we had 847 alerts. The ops team acknowledged 12 of them. The rest were noise — normal traffic variance, scheduled maintenance windows, known periodic patterns (usage always drops 40% between 2am and 4am).

Fixes that cut false positives by 85%:

• Separate baselines for peak hours, off-peak hours, and weekends
• Maintenance window suppression — alerts automatically silenced during planned maintenance
• Minimum event count threshold — don’t alert if the window has fewer than 100 events (statistical noise)
• Alert grouping — correlated alerts from the same network element within 10 minutes group into one incident

Results after 3 months

Mean time to detection for billing anomalies: reduced from ~14 hours to 6 minutes. Network element failure detection: from next-day to under 4 minutes. Alert volume: stabilised at 8–15 meaningful alerts per day, with 90%+ action rate from the ops team. One incident caught early saved an estimated 6 hours of incorrect premium rate charging — the business case for the project was proven in week two.