← Back to projects

Real-Time Tweet Sentiment Pipeline — Notes

Spark Structured Streaming + Delta Lake medallion architecture with live transformer inference and monitoring.

Spark Streaming Delta Lake Hugging Face MLflow
Repo
Tweet sentiment pipeline cover

Problem

Run sentiment inference on high-velocity tweet streams with low latency, reliable ingestion, and storage that supports downstream analytics.

What I Built

Built an end-to-end streaming pipeline using Spark Structured Streaming with Delta Lake (Bronze→Silver→Gold), Hugging Face transformer inference, and MLflow tracking.

System Overview

Streaming pipeline overview diagram
Ingest → Bronze (raw) → Silver (cleaned) → Inference → Gold (aggregates) → Monitoring.
  • Structured Streaming micro-batches + checkpointing
  • Delta medallion layers for versioned, query-ready tables
  • Transformer inference integrated in the stream
  • Monitoring via MLflow + lightweight dashboard

Performance & Validation

Validated pipeline behavior by measuring throughput and per-stage processing time across Bronze/Silver/Gold.

Rows processed over time
Rows processed over time (throughput)
Processing time over time
Processing time over time (latency)

Engineering Notes

  • Checkpointing + idempotent writes for reliable progress
  • Event-time watermarking + windowed aggregations for late data
  • Separation of concerns across Bronze/Silver/Gold layers
  • Designed for schema evolution without breaking downstream queries
  • MLflow tracking for reproducible runs and monitoring