/ overview
Moby Dublin ingests continuous GPS and occupancy events from a distributed vehicle fleet and turns them into live demand signals for the city.
The pipeline is built on Spark Structured Streaming with event-time windowing and watermarking, so late and out-of-order device readings still land in the correct time bucket instead of skewing the aggregate.
Cleaned records are written to partitioned Parquet on S3 for analytical replay, while rolling aggregates are served from MongoDB for the dashboard layer.
/ what it does
- 01Event-time windows with watermarks to absorb late-arriving device readings.
- 02Deduplication keyed on device and timestamp cut duplicate GPS records by 14%.
- 03Checkpointed, fault-tolerant aggregation that resumes cleanly after failure.
- 04Partitioned Parquet lake on S3 for cheap historical reprocessing.
- 05Hotspot and crowding scores exposed to a lightweight query API.
/ numbers
−14%
duplicate GPS records
event-time
processing model
checkpointed
recovery
/ stack
Processing
Apache SparkStructured StreamingPySpark
Storage
AWS S3ParquetMongoDB
Ops
CheckpointingPartitioningDocker