Top K Problem | Min Count Sketch | System Design
Design a system that computes top K hashtags on Instagram. Lambda architecture.
If you aren’t subscribed yet, join 1000+ engineers and technical managers learning Advanced System Design.
Tracking the top-k items—such as trending Instagram hashtags—requires a system that can deliver both real-time freshness and long-term accuracy at massive scale. To achieve this, modern architectures combine fast approximation techniques with slower but exact batch computations, ensuring that every time window is covered efficiently. This article walks through how a lambda architecture supports this dual-workflow design, covering a real-time pipeline built on Count-Min Sketches and a batch pipeline powered by distributed queues and MapReduce.
Lambda Architecture Overview
Separation of real-time and batch processing for balanced accuracy and freshness
Lambda Architecture cleanly splits the system into fast, approximate computation and slower, fully accurate computation. This ensures you can serve highly responsive “trending now” results while preserving the ability to produce precise …
