The problem this paper tries to tackle is triple-fold: application performs iterative processing on real-time data stream, and supports interactive queries on a fresh consistent view of results. And the authors say that no system can achieve all these requirements. The problem itself is real, and more-specified.
The main idea of this paper is timely dataflow. Timely dataflow is a computational model based on directed graph where vertices send/receive timestamped messages along directed edges. It can have cycles.
The unique character of Naiad is that, it can fulfill the triple-fold requirement in one system. In addition, Naiad introduce less overhead than Spark, etc. Since Spark and other systems requires centralized modification to dataflow graph. Compared with previous asynchronous systems, Naiad provides completeness notification in a global synchronization, whereas the previous asynchronous systems are lack of.
For the trade-off, I think without inheriting batch-computationa model, Naiad is not as robust as Spark on fault-tolerance. In addition, Naiad has a lot of checkpoints. These periodically checkpoints needs worker to pause and deliver message, which may slow down the performance. Thus the user needs to balance the checkpoints overhead and the system correctness.
I do think Naiad gives more freedom for the user to balance processing speed and system correctness. Thus Naiad can be adaptive in both streaming and batch processing. However, the Naiad system do not provide good fault tolerance. Therefore, I think this system is not influential in 10 years. It is because fault tolerance is a key issue in data processing.
No comments:
Post a Comment