Paper review for D-Stream (SOSP'13)

1) Is the problem real?
This paper tries to tickle the steaming problem in big data analysis. The problem indeed happens frequently in scenarios like social network etc. Thus the problem is real and important. In addition, current record-at-a-time model cannot work well on streaming. Thus it is urgent to propose a better solution.

2) What is the solution’s main idea (nugget)?
The key idea is still using batch-based system. They run each streaming computation as a series of deterministic batch computations on small time intervals

3) Why is solution different from previous work?

This problem is not new, but previous record-at-a-time model does not work well. In other words, record-at-a-time model can only support high-cost fault tolerant and cannot tickle with stragglers. Further, this model is hard to remain global consistency.
For batch-based system, none of the existing batch systems can achieve sub-second latencies. 

On the other hand, by incorporating RDD, D-stream can achieve sub-second latencies, and also support for fault tolerant, dealing with stragglers.


4) Does the paper (or do you) identify any fundamental/hard trade-offs? 
The fundamental tradeoff is using Batch-based system to provide fault tolerant & straggler problem. However, using batch-based system introduces high latency, even with RDD. 

In addition, with batch-based model, it lost many other properties of continuous operator model. For example, D-stream may suffer from back-pressure effect. More precisely, if the micro-batch takes longer in downstream processing than in batch operator, the micro-batch will take longer than configured. Thus it leads to batches queuing up or growing the batch size.

5) Do you think the paper will be influential in 10 years? Why or why not?
I think it will not be influential in 10 years. Indeed, D-stream is still batch-based system. Thus the latency issues is inevitable. I think we need to design system or model, which is specifically used for streaming data processing. I think system like record-at-a-time model will still be the mainstream of streaming data processing system. Or we may develop new technology to minimize latency while providing fault tolerant.