1) Is the problem real?
The problem indeed is real. Many proposed schemes, even though they have some good properties for data processing, all of them may lack of some crucial properties like exactly-once semantics, fault tolerance, etc. This DataFlow try to design a system that can balance correctness, latency, and cost in massive, unbounded, out of order data processing.
2) What is the solution’s main idea (nugget)?
Their window-based architecture is the nugget. It has two core functions ParDo and GroupbyKey.
In addition, they propose trigger & incremental processing to figure out window completeness.
3) Why is solution different from previous work?
For previous work on batch or stream processing, they either lack of fault tolerance, or lack of other important properties like exactly-once semantics, model for calculating event-time.
4) Does the paper (or do you) identify any fundamental/hard trade-offs?
While even the DataFlow can be fitted for processing all Batch, micro-batch, streaming workload. The processing efficiency is lower than other more specified schemes (e.g. spark for batch, etc).
5) Do you think the paper will be influential in 10 years? Why or why not?
Yeah, I think it will have impact. All previously systems have shortcomings. They are specifically designed for working with particular workload. On the other hand, DataFlow can be general enough to subsume the standard batch, micro-batch, and streaming model.
No comments:
Post a Comment