Fault tolerant is a crucial issue in stream processing. We cannot rollback and rerun the failed jobs since there is no begin/end in the failed part of stream. In addition, we need to handle state other than output.
2) What is the solution’s main idea (nugget)?
It draws snapshot of a running system topology periodically. And the checkpoint snapshot work in the background without stopping the steam process.
3) Why is solution different from previous work?
The most related work will be Apache Storm, it cannot guarantee the state consistency, whereas Flink can achieve that.
Another trend is to use micro-batching system, like spark streaming, Storm trident. It is indeed still batch-based system. This kind of system lost the good properties of continuous operator. For example, batch-based system cannot support window data, and have higher latency.
Goole Cloud Dataflow remains the good properties on stream operator, but do not fit well in batch jobs.
4) Does the paper (or do you) identify any fundamental/hard trade-offs?
The snapshot scheme needs I/O overhead. Thus for large state, the system need to make a trade-off between throughput and recovery time.
I do not think it will be influential in 10 years. First, the whole system is based on a model proposed in 1985. This mitigate the paper's novelty. Second, I think there is no significant advantage of Flink over Goole Cloud Dataflow. And Google Cloud Dataflow have much more users.
No comments:
Post a Comment