Review on Spark (hotcloud'10)

This paper point out the fact that MapReduce is not good at conducting iterative jobs and interactive analysis. The system Spark (aka, the implementation of RDD) try to achieve more efficiency in these two scenarios (e.g. iterative jobs, interactive analysis). The problem is real and urgent to be solved.

The nugget of this paper is Resilient Distributed Dataset. RDD enable user to cache RDD in memory and reuse it in multiple MapReduce rounds. And it can be fault tolerant.

RDD is related to DSM (Distributed Shared Memory). The difference is that RDD provides more restricted programming model and better fault tolerance. RDD achieves in memory MapReduce, rather than letting nodes access a global address space.

Compared with MapReduce, RDD can persist across operations. Different from Twister, RDD enable fault tolerance and more general.

For the tradeoff, even though RDD is more suitable for iterative jobs and interactive analysis, MapReduce performs better with common transaction workloads.

Another tradeoff is, RDD enable fault tolerance with parallel recovery, in exchange, RDD introduce overhead for centralized modifications to dataflow.

I think Spark will be influential in next 10 years. even though RDD is based on MapReduce, they developed in different ways. For RDD, it mainly focus on iterative jobs and interactive analysis. For MapReduce, it performs better in common transaction workloads.