review for MapReduce (osdi'04) paper

1) Is the problem real?

Based on the authors' experience on processing data at Google, they found out an interesting phenomenon that is twofold:

1) Nowadays the computation are quite straightforward. 

2) However, the quantity of data is huge. 

Therefore, how to process huge amount of data (aka, "big data") with simple computation is the key issue. More precisely, how to achieve parallel processing "big data" on distributed system, and also keep resistance to failure is the key challenge.

From my perspective, it is a fundamental issue needed to be solved in "big data" processing. 

First, currently the only efficient way we can process "big data" is to use distributed system and process the data in parallel. This paper is proposing an abstraction for efficient "big data" processing on distributed system (including data distribution, load balancing, etc.). 

Second, this paper also tries to tickle a key issue in running distributed system, that is the node failures.

2) What is the solution’s main idea (nugget)?

Two steps: Map & Reduce.

Map: shuffle and distribute data into parallel processing. 

Reduce: merge data with specified characteristic.

Another key component: Intermediate Key--link between Map & Reduce.

3) Why is solution different from previous work?

MapReduce is simpler and easier than many other restricted parallel programming models. In addition, MapReduce provide fault-tolerant on larger scale to thousands of processors.

4) Does the paper (or do you) identify any fundamental/hard trade-offs? 

1. Since MapReduce rely on atomic commits of map and reduce tasks, it can provide non-faulting execution when Map/Reduce operators are deterministic. However, MapReduce provides weak semantics when the Map/Reduce operators are non-deterministic.

 

2. By introducing Combiner function, it can mitigate the non-uniformly distributed workload on each reduce function. Thus, there are some nodes need to do both map and reduce functions. This one node play two role thing is not good for pipelining the whole process.

3. Since MapReduce need to replicate data during processing, the whole replication process is limited by the network bandwidth.

5) Do you think the paper will be influential in 10 years? Why or why not?

I think it will still have impacts in next 5-10 years. Even though Spark project works better in some specific scenarios (e.g. iterative algorithm, interactive data mining tools, etc.), MapReduce works well in common transaction workloads.