Review for Yarn (SoCC'13)

1) Is the problem real?
Currently the use of MapReduce is far beyond its original design purpose. And there are mainly two shortcomings.

1. tight coupling of programing model with source management infrastructure. It makes developers to abuse the MR model.

2. centralized job control flow, which limits the system's scalability.

2) What is the solution’s main idea (nugget)?

Separating resource management functions  from the programing model.

More precisely, Yarn replaces jobtracker & tasktracker which were used in Hadoop 1.0, with new components, namely, Resource Manager, Application Master, Node Manager.

RM: it runs as a daemon machine, serves as a central authority arbitrating resources among various applications.

AM: it manages each job. It monitors all lifecycle including resources consumption, flow execution etc.

NM: they are responsible for monitor resource availability on each node.

With this new architecture, it decentralizes the workload on JobTracker (Resource Manager in Yarn) in original Hadoop. More precisely, the job of task monitoring is assigned to AM instead of RM. Further, AM can be defined by users.

Container is also another novel framework that perform well for resource isolation in Yarn.

Another advantage of Yarn over Hadoop is that, Yarn represent resource with memory, which is more reasonable compared with Hadoop's number of map/reduce slots.

3) Why is solution different from previous work?

YARN mainly have 10 unique features to make it different from previous works.
1. YARN extends Hadoop's scalability.
2. YARN meets the requirement of Multi-tenancy needs.
3. YARN has the ability of decoupling the upgrade dependencies (which may be one of Mesos's job).
4. YARN can achieve locality awareness.
5. YARN can perform high cluster utilization
6. YARN needs to enable fault tolerance among multiple tenants Hadoop cluster.
7. YARN maintains secure and auditable operations.
8. YARN must support for programming model diversity.
9. YARN has a flexible resource model.
10. Since YARN is second version of Hadoop, it must achieve backward compatibility.

4) Does the paper (or do you) identify any fundamental/hard trade-offs? 


This two layer scheduling is a better way than centralized scheduling scheme. However, in the two lay scheduling architecture, each AM do not have a global view of the cluster resources, thus each AM could not achieve the best computational resources utilization.

5) Do you think the paper will be influential in 10 years? Why or why not?
Yes, since it is the second generation of Hadoop and has many novel attributes mentioned above, it definitely will be influential in 10 years.