Previous solutions for maximizing cluster utilization efficiency has been focusing on computing and storage usage. However, the network resources have been ignored.
In the paper, the author argue that in order to achieve good job performance, we should focus on transfer instead of each flow. A transfer is defined as all data flow transmitted between two stages of a job. They mainly focus on two transfer types, shuffle and broadcast. In intra-transfer activities, they design a Transfer Controller. For broadcast, they propose a transfer controller scheme called Cornet. For shuffle, they present weight shuffle scheduling. For inter-transfer, they design a simple ITC (inter-transfer controler), which can support FIFO, priority scheduling, etc. The basic idea of ITC is to achieve a weighted fair sharing.
Previous works have been analyzing and optimizing the network performance. They propose flow-level scheduling to improve the performance. However, it lacks of job-level semantics, thus cannot schedule collective behaviors.
The trade-off here is between utility and deployability. Orchestra can be implemented at the application level. Therefore it can be directly implemented into existing clusters without any modifications on hardware or management mechanism. However, this application-level transfer control cannot achieve perfect control over the network.
As a networking guy, I like the idea of Orchestra. And since it has been used in Apache Spark for shuffle process, I believe it will still be influential in 10 years.
No comments:
Post a Comment