Review for Scaling Distributed Machine Learning with the Parameter Server (OSDI'14)

Current powerful and complex machine learning model can have 10^9 to 10^12 parameters. The parameter sharing among work nodes in the cluster is challenging. There are mainly 3 challenges:
1 parameter access require high network bandwidth.
2 synchronization and machine latency may hurt the system performance.
3 fault tolerant cannot be guaranteed in the cloud computing environment.

The Parameter server (PS) is trying to manage the parameters among the nodes within the cluster. Briefly, there are mainly five unique characters: 1 Efficient communication, 2 Flexible consistency models, 3 Elastic Scalability, 4 Fault Tolerance and Durability, 5 Ease of Use. And this is the main difference between this scaling ML parameter server and other related works.

The parameter server that the authors proposed is the 3rd generation PS system. It mainly contains two parts: the server group and the work group. For each work group it only report updates to its own corresponding server node, instead of to all the node in the server group. All server nodes partition parameters keys with consistent hashing.

The tread-off here is that there is no feature balance within the cluster. More precisely, if a working node contains 90% or more features of the machine learning model. This is not a good load balance among all the workers.

In addition, there is another trade-off of asynchronous calls. Even it main use less time for data processing compared with synchronized system. However, it may need more iteration during data processing.

Since machine learning is vital to nowadays cluster computing, I think this parameter server can be influential within 10 years. Maybe there is some more features added in the Distributed parameter server.