The problem of graph processing is that it may not scale well and also it do not support for the fault tolerance. In addition, most of graph processing system are specially designed which is not well suited for general purpose graph processing. Furthermore, the graphs are often part of the data workflow which also contains unstructured and tabular data.
On the other side, general purpose distributed data flow frameworks are well suited for deal with the problems mentioned above. Thus, the author tries to convert previous specialized graph processing frameworks into a general purpose distributed dataflow framework. Therefore, the author build up a general purpose graph processing system on top of Apache Spark, called GraphX. By incorporating with spark, GraphX can enable users to easily and efficiently express the graph analytics pipeline.
The basic idea of GraphX is illustrated as follows. Basically, it first leverages flexible vertex-cut to split the graph into small pieces, and then encoding the split graphs as collections in Spark. In addition, since GraphX is based on Spark, it is naturally inherited of fault tolerant by leveraging the lineage and logical partitioning.
The difference of GraphX and other existing schemes is that, GraphX can achieve the composition of graphs with unstructured and tabular data. In addition, by leveraging logical partitioning and lineage in Spark, GraphX can achieve fault tolerance.
The trade off here is, since GraphX is a more general system designed for graph processing, compared with specialized graph processing system, GraphX may need more time for running the same job. It is because that specialized system are optimized for only a group of graph problem, thus it can achieve better optimization.
I think to design and implement a graph processing system which can serve as a general purpose architecture is very useful. And Spark is now on fire. Therefore, I think GraphX can also be influential within 10 years.
No comments:
Post a Comment