Review for Spark SQL(SIGMOD'15)

1) Is the problem real?
MapReduce is powerful, but the low-level programming for declarative query is complicated for programmer. This is true. Furthermore, users may want to query on both relational and complex procedural algorithms. And this is partially true. However, this is the key motivation of this paper.

2) What is the solution’s main idea (nugget)?
Spark SQL contains two parts: DataFrame API and Catalyst.

DataFrame SQL enables users intermix procedural and relational code.  DataFrame API is based on DataFrame, which is like a table in relational database. DataFrame SQL is the main part of SparkSQL.

Catalyst is a extensible optimizer. Catalyst is a tree of node objects. For rule-based optimization, Catalyst leverages pattern matching functions. And Catalyst conduct optimization in 4 aspects. 1) analyzing the logical plan to resolve references 2) logical plan optimization 3) physical planning (only in selecting join algorithm) 4) code generation using quasiquotes in scala to compile parts of queries to Java byte code

3) Why is solution different from previous work?
Spark SQL is based on Shark. Compared with Shark, Spark SQL has more AIP, DataFrames.

Another similar one is DryadLINQ. Compared with DryadLINQ, Spark SQL provides DataFrame interface, and support for iterative algorithm on Spark.

Compared with other system like Hive, Pig, etc, Spark SQL integrate more closely to Spark applications. In addition, Spark SQL enable mix procedural and relational API in the same language.

4) Does the paper (or do you) identify any fundamental/hard trade-offs? 


Spark SQL is still not good at processing low latency guaranteed queries (e.g. BI query, ETL SQL, etc.)

5) Do you think the paper will be influential in 10 years? Why or why not?

Compared with HiveQL, the performance gain is significant. However, as mentioned in the paper's evaluation part, when comparing with Impala, the latency is always higher than Imapla. At least, I think with incorporating with DataFrame, Spark SQL is at least good for more general query, with low latency. I think it will be still influential, since it is an more general one.