Spark standalone mode process analysis and data skew solutions

This paper is divided into two parts, the first part details the Spark standalone mode execution process, in-depth understanding of it will be the prerequisite and basis for our program tuning. The second part elaborates several scenes of Spark data skew, this paper presents six methods to solve the data skew according to different scenarios, including aggregated source data and filtering leading to skewed keys, increase the parallelism of the shuffle operation, use random keys to implement double aggregation, convert reduce join into map join, sampling skew key alone for join, use random numbers and expansion tables for join, and proved that the above method can effectively solve the problem of data skew.