Near-Storage Accelerator for Bulk Graph Ingestion

Due to the ever-increasing interest in using large-scale graph analytics on important applications including security, bioinformatics, and social network analysis, achieving high performance with graph databases and analytics platforms on such large datasets is also becoming an increasingly crucial issue. One important component of graph database performance is graph ingestion, because the rate of data collection is often vastly higher than what graph databases can support while managing the inherently complex data structures necessary to maintain a graph. In this work, we present a system for fast bulk ingestion of large graphs, while only using two relatively general-purpose near-storage accelerator modules: A JSON parser and a sorting accelerator. The sorting accelerator efficiently transforms random insertions into the index and sparse graph data structures into a sequential stream for efficient storage update. Using a prototype implementation on a Samsung SmartSSD, we demonstrate that our accelerator can reduce the host server utilization of processor and memory to almost zero, without performance impact. The fully in-storage operations also remove the memory capacity limitation, meaning it can also ingest much larger graphs than existing columnar and graph databases such as MonetDB and Neo4J.