A deep learning-based unsupervised learning method for spatially resolved transcriptomic data analysis

Spatially resolved transcriptomic data provide a large quantity of high-throughput gene expression and spatial structure information of tissues. Spatial clusters obtained by spatial transcriptome helps us to identify co-expressed regions and gene modules corresponding to cell types. In this study, we developed a Deep learning-based spatial clustering algorithm (RkDeep) by combining Ratio-cut and k-means. We first preprocessed spatial transcriptome data using graph neural network, and conducted dimensional reduction on the preprocessed data with denoising autoencoder. Finally, we clustered spatial transcriptome data by combining ratio cut and k-means. We compared our proposed RkDeep method with the other two spatial clustering methods, Seurat and Panoview. The results show that RkDeep computed the smallest Davide-Bouldin index and the largest Caliniski Harabaz index, adjusted rand index and normalized mutual information. Moreover, RkDeep was applied to analyze spatial transcriptome data of adult mouse brain, adult mouse kidney, and breast cancer. The results show that RkDeep can more accurately identify cell types from spatial transcriptome data.