Design and implementation of quickly building artificial intelligence training environment on high-performance server

In the artificial intelligence algorithm training process, it is a very time-consuming thing for researchers to configure the environment required by the code on the shared servers. In view of the difficulty to configure software operating environment on the server side and the pain that different researchers easy to conflict with each other in the research of big data and artificial intelligence when multiple people share multiple high-performance servers, a cooperation training platform between desktop and high-performance server based on SLURM, Singularity and Anaconda was proposed. At first, Anaconda was used to complete the software prototype development and simple running test in the desktop environment; Then a single file created by the Singularity software which packaged the existing environment and code was uploaded to a remote high-performance server. At last, SLURM and Singularity technology are used to start the operation of the single file on some high-performance servers and calculate the experimental results. A typical deep neural network training process was taken as an example to demonstrate the feasibility of the experimental data platform. The experiment shows that the experimental data platform can quickly configure the same experimental environment on the server and desktop, and has the characteristics of security, easy management and high efficiency.