Automatic Generation of Frontend Code from Design Interface

Internet technology is constantly evolving, and application software plays a crucial role in various fields. The development of various software relies heavily on the design of user interfaces and the corresponding code implementation. However, implementing front-end code is a highly repetitive and tedious task. Prior to designing the front-end interface based on the requirements document, designers usually create a simpler interface to assist in designing and implementing the actual interface. The specific focus of this study is to generate a domain-specific language (DSL) file based on a screenshot of the front-end interface using deep learning. The pix2code project was the first to propose using deep learning for DSL text generation. However, the pix2code project has several issues: firstly, the DSL language designed in the project lacks an adequate description vocabulary to meet the basic requirements of describing actual web pages; secondly, the project has poor scalability for the DSL language; and finally, the project's generation effectiveness using deep learning models is average. This study expands on the pix2code project by extending the design of the domain-specific language, introducing new descriptive vocabulary, and developing an algorithm to generate a new dataset. These enhancements allow the DSL to better describe various components and their combinations within the interface. Additionally, three improvements are made to the original deep learning model. Firstly, the convolutional network used for image feature extraction is deepened to enhance feature extraction capabilities. Secondly, a text encoder based on Transformer is designed for text feature extraction. Finally, a bidirectional LSTM network is used for decoding. Experimental results show that the proposed model in this study improves the BLEU score from 0.81 to 0.85 on a publicly available dataset, and from 0.547 to 0.575 on a newly created dataset.

I. Introduction

The software development process mainly includes requirements analysis, software design, code implementation, and project deployment. Software design encompasses an important aspect known as user interface (UI) design. The focus of this study is to utilize deep learning methods to assist frontend developers in more effectively and conveniently implement code for user interfaces.