Factorized models in neural architecture search: Impact on computational costs and performance

The current quest for environmentally sustainable AI models has led to NAS algorithms that not only pursue model accuracy, but also energy efficiency in a simultaneous manner, which increases the architecture optimization problem complexity. In this paper, we aim at simplifying this problem by using factorization. More specifically, we analyze the fitness landscapes of NAS from the point of view of search distributions related to the accuracy and computational complexity of the neural network architectures. For these search distributions, we extract probability factorizations and compute a number of descriptors of the search difficulty for methods based on probability distributions. Our results reveal that each objective can induce search distributions with different strengths of interactions between variables. Therefore, methods that intend to optimize the computational cost of NAS can require different types of factorizations than those focused on the accuracy.