Multimodal Machine Learning for Display Panel Defect Layer Identification

Process control in display mass production needs defect layer identification to estimate the process that causes fault conditions, which is crucial for quality fault prediction and monitoring. Defect layer identification is a labor-intensive task that requires domain knowledge of experts to make consistent decisions over multiple datasets. Sometimes, it also requires examining multiple sources of inspection images to make the final identification. In this paper, we propose a multimodal machine learning model for defect layer identification as a classification problem from multiple sources of input images. After training two single modal models from each input source, we develop a final joint fusion model to achieve the best classification performance. This work also demonstrates how to estimate the final fusion model performance without exhaustively trying all possible combinations of single modal models. In order to integrate the approach into an industrial display manufacturing defect data analytic, we propose the data coverage model, which guarantees human-level classification performance within a certain portion of data. The data coverage model improves the robustness of the multimodal model, utilizing confident learning to make a high-confidence predictions for the particle defect layer identification and adding a filter to assess whether data samples are within the coverage based on the model's latent space density estimation. This work uses particle defects data samples in 8 different layers as a combination of TEM and STEM images from display manufacturing fabrication lines. Our experimental results show that the classification accuracy improves substantially by deploying the proposed multimodal model. The results also show that it is possible to implement the data coverage model to achieve the human expert level of defect layer classification for a portion of data to automate the task.