Extraction and Optimization of Mathematical Knowledge Graph from Handwritten Math in Lecture Videos

Almost all existing techniques to do information retrieval from video containing handwritten mathematical expression, work on the static keyframe image extracted from the video. On the other hand, there is good progress in annotating video after converting it to a knowledge graph format. We combine both the techniques to achieve Mathematical Knowledge Graph (MKG) and optimize it to extract LATEX information to recognize mathematical expressions from the video at any point of time. Advantage of this technique is the consideration of the temporal aspect of the mathematical video. Additionally, this opens the paradigm of processing a mathematical video stream which is not prerecorded. Here we detect the discontinuation of ink-stoke in the video to process, recognize and add nodes to the MKG. Training for the classification of the mathematical symbol is done using the CHROME dataset and for typeset symbols, we used generated images.