A cross-modal transfer approach for histological images: A case study in aquaculture for disease identification using zero-shot learning

This research proposal is focused on using knowledge transfer and cross-modal learning, by tackling the problem of modeling biologist's skills to identify unseen objects on histological images solely based on the information obtained from domains such as text in scientific literature. Current techniques make use of deep learning approaches to tackle image classification problems, however, training a deep neural network involves a large number of images and, considering the high cost, time and requirement of highly specialized personnel, it's not convenient to focus efforts in increasing the size of the dataset. For this reason, this research proposal evaluates a cross-modal transfer approach, with the intention of generating alternative sources of information that complement the current knowledge provided by the images. The cross-modal approach starts from a semantic analysis performed with text mining techniques to a large text corpus of scientific documentation with textual description of shrimp diseases. Semantic associations (clusters of syntactic components in scientific documents) found in the documents will be used to generate vector representation of the diseases. Through a mapping process, using neural networks, vectors obtained exclusively from images will be associated with resulting vectors from the semantic grouping. This will allow that a new input data not belonging to any known class in the image domain, can have a vector representation obtained from the text domain.