Fine Tuning Idefic 9b With LORA for Multimodal Medical VQA

We introduce an innovative approach to Medical Visual Question Answering (VQA) by fine-tuning the Idefic 9b model on a combined dataset from SLAKE and VQARAD. Our methodology focuses on loading the model using a 4bit quantization method, striking a balance between computational efficiency and model performance. To enhance the model’s ability to adapt to specific tasks, we employ Low Rank Adaptation (LORA) to fine-tune the model’s attention layers. This technique contributes to reducing the model’s trainable parameters. For evaluation purposes, we employ the BLEU (Bilingual Evaluation Understudy) scores, a widely accepted metric in Natural Language Processing, to evaluate the generated responses quality. The results of our approach in the field of Medical VQA are promising, paving the way for the development of more advanced and efficient models in the future. This paper aims to offer insights into the effective application of fine-tuning techniques, showcasing their impact on enhancing the performance of VQA models in the medical domain.