Automatic Job Safety Report Generation using RAG-based LLMs

This study introduces an innovative approach to safety report generation using a Retrieval-Augmented Generation (RAG) framework, tailored to synthesize comprehensive reports from descriptions and logs of work sessions. The core contribution of our study is the comparison and optimization of various Large Language Model variants (based on LLaMA) and embedding models, aiming to identify the most effective combination for accurately capturing and reflecting the intricacies of safety-related data in a given domain. Our RAG-based system leverages the strengths of different LLaMA models and embedding techniques to process and contextualize the input data, which include detailed session descriptions and operational logs. By integrating these models, we aim to automate the generation of safety reports that are not only coherent and contextually relevant, but also adhere to the stringent requirements of safety documentation in professional environments. The validation of our approach is performed using an aviation safety dataset and classic metrics in the field, such as Recall@5, GLEU, METEOR, and BERTscore. Our findings demonstrate the potential of RAG-based systems in streamlining the process of safety report generation, offering significant improvements in efficiency and accuracy over traditional methods and non domain-specific tailored models.

I. Introduction

In modern times, safety regulations require continuous monitoring of the workers’ activities and their workplace, with the aim of improving the employees’ production and ensuring their safety. Indeed, some studies show the positive impact of the continuous monitoring of occupational safety on the health of workers [16]. In this context, human-centered Artificial Intelligence (AI) represents a new challenge for the organizations that research new tools and solutions to automate safety management procedures and perform accident prevention [9]. According to this, several studies have been conducted, in the last years, to evaluate AI-assistance systems based upon sensors [9] and of AI-based approaches for the analysis and generation of incident reports in different industrial domains (e.g., agriculture, aviation, medicine, construction, and railroad industry) [4], [9]. More recently, Large Language Models (LLMs) seem to enhance the efficiency of safety analyses and reduce the time required to process incident reports [4]. LLMs, trained on large datasets, can perform several language-related tasks without any specialized fine-tuning. This makes them potentially useful for the automatic analysis of safety reports, for the identification of patterns and anomalies, and for the detection of safety issues, risks, and strategies (also on the base of historical data). Finally, LLMs can be useful to complement and sometimes replace human expertise. They can also perform training activities and safety tests, since they can simulate several scenarios or generate synthetic data. Despite the great potentialities of LLMs in safety management, very few studies propose and evaluate these models in real scenarios.