Integrating Latent Dirichlet Allocation and Gradient Boosting Tree Methodology for Insurance Product Development Recommendation

In recent years, with the advent of the post-pandemic era, shifts in consumer behavior have presented challenges to financial conglomerates in effectively discerning market demands for financial products, thereby resulting in customer attrition. To efficiently manage customer requirements and support product managers in their decision-making processes, this study embarks on an exploration of machine learning algorithms and methodologies for unstructured data analysis. Moreover, it amalgamates the Latent Dirichlet Allocation (LDA) approach with diverse machine learning strategies to forecast demand for insurance products on e-commerce platforms. The findings of the research elucidate: (1) A two-stage model, integrating LDA and Gradient Boosting Tree, demonstrates the capacity to accurately assess customer product preferences. This model exhibits superior accuracy and robustness when compared to other benchmark models. (2) Through the analysis of unstructured data, invaluable insights into the industry are garnered, empowering the finance and insurance sectors to comprehend consumer requirements, pinpoint market opportunities, and expedite the development of novel products. (3) Within the contemporary economic milieu, the adoption of the Direct-to-Consumer (D2C) paradigm and the integration of consumer insights into decision-making frameworks are pivotal for fostering sustainable business growth in the finance and insurance domains.