Network Selection Over 5G-Advanced Heterogeneous Networks Based on Federated Learning and Cooperative Game Theory

5G-Advanced and Beyond claims a 3D ecosystem with cooperation between terrestrial and non-terrestrial networks to achieve seamless coverage, improve capacity, and enable advanced applications with strict quality of service (QoS) requirements. This complex environment requires a disaggregated Radio Access Network (RAN) deployment with open interfaces, such as the architecture promoted by the O-RAN Alliance. This architecture, supporting the slicing paradigm, is a prominent solution to guarantee dynamism and differentiated traffic management. Furthermore, intelligence is critical for future wireless networks to enable Machine Learning (ML)-based optimization for autonomous RANs, handling ultra-dense heterogeneous environments, and adapting to numerous scenarios. This paper presents an enhanced Dynamic Radio Access Network Selection (eDRANS) algorithm based on Federated Double Deep Q-Network (F-DDQN) and inserted in the novel O-RAN architecture. The proposal selects the most suitable base station (BS) to satisfy multiple service requests, optimizing QoS and slicing resource utilization. Moreover, the solution employs a Cooperative Game Theory (CGT) approach to manage resources in overload situations. This load-balancing process enables the acceptance of new clients without abruptly degrading the active users' perception. eDRANS is adapted to diverse network conditions, multiple service constraints, and several user types with different priorities and mobility behaviors. The proposal is validated through network-level simulations, recreating a heterogeneous environment composed of terrestrial-airborne nodes and using the Max-SINR criterion, a heuristic algorithm, and centralized and distributed ML solutions as benchmarks. Results show that eDRANS correctly learns during multiple trial-and-error interactions with the environment, fulfilling the Service Level Agreement (SLA) and maximizing user satisfaction.