I. Introduction
With the rapid development of Internet of Things (IoT), there is an urgent need for high-quality wireless communication systems. Among the potential enabling technologies for future IoT networks, multiple-input-multipleoutput (MIMO), a technology with high spectral efficiency, has been regarded as a crucial technology and has got broad application [1], [2], [3], [4]. In addition, MIMO systems usually combine with nonorthogonal multiple access (NOMA) for rational use of the spectrum, which is termed the MIMO- NOMA technology and has been widely studied and applied [5], [6], [7], [8]. However, in the long-distance communication, the feasible transmission distance between the base station and the IoT devices or users may not support a stable and efficient communication condition, in which case a relay station is required to improve link quality [9]. This promotes the development of cooperative MIMO- NOMA transmission research [10], [11], [12]. The relaying techniques can be classified into two types: 1) amplify-and-forward (AF) relaying and 2) decode-and-forward (DF) relaying. Compared with DF, AF has a simpler implementation and a lower dissipation [13], which makes it more attractive in practice. The precoding schemes of MIMO with an AF relay can be developed from conventional MIMO transmissions, which have been studied in [14], [15], and [16]. There are mainly two types for MIMO precoding schemes: 1) nonlinear precoding [17], [18], [19] and 2) linear precoding [20], [21]. The nonlinear precoding is too complex to be often employed in practice, while linear precoding schemes can better balance efficiency and complexity and thus have been extensively used. The generalized singular value decomposition (GSVD) has stood out among the linear precoding schemes because of its implementation simplicity and advantageous performance, especially when the signal-to-noise ratio (SNR) is high [22], [23], [24]. In the MIMO- NOMA- GSVD-based scheme, the channel matrices of two users are diagonalized simultaneously to achieve the purpose of transforming an MIMO channel into several parallel single-input–single-output (SISO) channels. Then, the NOMA principle can be applied to each SISO channel individually. Chen et al. [23] have studied a GSVD-based two-user MIMO- NOMA download communication system. Hanif and Ding [25] have analyzed the perturbation of GSVD- NOMA scheme with imperfect channel state information. Rao et al. [24] have considered the performance of GSVD- NOMA from the single-channel perspective, where the distribution characteristic of the ordered GSVD has been analyzed. In [26], the GSVD- NOMA scheme with the security constraint has been studied. Gerdes et al. [22] and Rao [27] have studied the GSVD-based MIMO- NOMA system with a DF relay.