I. Introduction
Digital backpropagation (DBP) has been proposed as a universal technique for jointly compensating for the intra-channel linear and nonlinear impairments in the coherent fiber-optic system [1]–[5]. As a result, the DBP has been used to benchmark schemes proposed in the literature [6]–[10]. The assumed optimality of DBP has spurred intense research in low-complexity variations, including weighted DBP, perturbation DBP, and filtered DBP [10], [11]. While the focus of the current paper is on single-channel systems, for wavelength division multiplexing (WDM) systems, DBP is typically employed for the center channel, thereby accounting only for the intra-channel effects. Inter-channel nonlinear effects in WDM systems can be modeled by taking the advantage of the temporal correlations of the nonlinear phase noise using a time-varying system with inter-symbol interference and thereby compensating for these inter-channel nonlinear effects [12]– [14]. While DBP has received a great deal of attention, it only deals with deterministic linear and nonlinear impairments and inherently does not consider noise. It is known that the nondeterministic nonlinear effects, such as nonlinear signal–noise interaction (NSNI) between the transmitted signal and the amplified spontaneous emission (ASE) noise, limit the transmission performance of a fiber-optic system [10], [15], [16]. Studies on the impact of NSNI reveal that there is a significant penalty due to NSNI for inline optical dispersion-managed (DM) links, and the severity of the NSNI is dependent on modulation formats and the symbol rate used in the system [17], [18]. It is often argued that NSNI cannot be compensated for in digital signal processing due to the nondeterministic nature of ASE noise [18] and as a result, none of the DBP methods account for NSNI. To deal with stochastic disturbances, Bayesian detection theory can be used to formulate maximum a posteriori probability (MAP) detectors, which are provably optimal in terms of minimizing the error probability. MAP detectors have been proposed for the discrete memoryless channel [19] assuming perfect chromatic dispersion compensation in a DM link, and a look-up table detector that can mitigate data-pattern-dependent nonlinear impairments [20]. A low-complexity Viterbi detector is suggested as an alternative or to complement DBP for combating fiber nonlinearities [21]. In [22], the stochastic digital backpropagation (SDBP) algorithm was proposed to compensate not only for deterministic linear and nonlinear effects, but also to account for the ASE noise. However, the decisions were taken on a symbol-by-symbol (SBS) basis after applying a matched filter (MF). This approach was later shown to be suboptimal [23].