1. Introduction
In the general sense the modulation technique allows coding to prevent transmission errors from occurring (‘line coding’) or to correct for already occurred transmission errors (‘error correcting coding’) [1]. By reaching transmission rate 40 Gb/s with utilizing of on-off keying modulation format at the turn of the millennium, optical high-order modulation formats and coding became very important. The steadily growing demands for capacity in fiber-optic communication networks forcing use new techniques which utilize more efficient the available bandwidth in optical fibers. High-order modulation formats have attracted much interest since they allow increased spectral efficiencies by encoding multiple bits per one transmitted symbol [1], [2]. Higher optical transmission capacity can be achieved by using high-order modulation formats, without deployment of new optical fibers. Generally, the implementation of high-order modulation formats in fiber optic transmission systems is very significant goal for the next generation of communication systems [3], [4]. The experimentally demonstrated record for the aggregate capacity over a single optical fiber is currently at 25.6 Tb/s transmission over 240 km using 160 WDM (wavelength division multiplex) channels on a 50 GHz grid in the C+L bands. Every channel contains two polarization-multiplexed 85.4 Gbit/s RZ-DQPSK (return-to-zero differential quadrature phase-shift keying) signals, yielding a spectral efficiency of 3.2 b/s/Hz in each wavelength [1].