I. Introduction
Optical multilevel modulation [1]–[3], which carries several information bits in a single symbol, will be one of the key technologies to significantly increase the channel capacity of high-speed optical signals beyond the limit of electrical modulation speed (such as 40 Gb/s) by lowering actual signaling speed (symbol rate). It is also expected to effectively improve the efficiency of the spectral bandwidth usage of existing optical amplifiers and fiber transmission lines. Since early times, the use of multilevel amplitude-shift keyings (ASKs), such as quaternary amplitude-shift keying (QASK), has received wide attention [1] because of their efficient bandwidth usage. However, they have not yet been widely used in commercial optical transmission systems. This is mainly due to their severe sensitivity penalty and vulnerability to the signal “intersymbol interference (ISI)” that is induced by practical high-speed electrical driver amplifiers, band-limited modulators, receiver circuits, etc.