1. Introduction
With the exponential increase of demand for network bandwidth, the transmission rate of optical fiber communication system is developing rapidly, even 1Tb/s will be realized in the near future. The polarization effect of optical fiber is always one of the most important factors that limit the performance of communication system. The polarization effect in optical fiber mainly includes polarization-mode dispersion (PMD) and rotation of state-ofpolarization (RSOP). PMD varies randomly with fiber length, frequency and time. It can be expressed as a threedimensional vector in Stokes space [1]. The magnitude of the vector is differential group delay (DGD), and the direction of the vector is the principal states of polarization (PSPs) [2]. RSOP causes random variation of the polarization state of the optical signal propagating in the fiber. In the actual fiber optic link, RSOP and PMD coexist. Moreover, RSOP affects the PSPs of PMD, which means that our analysis of the combined effects of RSOP and PMD can be equivalent to the analysis of PMD.