Indoor Localization Using K-Pairwise Light Emitting Diode Image-Sensor-Based Visible Light Positioning

A great deal of research has been devoted to studying indoor localization schemes for the development of location-based services. This study investigated an indoor localization system, including subsystems of nonline-of-sight (NLOS) image-sensor-based visible light communications (IS-VLCs) and the proposed K-pairwise light-emitting diode (LED) image-sensor-based visible light positioning (IS-VLP), in a unified lighting environment. A smartphone received multiple LED identifiers through the designed NLOS IS-VLC and successfully performed the proposed K-pairwise LED IS-VLP with K = 3 to achieve localization, with maximum localization errors of 2.92 and 3.54 cm being obtained for ceiling heights of 100 and 150 cm, respectively.

SECTION 1. Introduction

Recently, an increasing number of location-based services (LBS) have been implemented to facilitate the experience of human daily life. People spend approximately 80%–90% of their time indoors, which has become a significant factor for LBS applications [1], [2]. Highly precise localization systems are expected to be helpful for indoor LBS applications. Because wireless local access network and sensor networks have been installed in many buildings, numerous approaches have been made to use these networks for localization acquisition [3], [4]. The primary goal is to describe the distance between the receiver and a reference grid, i.e., a number of known reference points, as a function of the received signal strength (RSS), and then estimate the distance from the receiver to the reference grid by comparing the measured field strength with the actual field strength. The major challenge of these radio-based RSS localizations is the variations in the RSS that result from the fluctuating and unpredictable nature of radio signals, which can lead to a significant location estimation error in indoor environments. Furthermore, one downside of radio-based RSS localization is its high implementation cost resulting from the need for a wide deployment of network infrastructures.

Lighting plays an essential role in daily life and is considered a fundamental function of indoor environments. Solid-state lighting has been attracting considerable attention because of its versatility and advantages over conventional lighting sources in applications such as monitor backlighting, traffic indicators, and general illumination. Taking advantage of low energy consumption, long lifetime, small size, and easy production, light emitting diodes (LEDs) are replacing the incandescent lamp as a dominant part of the next generation of lighting sources. Indoor localization using visible light communication (VLC) based on LED lighting has been considered. In contrast to the phenomena of significant radio signal scattering and diffusing in indoor environments, optical signals from LED lighting are relatively stable because lighting sources are often fixed to ceilings, which facilitates line-of-sight (LOS) transmission. Localization systems must set a reference grid and then obtain the receiver position according to the distances between the receiver and various reference nodes. Several ranging methods have been developed to obtain the distance information required in the positioning algorithm, including time of arrival (TOA), time difference of arrival (TDOA) [5], [6] , and RSS [7], [8]. For propagation-time-based ranging techniques, including TOA and TDOA, the inaccuracy of time synchronization often leads to ranging errors. In the context of LED lighting, a photodetector (PD) can easily obtain the RSS without any additional hardware to calculate the position of the target by using the distance measurement from at least three reference nodes. However, the average RSS of every position in the region of interest (ROI) should be pre-established. This raises implementation costs and necessitates the inconvenient execution of numerous RSS measurements in the ROI in advance.

Currently, smart devices are typically equipped with camera modules that have been used to implement image-sensor-based VLC (IS-VLC) signal reception, specifically leading to the development of indoor localization technology based on the image sensors of camera modules [9]–​[11]. In [9], an indoor positioning system based on two LED luminaries was implemented with the assumption that LED coordinates were provided. Moreover, its localization accuracy was degraded because of misalignment between ceiling and floor coordinates. Image-sensor-based trilateration was developed and implemented in [10], [11]. The focal length of the camera module was assumed to be known in [9], [10], but the focal length of the camera module is a specified parameter in the module and might differ from one smartphone to another. In this paper, we propose and demonstrate indoor localization using $K$-pairwise LED image-sensor-based visible light positioning (IS-VLP), including transmission of the system coordinates of LED luminaries using IS-VLC and improvement of localization accuracy by combining positioning results with several pairwise LED IS-VLP. The proposed $K$-pairwise LED IS-VLP is capable of independently calculating position estimates based on $K$ distinct pairs of LED luminaries, and then combining these position estimates to determine a final localization using an arithmetic mean for localization accuracy improvement. In the scenarios of three LED luminaries and the ceiling positioned 100 cm or 150 cm away from the smartphone, the experimental results revealed that the maximum localization errors of the $K$-pairwise LED IS-VLP with $K = 3$ were 2.92 cm and 3.54 cm for the ceiling heights of 100 cm and 150 cm, respectively, which outperform the existing image-sensor-based trilateration and single pairwise LED IS-VLP.

SECTION 2. The Proposed Localization System

2.1 System Model

The system model of the proposed $K$-pairwise LED IS-VLP is presented in Fig. 1, consisting of ${N_{{\rm{LED}}}}$ LED luminaries ${L_i}$ installed on the ceiling and a smartphone$S$, where ${N_{{\rm{LED}}}} \geq 2$ and $h$ denotes the distance between the ceiling and the smartphone. The system assigns each LED luminary with a unique identity (ID) indicating its system coordinates $({{x_i},{y_i},{z_i}})$, and the system transmits the ID information associated with each LED luminary to the smartphone through IS-VLC. After receiving the ID associated with ${L_i}$, the smartphone was able to determine its approximate position. On the smartphone image sensor, the image of ${L_i}$, denoted as ${I_i}$, can help to more accurately calculate positioning. We can construct a coordinate system on the image sensor, termed as image sensor coordinates, and denote ${I_i}$ as $({i{x_i},i{y_i}})$. Moreover, ${I_S}$, denoted as $({i{x_S},i{y_S}})$, represents the image sensor coordinates of the center of the image sensor. Hence, the three-dimensional coordinates of the positioning system in Fig. 1 can be projected onto the two-dimensional $xy$-plane by viewing the positioning system from the top down, as illustrated in Fig. 4, which will be discussed later. Notably, the center of the image sensor and the position of the smartphone are regarded as the same point and are represented as $({i{x_S},i{y_S}})$ in image sensor coordination and $({{x_S},{y_S}})$ in system coordination, respectively.

Fig. 1.

System model of the proposed $K$ -pairwise LED IS-VLP.

2.2 The Principle of ID Assignment With Orientation

The block diagram of the proposed localization system, which consists of IS-VLC and $K$-pairwise LED IS-VLP, is displayed in Fig. 2. The smartphone captures a photo to obtain LED IDs through IS-VLC. In the ROI, the smartphone may receive more than one LED IDs but does not recognize the mapping between the received LED IDs and their images on the image sensor. For the indoor localization system prototype, we propose adopting a simple method of assigning IDs to LED luminaries in accordance with the orientation of LED luminaries relative to other ones. The ID number assigned to LED luminaries increased from the southwest to northeast in the system coordinates; in other words, the ID of the LED luminary ${L_i}$ was assigned a larger number than that of the LED luminary ${L_j}$ when ${L_i}$ was situated to the northeast of${L_j}$. Accordingly, the smartphone could recognize the relationship between the received IDs and their corresponding images on the image sensor.

Fig. 2.

Block diagram of the proposed localization system.

2.3 Indoor Localization With $K$ -Pairwise LED IS-VLP

Based on the principle of ID assignment with orientation, the smartphone can identify the mapping between the received LED IDs and their images on the image sensor and determine the system coordinates of those LED luminaries. The proposed $K$-pairwise LED IS-VLP calculated the exact localization of the smartphone by using the geometric relationship between the LED luminaries and their images on the smartphone image sensor. The operational steps of the proposed localization system are as follows:

1. Acquiring system coordinates through IS-VLC: The system coordinates of LED luminaries $({s{x_i},s{y_i}})$ are acquired through the subsystem of IS-VLC. Currently, most IS-VLC are discussed in the context of LOS. However, LOS transmission could cause strong image sensor blooming effect when detecting modulated optical signals [12] . Moreover, the smartphone camera should always align with the LED luminaries in the LOS scenario. This might limit the reception of system coordinates. To enable users to easily receive LED IDs, we implemented non-line-of-sight (NLOS) IS-VLC, in which the smartphone camera receives the optical modulated signal from surrounding reflective surfaces. The smartphone uses the rear camera to capture a photo with a size of ${N_r}$ rows and ${N_c}$ columns, and the modulated optical signal is indicated by bright and dark fringes on the image when we properly set the exposure time and ISO level of the camera.

   For further image processing to extract the LED IDs, the captured image was converted into a grayscale format. Notably, we only used the central width $w$ of the captured image to detect the logic values on the image. This not only reduces the computation of image processing but also prevents the effect of shadowing caused by objects surrounding the smartphone. As illustrated in Fig. 3(a), because of an NLOS transmission, the image was not subjected to the blooming effect. As portrayed in Fig. 3(b), for the central width $w$ of the image, the grayscale values were averaged for each row, resulting in an ${N_r} \times 1$ column vector ${{\bf G}} = [ {{{\bar{g}}_1},{{\bar{g}}_2}, \ldots,{{\bar{g}}_{{N_r}}}} ]$, in which ${\bar{g}_i}$ denotes the grayscale values of the central $w$ pixels on the $i$ -th row. The data were demodulated by using second-order polynomial fitting for the averaged row pixel values in ${\rm{G}}$. As represented in Fig. 3(c), these grayscale values were regressed to build a second-order polynomial decision threshold. Thus, the averaged row grayscale values could be demodulated to logic values and the IDs of LED luminaries could be retrieved according to the threshold.

2. Constructing a geometric relationship between system coordination and image sensor coordination: After acquiring the LED IDs, the smartphone identifies the system coordinates of each LED luminary and begins to construct a geometric relationship between system coordination and image sensor coordination. The smartphone begins to extract the image sensor coordinates $({i{x_j},i{y_j}})$ of the LED luminaries ${L_j}$ through its front camera. From the use of any two LED luminaries, for example, ${L_i}$, $i=1, 2$ , the geometric relationship among the system coordinates $({s{x_i},s{y_i}})$ of the LED luminaries ${L_i}$, the system coordinates $({{x_S},{y_S}})$ of the smartphone $S$, the image sensor coordinates $({i{x_S},i{x_S}})$ of the center on the image sensor ${I_S}$, and the image sensor coordinates $({i{x_i},i{y_i}})$ of images of the LED luminaries ${I_i}$ can be determined and shown in Fig. 4, from the top-to-down view along the $z$-axis. Notably, this geometrical interpretation can be viewed as an orthogonal projection of the localization system onto the xy-plane. Thus, the smartphone position $S$ and the center of the image center ${I_S}$ are viewed as the same point in Fig. 4, and the focal length between the image sensor and the camera lens is not included in this geometrical interpretation.

   

   Fig. 3.

   (a) One image showing the bright and dark fringe in the scenario of NLOS. (b) The central ${N_r} \times w$ matrix for data demodulation. (c) Grayscale values in ${{\bf G}}$ (blue line) and the threshold (red line).

   

   Fig. 4.

   Relationship between the system coordinate system and the image-sensor coordinate system.

   By using typical image processing, including the process of grayscale conversion, Gaussian filtering, image thresholding, and Canny contour detection, the image sensor coordinates of the LED luminaries $({i{x_i},i{y_i}})$, $i=1,\,2$, can be calculated. Based on the similar relationship between triangles formed by points $\{ {{L_1},M,S} \}$ and $\{ {{I_1},{I_M},{I_S}} \}$ or by the points $\{ {M,{L_2},S} \}$ and $\{ {{I_M},{I_2},{I_S}} \}$ displayed in Fig. 4, the smartphone system coordinates were estimated as follows:

   $$\begin{equation} \left\{ {\begin{array}{l} {{x_s} = s{x_1} + (i{x_1} - i{x_s}) \times \frac{{s{w_1}}}{{i{w_2}}}}\\ {{y_s} = s{y_1} + (i{y_1} - i{y_s}) \times \frac{{s{w_1}}}{{i{w_2}}}} \end{array}} \right.\quad {\rm{or}}\quad \left\{ {\begin{array}{l} {{x_s} = s{x_2} + (i{x_2} - i{x_s}) \times \frac{{s{w_1}}}{{i{w_2}}}}\\ {{y_s} = s{y_2} + (i{y_2} - i{y_s}) \times \frac{{s{w_1}}}{{i{w_2}}}} \end{array}} \right.\tag{1} \end{equation}$$

3. Indoor localization using $K$-pairwise LED IS-VLP: In a practical indoor space, ceilings are not often parallel to the floor, which results in a misalignment between the LED luminary and floor coordinates [9]. To compensate for the localization error caused by the misalignment between these coordinates, we propose the adoption of $K$-pairwise LED IS-VLP to possibly reduce the localization error by only using one pairwise LED IS-VLP. When the smartphone registers ${N_{{\rm{LED}}}}$ LED luminaries on its image sensor, where ${N_{{\rm{LED}}}} \geq 2$, we are able to direct $K$-pairwise LEDs to execute the positioning method in (1), in which $K = ({\begin{array}{c} {{N_{{\rm{LED}}}}}\\ 2 \end{array}})$. For example, three pairwise LEDs are available, including the ${L_1}$ and ${L_2}$, ${L_2}$ and ${L_3}$ , and ${L_1}$ and ${L_3}$ pairs in the scenario presented in Fig. 1. According to (1), we are able to acquire the estimated coordinates of the smartphone ${{{\bf p}}^{({i,j})}} = ({x_S^{({i,j})},y_S^{({i,j})}})$ by using the ${L_i}$ and ${L_j}$ pair. The final localization of the smartphone ${{{\bf p}}^{(f)}} = ({x_S^{(f)},y_S^{(f)}})$ could simply be calculated by using an arithmetic mean of all ${{{\bf p}}^{({i,j})}}$ as follows:

   $$\begin{align} x_S^{\left(f \right)} & = \frac{1}{K}\mathop \sum \limits_{\left({i,j} \right) \in Q} x_S^{\left({i,j} \right)},\nonumber\\ y_S^{\left(f \right)} & = \frac{1}{K}\mathop \sum \limits_{\left({i,j} \right) \in Q} y_S^{\left({i,j} \right)},\tag{2} \end{align}$$ where $Q$ denotes the set of all possible LED luminary pairs.

SECTION 3. Experimental Setup and Results

Two indoor lighting and positioning environments were used to verify our proposed method. First, three LED luminaries, ${L_1}$, ${L_2}$ and ${L_3}$ , were installed on coordinates (30, 20, 185), (60, 20, 185), and (45, 50, 185), respectively. Each LED luminary was realized using a phosphor-based white LED (Fedy FD-10 W-S45). We adopted the Arduino Mega 2560 microcontroller board to coordinate LED luminaries and assign each luminary with its own ID through universal asynchronous receiver/transmitter (UART) serial ports. Each LED luminary transmitted its unique ID in the format of on-off keying signaling. ID information emitted from the LED luminaries was registered through the captured photo using the rear camera of the smartphone (ASUS ZE552KL) with the exposure time and ISO settings at 0.2 ms and 3200, respectively. This smartphone is equipped with a SONY IMX 298 image sensor, which is a 16 Mega-pixel CMOS active pixel type stacked image sensor. Moreover, it has a $1080 \times 1920$ pixel resolution, 5.5-inch display, and a ppi density of approximately 401. Only the central $w=150$ pixels were used to extract ID information associated with the LED luminaries.

To evaluate the proposed indoor VLP system, the smartphone was placed at the locations of the predefined testing points (TPs) on the floor at a height of 85 cm. A total of ${N_{TP}}$ TPs were established by considering the following two cases. In the first instance, TPs were assigned to coordinates between (30, 20, 85) and (60, 20, 85), whereas TPs were allocated to form a closed triangle zone, with points at (30, 20, 85), (60, 20, 85), and (45, 50, 85) in the second case. TPs were spaced at 5-cm intervals in both horizontal and vertical directions; hence, ${N_{TP}} = 3 \times 5$ TPs, i.e., ${\rm{T}}{{\rm{P}}_i} = {[ {{x_{TP,i}}\ {y_{TP,i}}} ]^T}$ , $i=1,\ 2,\ \ldots,\ 15$, in both cases. The front camera of the smartphone was used to record the video frame with $640 \times 480$ pixel resolution of the LED luminaries, and these image frames were adopted to implement the proposed $K$-pairwise LED IS-VLP. Notably, the orientation of the smartphone did not align with the axis of the system coordinates in a practical situation. In the following experiments, the rotation angle of the smartphone with the axis of the system coordinates, ${\theta _{rot}}$, was calculated, and the image was inversely rotated to align the orientation of the smartphone with the system coordinates [9].

Fig. 5 presents the experimental results of the single pairwise LED IS-VLP method with four different ${\theta _{rot}}$ values. Without loss of generality, we assume that the ${\theta _{rot}}$ values are ${45^\circ }$, ${135^\circ }$, ${225^\circ }$, and ${315^\circ }$. In this experiment, we adopted two LEDs, ${L_1}$ and ${L_2}$, as reference nodes, and the TPs were set to those specified in case 1. In Fig. 5, the black square markers denote the LED luminary coordinates and the black circle markers represent the TP positions. The rhombus, upper triangle, lower triangle, and star markers represent the localization results when ${\theta _{rot}}$ was ${45^\circ }$, ${135^\circ }$, ${225^\circ}$, and ${315^\circ }$, respectively. The maximum localization error ${E_{\max}}$ was 4.50 cm.

Fig. 5.

Localization results of the $K$ -pairwise LED IS-VLP ($K=1$) with different ${\theta _{rot}}$.

Fig. 6 presents the localization results obtained by using the proposed $K$-pairwise LED IS-VLP, with $K=3$ and LEDs ${L_1}$, ${L_2}$, and ${L_3}$, where TPs were set to those specified in case 2 for ${\theta _{rot}}$ values of ${45^\circ }$, ${135^\circ }$, ${225^\circ }$, and ${315^\circ }$. The positioning results based on the $K$-pairwise LED IS-VLP $({K=3})$ are indicated as follows, the red lower triangle, green rectangle, and yellow diamond markers are the estimated positions based on the ${L_1}$ and ${L_2}$, ${L_2}$ and ${L_3}$ , and ${L_1}$ and ${L_3}$ pairs, respectively. Finally, the blue upper triangle markers denote the estimated localization in terms of an average of the aforementioned positioning results based on the three pairwise LED positions. The maximum localization error ${E_{\max}}$ was 2.92 cm. From the experimental results, we can observe that the pair of LEDs with the estimated positions that were closest to the corresponding TP varied at each TP. When the positions were estimated based on different LED pairs located on the same side of a TP, the proposed $K$-pairwise LED IS-VLP could reduce the maximum positioning error. Moreover, we were able to obtain more accurate localization results from the $K$-pairwise LED IS-VLP when the estimated positions based on different LED pairs located around a TP.

Fig. 6.

Localization results of the $K$ -pairwise LED IS-VLP ($K = 3$) with (a) ${\theta _{rot}} = {45^\circ }$, (b) ${\theta _{rot}} = {135^\circ }$, (c) ${\theta _{rot}} = {225^\circ }$, and (d) ${\theta _{rot}} = {315^\circ }$.

We also implemented the image-sensor-based trilateration (IS-based trilateration) presented in [10] and [11]. Table 1 presents a comparison of the maximum localization error ${E_{\max}}$ with the $K$-pairwise LED IS-VLP ($K = 1$), the $K$ -pairwise LED IS-VLP ($K = 3$), and IS-based trilateration. The results of localization in each of the TPs were different when ${\theta _{rot}}$ varied for every localization method. This was a result of the misalignment between the ceiling and floor coordinates. In the proposed $K$-pairwise LED IS-VLP, namely $K=3$ in the present case, the localization error caused by the misalignment between the ceiling and floor coordinates was successfully decreased.

TABLE 1 The Max Localization Error ${E_{\max}}$ With the $K$-Pairwise LED IS-VLP ($K=1$), the $K$-Pairwise LED IS-VLP ( $K=3$), and IS-Based Trilateration With Different ${\theta _{rot}}$ for 100 cm Ceiling Height

In the second experimental condition, three LED luminaries, ${L_1}$ , ${L_2}$ and ${L_3}$, were installed at coordinates (50, 15, 235), (100, 15, 235), and (75, 65, 235), respectively. TPs were allocated to form a closed triangle zone, with points at (50, 15, 85), (100, 15, 85), and (75, 65, 85), and were spaced at 10-cm intervals in both horizontal and vertical directions; hence, ${N_{TP}} = 4 \times 4$ TPs. Table 2 presents a comparison of the maximum localization error ${E_{\max}}$ with the $K$-pairwise LED IS-VLP for $K=1$, the $K$ -pairwise LED IS-VLP for $K=3$, and IS-based trilateration. Compared with the results in Table 1, it was observed that the ${E_{\max}}$ of IS-based trilateration was increased by 46.38% from 4.7 cm to 6.88 cm. The ${E_{\max}}$ values of the proposed $K$-pairwise LED IS-VLP for the cases of $K=1$ and $K=3$ were increased by 41.78% from 4.5 cm to 6.38 cm and by 21.23% from 2.92 cm to 3.54 cm, respectively. Intuitively, the ranging error, that is, the difference in the estimated distance and the true distance between the receiver and reference nodes, increases with increasing ceiling height. The fundamental idea underlying the use of trilateration to determine the receiver position consists of solving a system of equations that contains circle or sphere equations in the geometric space between the receiver and at least three reference nodes. When the system of equations has no solution, the least square solution has to be used, and hence, large localization errors were yielded due to increases in the ranging error. The proposed $K$ -pairwise LED IS-VLP with $K=1$ also suffered from an increase in ranging error because its localization result was based on only a pair of LED luminaries. The proposed $K$ -pairwise LED IS-VLP with $K=3$ is capable of independently calculating position estimates three times based on three different pairs of LED luminaries, and it then combines these position estimates into a final localization using an arithmetic mean, resulting in a moderate increase in ${E_{\max}}$ when the ceiling height is increased from 100 cm to 150 cm.

TABLE 2 The Max Localization Error ${E_{\max}}$ With the $K$-Pairwise LED IS-VLP ($K=1$), the $K$-Pairwise LED IS-VLP ( $K=3$), and IS-Based Trilateration With Different ${\theta _{rot}}$ for 150 cm Ceiling Height

SECTION 4. Conclusion

This study implements an indoor localization system by integrating NLOS IS-VLC and the proposed $K$-pairwise LED IS-VLP. The proposed $K$-pairwise LED IS-VLP does not require the focal length information of the camera module and can identify the mapping between multiple received LED IDs and their corresponding images on the image sensor according to the principle of ID assignment with orientation. The localization results revealed that the proposed $K$-pairwise LED IS-VLP ($K=3$) effectively reduced the localization error caused by the misalignment between the ceiling and floor coordinates as well as achieved maximum localization errors of 2.92 cm and 3.54 cm for ceiling heights of 100 cm and 150 cm, respectively.