Depth Super-Resolution via Joint Color-Guided Internal and External Regularizations

Depth information is being widely used in many real-world applications. However, due to the limitation of depth sensing technology, the captured depth map in practice usually has much lower resolution than that of color image counterpart. In this paper, we propose to combine the internal smoothness prior and external gradient consistency constraint in graph domain for depth super-resolution. On one hand, a new graph Laplacian regularizer is proposed to preserve the inherent piecewise smooth characteristic of depth, which has desirable filtering properties. A specific weight matrix of the respect graph is defined to make full use of information of both depth and the corresponding guidance image. On the other hand, inspired by an observation that the gradient of depth is small except at edge separating regions, we introduce a graph gradient consistency constraint to enforce that the graph gradient of depth is close to the thresholded gradient of guidance. We reinterpret the gradient thresholding model as variational optimization with sparsity constraint. In this way, we remedy the problem of structure discrepancy between depth and guidance. Finally, the internal and external regularizations are casted into a unified optimization framework, which can be efficiently addressed by ADMM. Experimental results demonstrate that our method outperforms the state-of-the-art with respect to both objective and subjective quality evaluations.

I. Introduction

Depth maps play a fundamental role in many computer vision and computational photography applications, such as 3D reconstruction [1], multi-view rendering [2], virtual reality [3], and robot vision [4] etc. With the progress of sensing technology, depth information of a scene can now be readily acquired by inexpensive cameras, such as Time of Flight (ToF) camera [5] and Microsoft Kinect [6]. Nowadays, RGB-D cameras are ubiquitous and have enabled a large suite of consumer applications. However, the captured depth maps in practice usually have much lower resolution compared with the companion color image. For instance, the depth maps captured by the Time-of-flight (ToF) camera are subject to low resolutions, e.g., and . Many applications, such as 3D object reconstruction, robot navigation and automotive driver assistance, require accurate depth information in all color pixel positions. Therefore, it is an essential task to develop an effective depth super-resolution strategy to bridge the resolution gap between depth and color images.