3D Hand Bones and Tissue Estimation from a Single 2D X-Ray Image Via a Two-Stream Deep Neural Network

The hand bones and soft tissue are essential for many applications such as clinical diagnosis, hand modeling and meta-verse. However, their 3D reconstruction from CT or MRI data is time consuming and requires a lot of computational resources from modern hardware. To address this issue, we propose a novel two-stream deep neural network to estimate the 3D hand bones and soft tissue from a single X-ray image. The first stream of the network is for the bone estimation, which incorporates a module component from other modalities. The second stream is for the soft tissue modeling, which utilizes a sub-network from hand pose estimation. After combining these two streams, we can successfully construct a 3D virtual hand from a single 2D X-ray image. We conduct several numerical experiments to validate the proposed two-stream network. Our method can be used for hand bone and soft tissue modeling from X-ray images.