Loading [MathJax]/extensions/MathMenu.js
Real-Time Full-Body Motion Capture from Video and IMUs | IEEE Conference Publication | IEEE Xplore
Subscribe
ADVANCED SEARCH
Conferences >2017 International Conference...

Real-Time Full-Body Motion Capture from Video and IMUs

PDF
Charles Malleson; Andrew Gilbert; Matthew Trumble; John Collomosse; Adrian Hilton; Marco Volino
All Authors

Abstract:

A real-time full-body motion capture system is presented which uses input from a sparse set of inertial measurement units (IMUs) along with images from two or more standa...Show More

Metadata

Abstract:

A real-time full-body motion capture system is presented which uses input from a sparse set of inertial measurement units (IMUs) along with images from two or more standard video cameras and requires no optical markers or specialized infra-red cameras. A real-time optimization-based framework is proposed which incorporates constraints from the IMUs, cameras and a prior pose model. The combination of video and IMU data allows the full 6-DOF motion to be recovered including axial rotation of limbs and drift-free global position. The approach was tested using both indoor and outdoor captured data. The results demonstrate the effectiveness of the approach for tracking a wide range of human motion in real time in unconstrained indoor/outdoor scenes.
Published in: 2017 International Conference on 3D Vision (3DV)
Date of Conference: 10-12 October 2017
Date Added to IEEE Xplore: 07 June 2018
ISBN Information:
Electronic ISSN: 2475-7888
DOI: 10.1109/3DV.2017.00058
Conference Location: Qingdao, China
Citations are not available for this document.
Contents

1. Introduction

Real-time capture of human motion is of considerable interest in various domains including entertainment and the life sciences. Recent advances in computer vision and the availability of commodity wireless inertial sensors [13], [3] are beginning to take motion capture from constrained studio settings to more natural, outdoor environments, and with less encumbrance of the performer from specialized costumes and optical marker setups traditionally required (e.g. [4], [2]), while still retaining a high level of capture fidelity.

Cites in Papers - |

Cites in Papers - IEEE (40)

Select All
1.
Yudi Dai, Zhiyong Wang, Xiping Lin, Chenglu Wen, Lan Xu, Siqi Shen, Yuexin Ma, Cheng Wang, "HiSC4D: Human-Centered Interaction and 4D Scene Capture in Large-Scale Space Using Wearable IMUs and LiDAR", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.46, no.12, pp.11236-11253, 2024.
Show Article
Google Scholar
2.
Mihaela Popescu, Kashmira Shinde, Proneet Sharma, Lisa Gutzeit, Frank Kirchner, "Fusion of Inertial Sensor Suit and Monocular Camera for 3D Human Pelvis Pose Estimation", 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), pp.160-167, 2024.
Show Article
Google Scholar
3.
Zehai Niu, Ke Lu, Jian Xue, Xiaoyu Qin, Jinbao Wang, Ling Shao, "From Methods to Applications: A Review of Deep 3D Human Motion Capture", IEEE Transactions on Circuits and Systems for Video Technology, vol.34, no.11, pp.11340-11359, 2024.
Show Article
Google Scholar
4.
Tom Van Wouwe, Seunghwan Lee, Antoine Falisse, Scott Delp, C. Karen Liu, "DiffusionPoser: Real-Time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion", 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.2513-2523, 2024.
Show Article
Google Scholar
5.
Jiye Lee, Hanbyul Joo, "Mocap Everyone Everywhere: Lightweight Motion Capture with Smartwatches and a Head-Mounted Camera", 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.1091-1100, 2024.
Show Article
Google Scholar
6.
Chengfeng Zhao, Juze Zhang, Jiashen Du, Ziwei Shan, Junye Wang, Jingyi Yu, Jingya Wang, Lan Xu, "I'M HOI: Inertia-Aware Monocular Capture of 3D Human-Object Interactions", 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.729-741, 2024.
Show Article
Google Scholar
7.
Guoan Wang, Hengyi Li, Liang Zhang, Xuefeng Wang, "Development of a Low-Cost and Portable Walker-Based Human Motion Estimation System", IEEE/ASME Transactions on Mechatronics, vol.29, no.5, pp.3901-3911, 2024.
Show Article
Google Scholar
8.
Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan José Zárate, Otmar Hilliges, "EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild", 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.14586-14597, 2023.
Show Article
Google Scholar
9.
Md Moniruzzaman, Zhaozheng Yin, Md Sanzid Bin Hossain, Hwan Choi, Zhishan Guo, "Wearable Motion Capture: Reconstructing and Predicting 3D Human Poses From Wearable Sensors", IEEE Journal of Biomedical and Health Informatics, vol.27, no.11, pp.5345-5356, 2023.
Show Article
Google Scholar
10.
Yudi Dai, Yitai Lin, Xiping Lin, Chenglu Wen, Lan Xu, Hongwei Yi, Siqi Shen, Yuexin Ma, Cheng Wang, "SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environments", 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.682-692, 2023.
Show Article
Google Scholar
11.
Qi Fang, Kang Chen, Yinghui Fan, Qing Shuai, Jiefeng Li, Weidong Zhang, "Learning Analytical Posterior Probability for Human Mesh Recovery", 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.8781-8791, 2023.
Show Article
Google Scholar
12.
Tong Li, Haoyong Yu, "Visual–Inertial Fusion-Based Human Pose Estimation: A Review", IEEE Transactions on Instrumentation and Measurement, vol.72, pp.1-16, 2023.
Show Article
Google Scholar
13.
Yiming Ren, Chengfeng Zhao, Yannan He, Peishan Cong, Han Liang, Jingyi Yu, Lan Xu, Yuexin Ma, "LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors", IEEE Transactions on Visualization and Computer Graphics, vol.29, no.5, pp.2337-2347, 2023.
Show Article
Google Scholar
14.
Yiming Bao, Xu Zhao, Dahong Qian, "FusePose: IMU-Vision Sensor Fusion in Kinematic Space for Parametric Human Pose Estimation", IEEE Transactions on Multimedia, vol.25, pp.7736-7746, 2023.
Show Article
Google Scholar
15.
Gongjin Lan, Yu Wu, Fei Hu, Qi Hao, "Vision-Based Human Pose Estimation via Deep Learning: A Survey", IEEE Transactions on Human-Machine Systems, vol.53, no.1, pp.253-268, 2023.
Show Article
Google Scholar
16.
Gong-Bin Chen, Lei Wang, Xun-Yu Liu, Long-Hua Hu, Jun Cheng, "Character animation and retargeting from video streams", 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp.3267-3272, 2022.
Show Article
Google Scholar
17.
Yuan Niu, Jinhua She, Chi Xu, "A Survey on IMU-and-Vision-based Human Pose Estimation for Rehabilitation", 2022 41st Chinese Control Conference (CCC), pp.6410-6415, 2022.
Show Article
Google Scholar
18.
Xinyu Yi, Yuxiao Zhou, Marc Habermann, Soshi Shimada, Vladislav Golyanik, Christian Theobalt, Feng Xu, "Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors", 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.13157-13168, 2022.
Show Article
Google Scholar
19.
Yudi Dai, Yitai Lin, Chenglu Wen, Siqi Shen, Lan Xu, Jingyi Yu, Yuexin Ma, Cheng Wang, "HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR", 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.6782-6792, 2022.
Show Article
Google Scholar
20.
Gizem Ateş, Martin Fodstad Stølen, Erik Kyrkjebø, "Force and Gesture-based Motion Control of Human-Robot Cooperative Lifting Using IMUs", 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp.688-692, 2022.
Show Article
Google Scholar
21.
Rui Li, Hongyu Wang, Zhenyu Liu, "Survey on Mapping Human Hand Motion to Robotic Hands for Teleoperation", IEEE Transactions on Circuits and Systems for Video Technology, vol.32, no.5, pp.2647-2665, 2022.
Show Article
Google Scholar
22.
Manuel Kaufmann, Yi Zhao, Chengcheng Tang, Lingling Tao, Christopher Twigg, Jie Song, Robert Wang, Otmar Hilliges, "EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers", 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.11490-11500, 2021.
Show Article
Google Scholar
23.
Yuanmin Zhou, Xiaoyan Hu, "3D Human Pose Estimation: Using Context Information in Monocular Video", 2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI), pp.1-7, 2021.
Show Article
Google Scholar
24.
Tomoya Kaichi, Tsubasa Maruyama, Mitsunori Tada, Hideo Saito, "Learning Sensor Interdependencies for IMU-to-Segment Assignment", IEEE Access, vol.9, pp.116440-116452, 2021.
Show Article
Google Scholar
25.
Vladimir Guzov, Aymen Mir, Torsten Sattler, Gerard Pons-Moll, "Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors", 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.4316-4327, 2021.
Show Article
Google Scholar
26.
Kristijan Bartol, David Bojanić, Tomislav Petković, Tomislav Pribanić, "A Review of Body Measurement Using 3D Scanning", IEEE Access, vol.9, pp.67281-67301, 2021.
Show Article
Google Scholar
27.
Young-Woon Cha, Husam Shaik, Qian Zhang, Fan Feng, Andrei State, Adrian Ilie, Henry Fuchs, "Mobile. Egocentric Human Body Motion Reconstruction Using Only Eyeglasses-mounted Cameras and a Few Body-worn Inertial Sensors", 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp.616-625, 2021.
Show Article
Google Scholar
28.
Zheng Chen, Jiliang Chen, "FRSK visual odometey based on FREAK", 2021 4th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), pp.1081-1085, 2021.
Show Article
Google Scholar
29.
Caterina Buizza, Yiannis Demiris, "Rotational Adjoint Methods for Learning-Free 3D Human Pose Estimation from IMU Data", 2020 25th International Conference on Pattern Recognition (ICPR), pp.7868-7875, 2021.
Show Article
Google Scholar
30.
Matthew Shere, Hansung Kim, Adrian Hilton, "Temporally Consistent 3D Human Pose Estimation Using Dual 360° Cameras", 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.81-90, 2021.
Show Article
Google Scholar

Cites in Papers - Other Publishers (30)

1.
Mingliang Chen, Guangxing Tan, "3D Human Pose Estimation Based on Wearable IMUs and Multiple Camera Views", Electronics, vol.13, no.15, pp.2926, 2024.
CrossRef Google Scholar
2.
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, Lan Xu, "InterGen: Diffusion-Based Multi-human Motion Generation Under Complex Interactions", International Journal of Computer Vision, 2024.
CrossRef Google Scholar
3.
Xuan Xiao, Jianjian Wang, Pingfa Feng, Ao Gong, Xiangyu Zhang, Jianfu Zhang, "Fast Human Motion reconstruction from sparse inertial measurement units considering the human shape", Nature Communications, vol.15, no.1, 2024.
CrossRef Google Scholar
4.
Sizhen Bian, Mengxi Liu, Bo Zhou, Paul Lukowicz, Michele Magno, "Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer Interaction", Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol.8, no.1, pp.1, 2024.
CrossRef Google Scholar
5.
Yinghao Huang, Omid Taheri, Michael J. Black, Dimitrios Tzionas, "InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction from Multi-view RGB-D Images", International Journal of Computer Vision, 2024.
CrossRef Google Scholar
6.
Liujun Liu, Jiewen Yang, Ye Lin, Peixuan Zhang, Lihua Zhang, "3D human pose estimation with single image and inertial measurement unit (IMU) sequence", Pattern Recognition, pp.110175, 2023.
CrossRef Google Scholar
7.
Xianhua Liao, Jiayan Dong, Kangkang Song, Jiangjian Xiao, "Three-Dimensional Human Pose Estimation from Sparse IMUs through Temporal Encoder and Regression Decoder", Sensors, vol.23, no.7, pp.3547, 2023.
CrossRef Google Scholar
8.
Meejin Kim, Sukwon Lee, "Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time", Sensors, vol.22, no.13, pp.4846, 2022.
CrossRef Google Scholar
9.
Anik Sarker, Don-Roberts Emenonye, Aisling Kelliher, Thanassis Rikakis, R. Michael Buehrer, Alan T. Asbeck, "Capturing Upper Body Kinematics and Localization with Low-Cost Sensors for Rehabilitation Applications", Sensors, vol.22, no.6, pp.2300, 2022.
CrossRef Google Scholar
10.
Angel Dacal-Nieto, Breogan Ra?a, Juan Moreno-Rodriguez, Juan Jose Areal, Victor Alonso-Ramos, "Self-training of Manufacturing Operators Using Finger-Tracking Wearable Technologies", Advanced Research in Technologies, Information, Innovation and Sustainability, vol.1675, pp.3, 2022.
CrossRef Google Scholar
11.
Tadele Belay Tuli, Martin Manns, Sebastian Zeller, "Human motion quality and accuracy measuring method for human–robot physical interactions", Intelligent Service Robotics, vol.15, no.4, pp.503, 2022.
CrossRef Google Scholar
12.
Vinh Bui, Alireza Alaei, "Virtual reality in training artificial intelligence-based systems: a case study of fall detection", Multimedia Tools and Applications, vol.81, no.22, pp.32625, 2022.
CrossRef Google Scholar
13.
Jackie (Junrui) Yang, Tuochao Chen, Fang Qin, Monica S. Lam, James A. Landay, "HybridTrak: Adding Full-Body Tracking to VR Using an Off-the-Shelf Webcam", CHI Conference on Human Factors in Computing Systems, pp.1, 2022.
CrossRef Google Scholar
14.
Vicent Girbés-Juan, Vinicius Schettino, Luis Gracia, J. Ernesto Solanes, Yiannis Demiris, Josep Tornero, "Combining haptics and inertial motion capture to enhance remote control of a dual-arm robot", Journal on Multimodal User Interfaces, 2022.
CrossRef Google Scholar
15.
Jan Stenum, Kendra M. Cherry-Allen, Connor O. Pyles, Rachel D. Reetzke, Michael F. Vignos, Ryan T. Roemmich, "Applications of Pose Estimation in Human Health and Performance across the Lifespan", Sensors, vol.21, no.21, pp.7315, 2021.
CrossRef Google Scholar
16.
Ashok Kumar Patil, Adithya Balasubramanyam, Jae Yeong Ryu, Bharatesh Chakravarthi, Young Ho Chai, "An Open-Source Platform for Human Pose Estimation and Tracking Using a Heterogeneous Multi-Sensor System", Sensors, vol.21, no.7, pp.2340, 2021.
CrossRef Google Scholar
17.
Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, Wenjun Zeng, "AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild", International Journal of Computer Vision, vol.129, no.3, pp.703, 2021.
CrossRef Google Scholar
18.
Manh-Hung Nguyen, Ching-Chun Hsiao, Wen-Huang Cheng, Ching-Chun Huang, "Practical 3D human skeleton tracking based on multi-view and multi-Kinect fusion", Multimedia Systems, 2021.
CrossRef Google Scholar
19.
Gizem Ateş, Erik Kyrkjebø, "Human-Robot Cooperative Lifting Using IMUs and Human Gestures", Towards Autonomous Robotic Systems, vol.13054, pp.88, 2021.
CrossRef Google Scholar
20.
Eni Halilaj, Soyong Shin, Eric Rapp, Donglai Xiang, "American society of biomechanics early career achievement award 2020: Toward portable and modular biomechanics labs: How video and IMU fusion will change gait analysis", Journal of Biomechanics, vol.129, pp.110650, 2021.
CrossRef Google Scholar
21.
Xinyu Yi, Yuxiao Zhou, Feng Xu, "TransPose", ACM Transactions on Graphics, vol.40, no.4, pp.1, 2021.
CrossRef Google Scholar
22.
Xinyu Yi, Yuxiao Zhou, Feng Xu, "TransPose", ACM Transactions on Graphics, vol.40, no.4, pp.1, 2021.
CrossRef Google Scholar
23.
Tomoya Kaichi, Tsubasa Maruyama, Mitsunori Tada, Hideo Saito, "Resolving Position Ambiguity of IMU-Based Human Pose with a Single RGB Camera", Sensors, vol.20, no.19, pp.5453, 2020.
CrossRef Google Scholar
24.
Matthias Seuter, Alexandra Pollock, Gernot Bauer, Christian Kray, "Recognizing Running Movement Changes with Quaternions on a Sports Watch", Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol.4, no.4, pp.1, 2020.
CrossRef Google Scholar
25.
Charles Malleson, John Collomosse, Adrian Hilton, "Real-Time Multi-person Motion Capture from Multi-view Video and IMUs", International Journal of Computer Vision, vol.128, no.6, pp.1594, 2020.
CrossRef Google Scholar
26.
Andrew Gilbert, Matthew Trumble, Charles Malleson, Adrian Hilton, John Collomosse, "Fusing Visual and Inertial Sensors with Semantics for 3D Human Pose Estimation", International Journal of Computer Vision, vol.127, no.4, pp.381, 2019.
CrossRef Google Scholar
27.
Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll, "Deep inertial poser", ACM Transactions on Graphics, vol.37, no.6, pp.1, 2019.
CrossRef Google Scholar
28.
Timo von Marcard, Roberto Henschel, Michael J. Black, Bodo Rosenhahn, Gerard Pons-Moll, "Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera", Computer Vision ? ECCV 2018, vol.11214, pp.614, 2018.
CrossRef Google Scholar
29.
Matthew Trumble, Andrew Gilbert, Adrian Hilton, John Collomosse, "Deep Autoencoder for Combined Human Pose Estimation and Body Model Upscaling", Computer Vision ? ECCV 2018, vol.11214, pp.800, 2018.
CrossRef Google Scholar
30.
Zerong Zheng, Tao Yu, Hao Li, Kaiwen Guo, Qionghai Dai, Lu Fang, Yebin Liu, "HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs", Computer Vision – ECCV 2018, vol.11213, pp.389, 2018.
CrossRef Google Scholar
Contact IEEE to Subscribe
More Like This
Three-dimensional display rendering acceleration using occlusion camera reference images

Journal of Display Technology

Published: 2006

A real-time system for object detection and location reminding with RGB-D camera

2014 IEEE International Conference on Consumer Electronics (ICCE)

Published: 2014

Show More

References

References is not available for this document.

IEEE Account

Purchase Details

Profile Information

Need Help?

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity.
© Copyright 2025 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.