Abstract:
This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). ...Show MoreMetadata
Abstract:
This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.
Published in: IEEE Transactions on Circuits and Systems for Video Technology ( Volume: 31, Issue: 10, October 2021)

Qualcomm Inc., San Diego, CA, USA
Wei-Jung Chien (Member, IEEE) received the B.S. and M.S. degrees in electronics engineering from the National Chiao Tung University, Hsinchu, Taiwan, in 1995 and 1999, respectively, and the Ph.D. degree in electrical engineering from Arizona State University, Tempe, AZ, USA, in 2009. Since 2008, he has been actively involved in the development of HEVC and VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG...Show More
Wei-Jung Chien (Member, IEEE) received the B.S. and M.S. degrees in electronics engineering from the National Chiao Tung University, Hsinchu, Taiwan, in 1995 and 1999, respectively, and the Ph.D. degree in electrical engineering from Arizona State University, Tempe, AZ, USA, in 2009. Since 2008, he has been actively involved in the development of HEVC and VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG...View more

Bytedance Inc., San Diego, CA, USA
Li Zhang (Senior Member, IEEE) received the Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, in 2009.
From 2009 to 2011, she held a post-doctoral position at the Institute of Digital Media, Peking University, Beijing. From 2011 to 2018, she was a Senior Staff Engineer with Multimedia R&D and Standards Group, Qualcomm, Inc., San Diego, CA, USA. She was...Show More
Li Zhang (Senior Member, IEEE) received the Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, in 2009.
From 2009 to 2011, she held a post-doctoral position at the Institute of Digital Media, Peking University, Beijing. From 2011 to 2018, she was a Senior Staff Engineer with Multimedia R&D and Standards Group, Qualcomm, Inc., San Diego, CA, USA. She was...View more
Fraunhofer HHI, Berlin, Germany
Martin Winken received the Dipl.-Ing. and Dr.-Ing. degrees from the Technical University of Berlin, Germany, in 2006 and 2015, respectively. Since 2005, he has been active in video coding standardization. He is currently a Research Associate with Fraunhofer HHI, Berlin.
Martin Winken received the Dipl.-Ing. and Dr.-Ing. degrees from the Technical University of Berlin, Germany, in 2006 and 2015, respectively. Since 2005, he has been active in video coding standardization. He is currently a Research Associate with Fraunhofer HHI, Berlin.View more

Tencent, Palo Alto, CA, USA
Xiang Li (Senior Member, IEEE) received the B.Sc. and M.Sc. degrees in electronic engineering from Tsinghua University, Beijing, China, and the Dr.-Ing. degree in electrical, electronic and communication engineering from the University of Erlangen-Nuremberg, Germany.
He is currently a Senior Principal Researcher and the Head of video coding standards at Tencent’s Media Lab. Before joining Tencent, he was with Qualcomm, Med...Show More
Xiang Li (Senior Member, IEEE) received the B.Sc. and M.Sc. degrees in electronic engineering from Tsinghua University, Beijing, China, and the Dr.-Ing. degree in electrical, electronic and communication engineering from the University of Erlangen-Nuremberg, Germany.
He is currently a Senior Principal Researcher and the Head of video coding standards at Tencent’s Media Lab. Before joining Tencent, he was with Qualcomm, Med...View more

Alibaba, Beijing, China
Ru-Ling Liao received the B.S. and M.S. degrees in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2013 and 2016, respectively.
She has been participated in the development of VVC standard since 2016. After 2019, she joined XG Laboratory, Alibaba Inc., Beijing, China. She is currently a senior engineer. Her research interests include video processing and neural network-based vid...Show More
Ru-Ling Liao received the B.S. and M.S. degrees in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2013 and 2016, respectively.
She has been participated in the development of VVC standard since 2016. After 2019, she joined XG Laboratory, Alibaba Inc., Beijing, China. She is currently a senior engineer. Her research interests include video processing and neural network-based vid...View more

Huawei Technologies, Munich, Germany
Han Gao (Graduate Student Member, IEEE) received the B.Sc. degree in electrical engineering from Xidian University, Xi’an, China, in 2012, and the M.Sc. degree in electrical engineering and information technology from the Technical University of Munich (TUM), Munich, Germany, in 2015, where he is currently pursuing the Ph.D. degree. In 2016, he joined the Chair of Media Technology, TUM, and Audiovisual Technology Laborato...Show More
Han Gao (Graduate Student Member, IEEE) received the B.Sc. degree in electrical engineering from Xidian University, Xi’an, China, in 2012, and the M.Sc. degree in electrical engineering and information technology from the Technical University of Munich (TUM), Munich, Germany, in 2015, where he is currently pursuing the Ph.D. degree. In 2016, he joined the Chair of Media Technology, TUM, and Audiovisual Technology Laborato...View more

MediaTek Inc., Hsinchu, Taiwan
Chih-Wei Hsu received the B.S. degree in electrical engineering and the M.S. degree in electronics engineering from the National Taiwan University (NTU), Taipei, Taiwan, in June 2001 and June 2003, respectively.
He joined MediaTek Inc., Hsinchu, Taiwan, in October 2003, and is currently a Senior Manager of the Multimedia Technology Development Division. From 2003 to 2010, he researched and developed video encoding algorith...Show More
Chih-Wei Hsu received the B.S. degree in electrical engineering and the M.S. degree in electronics engineering from the National Taiwan University (NTU), Taipei, Taiwan, in June 2001 and June 2003, respectively.
He joined MediaTek Inc., Hsinchu, Taiwan, in October 2003, and is currently a Senior Manager of the Multimedia Technology Development Division. From 2003 to 2010, he researched and developed video encoding algorith...View more

Bytedance Inc., Beijing, China
Hongbin Liu received the B.S., M.S., and Ph.D. degrees in computer science and technology from Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively.
He is currently a Video Coding Engineer at Bytedance Inc., Beijing. He was actively participating in the research of 3D-HEVC and VVC standards. His current research interest includes video coding and processing.
Hongbin Liu received the B.S., M.S., and Ph.D. degrees in computer science and technology from Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively.
He is currently a Video Coding Engineer at Bytedance Inc., Beijing. He was actively participating in the research of 3D-HEVC and VVC standards. His current research interest includes video coding and processing.View more

Qualcomm Inc., San Diego, CA, USA
Chun-Chi Chen received the Ph.D. degree in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2017. Since 2011, he has been participating in the video coding standardization process of ISO/IEC MPEG. In 2014–2015, he has been an active contributor to HEVC-SCC and has co-chaired several core experiments. During his internship with InterDigital, San Diego, in 2015–2016, he initiated ...Show More
Chun-Chi Chen received the Ph.D. degree in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2017. Since 2011, he has been participating in the video coding standardization process of ISO/IEC MPEG. In 2014–2015, he has been an active contributor to HEVC-SCC and has co-chaired several core experiments. During his internship with InterDigital, San Diego, in 2015–2016, he initiated ...View more

Qualcomm Inc., San Diego, CA, USA
Wei-Jung Chien (Member, IEEE) received the B.S. and M.S. degrees in electronics engineering from the National Chiao Tung University, Hsinchu, Taiwan, in 1995 and 1999, respectively, and the Ph.D. degree in electrical engineering from Arizona State University, Tempe, AZ, USA, in 2009. Since 2008, he has been actively involved in the development of HEVC and VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG. Since 2009, he has been a Senior Engineer with Qualcomm Technologies Inc., San Diego, CA, USA. He is currently the director of engineering. His current research interests include areas of image and video processing and image and video coding.
Wei-Jung Chien (Member, IEEE) received the B.S. and M.S. degrees in electronics engineering from the National Chiao Tung University, Hsinchu, Taiwan, in 1995 and 1999, respectively, and the Ph.D. degree in electrical engineering from Arizona State University, Tempe, AZ, USA, in 2009. Since 2008, he has been actively involved in the development of HEVC and VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG. Since 2009, he has been a Senior Engineer with Qualcomm Technologies Inc., San Diego, CA, USA. He is currently the director of engineering. His current research interests include areas of image and video processing and image and video coding.View more

Bytedance Inc., San Diego, CA, USA
Li Zhang (Senior Member, IEEE) received the Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, in 2009.
From 2009 to 2011, she held a post-doctoral position at the Institute of Digital Media, Peking University, Beijing. From 2011 to 2018, she was a Senior Staff Engineer with Multimedia R&D and Standards Group, Qualcomm, Inc., San Diego, CA, USA. She was a Software Coordinator for Audio and Video Coding Standard (AVS) and the 3D extensions of High Efficiency Video Coding (HEVC). She has been an Active Contributor to the Versatile Video Coding (VVC), Advanced AVS, the IEEE 1857, 3D Video (3DV) coding extensions of H.264/AVC, HEVC, and HEVC screen content coding extensions. During the development of those video coding standards, she co-chaired several ad hoc groups and core experiments. She is currently the Head of Advanced Video Group, Bytedance Inc., San Diego. She has authored more than 450 standardization contributions, more than 170 granted U.S. patents, and more than 70 technical articles in related book chapters, journals, and proceedings in image/video coding and video processing. Her research interests include 2D/3D image/video coding, video processing, and transmission. She has been appointed as an Editor of AVS and the Main Editor of the Software Test Model for 3DV Standards.
Li Zhang (Senior Member, IEEE) received the Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, in 2009.
From 2009 to 2011, she held a post-doctoral position at the Institute of Digital Media, Peking University, Beijing. From 2011 to 2018, she was a Senior Staff Engineer with Multimedia R&D and Standards Group, Qualcomm, Inc., San Diego, CA, USA. She was a Software Coordinator for Audio and Video Coding Standard (AVS) and the 3D extensions of High Efficiency Video Coding (HEVC). She has been an Active Contributor to the Versatile Video Coding (VVC), Advanced AVS, the IEEE 1857, 3D Video (3DV) coding extensions of H.264/AVC, HEVC, and HEVC screen content coding extensions. During the development of those video coding standards, she co-chaired several ad hoc groups and core experiments. She is currently the Head of Advanced Video Group, Bytedance Inc., San Diego. She has authored more than 450 standardization contributions, more than 170 granted U.S. patents, and more than 70 technical articles in related book chapters, journals, and proceedings in image/video coding and video processing. Her research interests include 2D/3D image/video coding, video processing, and transmission. She has been appointed as an Editor of AVS and the Main Editor of the Software Test Model for 3DV Standards.View more
Fraunhofer HHI, Berlin, Germany
Martin Winken received the Dipl.-Ing. and Dr.-Ing. degrees from the Technical University of Berlin, Germany, in 2006 and 2015, respectively. Since 2005, he has been active in video coding standardization. He is currently a Research Associate with Fraunhofer HHI, Berlin.
Martin Winken received the Dipl.-Ing. and Dr.-Ing. degrees from the Technical University of Berlin, Germany, in 2006 and 2015, respectively. Since 2005, he has been active in video coding standardization. He is currently a Research Associate with Fraunhofer HHI, Berlin.View more

Tencent, Palo Alto, CA, USA
Xiang Li (Senior Member, IEEE) received the B.Sc. and M.Sc. degrees in electronic engineering from Tsinghua University, Beijing, China, and the Dr.-Ing. degree in electrical, electronic and communication engineering from the University of Erlangen-Nuremberg, Germany.
He is currently a Senior Principal Researcher and the Head of video coding standards at Tencent’s Media Lab. Before joining Tencent, he was with Qualcomm, MediaTek, the Institute of Communications Engineering, RWTH Aachen University, and Siemens. He has been working in the field of video compression for years. He is an active contributor to international video coding standards. He served as the chair and the co-chair in a number of ad hoc groups, core experiments, including the Co-Chair of JEM Reference Software and VVC Reference Software. He has published over 50 journals and conference papers, more than 300 standard contributions, and holds more than 240 U.S. granted and pending patents. His research interests include video coding and processing. He is a Co-Editor of MPEG-5 EVC.
Xiang Li (Senior Member, IEEE) received the B.Sc. and M.Sc. degrees in electronic engineering from Tsinghua University, Beijing, China, and the Dr.-Ing. degree in electrical, electronic and communication engineering from the University of Erlangen-Nuremberg, Germany.
He is currently a Senior Principal Researcher and the Head of video coding standards at Tencent’s Media Lab. Before joining Tencent, he was with Qualcomm, MediaTek, the Institute of Communications Engineering, RWTH Aachen University, and Siemens. He has been working in the field of video compression for years. He is an active contributor to international video coding standards. He served as the chair and the co-chair in a number of ad hoc groups, core experiments, including the Co-Chair of JEM Reference Software and VVC Reference Software. He has published over 50 journals and conference papers, more than 300 standard contributions, and holds more than 240 U.S. granted and pending patents. His research interests include video coding and processing. He is a Co-Editor of MPEG-5 EVC.View more

Alibaba, Beijing, China
Ru-Ling Liao received the B.S. and M.S. degrees in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2013 and 2016, respectively.
She has been participated in the development of VVC standard since 2016. After 2019, she joined XG Laboratory, Alibaba Inc., Beijing, China. She is currently a senior engineer. Her research interests include video processing and neural network-based video compression technology.
Ru-Ling Liao received the B.S. and M.S. degrees in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2013 and 2016, respectively.
She has been participated in the development of VVC standard since 2016. After 2019, she joined XG Laboratory, Alibaba Inc., Beijing, China. She is currently a senior engineer. Her research interests include video processing and neural network-based video compression technology.View more

Huawei Technologies, Munich, Germany
Han Gao (Graduate Student Member, IEEE) received the B.Sc. degree in electrical engineering from Xidian University, Xi’an, China, in 2012, and the M.Sc. degree in electrical engineering and information technology from the Technical University of Munich (TUM), Munich, Germany, in 2015, where he is currently pursuing the Ph.D. degree. In 2016, he joined the Chair of Media Technology, TUM, and Audiovisual Technology Laboratory, Huawei Technologies, Munich. Since 2016, he has been actively involved in the development of the VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG. His research focuses on image and video processing, traditional and neural network-based video compression, and coding.
Han Gao (Graduate Student Member, IEEE) received the B.Sc. degree in electrical engineering from Xidian University, Xi’an, China, in 2012, and the M.Sc. degree in electrical engineering and information technology from the Technical University of Munich (TUM), Munich, Germany, in 2015, where he is currently pursuing the Ph.D. degree. In 2016, he joined the Chair of Media Technology, TUM, and Audiovisual Technology Laboratory, Huawei Technologies, Munich. Since 2016, he has been actively involved in the development of the VVC standard that was jointly issued by ITU-T VCEG and ISO/IEC MPEG. His research focuses on image and video processing, traditional and neural network-based video compression, and coding.View more

MediaTek Inc., Hsinchu, Taiwan
Chih-Wei Hsu received the B.S. degree in electrical engineering and the M.S. degree in electronics engineering from the National Taiwan University (NTU), Taipei, Taiwan, in June 2001 and June 2003, respectively.
He joined MediaTek Inc., Hsinchu, Taiwan, in October 2003, and is currently a Senior Manager of the Multimedia Technology Development Division. From 2003 to 2010, he researched and developed video encoding algorithms, hardware architectures, and related IC designs for consumer electronics products. He started attending international video coding standard meetings held by ITU-T VCEG and ISO/IEC MPEG in 2011 and is an active contributor. During the development of VVC, he has made several technical contributions and was the coordinator of the core experiments on combined and multi-hypothesis prediction. His current research interests include image and video coding, image and video processing, and related hardware architectures.
Chih-Wei Hsu received the B.S. degree in electrical engineering and the M.S. degree in electronics engineering from the National Taiwan University (NTU), Taipei, Taiwan, in June 2001 and June 2003, respectively.
He joined MediaTek Inc., Hsinchu, Taiwan, in October 2003, and is currently a Senior Manager of the Multimedia Technology Development Division. From 2003 to 2010, he researched and developed video encoding algorithms, hardware architectures, and related IC designs for consumer electronics products. He started attending international video coding standard meetings held by ITU-T VCEG and ISO/IEC MPEG in 2011 and is an active contributor. During the development of VVC, he has made several technical contributions and was the coordinator of the core experiments on combined and multi-hypothesis prediction. His current research interests include image and video coding, image and video processing, and related hardware architectures.View more

Bytedance Inc., Beijing, China
Hongbin Liu received the B.S., M.S., and Ph.D. degrees in computer science and technology from Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively.
He is currently a Video Coding Engineer at Bytedance Inc., Beijing. He was actively participating in the research of 3D-HEVC and VVC standards. His current research interest includes video coding and processing.
Hongbin Liu received the B.S., M.S., and Ph.D. degrees in computer science and technology from Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively.
He is currently a Video Coding Engineer at Bytedance Inc., Beijing. He was actively participating in the research of 3D-HEVC and VVC standards. His current research interest includes video coding and processing.View more

Qualcomm Inc., San Diego, CA, USA
Chun-Chi Chen received the Ph.D. degree in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2017. Since 2011, he has been participating in the video coding standardization process of ISO/IEC MPEG. In 2014–2015, he has been an active contributor to HEVC-SCC and has co-chaired several core experiments. During his internship with InterDigital, San Diego, in 2015–2016, he initiated works on generalized bi-prediction which has been included as a coding tool in VVC. After 2018, he focused on inter-prediction techniques and conducted quarterly service works as a core experiment coordinator of VVC. He is currently a Senior Engineer with Multimedia R&D and Standards Group, Qualcomm Technologies, Inc., San Diego, CA, USA. He has authored/coauthored more than 70 technical papers and video standards contributions. His research interests include image/video compression and screen content coding.
Chun-Chi Chen received the Ph.D. degree in computer science and engineering from the National Chao Tung University, Hsinchu, Taiwan, in 2017. Since 2011, he has been participating in the video coding standardization process of ISO/IEC MPEG. In 2014–2015, he has been an active contributor to HEVC-SCC and has co-chaired several core experiments. During his internship with InterDigital, San Diego, in 2015–2016, he initiated works on generalized bi-prediction which has been included as a coding tool in VVC. After 2018, he focused on inter-prediction techniques and conducted quarterly service works as a core experiment coordinator of VVC. He is currently a Senior Engineer with Multimedia R&D and Standards Group, Qualcomm Technologies, Inc., San Diego, CA, USA. He has authored/coauthored more than 70 technical papers and video standards contributions. His research interests include image/video compression and screen content coding.View more