Loading [MathJax]/extensions/MathZoom.js
ToonMeet: A Real-time Portrait Toonification Framework with Frame Interpolation Fine-tuned for Online Meeting | IEEE Conference Publication | IEEE Xplore

ToonMeet: A Real-time Portrait Toonification Framework with Frame Interpolation Fine-tuned for Online Meeting


Abstract:

In this paper, we propose ToonMeet, a hybrid frame-work for high-resolution and style-controllable online meeting toonification that ensures real-time operation speed. To...Show More

Abstract:

In this paper, we propose ToonMeet, a hybrid frame-work for high-resolution and style-controllable online meeting toonification that ensures real-time operation speed. ToonMeet applies video frame interpolation to traditional portrait toonification pipelines, allowing for the synthesis of intermediate frames between adjacent toonified keyframes, significantly accelerating the overall process and saving computational resources. However, this approach brings a new problem, where prevailing flow-based video frame interpolation methods tend to cause more ghost and blur artifacts in toonified scenes compared to non-toonified scenes, especially when fast-moving objects exist. We study this previously undiscussed problem and explore its causes. To address this, we introduce a new dataset called TM3B (Toonified Multi-modal Meeting Behaviors), offering high-resolution and cross-platform multi-modal stylized meeting data of Japanese youth in various scenarios. Then, we fine-tune ToonMeet on these tailored data and the resulting model presents improved optical flow estimation ability on toonified videos. Extensive experiments demonstrate that ToonMeet can achieve great spatiotemporal performance and perform high-quality toonification of online meetings with real-time operation speed.
Date of Conference: 06-08 November 2023
Date Added to IEEE Xplore: 20 December 2023
ISBN Information:

ISSN Information:

Conference Location: Atlanta, GA, USA

I. Introduction

Online meeting, also known as video conference, has experienced a surge in popularity over the past few years due to the COVID-19 pandemic. According to a report [1] by Zoom, a popular online meeting platform, in April 2020, the platform hosted approximately 300 million daily meeting participants, which is considerably higher than the 10 million daily meeting participants in December 2019. With explosive user base growth, online meeting platforms has expanded more features to meet user needs.

References

References is not available for this document.