Abstract:
Emotion is precious, useful for many applications such as public opinion detection and psychological disease prediction. Emotion recognition on multimodal data has attrac...Show MoreMetadata
Abstract:
Emotion is precious, useful for many applications such as public opinion detection and psychological disease prediction. Emotion recognition on multimodal data has attracted extensive attention. Although modifying model structure or multimodal feature fusion methods have contributed a lot to emotion recognition, little attention is paid to mining implicit emotion relationship. In this article, implicit emotion relationship consists of emotion distribution, confusion, and transfer. Emotion distribution allows multiple emotions in one sample, while confusion and transfer imply the prediction confusion and bias. In order to mine implicit emotion relationship in multimodal data, this article employs three image and two text classification models to recognize emotions, respectively. Two prediction emotion synthesis methods (optimal prediction emotion synthesis and majority prediction emotion synthesis) are proposed to synthesize the outputs of multiple models. Based on the results of two emotion synthesis methods, emotion distribution on samples is obtained. Emotion confusion and transfer among different emotion samples are analyzed by relative entropy and Jensen–Shannon divergence. Implicit emotion relationship mining has potential not only in the interpretation of model performance, but also in guiding the development of emotion recognition as prior knowledge. Finally, we take topic scenario as an instance to mine implicit emotion relationships.
Published in: IEEE MultiMedia ( Volume: 28, Issue: 2, 01 April-June 2021)
Funding Agency:
Shanghai University, Shanghai, China
Xinzhi Wang is currently an Assistant Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. She ever visited Carnegie Mellon University (CMU) for one year in 2017–2018. Her recently researches focus on natural language and image processing, including sentiment analysis, AI transparency, agent action planning, and agent action intervention. She received the Ph.D. degree from T...Show More
Xinzhi Wang is currently an Assistant Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. She ever visited Carnegie Mellon University (CMU) for one year in 2017–2018. Her recently researches focus on natural language and image processing, including sentiment analysis, AI transparency, agent action planning, and agent action intervention. She received the Ph.D. degree from T...View more
Shanghai University, Shanghai, China
Yudong Chang is currently working toward the master’s degree with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. His main research field includes natural language processing, sentiment analysis, and AI transparency. He received the bachelor’s degree from the School of Information Science and Engineering, Shandong Agricultural University, Tai’an, China, in 2019. Contact him at cydshu@...Show More
Yudong Chang is currently working toward the master’s degree with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. His main research field includes natural language processing, sentiment analysis, and AI transparency. He received the bachelor’s degree from the School of Information Science and Engineering, Shandong Agricultural University, Tai’an, China, in 2019. Contact him at cydshu@...View more
Oakland University, Rochester, MI, USA
Vijayan Sugumaran is currently a Distinguished Professor of management information systems, the Chair of the Department of Decision and Information Sciences, and the Co-Director of the Center for Data Science and Big Data Analytics, Oakland University, Rochester, MI, USA. He has authored and coauthored more than 250 peer-reviewed articles in Journals, Conferences, and Books such as Information Systems Research, ACM Transa...Show More
Vijayan Sugumaran is currently a Distinguished Professor of management information systems, the Chair of the Department of Decision and Information Sciences, and the Co-Director of the Center for Data Science and Big Data Analytics, Oakland University, Rochester, MI, USA. He has authored and coauthored more than 250 peer-reviewed articles in Journals, Conferences, and Books such as Information Systems Research, ACM Transa...View more
Shanghai University, Shanghai, China
Xiangfeng Luo is currently a Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. He has ever visited Purdue University, USA as a Visiting Professor. He has authored or co-authored more than 100 publications and his publications have appeared in the IEEE Transactions on Systems, Man, and Cybernetics-Part C and IEEE Transactions on Automation Science and Engineering. His main...Show More
Xiangfeng Luo is currently a Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. He has ever visited Purdue University, USA as a Visiting Professor. He has authored or co-authored more than 100 publications and his publications have appeared in the IEEE Transactions on Systems, Man, and Cybernetics-Part C and IEEE Transactions on Automation Science and Engineering. His main...View more
Shanghai University, Shanghai, China
Peng Wang is currently a Professor with the School of Computer Engineer and Science, Shanghai University, Shanghai, China. He was previously a research staff member with the Shanghai Advanced Research Institute, Chinese Academy of Science. He has been PI on projects worth more than 25 million funded by NSFC, SSTC, Intel, and IBM. He is an expert in the development of integrated circuit standards of the Ministry of Industr...Show More
Peng Wang is currently a Professor with the School of Computer Engineer and Science, Shanghai University, Shanghai, China. He was previously a research staff member with the Shanghai Advanced Research Institute, Chinese Academy of Science. He has been PI on projects worth more than 25 million funded by NSFC, SSTC, Intel, and IBM. He is an expert in the development of integrated circuit standards of the Ministry of Industr...View more
Tsinghua University, Beijing, China
Hui Zhang is currently the Vice Director of the Institute of Public Safety Research, Tsinghua University, Beijing, China. His recently researches most focus on public safety and emergency management, including preparedness and response to emergencies, sentiment analysis, decision-making support, applications of social media and big data in emergency management, etc. He received the National Science Foundation CAREER Award...Show More
Hui Zhang is currently the Vice Director of the Institute of Public Safety Research, Tsinghua University, Beijing, China. His recently researches most focus on public safety and emergency management, including preparedness and response to emergencies, sentiment analysis, decision-making support, applications of social media and big data in emergency management, etc. He received the National Science Foundation CAREER Award...View more
Contents Shanghai University, Shanghai, China
Xinzhi Wang is currently an Assistant Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. She ever visited Carnegie Mellon University (CMU) for one year in 2017–2018. Her recently researches focus on natural language and image processing, including sentiment analysis, AI transparency, agent action planning, and agent action intervention. She received the Ph.D. degree from Tsinghua University, Beijing, China, in 2019. She is the corresponding author. Contact her at wxz2017@shu.edu.cn.
Xinzhi Wang is currently an Assistant Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. She ever visited Carnegie Mellon University (CMU) for one year in 2017–2018. Her recently researches focus on natural language and image processing, including sentiment analysis, AI transparency, agent action planning, and agent action intervention. She received the Ph.D. degree from Tsinghua University, Beijing, China, in 2019. She is the corresponding author. Contact her at wxz2017@shu.edu.cn.View more
Shanghai University, Shanghai, China
Yudong Chang is currently working toward the master’s degree with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. His main research field includes natural language processing, sentiment analysis, and AI transparency. He received the bachelor’s degree from the School of Information Science and Engineering, Shandong Agricultural University, Tai’an, China, in 2019. Contact him at cydshu@shu.edu.cn.
Yudong Chang is currently working toward the master’s degree with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. His main research field includes natural language processing, sentiment analysis, and AI transparency. He received the bachelor’s degree from the School of Information Science and Engineering, Shandong Agricultural University, Tai’an, China, in 2019. Contact him at cydshu@shu.edu.cn.View more
Oakland University, Rochester, MI, USA
Vijayan Sugumaran is currently a Distinguished Professor of management information systems, the Chair of the Department of Decision and Information Sciences, and the Co-Director of the Center for Data Science and Big Data Analytics, Oakland University, Rochester, MI, USA. He has authored and coauthored more than 250 peer-reviewed articles in Journals, Conferences, and Books such as Information Systems Research, ACM Transactions on Database Systems, IEEE Transactions on Big Data, and IEEE Transactions on Education. He has edited 20 books and serves on the Editorial Board of 8 journals. His research interests include big data management and analytics, ontologies, and semantic web, intelligent agent, and multiagent systems. Contact him at sugumara@oakland.edu.
Vijayan Sugumaran is currently a Distinguished Professor of management information systems, the Chair of the Department of Decision and Information Sciences, and the Co-Director of the Center for Data Science and Big Data Analytics, Oakland University, Rochester, MI, USA. He has authored and coauthored more than 250 peer-reviewed articles in Journals, Conferences, and Books such as Information Systems Research, ACM Transactions on Database Systems, IEEE Transactions on Big Data, and IEEE Transactions on Education. He has edited 20 books and serves on the Editorial Board of 8 journals. His research interests include big data management and analytics, ontologies, and semantic web, intelligent agent, and multiagent systems. Contact him at sugumara@oakland.edu.View more
Shanghai University, Shanghai, China
Xiangfeng Luo is currently a Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. He has ever visited Purdue University, USA as a Visiting Professor. He has authored or co-authored more than 100 publications and his publications have appeared in the IEEE Transactions on Systems, Man, and Cybernetics-Part C and IEEE Transactions on Automation Science and Engineering. His main research interests include web wisdom, cognitive informatics, and text understanding. Contact him at luoxf@shu.edu.cn.
Xiangfeng Luo is currently a Professor with the School of Computer Engineering and Science, Shanghai University, Shanghai, China. He has ever visited Purdue University, USA as a Visiting Professor. He has authored or co-authored more than 100 publications and his publications have appeared in the IEEE Transactions on Systems, Man, and Cybernetics-Part C and IEEE Transactions on Automation Science and Engineering. His main research interests include web wisdom, cognitive informatics, and text understanding. Contact him at luoxf@shu.edu.cn.View more
Shanghai University, Shanghai, China
Peng Wang is currently a Professor with the School of Computer Engineer and Science, Shanghai University, Shanghai, China. He was previously a research staff member with the Shanghai Advanced Research Institute, Chinese Academy of Science. He has been PI on projects worth more than 25 million funded by NSFC, SSTC, Intel, and IBM. He is an expert in the development of integrated circuit standards of the Ministry of Industry and Information Technology, and he is an expert in the evaluation of the maturity of integrated circuits evolution of the Encore Alliance. Contact him at pengwang@shu.edu.cn.
Peng Wang is currently a Professor with the School of Computer Engineer and Science, Shanghai University, Shanghai, China. He was previously a research staff member with the Shanghai Advanced Research Institute, Chinese Academy of Science. He has been PI on projects worth more than 25 million funded by NSFC, SSTC, Intel, and IBM. He is an expert in the development of integrated circuit standards of the Ministry of Industry and Information Technology, and he is an expert in the evaluation of the maturity of integrated circuits evolution of the Encore Alliance. Contact him at pengwang@shu.edu.cn.View more
Tsinghua University, Beijing, China
Hui Zhang is currently the Vice Director of the Institute of Public Safety Research, Tsinghua University, Beijing, China. His recently researches most focus on public safety and emergency management, including preparedness and response to emergencies, sentiment analysis, decision-making support, applications of social media and big data in emergency management, etc. He received the National Science Foundation CAREER Award (USA) in 1999 when he was in SUNY Stony Brook, USA, and Changjiang Scholar (China) in 2008. Contact him at zhhui@mail.tsinghua.edu.cn.
Hui Zhang is currently the Vice Director of the Institute of Public Safety Research, Tsinghua University, Beijing, China. His recently researches most focus on public safety and emergency management, including preparedness and response to emergencies, sentiment analysis, decision-making support, applications of social media and big data in emergency management, etc. He received the National Science Foundation CAREER Award (USA) in 1999 when he was in SUNY Stony Brook, USA, and Changjiang Scholar (China) in 2008. Contact him at zhhui@mail.tsinghua.edu.cn.View more
