Chartem: Reviving Chart Images with Data Embedding

In practice, charts are widely stored as bitmap images. Although easily consumed by humans, they are not convenient for other uses. For example, changing the chart style or type or a data value in a chart image practically requires creating a completely new chart, which is often a time-consuming and error-prone process. To assist these tasks, many approaches have been proposed to automatically extract information from chart images with computer vision and machine learning techniques. Although they have achieved promising preliminary results, there are still a lot of challenges to overcome in terms of robustness and accuracy. In this paper, we propose a novel alternative approach called Chartem to address this issue directly from the root. Specifically, we design a data-embedding schema to encode a significant amount of information into the background of a chart image without interfering human perception of the chart. The embedded information, when extracted from the image, can enable a variety of visualization applications to reuse or repurpose chart images. To evaluate the effectiveness of Chartem, we conduct a user study and performance experiments on Chartem embedding and extraction algorithms. We further present several prototype applications to demonstrate the utility of Chartem.

1 Introduction

As an effective and efficient means to convey quantitative information [39], charts have become an increasingly pervasive type of content widely adopted in newspapers, textbooks, websites, academic papers, etc. Nowadays, there are many tools, such as Excel, Tableau, and Power BI, to help users convert data into charts or graphs effortlessly. During the authoring process, a chart object is often created to maintain relationships between data and visual elements. After the authoring process, it is common to save the created chart as a bitmap image, for easy typesetting or sharing. In many cases, the resulting image is then disconnected from its chart object and becomes the only representation available for the underlying data. This may cause several issues in the long run. First, since the carried information and visual style are locked in a chart image, it is hard to reuse or repurpose the chart in the future. For example, if Alice wants to change the chart type or style for a different story or document, she often has to do it manually as a chart image is generally not machine readable. To assist with this task, many image-recognition-based techniques have been proposed to automatically recover data and visual design information from chart images [5], [6], [10], [11], [20], [21], [25]. However, this is still a relatively new research direction, and a robust solution that can accurately recover the full information of a chart image has not been accomplished yet due to diversity and complexity of chart content. Second, in many cases, the message conveyed by a chart is distilled from a bigger dataset via a series of aggregation and filtering operations. If a user likes to perform a different analysis on the same underlying dataset for a different purpose, it will be impossible since the information carried by the chart is limited and the original dataset is completely lost after the conversion.