Data Profiling Method for Metadata Management

To improve utilization of existing data assets in an enterprise, it is important to reduce the time required for data preparation, which is currently time-consuming for data scientists. In recent years, data catalogs have attracted attention as a solution for this problem. On a data catalog search system, it is important to manage the quality of metadata, which is the contents of the data catalog. However, traditional data profiling methods lack correctness verification of metadata so they don't detect incorrect metadata. Therefore, we propose novel metadata profiling method focused on structural similarity of meta-graphs which represent the relationships between data and metadata. In this method, the graph representation generator creates a metagraph, the similarly calculator measures the similarity of data sources using structural similarity, and the metadata profiler detects wrong metadata through conflicts where different metadata are assigned when similarity is high.