Corddl: An Efficient and Extensible Connector between Relational Databases and Data Lakes

In the era of big data, data lakes have been widely used. However, enterprise applications generally do not interact directly with data lakes, but with relational databases. Relational databases may become data silos because they can only store structured data. In this paper, we proposed Corddl, a connector between relational databases and data lakes. For ingesting data from a relational database into a data lake, first, Corddl uses configuration files and parameters to match the data source class and data source instance for the analysis engine. Then, Corddl uses the JDBC driver to realize the connection to the relational database, and do data conversion according to the data type configuration file. Finally, the ingested data is written into the data lake. There is a similar process for ingesting data from a data lake into a relational database. Experimental results show that Corddl has high efficiency and strong extensibility.