Textual Data Quality at Scale for High Dimensionality Data

One of the challenges of building a big data ecosystem is the lack of scale in management of textual data as present in various attributes. This is further compounded by growth in the raw data dimensions as the data stream grows in breadth, including more attributes as the product suite grows. A key part of the challenge in this management of growing textual data in newer dimensions is keeping a check on data quality. As the dimensions grow, the underlying system is always under strain, not only from the increased storage needs, but also from the likelihood of these new dimensions having poor data quality. Such instances of poor data quality on the new dimensions can lead to erroneous insights into launch metrics of the new products. Due to high dimensionality, even the older products on preexisting textual dimensions could suffer if the big data ingestion and data wrangling jobs do not offer an automated way to profile the textual columns. Lack of automation could cause lack of trust in such a growing data streaming and persistence system, with data consumers unable to be sure if the textual data received in recent history matches the data quality patterns seen historically. Our system intends to solve this particular challenge by a unique combination of methods of catching textual data anomalies at scale along with lightweight time series based anomaly detection systems which can run linearly on expanding dimensionality data thereby handling the strain on additional data to analyze without requirement of extensive retraining on past data and or having a need for large amount of historical data in a new dimension to begin offering insights on data quality. We will show that with this sequential combination of old and new tools for textual analysis, a robust data quality tracking system can be built to handle incoming requests for newer dimensions in such a big data platform.