A Framework for the Design, Development, Testing and Deployment of Reliable Big Data Platforms

We consider the problem of reliability in big data science projects that are comprised of multiple computing platforms and complex architectures that harness data. Specifically on their ability to capture, process and analyze streaming high frequency data from vast complex systems reliably with effective scalability for deployment in vast domains such as clinical care, smart cities or within extreme climatic work environments. This paper introduces a framework to enable reliable data science projects by integrating multiple computing principles of autonomy, local responsibility, fault tolerance, symmetry, decentralization, well-understood building blocks, and simplicity. The designed framework is applied in the development of a decoupled data pipeline demonstrated through a case study on pre-deployment acclimation strategies that is continuously monitored to ensure reliability and availability is effectively quantified.