Path-Based Degraded Network Interface Card Detection at Mapreduce Startup Phase

The use of distributed system nowadays has vastly increased because of the ability to utilize multiple nodes efficiently on big data processing. One of the widely used distributed system framework is Hadoop. For more than a decade, Hadoop had overcome a single point of failure (SPOF) by using speculative execution (Base SE). A recent study entitled Limplock exposed Base SE's vulnerability on handling node failure caused by damaged network interface card (NIC) in which results in more than 60x performance degradation. This problem is said to be solved with a new version of Base SE, that is Path Based Speculative Execution (PBSE). Our study, however, has found out that PBSE is still unable to detect the existence of nodes with slow NIC at Mapreduce startup and cleanup phase in which leads to up to 300% slowdown. This paper tries to improve the PBSE solution by creating a new node with a slow NIC detection mechanism in the Mapreduce startup phase. The solution is comprised of creation of a new heartbeat to monitor localization process during the Mapreduce startup, adopting path diversity mechanism by PBSE and implementing straggler node detection algorithm. There are still special cases that cannot be handled by this solution, however those special cases hardly occur in big clustered Hadoop. Furthermore, the performance cost of conducting additional this detection mechanism is negligible. Finally, the proposed solution has successfully improved the PBSE solution in detecting the nodes with slow NIC during the Mapreduce startup phase.