Availability Of The Jobtracker Machine In Hadoop/Map-Reduce Implementations

Abstract

Due to the growing demand for Cloud Computing services, the need and importance of Distributed Systems cannot be underestimated. However, it is difficult to use the traditional Message Passing Interface (MPI) approach to implement synchronization, coordination,and prevent deadlocks in distributed systems. This difficulty is lessened by the use of Apache’s Hadoop/MapReduce and Zookeeper to provide Fault Tolerance in a Homogeneously Distributed Hardware/Software environment. In this thesis, a mathematical model for the availability of the JobTracker in Hadoop/MapReduce using Zookeeper’s Leader Election Service is examined. Though the availability is less than what is expected in a k Fault Tolerance system for higher values of the hardware failure rate, this approach makes coordination and synchronization easy, reduces the effect of Crash failures, and provides Fault Tolerance for distributed systems. The availability model starts with a Markov state diagram for a general case of N Zookeeper servers followed by specific cases of 3,4,and 5 servers. Both software and hardware faults are considered in addition to the effect of hardware and software repair rates. Comparisons show that, the system availability changes with change in the number of Zookeeper servers, with 3 servers having the highest availability. The model presented in this study can be used to decide on how many servers are optimal for maximum availability and from which vendor they must be purchased. It can also help determine what time to use a Zookeeper coordinated Hadoop cluster to perform critical tasks.