Abstract:
Hadoop utilizes RPC heartbeat mechanism with fixed time interval and fixed timeout value to identify node faults, which affects the completion time of node jobs and accuracy of node fault detection. Two methods to optimize Hadoop performance are proposed from the perspective of distributed system master- slave architecture. The RPC heartbeat policy based on access popularity perception dynamically adjusted the heartbeat sending interval according to node access popularity. A fault tolerant model based on the interconnection of hot and cold nodes was proposed, and the traditional fault tolerant model was optimized by using the states of different nodes in the same rank. The experimental results show that the heat sensing RPC heartbeat method significantly improves the I/O performance of read and write data. When reading small files, the adaptive heartbeat mechanism reduces the task completion time by 8% on average. The fault tolerant model of cold and hot node interconnection can find the failure node faster, and improve the performance of the existing mechanism by 2%~4%.