大数据系统中高效容错的热度感知RPC心跳机制

AN EFFICIENT FAULT-TOLERANCE ACCESS-AWARE RPC HEARTBEAT MECHANISM IN BIG DATA SYSTEMS

  • 摘要: Hadoop采用固定时间间隔的RPC心跳机制和固定超时时间来判断节点故障,影响节点作业完成时间和节点故障检测的时效性和准确性。从分布式系统主从架构角度出发,提出两种优化Hadoop性能的方法。首先,提出基于访问热度感知的RPC心跳策略,根据节点访问热度动态调整心跳发送间隔。其次,提出基于冷热节点互联的容错模型,在同机架中利用不同节点的状态优化传统容错模型。实验结果表明,基于热度感知的RPC心跳方法显著提高了读写数据的I/O性能,在读取小文件时平均减少8%的任务完成时间。冷热节点互联的容错模型能够更快地发现失效节点,相对现有机制平均提高了2%~4%的性能。

     

    Abstract: Hadoop utilizes RPC heartbeat mechanism with fixed time interval and fixed timeout value to identify node faults, which affects the completion time of node jobs and accuracy of node fault detection. Two methods to optimize Hadoop performance are proposed from the perspective of distributed system master- slave architecture. The RPC heartbeat policy based on access popularity perception dynamically adjusted the heartbeat sending interval according to node access popularity. A fault tolerant model based on the interconnection of hot and cold nodes was proposed, and the traditional fault tolerant model was optimized by using the states of different nodes in the same rank. The experimental results show that the heat sensing RPC heartbeat method significantly improves the I/O performance of read and write data. When reading small files, the adaptive heartbeat mechanism reduces the task completion time by 8% on average. The fault tolerant model of cold and hot node interconnection can find the failure node faster, and improve the performance of the existing mechanism by 2%~4%.

     

/

返回文章
返回