A BATCH SCHEDULING OPTIMIZATION METHOD AND IMPLEMENTATION OF DNN INFERENCE TASKS FOR HETEROGENEOUS CLUSTERS

With the development of artificial intelligence industry, a large number of DNN-based applications generate large-scale DNN inference tasks every day. These tasks are aggregated into large heterogeneous clusters for inference. How to design an efficient batch scheduling optimization method of DNN inference tasks for heterogeneous clusters and its implementation is a key issue. To solve this problem, a scheduling optimization method based on a policy parameter characterization mechanism was proposed. It used a combination of policies with embedded parameters for scheduling results generation and continuously searches for optimal solutions to these parameters using a meta-heuristic algorithm. A scheduling system for inference tasks in heterogeneous computing clusters was implemented based on this optimization method. Experiments show that the proposed optimization method has significant performance improvements over traditional scheduling methods.