Tensor language model enables generative scheduling for efficient tensor compilation

Abstract

Abstract The high rate of increase in the deep learning tasks as well as heterogeneous computing systems necessitates compilers that achieve low compile time and high performance. The current state of the art in the use of tensor compilers is based on exhaustive search, a slow and prohibitive process, or heuristics at the expense of generality and quality of optimisation. The paper presents the Tensor Language Model (TLM), a generative framework of a compiler that redefines the optimisation of tensor programmes as a language modelling problem. TLM is also based on a GPT-2 architecture that is pre-trained on millions of tensor programs coded as compact tensor code capturing operator graphs, hardware metadata, and reconfiguration choices as a sequence of tokens. The model creates optimised schedules of tensors, thus avoiding any search or reinforcement learning at run time. Experimental results using ResNet-50, BERT, GPT-2 and LLAMA-7B indicate that TLM compiles up to 61 times faster than search-based compilers (e.g., Ansor, MetaSchedule) and is up to 2.25 times faster than heuristic models (e.g. Roller), and has similar runtime efficiency. TLM then selects a trade-off seemingly traditional between compilation time and execution performance, a scalable, hardware-agnostic and reproducible generative paradigm of next-generation deep learning compilers.