Real Post-Training Quantization Framework for Resource-Optimized Multiplier in LLMs

Large language models (LLMs) offer unparalleled natural language processing capabilities but come with massive memory footprint and computation requirements. Post Training Quantization (PTQ) is widely adopted to tackle the challenges without expensive LLMs’ fine-tuning. However, quantizing LLMs with PTQ typically requires a scale and bias for accuracy and focuses on minimizing the total number of bits rather than hardware resources. To address the problem, we propose a new PTQ framework. Our method primarily focuses on minimizing the mantissa bits in floating-point multiplications, identified as the most resource-intensive components. More importantly, to mitigate large accuracy degradation, we introduce a distribution-based strategy for selecting quantization-target parameters and a Mean Squared Error-based quantization approach with extra exponent bias. The experimental results show that our framework outputs a new sub-8bit format, i.e., FP6 with four exponent bits and one mantissa bit, reducing GPT-2 models by 5.3 times with only a minimal increase in perplexity (0.36). In addition, we present a custom-designed multiplier, greatly enhancing resource-efficiency. Our multiplier design significantly reduces resource utilization by 53.2 times over a FP32 multiplier.