Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, Xubo Liu

Stability AI

Abstract

The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models have concentrated on low parameter-count architectures using only components with strong inductive biases. In this work we show that by scaling a transformer architecture with large parameter count to this problem, and applying a flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to reach state-of-the-art speech quality at extremely low bit-rates of $400$ or $700$ bits-per-second. The trained models strongly out-perform existing baselines in both objective and subjective tests.

Model Architecture

Architecture of the proposed model TAAE (Transformer Audio AutoEncoder). Detail is shown for the encoder block and sub-blocks. The decoder block is configured identically to the encoder block, with the exception of the strided convolution, which is replaced with its transposed equivalent and moved to the end of the $T_m$ blocks.

Performance Comparison

The results of the MUSHRA subjective test indicate that TAAE obtains state-of-the-art results that outperform, by a significant margin, recently published speech codecs. Importantly, the proposed model obtains results close to the ground truth. Comparing these evaluation results indicates the potential of scaling transformer-based codec architectures to achieve new benchmarks in terms of speech quality and compression.