TempoTokens

TempoTokens is a research project from the Hebrew University of Jerusalem and collaborators, presented at AAAI 2024. It introduces a method for diverse and aligned audio-to-video generation by adapting an existing text-to-video (T2V) generation model. The core innovation is a lightweight adaptor network that maps audio representations from a pretrained audio encoder into the input space expected by the T2V model. This enables conditioning on audio alone, text alone, or both text and audio simultaneously—a first for this task. The method addresses two alignment levels: global semantic alignment (the video content matches the audio class, e.g., a dog barking video with barking sound) and temporal alignment (energy peaks in audio correspond to visual events). To quantify temporal alignment, the authors propose a novel metric called AV-Align, which detects and compares energy peaks across modalities. TempoTokens is validated on three datasets (VGGSound, Landscape, AudioSet Drum), demonstrating higher visual quality, better alignment, and greater diversity than prior work like MM-Diffusion and TATS. TempoTokens is designed for researchers and developers exploring multimodal generation, particularly those working on audio-conditioned video creation. The project provides pretrained models and code for inference, but is not a commercial product. It is suitable for academic use and experimentation with diverse audio-visual datasets. Compared to existing solutions, TempoTokens uniquely supports joint text+audio conditioning and a lightweight adaptor that avoids full finetuning of large T2V models. While not a polished product, its code and evaluation metrics provide a valuable baseline for multimodal generation. The method is compatible with zeroscope_v2_576w as the T2V backbone, producing 576p resolution videos. Researchers can leverage the public code and pretrained models to experiment with audio-conditioned video synthesis.