Summary

AudioSep is an open-domain sound separation foundation model that performs audio source separation based on natural language queries. It can separate various sound sources such as audio events, musical instruments, and speech enhancement without task-specific training.

Features

• Strong zero-shot generalization ability on multiple sound separation tasks.
• Can separate sounds from natural language descriptions, enabling flexible and user-defined separation.
• Supports running inference with chunk-based processing to save memory.
• Open source implementation with training scripts and benchmark evaluation tools available.
• Processes audio at 32 kHz sampling rate using STFT with specified window and hop sizes.

Usage

The model can be set up via cloning the repository and installing dependencies with conda. Model weights are available for download. Inference can be performed by providing an audio file and a textual description of the desired sound source to be separated.

Training is possible with custom audio-text paired datasets formatted in a provided JSON structure. Users can train the model from scratch or fine-tune from pretrained checkpoints.

Evaluation

Benchmark evaluation scripts and datasets are provided to measure separation quality on diverse audio datasets with reported results such as SDR improvements on VGGSound, MUSIC, ESC-50, AudioSet, AudioCaps, and Clotho.

Citing

The repository provides references for academic citation if the tool is used.