A contrastive learning framework for music–emotion alignment
Music information retrieval systems typically encode audio through genre or tempo tags, these structural attributes fail to capture the affective experience of listening. This project introduces music-CLAP, a contrastive learning framework that aligns music audio with affect-rich natural language descriptions in a shared embedding space. Building on the CLIP paradigm applied to audio, it pairs a frozen MERT-v1-95M music encoder with a BGE-base-en-v1.5 text encoder, both projecting to a shared 512-dimensional space. Training uses a custom 5,000-sample dataset of music clips annotated with emotional descriptions partly generated with Music Flamingo.
I was working on a music recommendation system for a project and found that traditional metadata-based approaches (genre, tempo, etc.) were not sufficient to capture the emotional qualities users often seek.
I recently studied the CLIP paradigm for vision–language alignment and realized there is an audio adaptation of this approach: CLAP. However, most existing CLAP models are trained on large-scale datasets of short audio clips for general audio understanding, such as:
The only open-weight model trained specifically on music that I found was:
In my experiments, it did not perform as well as I hoped on emotion-related retrieval tasks. This motivated me to implement my own CLAP variant specifically for music–emotion alignment, using a smaller and more focused dataset while leveraging strong pretrained encoders.
graph LR
A["Music Waveform\n24kHz mono"] --> B["MERT-v1-95M\nfrozen"]
B --> C["Weighted Layer Fusion\n13 layers × learned softmax weights"]
C --> D["Projection Head\nLinear + GELU + Residual + LayerNorm"]
D --> E["512D Audio Embedding"]
F["Emotional Text"] --> G["BGE-base-en-v1.5"]
G --> H["CLS Pooling"]
H --> I["Projection Head\nshared design"]
I --> J["512D Text Embedding"]
E --> K{"Contrastive Loss\nSigLIP / InfoNCE"}
J --> K
After searching for a suitable pretrained music encoder, I chose MERT-v1-95M for its strong performance on music understanding tasks and its manageable size. For the text encoder, I selected BGE-base-en-v1.5 for its high-quality sentence embeddings and open availability. I froze both encoders to leverage their pretrained knowledge without the computational cost of fine-tuning to my relatively small dataset.
The fused representation is projected from 768D to 512D through a shared projection head with a residual connection on the second linear. Biases are omitted throughout, following modern best practices for contrastive projection heads.
| Source | Size | Annotation method | Key property |
|---|---|---|---|
| MTG-Jamendo | ~2,000 | Music Flamingo | Human mood/theme tags used as context for caption generation |
| Jamendo-QA | ~1,500 | Pre-existing captions | High-quality natural language descriptions, no inference needed |
| FMA | ~1,500 | Music Flamingo | Genre-balanced subset from the medium split |
Total: ~5,000 samples, split into a 90/10 train/val partition, stratified by source.
MTG-Jamendo and FMA tracks carry no natural language descriptions. They are annotated with Music Flamingo (4-bit quantized inference on GPU) using the following prompt:
"Describe how this music makes you feel. Focus on emotions, mood, and atmosphere rather than technical characteristics."
This produces affective captions aligned with the retrieval use case: a query like "melancholic and introspective" will match tracks whose caption contains semantically similar language.
The annotation pipeline writes to a crash-resilient JSONL file, enabling resumption after OOM errors or interruption. Jamendo-QA captions are written directly without inference.
subset_selector.py → manifest.parquet (stratified sampling, 90/10 train/val split)
annotate.py → dataset.parquet (Music Flamingo captions + Jamendo-QA captions)
music_clap_dataset.py → PyTorch Dataset (resample to 24kHz, random/center crop)
src/
├── music_clap/
│ ├── audio.py # MERT encoder with weighted layer fusion
│ ├── clap.py # Joint CLAP model
│ ├── config.py # Hyperparameters and training configuration
│ ├── checkpoint.py # Checkpoint saving/loading utilities
│ ├── collate.py # Batch collation (Wav2Vec2 processor + BGE tokenizer)
│ ├── loss.py # Contrastive loss (SigLIP / InfoNCE)
│ └── train.py # Training loop
└── dataset/
├── subset_selector.py # Manifest builder
├── annotate.py # Music Flamingo annotation pipeline
├── music_clap_dataset.py # PyTorch Dataset
└── create_5k_dataset.py # Full pipeline orchestrator
The src/original directory contains my initial experimentation with the msclap codebase, which I used to understand and learn the CLAP framework before implementing my own version. The final music_clap implementation is a complete rewrite for my specific use case.
git clone https://github.com/GabinVr/music-CLAP.git
cd music-CLAP
uv venv && source .venv/bin/activate
uv run pip install -r requirements.txt
# Download and configure dataset paths
cp .env.example .env
# Edit .env: set JAMENDO_MTG_PATH, FMA_AUDIO_PATH, DATA_OUTPUT_PATH, etc.
# Build the annotated dataset (requires GPU for Music Flamingo annotation)
uv run src/dataset/create_5k_dataset.py
# Train the model
uv run main.pyI am actively training the model and trying different hyperparameters and looking for an opportunity to train the model on a more powerful system to be able to increase the batch size and number of epochs, which is currently limited by my local GPU.
-
Elizalde, B., Deshmukh, S., Al Ismail, M., & Wang, H. (2023).
CLAP: Learning Audio Concepts from Natural Language Supervision.
In ICASSP 2023 - IEEE International Conference on Acoustics, Speech and Signal Processing, 1–5.@inproceedings{elizalde2023clap, title={Clap learning audio concepts from natural language supervision}, author={Elizalde, Benjamin and Deshmukh, Soham and Al Ismail, Mahmoud and Wang, Huaming}, booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={1--5}, year={2023}, organization={IEEE} }
-
Li, Y., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Xiao, C., Lin, C., Ragni, A., Benetos, E., et al. (2023).
MERT: Acoustic Music Understanding Model with Large-Scale Self-Supervised Training.
arXiv preprint arXiv:2306.00107.@article{li2023mert, title={Mert: Acoustic music understanding model with large-scale self-supervised training}, author={Li, Yizhi and Yuan, Ruibin and Zhang, Ge and Ma, Yinghao and Chen, Xingran and Yin, Hanzhi and Xiao, Chenghao and Lin, Chenghua and Ragni, Anton and Benetos, Emmanouil and others}, journal={arXiv preprint arXiv:2306.00107}, year={2023} }
- Music Flamingo (m-a-a-p / NVIDIA): Hugging Face
- Original msclap implementation: microsoft/CLAP