Local Music Recommendation System

This repository contains a fully local music recommendation pipeline inspired by Shazam‐style audio identification and recommendation systems. It is designed to run efficiently on a consumer laptop (e.g. Intel® Ultra 7 155H, 32 GB RAM, NVIDIA RTX 4070) without requiring any cloud services. The goal is to identify what song is playing, extract meaningful audio features (style, mood, tempo, etc.), find similar or contrasting tracks from your local library, and provide short natural–language explanations using a lightweight local large language model (LLM) running via Ollama.

Overview

The pipeline is composed of several stages:

1. Fingerprinting: Each track in your library is fingerprinted once using Dejavu (a pure‑Python Shazam‑style recogniser). Fingerprints are stored in a local SQLite database so that unknown audio snippets can be matched quickly.
2. Feature extraction: For every track, we compute high‑level descriptors such as mood (valence/arousal), tempo, key, energy and a timbre/style embedding. This uses the open‑source Essentia streaming extractor and MusicNN for multi‑label genre/instrument/mood tags. Thanks to Institute of Computational Perception!
3. Audio embedding: A 512‑dimension audio embedding is produced using the CLAP HTSAT‑Tiny model (loaded via the laion_clap package). These vectors capture timbre and “sounds‑like” similarity across genres. All embeddings are stored in a NumPy array and indexed with FAISS for fast nearest‑neighbour search.
4. Sequencing: Given a current track and a pool of candidates, a simple cost function decides which track to play next. The cost includes style similarity/dissimilarity, mood differences, tempo and key distances, and energy changes. Two example policies are provided:
   • Smooth mode: emphasises continuity (small mood/tempo/key shifts) and slight stylistic similarity.
   • Contrast mode: every few tracks, encourages a large stylistic jump while constraining mood/tempo/key to avoid whiplash.
5. Explanation: A local LLM (e.g. Qwen 2.5 VL 3B in 4‑bit quantisation) is prompted via Ollama to explain why the chosen transition is musically appropriate. Prompts are kept under 160 tokens to control memory usage.

This repository provides a minimal but complete implementation of the above pipeline. It is organised so that each component can be swapped out or extended without affecting the rest of the system.

Folder structure

music_rec_system/
├── README.md            # this file
├── requirements.txt      # Python dependencies
├── config.yaml           # global configuration (paths, model settings, weights)
├── .gitignore            # ignore transient files (build artefacts, data, caches)
├── data/                 # data files (fingerprint DB, embeddings, features, indices)
│   ├── README.md         # explanation of each data artefact
├── scripts/              # Python scripts implementing each stage
│   ├── fingerprint.py    # fingerprinting your music library
│   ├── embed_clap.py     # computing CLAP embeddings
│   ├── extract_features.py # extracting mood/tempo/key/energy and genre tags
│   ├── build_faiss_index.py # building the FAISS index from embeddings
│   ├── sequencer.py      # transition cost functions and playlist sequencing
│   ├── explanation.py    # LLM prompts for explaining transitions
│   └── generate_playlist.py # end‑to‑end pipeline demo
└── tests/                # minimal unit tests (optional)

Installation

This project targets Python 3.9+ on Linux or Windows. Use a virtual environment (e.g. venv or conda) to avoid polluting your system Python. Then install the dependencies:

python -m venv .venv
source .venv/bin/activate  # on Windows use .venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt

# You also need Essentia and its models (see below), and to set up Ollama.

Additional dependencies

• Essentia: The Essentia streaming extractor provides high‑level audio descriptors (tempo, key, energy, valence, arousal). Follow the Essentia installation guide for your platform, or install via conda (conda install -c conda-forge essentia). You also need the pretrained streaming_extractor_music model and configuration files; these can be downloaded from UPF’s website.
• MusicNN: Install the MusicNN models for tag extraction by running pip install musicnn and downloading the pretrained weights via the MusicNN API (see MusicNN).
• Ollama: Install Ollama to run local LLMs. Make sure to download the models you intend to use (e.g. ollama pull qwen2.5vl:3b) and adjust the settings in config.yaml accordingly.

Configuration

The file config.yaml centralises paths and hyperparameters. Key fields include:

• database_path: location of the SQLite database for fingerprints.
• embedding_file: path to a .npy file containing all CLAP embeddings (shape (n_tracks, 512)).
• feature_file: path to a JSON or CSV file with extracted features per track (IDs, tags, mood, tempo, key, energy).
• faiss_index_file: location to store the serialized FAISS index.
• ollama_model and ollama_options: specify the Ollama model tag and runtime options (max output length, temperature, stop token, etc.).
• smooth_policy and contrast_policy: weight parameters used by the sequencer to compute transition costs.

Feel free to adjust these fields to match your local environment (e.g. if your music is stored on a different drive).

Usage

1. Fingerprint your library

Run the fingerprinting script to analyse all audio files under a directory and populate the fingerprint database:

python scripts/fingerprint.py --music-dir /path/to/music --extensions .mp3 .flac .wav

This creates the SQLite database specified in config.yaml (by default data/fingerprints.db).

2. Extract features

Use the Essentia streaming extractor and MusicNN to compute high‑level descriptors and save them to data/features.json:

python scripts/extract_features.py --music-dir /path/to/music

This will output a JSON/CSV file with one entry per track containing mood (valence and arousal), tempo (BPM), key (as an integer 0–11), energy, and a vector of genre/instrument tags.

3. Compute CLAP embeddings

Generate 512‑dimensional CLAP embeddings for each track and save them into a single NumPy file:

python scripts/embed_clap.py --music-dir /path/to/music

Running this script requires a GPU with at least 2 GB of VRAM (e.g. your RTX 4070). The script streams audio from disk and processes it in batches to minimise memory usage.

4. Build the FAISS index

Once all embeddings have been generated, build a FAISS index for fast nearest‑neighbour search:

python scripts/build_faiss_index.py

The resulting index is stored at the path specified by faiss_index_file in the configuration.

5. Generate playlists

To generate a playlist from a recorded snippet and receive LLM explanations, run:

python scripts/generate_playlist.py --input-audio path/to/snippet.wav \
    --length 20 --policy smooth

This script will:

1. Recognise the current track via Dejavu.
2. Look up its features and embedding.
3. Query FAISS for similar tracks (or sample dissimilar ones during contrast slots).
4. Use the sequencer cost function and the selected policy to build a list of track IDs.
5. Prompt the local LLM to explain each transition concisely.

The resulting playlist (list of track IDs, titles, and explanations) is printed to the console.