[FEATURE] Missing Audio Matching Evaluation Framework

[Verryx-02]

Related to #11

Problem:

Right now, we have no reliable way to compare how well different audio matching algorithms work.

This means we can't tell:

• if a newer version of spot_downloader actually improves accuracy over an older version
• if spot_downloader is better than other tools like spotDL.

The core problem is that we don't have a clear definition of what a "correct match" is.

---

Definition of a correct match

Let:

• a be a Spotify track
• b be a YouTube track returned by a matching algorithm

A match match(a, b) is correct if:

a and b are perceptually equivalent, i.e. they are indistinguishable to a human listener, even if they differ in:

• recording device
• compression
• upload quality
• minor mastering differences

Important: Robust audio fingerprinting (like Shazam) doesn't work here, because it ignores these differences and considers tracks identical even if they're re-recorded through speakers and a microphone.

Perceptual Audio Similarity Function

Introduce a perceptual audio similarity function:

$$D(a, b) \in [0, 1]$$

where:

• $D(a, b) \approx 1$ → tracks are perceptually equivalent
• $D(a, b) \approx 0$ → tracks are clearly different

Possible implementations:

• Chroma-based similarity (pitch content comparison)
• MFCCs with temporal alignment (DTW) (spectral features)
• Learned audio embeddings (e.g., OpenL3, VGGish, CLAP) with cosine similarity

A match is considered correct if:

$$D(a, b) \geq \tau$$

where $\tau$ is a threshold approximating human auditory equivalence (to be calibrated using manual verification).

Benchmark Algorithm

Given:

• Spotify playlist $S = {s_1, s_2, ..., s_n}$
• Algorithm A producing audio files: $A = {a_1, a_2, ..., a_n}$
• Algorithm B producing audio files: $B = {b_1, b_2, ..., b_n}$

For each track $i$:

$$\text{difference}_i = \begin{cases} \text{true} & \text{if } D(a_i, b_i) < \tau \\ \text{false} & \text{if } D(a_i, b_i) \geq \tau \end{cases}$$

Output: List of track indices where $\text{difference}_i = \text{true}$

This allows us to:

• Focus manual effort only on disagreements
• Build a ground truth dataset efficiently
• Calculate precision metrics objectively

Evaluation Metric

Given a set of Spotify tracks $S$ and a matching algorithm $A$:

$$\text{Precision}(A) = \frac{\text{number of correct matches}}{\text{total number of matches}}$$

This allows objective comparison between:

• different versions of the same algorithm (e.g., spot_downloader v1.0 vs v1.1)
• different matching tools (e.g., spot_downloader vs spotDL)

---

Implementation Plan

Phase 1: Benchmark Tool Development

1. Research perceptual similarity algorithms

   • Evaluate Chroma, MFCC+DTW, and audio embeddings approaches
   • Test on sample audio pairs to understand behavior
   • Select the most appropriate method

2. Implement the benchmark tool

   • Input: Two directories with audio files (from Algorithm A and B)
   • Process: Compare corresponding audio files using perceptual similarity
   • Output: List of track IDs where algorithms disagree (D(a, b) < τ)

3. Calibrate threshold τ

   • Test with known correct/incorrect matches
   • Find threshold that best separates similar from different

Phase 2: Evaluation Dataset Creation

4. Create evaluation playlist

   • Select ~100-200 diverse tracks
   • Include edge cases (live, acoustic, remixes, duplicates)
   • Document playlist creation criteria

5. Download with multiple algorithms

   • Run spot_downloader current version
   • Run comparison algorithm (spotDL or older version)
   • Store audio files separately

6. Run benchmark and manual verification

   • Use benchmark tool to identify differences
   • Listen only to flagged tracks
   • Label which algorithm produced better match
   • Create ground truth dataset