Synesthesia

Code and media samples for the paper Automated Music Generation for Visual Art through Emotion.

Generated Samples

Input
Output	midi	midi	midi	midi
Input
Output	midi	midi	midi	midi
Input
Output	midi	midi	midi	midi
Input
Output		midi	midi

Sources of the images: https://www.imageemotion.org/

Instructions

Dependencies

• python3.7
• cuda10
• musescore

The remaining python dependencies can be installed with:

pip install -r requirements.txt

Training dataset

The full dataset can be downloaded from https://www.cs.rochester.edu/u/qyou/deepemotion/ .

PerformanceRNN

cd rnn
python preprocess.py ../dataset/midi/train ../dataset/midi/rnn # preprocess data
python train.py -s ../model/rnn_example.sess -d ../dataset/midi/rnn -i 10 # train model
python generate.py -i ../dataset/image/test/1.jpg #generate music from a given image

The implementation of performance RNN is modified from https://github.com/djosix/Performance-RNN-PyTorch .

Transformer

cd transformer
python preprocess.py # preprocess data
python train.py # train model
python generate.py -i ../dataset/image/test/1.jpg #generate music from a given image

The implementation of transformer is modified from https://github.com/bearpelican/musicautobot .