Add compatibility for attention applied only at single layer of MusicRNN.

[naotokui]

Hi,

I've trained a DrumsRNN model with my own drum sequence dataset and been trying to use it with magenta-js. When I load the model, I get errors apparently caused by the differences in the layer structure described in weights_manifest.json. I suspect that it is a compatibility issue.

Which tensorflow version is compatible with magenta-js?

FYI, I used tensorflow 1.4.1 to train the DrumsRNN model.

Thanks

[adarob]

What is the error that you saw?

[naotokui]

I have a CodePen sketch based on Tero's amazing Neural Drum Machine.

Every time I tried to generate sequences (sampleRnn()) using my own model, I got this error:

magentamusic.js:45133 Uncaught (in promise) TypeError: Cannot read property 'matMul' of undefined
    at t.sampleRnn (magentamusic.js:45133)
    at magentamusic.js:45098
    at Object.t.tidy (magentamusic.js:12849)
    at t.<anonymous> (magentamusic.js:45088)
    at magentamusic.js:44957
    at Object.next (magentamusic.js:44938)
    at magentamusic.js:44932
    at new Promise (<anonymous>)
    at r (magentamusic.js:44928)
    at t.continueSequence (magentamusic.js:45069)

Then I figured out that my instance of musicRNN class has 0 lstmCells after the initialization (len(this.lstmCells) == 0). I believe it's due to the differences in the network structure described in weights_manifes.json.

Here is the weights_manifes.json of the original DrumsRNN model.
https://gist.github.com/naotokui/847394f037c3481c27600da13a51728a

and this is mine.
https://gist.github.com/naotokui/aa64abf09e748c0d5dbc14d7009731d4

For example, one of the lstm layers in the original model has a name like this: rnn/attention_cell_wrapper/multi_rnn_cell/cell_0/basic_lstm_cell/bias

but in my model, the same layer iss named:
rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/basic_lstm_cell/bias

I suspect this is the cause of this issue.

[adarob]

Thanks for pointing this out. There is indeed a compatibility issue due to a change we made in the Python code. Can you share your checkpoint with me so that I can make the required changes to magenta.js to support both types of checkpoints?

…

On Tue, Jul 17, 2018, 8:09 PM Nao Tokui ***@***.***> wrote: I have a CodePen sketch based on Tero's amazing Neural Drum Machine. Every time I tried to generate sequences (sampleRnn()) using my own model, I got this error: magentamusic.js:45133 Uncaught (in promise) TypeError: Cannot read property 'matMul' of undefined at t.sampleRnn (VM510 magentamusic.min.js:7) at VM510 magentamusic.min.js:7 at Object.t.tidy (VM510 magentamusic.min.js:7) at t.<anonymous> (VM510 magentamusic.min.js:7) at VM510 magentamusic.min.js:7 at Object.next (VM510 magentamusic.min.js:7) at VM510 magentamusic.min.js:7 at new Promise (<anonymous>) at r (VM510 magentamusic.min.js:7) at t.continueSequence (VM510 magentamusic.min.js:7) t.sampleRnn @ magentamusic.js:45133 (anonymous) @ magentamusic.js:45098 t.tidy @ magentamusic.js:12849 (anonymous) @ magentamusic.js:45088 (anonymous) @ magentamusic.js:44957 (anonymous) @ magentamusic.js:44938 (anonymous) @ magentamusic.js:44932 r @ magentamusic.js:44928 t.continueSequence @ magentamusic.js:45069 generatePattern @ pen.js:94 regenerate @ pen.js:261 (anonymous) @ pen.js:485 Promise.then (async) (anonymous) @ pen.js:485 Then I figured out that my instance of musicRNN class has 0 lstmCells after the initialization (len(this.lstmCells) == 0). I believe it's due to the differences in the network structure described in weights_manifes.json. Here is the weights_manifes.json of the original DrumsRNN model. https://gist.github.com/naotokui/847394f037c3481c27600da13a51728a and this is mine. https://gist.github.com/naotokui/aa64abf09e748c0d5dbc14d7009731d4 For example, one of the lstm layers in the original model has a name like this: rnn/attention_cell_wrapper/multi_rnn_cell/cell_0/basic_lstm_cell/bias but in my model, the same layer iss named: rnn/multi_rnn_cell/cell_0/attention_cell_wrapper/basic_lstm_cell/bias I suspect this is the origin of this issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCa6C3d3VtvkKbh_U0qE2iGEcJepDvuks5uHqb9gaJpZM4VMtv-> .

[naotokui]

Thank you! here are my checkpoints:
https://www.dropbox.com/sh/84ohnr9ee8yvzvm/AADn70s7HGzcLhQkLAfEaRbna?dl=0

[adarob]

Hi Nao, Unfortunately I won't be able to look into this until early next week. If you want something that works before then, it should be possible to retrain your model with "--hparams=attn_length=0". The resulting checkpoint should then work, although it may not be perform quite as well without attention.

…

-Adam

On Thu, Jul 19, 2018 at 7:39 AM Nao Tokui ***@***.***> wrote: Thank you! here are my checkpoints: https://www.dropbox.com/sh/84ohnr9ee8yvzvm/AADn70s7HGzcLhQkLAfEaRbna?dl=0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#106 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCa6M6GuqQrMsqqxQrD5VLriaUTclCTks5uIJo8gaJpZM4VMtv-> .

[mjxmusic]

hi @adarob - any idea if there was ever a fix implemented for this w/ Magenta.JS?

[adarob]

Sorry. but I am not sure if/when we will have time to fix this on our end. However, we would absolutely accept PR hhat does!