Generating midi in Magenta-js with own trained DrumsRNN model

[jonasdoevenspeck]

Hi,

I want to train my own Drums_RNN model and use it to generate MIDI in a web-browser with magenta-js.
I have the midi generation and synthesis working with the pretrained drums_rnn model but have some issues when using my own trained model.

1) First, I trained the model as follows:

drums_rnn_train --config="drum_kit" --run_dir="path1" --sequence_example_file="path2" --hparams="batch_size=64,rnn_layer_sizes=[64,64]" --num_training_steps=3000

The model is then used in magenta-js as follows:

  import { Player, MusicRNN } from '@magenta/music';

  const model = new MusicRNN("local_http_link");
  const player = new Player();
  console.log(player);
  await model.initialize();
  await player.initialize();

//manually generate priming sequence here
sequence = {{{pitch: 36, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true}.
                     {pitch: 36, quantizedStartStep: 4, quantizedEndStep: 5, isDrum: true},
                     {pitch: 36, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true},
                      ...
                      },
      quantizationInfo: {stepsPerQuarter: 4},
      tempos: [{time: 0, qpm: 120}],
      totalQuantizedSteps: 9}

//generate new sequence by feeding priming sequence
  const samples = await model.continueSequence(sequence, 50);
  player.resumeContext();
  await player.start(samples);

This results in the following error as decribed already in #106

Unhandled Rejection (TypeError): Cannot read properties of undefined (reading 'matMul')

2) As suggested in the issue linked above, I solved this by including "--hparams=attn_length=0"

So the new train cmd is:
drums_rnn_train --config="drum_kit" --run_dir="path1" --sequence_example_file="path2" --hparams="batch_size=64,rnn_layer_sizes=[64,64],attn_length=0" --num_training_steps=3000

I used the same magenta-js code to generate new MIDI and now get the following error:

Unhandled Rejection (Error): Error in matMul: inner shapes (74) and (582) of Tensors with shapes 1,74 and 582,256 and transposeA=false and transposeB=false must match.

I also add the entire error trace at the bottom of this issue.
I have the feeling the input dimension of my priming sequence is incompatible with the dimension of the trained network.
However, I have no idea how to solve it.

Extra information:

1. I'm able to generate new midi sequences with this model with the CLI with the following cmd:
   drums_rnn_generate --config="drum_kit" --run_dir="model_path" --hparams="batch_size=64,rnn_layer_sizes=[64,64],attn_length=0" --output_dir="path_2"

However, this cmd doesn't require a priming sequence in contrary to the model.continueSequence() from magenta-js.

2. When I use the pretrained DrumsRNN model to generate midi in magenta-js, I don't have any errors.

Full error trace:
Unhandled Rejection (Error): Error in matMul: inner shapes (74) and (582) of Tensors with shapes 1,74 and 582,256 and transposeA=false and transposeB=false must match.
Module.assert
src/util_base.ts:108
105 | assert(a != null, () => The input to the tensor constructor must be a non-null value.);
106 | }
107 | // NOTE: We explicitly type out what T extends instead of any so that

108 | // util.flatten on a nested array of number doesn't try to infer T as a
109 | // number[][], causing us to explicitly type util.flatten().
110 | /**
111 | * Flattens an arbitrarily nested array.
View compiled
batchMatMul [as kernelFunc]
src/kernels/BatchMatMul.ts:64
61 | [a3dStrides[0], 1, a3dStrides[1]] :
62 | [a3dStrides[0], a3dStrides[1], 1];
63 | const [bInnerStep, bOuterStep, bBatch] = transposeB ?
64 | [1, b3dStrides[1], b3dStrides[0]] :
| ^ 65 | [b3dStrides[1], 1, b3dStrides[0]];
66 | const size = leftDim * rightDim;
67 | const result = buffer([batchDim, leftDim, rightDim], a3d.dtype);
View compiled
kernelFunc
src/engine.ts:598
595 | }
596 | const dataId = backend.write(backendVals, shape, dtype);
597 | const t = new Tensor(shape, dtype, dataId, this.nextTensorId());
598 | this.incRef(t, backend);
| ^ 599 | // Count bytes for string tensors.
600 | if (dtype === 'string') {
601 | const info = this.state.tensorInfo.get(dataId);
View compiled
(anonymous function)
src/engine.ts:668
665 | }
666 | this.state.numTensors--;
667 | if (a.dtype === 'string') {
668 | this.state.numStringTensors--;
| ^ 669 | }
670 | const info = this.state.tensorInfo.get(a.dataId);
671 | const refCount = info.refCount;
View compiled
Engine.scopedRun
src/engine.ts:453
450 | if (this.shouldCheckForMemLeaks()) {
451 | this.checkKernelForMemLeak(kernelName, numDataIdsBefore, outInfos);
452 | }
453 | const outTensors = outInfos.map((outInfo) => {
| ^ 454 | // todo (yassogba) remove this option (Tensor) when node backend
455 | // methods have been modularized and they all return tensorInfo.
456 | // TensorInfos do not have a rank attribute.
View compiled
Engine.runKernelFunc
src/engine.ts:665
662 | disposeTensor(a) {
663 | if (!this.state.tensorInfo.has(a.dataId)) {
664 | return;
665 | }
| ^ 666 | this.state.numTensors--;
667 | if (a.dtype === 'string') {
668 | this.state.numStringTensors--;
View compiled
Engine.runKernel
src/engine.ts:522
519 | outputs = kernelProfile.outputs;
520 | }
521 | });
522 | if (isTapeOn) {
| ^ 523 | this.addTapeNode(kernelName, inputs, outputs, backwardsFunc, saved, attrs);
524 | }
525 | if (this.state.profiling) {
View compiled
matMul_
src/ops/mat_mul.ts:54
matMul__op
src/ops/operation.ts:51
48 | ENGINE.endScope(result);
49 | return result;
50 | }
51 | catch (ex) {
| ^ 52 | ENGINE.endScope(null);
53 | throw ex;
54 | }
View compiled
basicLSTMCell_
src/ops/basic_lstm_cell.ts:61
58 | const f = slice(res, [0, sliceCols * 2], sliceSize);
59 | const o = slice(res, [0, sliceCols * 3], sliceSize);
60 | const newC = add(mul(sigmoid(i), tanh(j)), mul($c, sigmoid(add($forgetBias, f))));
61 | const newH = mul(tanh(newC), sigmoid(o));
62 | return [newC, newH];
63 | }
64 | export const basicLSTMCell = op({ basicLSTMCell_ });
View compiled
Module.basicLSTMCell__op
src/ops/operation.ts:51
48 | ENGINE.endScope(result);
49 | return result;
50 | }
51 | catch (ex) {
| ^ 52 | ENGINE.endScope(null);
53 | throw ex;
54 | }
View compiled
Array.
src/music_rnn/model.ts:165
162 | else {
163 | sampledOutput = logits.argMax().as1D();
164 | }
165 | if (returnProbs) {
| ^ 166 | probs.push(tf.softmax(logits));
167 | }
168 | nextInput =
View compiled
multiRNNCell_
src/ops/multi_rnn_cell.ts:56
Module.multiRNNCell__op
src/ops/operation.ts:51
48 | ENGINE.endScope(result);
49 | return result;
50 | }
51 | catch (ex) {
| ^ 52 | ENGINE.endScope(null);
53 | throw ex;
54 | }
View compiled
MusicRNN.sampleRnn
src/music_rnn/model.ts:375
(anonymous function)
src/music_rnn/model.ts:272
(anonymous function)
src/engine.ts:442
439 | // backend and set properties like this.backendName
440 | // tslint:disable-next-line: no-unused-expression
441 | this.backend;
442 | }
| ^ 443 | const kernel = getKernel(kernelName, this.backendName);
444 | let out;
445 | if (kernel != null) {
View compiled
Engine.scopedRun
src/engine.ts:453
450 | if (this.shouldCheckForMemLeaks()) {
451 | this.checkKernelForMemLeak(kernelName, numDataIdsBefore, outInfos);
452 | }
453 | const outTensors = outInfos.map((outInfo) => {
| ^ 454 | // todo (yassogba) remove this option (Tensor) when node backend
455 | // methods have been modularized and they all return tensorInfo.
456 | // TensorInfos do not have a rank attribute.
View compiled
Engine.tidy
src/engine.ts:440
437 | // can be deferred until an op/ kernel is run).
438 | // The below getter has side effects that will try to initialize the
439 | // backend and set properties like this.backendName
440 | // tslint:disable-next-line: no-unused-expression
| ^ 441 | this.backend;
442 | }
443 | const kernel = getKernel(kernelName, this.backendName);
View compiled
Module.tidy
src/globals.ts:192
189 | const tensors = getTensorsInContainer(container);
190 | tensors.forEach(tensor => tensor.dispose());
191 | }
192 | /**
193 | * Keeps a tf.Tensor generated inside a tf.tidy from being disposed
194 | * automatically.
195 | *
View compiled
MusicRNN.continueSequenceImpl
src/music_rnn/model.ts:258
MusicRNN.continueSequence
src/music_rnn/model.ts:215
playGen
src/magenta/magenta.js:137
134 |
135 |
136 |
137 | const samples = await model.continueSequence(sequence, 50);
| ^ 138 |
139 | player.resumeContext();
140 | await player.start(samples);

[mjxmusic]

hi @jonasdoevenspeck - did you ever work through this issue?