error during training with opso 0.10.0

[sammlapp]

HI Samm,

Thank you for your help with the decibel_imits issue I experienced on the Spectrogram preprocessor. I am now having difficulty training my model in the new OpSo version. My code below:

model.logging_level = 2 # request lots of logged content
model.log_file = './multispecies_model_train_3/training_log_20102023.txt' #specify a file to log output to
Path(model.log_file).parent.mkdir(parents=True,exist_ok=True) #make the folder ./

model.verbose = 1 #don't print anything to the screen during training

#learning rate schedule
model.lr_cooling_factor = 0.3
model.lr_update_interval = 4

#Regularization weight decay
model.optimizer_params['weight_decay']=0.001

# Train the model on the full training set and validate on the validation set. Save the model to directory.
model.train(
    balanced_train_df,
    #train_df,
    valid_df,
    epochs= 1,
    batch_size= 64,
    log_interval = 5,
    num_workers= 4, # if multiple processors are available utilise them to speed up training
    #wandb_session = wandb_session,
    save_interval= 10,
    save_path= checkpoint_folder #location to save checkpoints
)

#Let Wandb know that we finished training successfully
wandb.unwatch(model.network)
wandb.finish()

Throws us the following error:

raining Epoch 0
0%
0/46 [00:42<?, ?it/s]
<frozen importlib._bootstrap>:1047: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
---------------------------------------------------------------------------
Empty                                     Traceback (most recent call last)
File c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\site-packages\torch\utils\data\dataloader.py:1132<file:///C:/Users/lscala/AppData/Local/anaconda3/envs/opensoundscape/Lib/site-packages/torch/utils/data/dataloader.py:1132>, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1131 try:
-> 1132     data = self._data_queue.get(timeout=timeout)
   1133     return (True, data)

File c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\multiprocessing\queues.py:114<file:///C:/Users/lscala/AppData/Local/anaconda3/envs/opensoundscape/Lib/multiprocessing/queues.py:114>, in Queue.get(self, block, timeout)
    113     if not self._poll(timeout):
--> 114         raise Empty
    115 elif not self._poll():

Empty:

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
c:\Users\lscala\Python_learning\multispecies_trial_chris.ipynb<file:///C:/Users/lscala/Python_learning/multispecies_trial_chris.ipynb> Cell 19 line 1
     12<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=11> model.optimizer_params['weight_decay']=0.001
     14<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=13> # Train the model on the full training set and validate on the validation set. Save the model to directory.
---> 15<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=14> model.train(
     16<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=15>     balanced_train_df,
     17<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=16>     #train_df,
     18<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=17>     valid_df,
     19<vscode-notebook-cell://c%3A/Users/lscala/Python_learning/multispecies_trial_chris.ipynb#X22sZmlsZQ%3D%3D?line=18>     epochs= 1,
...
-> 1145     raise RuntimeError(f'DataLoader worker (pid(s) {pids_str}) exited unexpectedly') from e
   1146 if isinstance(e, queue.Empty):
   1147     return (False, None)

RuntimeError: DataLoader worker (pid(s) 18868, 18184, 19292, 20324) exited unexpectedly
Output is truncated. View as a scrollable element<command:cellOutput.enableScrolling?c9cfdb8a-3cdc-4a1f-a618-229ff125be40> or open in a text editor<command:workbench.action.openLargeOutput?c9cfdb8a-3cdc-4a1f-a618-229ff125be40>. Adjust cell output settings<command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D>...

You’ll notice I have also commented out the WandB_session = WandBsession argument, as this also throws up an error related to missing samplez / training samples dir.

I have tried various tweaks to prevent the RuntimeErrorDataLoader error but without success. Have you come across this problem before?

Thanks very much,
Lorenzo

[sammlapp]

Hi Lorenzo, it's hard to tell what's happening here but I'm happy to try to help. The error message doesn't contain any information about opensoundscape, only that the "workers" failed. The "workers" in this context are the multi-threaded CPU processes that the DataLoader is launching for preprocessing when you use num_workers>1. I would try setting num_workers=0 to only use the root process, and check if any more intelligible errors are given.

[canihaveabravo]

I changed num_workers to 0 and now get the following error:

---

RuntimeError Traceback (most recent call last)
c:\Users\lscala\Python_learning\multispecies_trial_chris.ipynb Cell 19 line 1
12 model.optimizer_params['weight_decay']=0.001
14 # Train the model on the full training set and validate on the validation set. Save the model to directory.
---> 15 model.train(
16 balanced_train_df,
17 valid_df,
18 epochs= 1,
19 batch_size= 64,
20 log_interval = 5,
21 num_workers= 0, # if multiple processors are available utilise them to speed up training
22 #wandb_session = wandb_session,
23 save_interval= 10,
24 save_path= checkpoint_folder #location to save checkpoints
25 )
27 #Let Wandb know that we finished training successfully
28 #wandb.unwatch(model.network)
29 #wandb.finish()

File c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\site-packages\opensoundscape\ml\cnn.py:909, in CNN.train(self, train_df, validation_df, epochs, batch_size, num_workers, save_path, save_interval, log_interval, validation_interval, invalid_samples_log, raise_errors, wandb_session, progress_bar)
903 for epoch in range(epochs):
904 # 1 epoch = 1 view of each training file
905 # loss fn & backpropogation occurs after each batch
906
...
455 _pair(0), self.dilation, self.groups)
--> 456 return F.conv2d(input, weight, bias, self.stride,
457 self.padding, self.dilation, self.groups)

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 1, 103, 57] to have 3 channels, but got 1 channels instead
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[canihaveabravo]

I managed to fix this error by setting sample shape to sample_shape=(224, 224, 1) in my model = CNN()

[sammlapp]

Great, yes the error occurs because the default preprocessing in v0.10.0 changed to use the original spectrogram shape rather than some arbitrary shape like 224x224. If the audio samples are of different sampling rates, they will create spectrograms of different dimensions. Choosing the sample shape is the correct way to fix your issue.