Compatibility with other experiment managers (loggers) for model training

[paulpeyret-biophonia]

Hi OPSO community,

First, thanks again for this great package!

I have successfully trained a model with OPSO which I am very happy with. The experiment tracking is currently done with Wandb.
However, I am wondering if it would be possible in the future to use other experiment managers like MLFlow ? I have been using it in the past and I think its a great tool which is opensource and free for any user (even non-academic users like me).

It seems that some works/discussions are ongoing on the topic of switching to pytorch lightning framework for the models (see issues #978 and #917). Is this really the case ? (Tagging @sammlapp as you opened these issues).
If there is some work planned on this, I would be pleased to see this new architecture having the flexibility to allow various experiment managers ! 😃

It seems lightning is making it much easier than pytorch framework to use any experiment managers. You can check it out here:
https://lightning.ai/docs/pytorch/stable/visualize/experiment_managers.html

Do you think there is a chance to make OPSO model training architecture compatible with MLFLow or other loggers in the near future ?

Thank you for you time!

All the best!

[sammlapp]

Hi @paulpeyret-biophonia great question. I'm currently working on a major refactor which will provide a Lightning as well as pure PyTorch API within OpenSoundscape. I'd love to get some feedback on this lightning branch if you want to try it out. This refactor also implements all of the modularity enforced by Lightning in the pure PyTorch API, which will resolve issues like #917.

If you use the lightning.LightningSpectrogramModule.fit_with_trainer() method to train and .predict_with_trainer() method for inference, you have full access to all of the lightning Trainer features, including integration with ML Flow and other logging platforms, plus all the fancy lightning features like automatic mixed precision, multi-divice training, etc.

The reason for the odd names of the methods is to avoid conflict with other methods. For instance, the LightningModule needs the .train() method to just set a boolean flag of whether the object is in training mode or not. The "Trainer" is Lightning's (IMO awkward) way of using a separate object/class from the model object itself for training and inference. In my implementation, fit_with_trainer and predict_with_trainer internally create and use a lightning.Trainer object.

Happy to hear feedback if you try out the opso + lightning api