Training Loop Accessibility

[Ruining0916]

Hi Excellent Authors,

I noticed that you are using xtuner.train() for training. I wonder is there any way that we can access the main training loop such that we can monitor the pred_mask and gt_mask from each iteration, and also monitor ciou score for each iteration? Additionally, I wonder if there is any way to visualize training loss with mmengine (such as the tensorboard visualization)?

Thanks,
Ruining

[HarborYuan]

Hi @Ruining0916 ,

The loss is calculated here:

Sa2VA/projects/llava_sam2/models/llava_sam2.py

Line 416 in 6f59e55

sam_loss_mask = self.loss_mask(pred_masks, gt_masks)

You can add some code here to generate some information to monitor.

For the Tensorboard, I think Xtuner already supports Tensorboard. (https://github.com/InternLM/xtuner/blob/main/docs/zh_cn/training/visualization.rst)