CAM statistics

[markitantov]

Hello,
Thank you for providing this repository.
I have one question.
How can I get statistics from CAM?
For example, in the task of sound event detection. From each sound file I extracted a set of spectrograms. I drew activation map on each spectrograms, but unfortunately it is low informative. I want to get statistics on specific axes (eg frequency axis). I want to draw a graph showing the relationship between a sound file class and the most informative frequencies obtained from CAM (in terms of that class). It should be just line for each class (X - frequency, Y - activation value)
I hope, that you understand me:)

[frgfm]

Hi @markitantov 👋

So, from what I understood, here are a few points:

• this library is made for CAM extraction specifically
• about your model, what is the shape of the output tensor? do you have a target layer in mind? What types of layer are in your model? what are the shape of feature maps at the target layer?

So to make this simple: if your model is a classification model for "does the event occur in the sound sample? Y/N", then most likely we'll be able to pinpoint the part of the input spectrum that steered the prediction.

Cheers ✌️

[markitantov]

My model is PANN based. It has convolution and fully-connected layers, and some transformations like max, mean.
The input size is (B, Time_steps, Mel_bins). The output size is (B, C_num), where C_num is the number of classes. I have 6 classes.

SpeechCNN(
  (base): Cnn14_16k(
    (bn0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv_block1): ConvBlock(
      (conv1): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_block2): ConvBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_block3): ConvBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_block4): ConvBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_block5): ConvBlock(
      (conv1): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_block6): ConvBlock(
      (conv1): Conv2d(1024, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (conv2): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (bn2): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    -> here mean, max, sum transformation
    (fc1): Linear(in_features=2048, out_features=2048, bias=True)
    (fc_audioset): Linear(in_features=2048, out_features=527, bias=True)  # not used
  )
  (fc_transfer): Linear(in_features=2048, out_features=6, bias=True)
)

Transformations, which I mentioned after conv_block6:

  x = self.conv_block6(x)
  x = F.dropout(x, p=0.2, training=self.training)
  x = torch.mean(x, dim=3)
  (x1, _) = torch.max(x, dim=2)
  x2 = torch.mean(x, dim=2)
  x = x1 + x2
  x = F.dropout(x, p=0.5, training=self.training)
  x = F.relu_(self.fc1(x))

Perhaps because of this transformations CAM does not work correctly?
What target layer should I use? conv_block6 and conv_block6.bn2 have two different results.
And what about fc layer? fc1 or fc_transfer?
Now my configuration is:
CAM(model, target_layer='base.conv_block6.conv2', fc_layer='base.fc1'), input_shape=(201,64))

[markitantov]

Do you have any suggestions? How can I get the graph of CAM statistics, f.e. value of activation vs frequencies?

[frgfm]

Hey there :)

Sorry about the late reply!

Regarding the CAM method, definitely discard base CAMs (since you have several Linear layers) and go for LayerCAM (quite fast and very good performances) !

So, with that, at best, given your input/output shapes, you should target a layer that has an output shape of (N, C, T, M). The good thing is that then it's analog to an image tensor, so you should be able to visualize it the same way with time steps vertically 👍

Let me know if it's unclear 👌