Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modality settings. #6

Open
zgp123-wq opened this issue Feb 21, 2024 · 20 comments
Open

modality settings. #6

zgp123-wq opened this issue Feb 21, 2024 · 20 comments

Comments

@zgp123-wq
Copy link

I'm encountering several challenges while training my model and would appreciate guidance on resolving them, particularly regarding the modality settings.

  1. Error with Batch Size Setting at 12:
    When setting the batch size to 12, I encountered a RuntimeError due to inconsistent tensor sizes during stacking. The error message states: "stack expects each tensor to be equal size, but got [1919, 3, 40, 40] at entry 0 and [3673, 3, 40, 40] at entry 1."

  2. Out-of-Memory Error during Training with Batch Size 1:
    Despite reducing the batch size to 1 during training and selecting modality as "video," "mfcc," "vggish," and "VA_continuous_label," I faced persistent out-of-memory errors. It seems that the memory consumption remains high. Could you suggest strategies to mitigate memory usage during training, considering the specified modalities?

  3. Calculation Issue with Concordance Correlation Coefficient (CCC) during Validation:
    During the validation process, I encountered difficulties in calculating the Concordance Correlation Coefficient (CCC). Despite employing standard procedures and specifying modality as "mfcc," "vggish," "bert," and "VA_continuous_label," the calculation consistently fails. Are there any specific considerations regarding the modality setting that might affect CCC calculation? I would appreciate any advice on ensuring accurate CCC calculation during validation.
@Jesayy
Copy link

Jesayy commented Feb 21, 2024

Hello, may I ask what error you encountered with the last question? When I ran it, I got an error like "No such file or directory: '/aff_wild2_abaw5/mean_std_info.pkl". Do you have this file?

@sucv
Copy link
Owner

sucv commented Feb 22, 2024

Hi zgp123-wq,

  1. Does it happen in batchsize8 or 2? If 8 or 2 works, then the only way to find out is to debug the code, set a breakpoint in getitem(self, index) for the dataset.py, and see which trial caused the error, then maybe go back to dataset_info.pkl, check the length recorded there, and check the actual mp4 length, etc.
  2. I won't run the code in my own desktop. I run the code in computing servers, usually having hundreds of RAM and Video-RAM. If you choose to run in your local PC (having 32 GB RAM, it may lead to this error). However, in case it is caused by some problematic code, please still set break points in anywhere, probably dataset.py, to see what exactly caused the error and if it's code issue or truly insufficient RAM.
  3. The error only occurs for the three modalities? It should have nothing to do with the modalities. Please set a break point in the line calculating the CCC, to make sure the preds and labels have correct shapes, e.g. Nx1 and Nx1.

In short, I cannot answer your questions directly, only way is to debug line by line. Sorry I wrote the code in such a poor manner.

@Jesayy
Copy link

Jesayy commented Mar 4, 2024

Hello, I've encountered a similar error to the first problem you mentioned. Have you resolved it?

@praveena2j
Copy link

I think there is a bug in the code. torch data loader takes samples with equal shape but in the code, each trial is considered as a sample from getitem. SInce each trial has different shapes it throws this error

@praveena2j
Copy link

From the paper, I think the trials are divided into temporal sequences of 300 and each samples is considered as a sample of 300 samples, can you please confirm this ?

@praveena2j
Copy link

I think to fix the issue we need to change the windowing parameter to True in base/experiment.py line

self.data_arranger.generate_partitioned_trial_list(window_length=self.window_length,
hop_length=self.hop_length, fold=fold,
windowing=True)

@sucv
Copy link
Owner

sucv commented Mar 7, 2024

From the paper, I think the trials are divided into temporal sequences of 300 and each samples is considered as a sample of 300 samples, can you please confirm this ?

confirmed.

@sucv
Copy link
Owner

sucv commented Mar 7, 2024

I think to fix the issue we need to change the windowing parameter to True in base/experiment.py line

self.data_arranger.generate_partitioned_trial_list(window_length=self.window_length, hop_length=self.hop_length, fold=fold, windowing=True)

Thanks for your debugging and explanation. Indeed, setting windowing=True would use sliding window to sample each trial. Whereas setting windowing=False would load a complete trial. The latter is useful when you want to generate the output for the test trials with batch_size = 1.

But even when windowing=True, the output will still be restored to its original length (average if one time step has multiple output due to window overlapping) and the epoch CCC is then calculated over the restored output and VA labels.

@praveena2j
Copy link

Thank you for the confirmation

@zgp123-wq
Copy link
Author

Thank you very much for your reply. However, I have noticed a strange phenomenon: in the training set evaluation, the CCC score is only slightly above 50, while in the validation set evaluation, it reaches 65. In theory, the CCC score in the training set evaluation should be very close to 100.

@praveena2j
Copy link

May I know where do you load data annotations in the script ?

@sucv
Copy link
Owner

sucv commented Mar 8, 2024

Thank you very much for your reply. However, I have noticed a strange phenomenon: in the training set evaluation, the CCC score is only slightly above 50, while in the validation set evaluation, it reaches 65. In theory, the CCC score in the training set evaluation should be very close to 100.

Your "theory" has flaw then. Let me ask you,

Training CCC: 100
Validation CCC: 55

Training CCC: 55
Validation CCC: 54

Which model instance would you choose for submission?

Actually, you should say "strange" had the code reached CCC 100 in training.

@praveena2j
Copy link

can you share the "dataset_info.pkl" and "mean_std_info.pkl" ?

@praveena2j
Copy link

Also can you please let me know whether you trim the videos because "trim_video_fn" is not being called in preprocessing.py script as there is "trim_video" param is not set in config.py script

@zgp123-wq
Copy link
Author

train_ccc_and_val_ccc [datasetinfo_mean_sttdd.zip](https://github.com/sucv/ABAW3/files/14544763/datasetinfo_mean_sttdd.zip) I'm sorry, I just saw the message. I have provided the train_ccc and val_ccc, as well as the compressed file containing "dataset_info.pkl" and "mean_std_info.pkl" in the attachment. @sucv @praveena2j

@sucv
Copy link
Owner

sucv commented Mar 9, 2024 via email

@sucv
Copy link
Owner

sucv commented Mar 9, 2024 via email

@praveena2j
Copy link

Thanks for sharing the dataset_info.pkl files, in this file there is only 418 videos, I guess it is supposed to have 594 videos

@sucv
Copy link
Owner

sucv commented Mar 10, 2024 via email •
edited
Loading

@praveena2j
Copy link

Thanks for the update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sucv @praveena2j @zgp123-wq @Jesayy