Cannot Reproduce Results. Release Evaluation Code

[wjc2830]

Thank you for your work on this paper.
However, I am unable to reproduce the main results reported in your paper, including the FID, onset detection accuracy, and AP.
For evaluating FID, I used the SpecVQGAN code, and for onset performance, I used the CondFoleyGen code.
Despite using these resources, the results obtained from this repository's inference code do not match those reported in your paper.
Could you please release your evaluation scripts to facilitate further investigation and ensure reproducibility?

[ymzhang0319]

Hi @wjc2830, thanks for your interest.

We use the same evaluation tools.
Could you please provide more details in your evaluation settings (semantic weight 1.0, temporal weight 0.2,... in our experiments)?
Then I can try to help you figure it out.

[wjc2830]

Yes, ip_adapter_weight is set to 1.0 and controlnet_conditioning_scale is set to 0.2. I opted not to use a class prompt like "machine gun shooting" because I observed that the results generated without it were superior to those with it. With these settings, I got onset acc: 0.1213, detection acc: 0.1347, detection ap: 0.6893 FID: 47.498411865917966, MKL: 5.17888476451238, KID: [0.046363522010469276-1.8900277153235862e-07] on AVSync15 (1500 samples).
Regarding FID computation, I wanna clarify if the reported FID score represents an average across all 15 classes or no concept of class here.

[ymzhang0319]

Thanks for your information! The evaluation experiments is conducted on the AVSync15 test set (150 samples.) You can refer to the official link of AVSync15.
If you have any other questions, please feel free to contact us.

[wjc2830]

Thank you for the prompt reply. I have re-implemented the evaluation and obtained the following results: FID: 33.87400189673342 MKL: 5.159568889935811 KID: [0.053455384736237746-1.3165596698642787e-07] #onset acc: 0.1007, detection acc: 0.1209, detection ap: 0.6936.
There are still discrepancies in metrics such as MKL and detection accuracy. To ensure consistency, I recommend releasing your evaluation script.

[Gloria2tt]

@wjc2830 Can you provide your test code. I also try to do the same thing you do.