Any plan for releasing Modeling experiments codes?

[Wuziyi616]

Hi, thanks for opening source this great work! I'm currently working on a CSWM-like object-centric model and want to test its performance on the Physion dataset. It would be very great if you can release the code of training/testing CSWM on this dataset, so that I can easily integrate my model to the train/test protocols here. So I'm just wondering is there any plan for releasing any of the baseline models benchmarked in your paper? Thanks!

EDIT: okay so I found the code for particle models #45. But they're quite different from e.g. object-centric models because particle models are directly evaluated by calculating the min distance between patient/agent points. They don't involve the readout phase, while my question is mainly regarding the readout phase...

[Wuziyi616]

Also, if I want to train/test my model, which subset of the dataset do I need to download? I assume:

• PhysionTest-Core for testing. Specifically, mp4s-redyellow is for testing the models right?
• PhysionTrain-Dynamics for training. And here is what I'm confused about, there is also readout-related data (e.g. Collide_readout_MP4s) in the unzipped files, are they the data we use to train the linear readout models? If so, what is PhysionTrain-Readout and those links in the Table? Since they're HDF5 I assume they're additional visual inputs like segmentation masks? If that's true then I don't need to download them.
• Finally, where is the labels for training the readout model? I.e. labels of whether agent and patient contacts.

Thanks again!

[Wuziyi616]

Finally I have a question regarding the experimental setting. If the model doesn’t see objects in red/yellow at training time, but at test time the agent and patient are colored in red/yellow, wouldn’t this affect the model performance? From a domain shift perspective, it’s difficult for object-centric models to detect these out-of-distribution objects during test time, as proved by this ICML’22 paper Section 4.3.

[Wuziyi616]

BTW I find some strange data in the test set. For Roll videos with ledge in the names, I don't think any of them end up with contacts, see the three attached videos. However the labels for them are True, False, True. The names of these videos are:

• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0000_img.mp4
• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0001_img.mp4
• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0002_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0000_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0001_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0002_img.mp4

[felixbinder]

BTW I find some strange data in the test set. For Roll videos with ledge in the names, I don't think any of them end up with contacts, see the three attached videos. However the labels for them are True, False, True. The names of these videos are:

• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0000_img.mp4
• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0001_img.mp4
• pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0002_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0000_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0001_img.mp4

pilot_it2_rollingSliding_simple_ledge_box_1_dis_1_occ-redyellow_0002_img.mp4

Thanks a lot for pointing these out! These indeed do look like mistakes that the physics engine has made. We have conducted a manual audit of the dataset, the list of excluded stimuli can be found here. We've missed those stimuli during our audit, but I have added the stimuli to the list of exclusions.

[Wuziyi616]

@felixbinder thanks for the prompt reply. These three videos are just some samples I pick up from the readout set. I'm not sure if there are other videos we need to exclude... Will you also go over other videos and do a thorough update of the exclusion list?