Tips for multi-room scenes

[LaFeuilleMorte]

Hi, I've tried your mvd plus stage 2 model. My dataset was a two room scene connected. I extracted 20 sequential images from the scene and found the results were not so good as in the paper. Here's the confidence map that might be helpful for your debugging:

The first frame:

The last frame:

I found the confidence map in the first 10 frames looks normal, but the confidence map in the last half looks not so good. And the reconstruction point cloud further confirm this conclusion.From the point cloud, The first half 10 frames have good shapes , the last half frames are messy.

[LaFeuilleMorte]

I reverted the order of image sequences. And the results are here:

Looks like the model can construct good results in the first 10 frames, But performs bad on extra frames. And confidence map has similar distribution like the above: First 10 frames normal, last 10 frames bad. I guess the model has only a short term of memory about the scene. So maybe it's better to reconstruct the scene by dividing it into sub-parts and merge them?

[recordmp3]

The model do not process images in some order. So changing the order should not change the result. Let me check is this because of some bugs of demo. can you show your raw data of 20 images? like a link of them?

[LaFeuilleMorte]

livingroom_sparse.zip

Hi, this is my image collection.

[LaFeuilleMorte]

I tried the MVD pretrained weights, the results looks much better. It's weird that mvd are better than mvd++:

[recordmp3]

I changed the strategy to select the reference view and put all images in ascending order and get this.

It looks better now, and we admit that in some case it's still challenging for our method, especially for multiple rooms.

Some tips to improve the quality:

1. Try to select the view that is in the center or that covers more region as the reference view (The current demo use the first view as the reference view, so put the proper view as first view)
2. Try to distribute the reference views of other paths to all around the scene. This is how I heuristically select the other-path reference views in demo.py and it indeed helps:
   
3. Maybe you can design your own view selection heuristics from the confidence map. you need change demo.py. for example: go through MV-dust3r, select the 4 views with least confidence. select them as reference views for another round of MV-dust3r+.
   Good luck!

[recordmp3]

I'll push the view selection code change today.

[LaFeuilleMorte]

Thanks for your quick response. I'll try your new strategy. I also noticed that the camera pose seemed not very accurate. I also tried mast3r-sfm which uses mast3r as the feature matching method and uses colmap as backend to calibrate the images. The point cloud and camera pose are much better and robust (though it's a lot slower), I think the main problem might be the lack of global bundle adjustment. In complex scenes, the error will accumulate, so maybe some global optimization or loop closure should be adopted to eliminate the accumulated error if using in custom datasets.

[recordmp3]

thanks for your suggestion and I believe extra global optimization can improve performance. dust3r and mast3r are better when local align is correct but sometimes are totally ruined due to wrong matching, which is not fixable by global optimization. Our method can put views in the coarse correct position, but yeah, it looks jittering and blurred on edges. If you would like, you can combine both sequentially and get a result better than all while not too slow.