About the floaters

[olicy666]

Thank you for this great work. While I was working on it, I noticed that there are many floaters in the reconstructed scene. Could you please explain what causes these floaters and if there are any ways to resolve them?

[brandonsmartdev]

Hello @olicy666,

These floaters are certainly one of the key challenges with Splatt3R at the moment. When there is large distance between adjacent pixels in one of the input images (i.e. one is in the foreground and the other is in the background), the network has trouble predicting this discontinuous jump, and instead places several Gaussians between the foreground and background. We inherit this issue from MASt3R, but it is a common problem with monocular depth prediction models (eg. see some of the visualisations here).

There are a few potential causes of this during training, such as the upscaling layers used in the network, but I think the core issue is poor quality depth from 3D datasets. Focusing more on synthetic data, such as in Depth Pro might provide better results if the network were to be retrained.

If you don't want to retrain the network, you may need to use a heuristic to remove the floaters, such as by removing Gaussians which are located far from their neighbours.

If you find success with this, please let us know, as this is one of the limitations of the current model.