ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion

Zitian Zhang, Frédéric Fortier-Chouinard, Mathieu Garon, Anand Bhattad, Jean-François Lalonde

WACV 2025

[Website] [Paper] [Supplementary]

Environments

1. Clone the repo and submodules

git clone --recursive https://github.com/zzt76/zerocomp.git

Make sure you also clone the submodule predictors.

2. Install required wheels, note that different intrinsic predictors may require different libraries.

pip install -r requirements.txt

3. Download Stable Diffussion 2.1 from huggingface, and modify the pretrained_model_name_or_path variable in configs/sg_labo.yaml.

Pretrained weights

Pretrained weights are only available for non-commercial use under CC BY-NC-SA 4.0 license.

Depth, normals, albedo Openrooms 7days: link

Depth, normals, albedo, roughness, metallic Interior Verse 2days: link Interior Verse 7days: link

Predictors

You can get the intrinsic predictor weights from the original repos, the links are provided in the following. After downloading, move them to .cache/checkpoints folder.

Depth

ZoeDepth: ZoeD_M12_NK.pt DepthAnything: depth_anything_metric_depth_indoor.pt DepthAnythingV2: depth_anything_v2_vitl.pth

Normals

OmniDataV2: omnidata_dpt_normal_v2.ckpt StableNormal: stable-normal-v0-1

Materials

For diffuse only, you can use our own not that good model dfnet. For diffuse, roughness and metallic, you can precompute these maps by IntrinsicImageDiffusion, RGB<->X or other predictors. Name them as in the provided test dataset and load them by changing predictor_names to precompute.

Custom predictors

You can use other predictors you prefer by modifying controlnet_input_handle.py. Implement handle_*** functions and modify ToPredictors class.

All predictors are subject to their own licenses. Please check the relating conditions carefully.

ZeroComp Test dataset

You can download the ZeroComp test dataset here.

Inference

To run the evaluations as in the paper:

python eval_controlnet_composite.py --config-name sg_labo

Live demo

ZeroComp trained on Openrooms:

python gradio_composite.py

ZeroComp trained on InteriorVerse, with roughness and metallic:

python gradio_composite_w_rm.py

Acknowledgements

This research was supported by NSERC grants RGPIN 2020-04799 and ALLRP 586543-23, Mitacs and Depix. Computing resources were provided by the Digital Research Alliance of Canada. We also thank Louis-Étienne Messier and Justine Giroux for their help as well as all members of the lab for discussions and proofreading help.

This implementation builds upon Hugging Face’s Diffusers library. We also acknowledge Gradio for providing a developer-friendly tool to create the interative demos for our models.

BibTex

If you find it useful, please consider citing ZeroComp:

@article{zhang2024zerocomp,
  title={Zerocomp: Zero-shot object compositing from image intrinsics via diffusion},
  author={Zhang, Zitian and Fortier-Chouinard, Fr{\'e}d{\'e}ric and Garon, Mathieu and Bhattad, Anand and Lalonde, Jean-Fran{\c{c}}ois},
  journal={arXiv preprint arXiv:2410.08168},
  year={2024}
}

License

The codes, pretrained weights and test dataset are all for non-commercial use only.

ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion by Zitian Zhang, Frédéric Fortier-Chouinard, Mathieu Garon, Anand Bhattad, Jean-François Lalonde is licensed under CC BY-NC-SA 4.0