Visual Effects
-- Using nerfies you can create fun visual effects. This Dolly zoom effect - would be impossible without nerfies since it would require going through a wall. -
- -
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Abstract
+
+
+ + The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, have been quickly ex- panding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. Websites are designed to be rendered visually for easy consumption by humans. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. +
++ We evaluate on the recent MIND2WEB benchmark. In addition to offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents—it can successfully complete 50% of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out not effective for web agents, and the best grounding strategy leverages both the HTML text and visuals, yet there is still a substantial gap with oracle grounding. +
+
@@ -243,158 +277,104 @@
-
-
-
- Abstract
-
-
- We present the first method capable of photorealistically reconstructing a non-rigidly
- deforming scene using photos/videos captured casually from mobile phones.
-
-
- Our approach augments neural radiance fields
- (NeRF) by optimizing an
- additional continuous volumetric deformation field that warps each observed point into a
- canonical 5D NeRF.
- We observe that these NeRF-like deformation fields are prone to local minima, and
- propose a coarse-to-fine optimization method for coordinate-based models that allows for
- more robust optimization.
- By adapting principles from geometry processing and physical simulation to NeRF-like
- models, we propose an elastic regularization of the deformation field that further
- improves robustness.
-
-
- We show that Nerfies can turn casually captured selfie
- photos/videos into deformable NeRF
- models that allow for photorealistic renderings of the subject from arbitrary
- viewpoints, which we dub "nerfies". We evaluate our method by collecting data
- using a
- rig with two mobile phones that take time-synchronized photos, yielding train/validation
- images of the same pose at different viewpoints. We show that our method faithfully
- reconstructs non-rigidly deforming scenes and reproduces unseen views with high
- fidelity.
-
-
-
-
-
-
-
-
-
- Video
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Matting
-
-
-
- As a byproduct of our method, we can also solve the matting problem by ignoring
- samples that fall outside of a bounding box during rendering.
-
-
-
-
-
-
-
-
-
-
-
- Animation
-
-
- Interpolating states
-
-
- We can also animate the scene by interpolating the deformation latent codes of two input
- frames. Use the slider here to linearly interpolate between the left frame and the right
- frame.
-
-
-
-
- 
- Start Frame
-
-
-
- Loading...
-
-
-
-
- 
- End Frame
-
-
-
-
-
-
- Re-rendering the input video
-
-
- Using Nerfies, you can re-render a video from a novel
- viewpoint such as a stabilized camera by playing back the training deformations.
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
-
-
-
-
-
-
-
-
- Abstract
-
-
- - We present the first method capable of photorealistically reconstructing a non-rigidly - deforming scene using photos/videos captured casually from mobile phones. -
-- Our approach augments neural radiance fields - (NeRF) by optimizing an - additional continuous volumetric deformation field that warps each observed point into a - canonical 5D NeRF. - We observe that these NeRF-like deformation fields are prone to local minima, and - propose a coarse-to-fine optimization method for coordinate-based models that allows for - more robust optimization. - By adapting principles from geometry processing and physical simulation to NeRF-like - models, we propose an elastic regularization of the deformation field that further - improves robustness. -
-- We show that Nerfies can turn casually captured selfie - photos/videos into deformable NeRF - models that allow for photorealistic renderings of the subject from arbitrary - viewpoints, which we dub "nerfies". We evaluate our method by collecting data - using a - rig with two mobile phones that take time-synchronized photos, yielding train/validation - images of the same pose at different viewpoints. We show that our method faithfully - reconstructs non-rigidly deforming scenes and reproduces unseen views with high - fidelity. -
-
-
-
-
-
- Video
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Matting
-
-
-
-
- - As a byproduct of our method, we can also solve the matting problem by ignoring - samples that fall outside of a bounding box during rendering. -
- -
-
-
- - - -
Animation
- - -Interpolating states
-
-
- - We can also animate the scene by interpolating the deformation latent codes of two input - frames. Use the slider here to linearly interpolate between the left frame and the right - frame. -
-
-
-
-
-
-
- Start Frame
-
-
-
- Loading...
-
-
-
-
-
-
- End Frame
-- - - -
Re-rendering the input video
-
-
- - Using Nerfies, you can re-render a video from a novel - viewpoint such as a stabilized camera by playing back the training deformations. -
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+