Skip to content

Latest commit

 

History

History
214 lines (189 loc) · 8.75 KB

visualize_smpl.md

File metadata and controls

214 lines (189 loc) · 8.75 KB

Visualize SMPL Mesh

  • fast visualize smpl(x) pose without background images:

    You have smpl pose tensor or array shape of which is (frame, 72)

    from mmhuman3d.core.visualization.visualize_smpl import visualize_smpl_pose
body_model_config = dict(
    type='smpl', model_path=model_path)
visualize_smpl_pose(
    poses=poses,
    output_path='smpl.mp4',
    resolution=(1024, 1024))

    Or you have smplx pose tensor or array shape of which is (frame, 165)

    body_model_config = dict(
    type='smplx', model_path=model_path)
visualize_smpl_pose(
    poses=poses,
    body_model_config=body_model_config,
    output_path='smplx.mp4',
    resolution=(1024, 1024))

    You could also feed dict tensor of smplx definitions. You could check that in visualize_smpl or original smplx.

  • visualize T-pose: If you want to visualize a T-pose smpl or your poses do not have global_orient, you can do:

    import torch
from mmhuman3d.core.visualization.visualize_smpl import visualize_T_pose
body_model_config = dict(
    type='smpl', model_path=model_path)
visualize_T_pose(
    num_frames=100,
    body_model_config=body_model_config,
    output_path='smpl_tpose.mp4',
    orbit_speed=(1, 0.5),
    resolution=(1024, 1024))

  • visualize smpl with predicted VIBE camera: You have poses (numpy/tensor) of shape (frame, 72), betas of shape (frame, 10), pred_cam of shape (10, 4). E.g., we use vibe sample_video.mp4 as an example.

    import pickle
from mmhuman3d.core.visualization.visualize_smpl import visualize_smpl_vibe
with open('vibe_output.pkl', 'rb') as f:
    d = pickle.load(f, encoding='latin1')
poses = d[1]['pose']
orig_cam = d[1]['orig_cam']
pred_cam = d[1]['pred_cam']
bbox = d[1]['bboxes']
gender = 'female'

# pass pred_cam & bbox
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_vibe(
    poses=poses,
    betas=betas,
    body_model_config=body_model_config,
    pred_cam=pred_cam,
    bbox=bbox,
    output_path='vibe_demo.mp4',
    origin_frames='sample_video.mp4',
    resolution=(1024, 1024))

# or pass orig_cam
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_vibe(
    poses=poses,
    betas=betas,
    body_model_config=body_model_config,
    orig_cam=orig_cam,
    output_path='vibe_demo.mp4',
    origin_frames='sample_video.mp4',
    resolution=(1024, 1024))

  • visualize smpl with predicted HMR/SPIN camera: You have poses (numpy/tensor) of shape (frame, 72), betas of shape (frame, 10), cam_translation of shape (10, 4). E.g., we use vibe sample_video.mp4 as an example.

    import pickle
from mmhuman3d.core.visualization.visualize_smpl import visualize_smpl_hmr
gender = 'female'
focal_length = 5000
det_width = 224
det_height = 224

# you can pass smpl poses & betas & gender
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_hmr(
    poses=poses,
    betas=betas,
    bbox=bbox,
    body_model_config=body_model_config,
    focal_length=focal_length,
    det_width=det_width,
    det_height=det_height,
    T=cam_translation,
    output_path='hmr_demo.mp4',
    origin_frames=image_folder,
    resolution=(1024, 1024))

# or you can pass verts
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_hmr(
    verts=verts,
    bbox=bbox,
    focal_length=focal_length,
    body_model_config=body_model_config,
    det_width=det_width,
    det_height=det_height,
    T=cam_translation,
    output_path='hmr_demo.mp4',
    origin_frames=image_folder,
    resolution=(1024, 1024))

# you can also pass kp2d in replace of bbox.
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_hmr(
    verts=verts,
    body_model_config=body_model_config,
    kp2d=kp2d,
    focal_length=focal_length,
    det_width=det_width,
    det_height=det_height,
    T=cam_translation,
    output_path='hmr_demo.mp4',
    origin_frames=image_folder,
    resolution=(1024, 1024))

  • visualize smpl with opencv camera: You should pass the opencv defined intrinsic matrix K and extrinsic matrix R, T.

    from mmhuman3d.core.visualization.visualize_smpl import visualize_smpl_calibration
body_model_config = dict(
    type='smpl', model_path=model_path, gender=gender)
visualize_smpl_calibration(
    poses=poses,
    betas=betas,
    transl=transl,
    body_model_config=body_model_config,
    K=K,
    R=R,
    T=T,
    output_path='opencv.mp4',
    origin_frames='bg_video.mp4',
    resolution=(1024, 1024))

Different render_choice:

  • visualize mesh: This is independent of cameras and you could directly set render_choice as hq(high quality), mq(medium quality) or lq(low quality).

  • visualize binary silhouettes: This is independent of cameras and you could directly set render_choice as silhouette. The output video/images will be binary masks.

  • visualize body part silhouette: This is independent of cameras and you could directly set render_choice as part_silhouette. The output video/images will be body part segmentation masks.

  • visualize depth map: This is independent of cameras and you could directly set render_choice as depth. The output video/images will be gray depth maps.

  • visualize normal map: This is independent of cameras and you could directly set render_choice as normal. The output video/images will be colorful normal maps.

  • visualize point clouds: This is independent of cameras and you could directly set render_choice as pointcloud. The output video/images will be point clouds with keypoints.

  • Choose your color: Set palette as 'white', 'black', 'blue', 'green', 'red', 'yellow', and pass a list of string with the length of num_person. Or send a numpy.ndarray of shape (num_person, 3). Should be normalized color: (1.0, 1.0, 1.0) represents white. The color channel is RGB.

  • Differentiable render: Set no_grad=False and return_tensor=True.

Important parameters:

  • background images: You could pass image_array(numpy.ndarray of shape (frame, h, w, 3)) or frame_list(list of paths of images(.png or .jpg)) or origin_frames(str of video path or image folder path). The priority order is image_array > frame_list > origin_frames. If the background images are too big, you should set read_frames_batch as True to relieve the IO burden. This will be done automatically in the code when you number of frame is large than 500.

  • smpl pose & verts: There area two ways to pass smpl mesh information: 1). You pass poses, betas(optional) and transl(optional) and gender(optional). 2). You pass verts directly and the above three will be ignored. The body_model or model_path is still required if you passverts since we need to get the faces. The priority order is verts > (poses & betas & transl & gender). Check the docstring for details. 3). for multi-person, you should have an extra dim for num_person. E.g., shape of smpl verts should be (num_frame, num_person, 6890, 3), shape of smpl poses should be (num_frame, num_person, 72), shape of smpl betas should be (num_frame, num_person, 10), shape of vibe pred_cam should be (num_frame, num_person, 3). This doesn't have influence on K, R, T since they are for every frame.

  • body model: There are two ways to pass body model: 1). You pass a dict body_model_config which containing the same configs as build_body_model 2). You pass body_model directly and the above three will be ignored. The priority order is body_model > (model_path & model_type & gender). Check the docstring for details.

  • output path: Output_path could be None or str of video path or str of image folder path. 1). If None, no output file will be wrote. 2). If a video path like xxx.mp4, a video file will be wrote. Make sure you have enough space for temporal images. The images will be removed automatically. 3). If a image folder path like xxx/, a folder will be created and the images will be wrote into it.