Capture Depth Image without opening Visualizer in Open3D?

Question

I'm looking to batch process 50,000+ 3D models. I need to capture depth maps from different angles. Using Open3D, How can I capture this information without launching the visualizer? In hopes of speeding up the process?

Answer 1

You could check Open3D OffScreen

It will render depth map or images without window, in other words, it can run batches in background.

Here are the example.

import copy
from cv2 import cv2
import numpy as np
import open3d as o3d
import matplotlib.pyplot as plt

img_width, img_height = (1920, 1080)
pcd = o3d.io.read_triangle_mesh('bunny.ply')

mat = o3d.visualization.rendering.Material()
mat.shader = 'defaultUnlit'

renderer_pc = o3d.visualization.rendering.OffscreenRenderer(img_width, img_height)
renderer_pc.scene.set_background(np.array([0, 0, 0, 0]))
renderer_pc.scene.add_geometry("pcd", pcd, mat)

# Optionally set the camera field of view (to zoom in a bit)
vertical_field_of_view = 15.0  # between 5 and 90 degrees
aspect_ratio = img_width / img_height  # azimuth over elevation
near_plane = 0.1
far_plane = 50.0
fov_type = o3d.visualization.rendering.Camera.FovType.Vertical
renderer_pc.scene.camera.set_projection(vertical_field_of_view, aspect_ratio, near_plane, far_plane, fov_type)

# Look at the origin from the front (along the -Z direction, into the screen), with Y as Up.
center = [0, 0, 0]  # look_at target
eye = [0, 0, 2]  # camera position
up = [0, 1, 0]  # camera orientation
renderer_pc.scene.camera.look_at(center, eye, up)

depth_image = np.asarray(renderer_pc.render_to_depth_image())
np.save('depth', depth_image)

normalized_image = (depth_image - depth_image.min()) / (depth_image.max() - depth_image.min())
plt.imshow(normalized_image)
plt.savefig('depth.png')

Answer 2

o3d.visualization.rendering.OffscreenRenderer is not currently working in Open3D versions 0.16.0 and 0.17.0, but works on version 0.15.1. There's an open issue here.

An alternative would be to build Open3D headless: http://www.open3d.org/docs/latest/tutorial/Advanced/headless_rendering.html