3d point reconstruction from depth map (with camera parameters)

Question

General introduction:

I have images A and B taken with the same camera (of which i know the internal parameters K). I then proceded to find the F matrix and from that the essential matrix. From that i recovered the complete camera matrices P and P'.

I then rectified the two images and computed a depth map, like the following one:

My question

Now i'd like to perform a dense 3d reconstruction using the depth map. The thing that is not really clear is what camera matrix should i use to reproject the points, P or P'?

Or yet another transform? P and P' were the camera matrices for A and B but now i'm considering a depth map created by using the rectified versions of A and B

Answer 1

When you rectify the images, you define new intrinsics for both. From that you get a homogeneous 4x4 matrix Q, which relates the image coordinates and disparity [x y d] to the world coordinates [X Y Z]. The details are in the "Learning OpenCV" book.