Just to add.
My main line of work is in 3D animation. I use flat 2D images projected on to a 3D object all the time to give the perspective of 3D for that object.
Example: I want to create a 3D model of the Earth. I want to use one of the famously used NASA images of the Earth taken by the Apollo mission. I would create a 3D sphere then planar map the flat 2D image of the Earth on to the 3D object. Voila!! A 3D planet Earth. (After adding atmosphere glow etc..) I can lock together 2 cameras in my 3d software or use an available plugin that will render out 3D image (for use with 3d glasses). I would most likely render out 2 separate camera views then edit in post.
The Depth map will create your 3D modeling, the 2D colour video image will be your paint to the 3D model. What then comes from this is then up to you, either 2D or 3D stereoscopic images. The single perspective is destroyed when outputting to 3D as the software will split itself out from the centerline with 1 camera for left and 1 for right.
I think a Youtube video may be necessary for people to visualise my thinking. You may think I'm way off on another planet! I'm not, I'm just forward thinking. 2 cameras, or 1 camera with 2 lenses just seem a simple way out to create 3D, its not my thinking. 2 camera filming is not something new. Its like the internal combustion engine, because it works doesn't been we have to stick to it. Lets think out of the box and invent!!