Depthmaps are typically greyscale images that represent how far an object is by the degree of shade.  They can be white-near, black-near, or some other color scale such as ironbar.  Depthmaps are useful for creating renders or systems that have a distance sensing need.

In this post I’ll be using footage that I shot using 2 Innovv cameras.  The rig was not precise, so I also post processed it to correct distortion and alignment issues.  The post processing topic was covered in last weeks post.

What we’ll do is take a segment from the left and right cameras and use that for our depthmapping.  Here are the commands to get the segments.

ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Right-Defish-Rotate-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV"
ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Left-Defish-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV"

The OpenCV script is based off of the work demonstrated here, http://docs.opencv.org/3.1.0/dd/d53/tutorial_py_depthmap.html

The python script only works with single images.  I have modified it so it will process a sequence of image files that have incremental numbering in the file name.  First I’ll convert our video segments to images using this command.

ffmpeg -i "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Right/Right_Seg%d.png"
ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Left/Left_Seg%d.png"

Now that I have the images, the python script can be run to process the images.  Here is the python code.

==== DepthMap.py

import numpy as np
import cv2
from matplotlib import pyplot as plt

count = 0
Rimg = '/home/local/Desktop/DepthMap/Right/Right_Seg'
Limg = '/home/local/Desktop/DepthMap/Left/Left_Seg'
Dimg = '/home/local/Desktop/DepthMap/Depth/Depth_Seg'
Ext = '.png'

# The values in range(x,y,z) are x=start, y=end, z=step
for N in range(1,453,1):
count = count + 1

imgR = cv2.imread(Rimg + str(count + 4) + Ext, 0) # Frames not alligned, this compinsates
imgL = cv2.imread(Limg + str(count) + Ext, 0)

stereo = cv2.StereoBM(1, 16, 15)
disparity = stereo.compute(imgL,imgR)

plt.imshow(disparity,'gray')
plt.savefig(Dimg + str(count) + Ext)
====

One thing that I noticed with my version was that the python script tanked after 142 images.  To continue on, I changed the value for the count variable to 142 and re-ran the python script.  Eventually it finished.  I just want to make that clear for anyone reproduction the effort.

This python script will create the file list needed for the image to video conversion.

==== FileList.py

count = 0
Dimg = 'file \'/home/local/Desktop/DepthMap/Depth/Depth_Seg'
Ext = '.png\''
text_file = open("/home/local/Desktop/DepthMap/Depth/Output.txt", "w")
# The values in range(x,y,z) are x=start, y=end, z=step
for N in range(1,453,1):
count = count + 1
text_file.write (Dimg + str(count) + Ext)
text_file.write ('\n')
text_file.close()
====

Now we can convert the image sequence into a video using this command.

ffmpeg -y -r 30 -f concat -safe 0 -i "/home/local/Desktop/DepthMap/Depth/Output.txt" -c:v libx264 -vf "fps=30,format=yuv420p" "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov"

The OpenCV process introduced a scale background which I would like to remove.  After finding the cropping locations using GIMP, I ran this command.

ffmpeg -i "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov" -filter:v "crop=621:351:100:125" -c:a copy "/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov"

To give a better presentation I decided to inlay the depthmap on top of the original left camera footage.  I desaturated the original video to draw more attention to the depthmap inlay.

ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf "eq=saturation=0" "/home/local/Desktop/DepthMap/De-Saturate.mov"

Then I inserted the depthmap as a picture in picture.

ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate.mov" -vf "movie=/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov, scale=621:351 [vid2]; [in][vid2] overlay=main_w-overlay_w-20:main_h-overlay_h-20" "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov"

Finally, I drew a red boarder around the depthmap.

ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov" -vf drawbox=x=1197:y=663:w=621:h=351:color=red@1 "/home/local/Desktop/DepthMap/De-Saturate_PicNPic_Final.mov"

That completes the process for creating depthmaps using FFMpeg and OpenCV.  The results are far from ideal.  This is mainly a proof of concept using existing footage.  The efforts of creating depthmaps are still maturing.  One effort by  University College London is producing some striking results.