Depthmaps are typically greyscale images that represent how far an object is by the degree of shade. They can be white-near, black-near, or some other color scale such as ironbar. Depthmaps are useful for creating renders or systems that have a distance sensing need.
In this post I’ll be using footage that I shot using 2 Innovv cameras. The rig was not precise, so I also post processed it to correct distortion and alignment issues. The post processing topic was covered in last weeks post.
What we’ll do is take a segment from the left and right cameras and use that for our depthmapping. Here are the commands to get the segments.
ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Right-Defish-Rotate-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV" ffmpeg -ss 00:00:00 -i "/home/local/Desktop/STP 2017/Anaglyph/L0000004-Left-Defish-Trim.MOV" -t 00:00:15 -acodec copy -vcodec copy -async 1 "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV"
The OpenCV script is based off of the work demonstrated here, http://docs.opencv.org/3.1.0/dd/d53/tutorial_py_depthmap.html
The python script only works with single images. I have modified it so it will process a sequence of image files that have incremental numbering in the file name. First I’ll convert our video segments to images using this command.
ffmpeg -i "/home/local/Desktop/DepthMap/Right/Right_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Right/Right_Seg%d.png" ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf fps=30 "/home/local/Desktop/DepthMap/Left/Left_Seg%d.png"
Now that I have the images, the python script can be run to process the images. Here is the python code.
==== DepthMap.py import numpy as np import cv2 from matplotlib import pyplot as plt count = 0 Rimg = '/home/local/Desktop/DepthMap/Right/Right_Seg' Limg = '/home/local/Desktop/DepthMap/Left/Left_Seg' Dimg = '/home/local/Desktop/DepthMap/Depth/Depth_Seg' Ext = '.png' # The values in range(x,y,z) are x=start, y=end, z=step for N in range(1,453,1): count = count + 1 imgR = cv2.imread(Rimg + str(count + 4) + Ext, 0) # Frames not alligned, this compinsates imgL = cv2.imread(Limg + str(count) + Ext, 0) stereo = cv2.StereoBM(1, 16, 15) disparity = stereo.compute(imgL,imgR) plt.imshow(disparity,'gray') plt.savefig(Dimg + str(count) + Ext) ====
One thing that I noticed with my version was that the python script tanked after 142 images. To continue on, I changed the value for the count variable to 142 and re-ran the python script. Eventually it finished. I just want to make that clear for anyone reproduction the effort.
This python script will create the file list needed for the image to video conversion.
==== FileList.py count = 0 Dimg = 'file \'/home/local/Desktop/DepthMap/Depth/Depth_Seg' Ext = '.png\'' text_file = open("/home/local/Desktop/DepthMap/Depth/Output.txt", "w") # The values in range(x,y,z) are x=start, y=end, z=step for N in range(1,453,1): count = count + 1 text_file.write (Dimg + str(count) + Ext) text_file.write ('\n') text_file.close() ====
Now we can convert the image sequence into a video using this command.
ffmpeg -y -r 30 -f concat -safe 0 -i "/home/local/Desktop/DepthMap/Depth/Output.txt" -c:v libx264 -vf "fps=30,format=yuv420p" "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov"
The OpenCV process introduced a scale background which I would like to remove. After finding the cropping locations using GIMP, I ran this command.
ffmpeg -i "/home/local/Desktop/DepthMap/Depth/Depth_Seg.mov" -filter:v "crop=621:351:100:125" -c:a copy "/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov"
To give a better presentation I decided to inlay the depthmap on top of the original left camera footage. I desaturated the original video to draw more attention to the depthmap inlay.
ffmpeg -i "/home/local/Desktop/DepthMap/Left/Left_Seg.MOV" -vf "eq=saturation=0" "/home/local/Desktop/DepthMap/De-Saturate.mov"
Then I inserted the depthmap as a picture in picture.
ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate.mov" -vf "movie=/home/local/Desktop/DepthMap/Depth_Seg_Crop.mov, scale=621:351 [vid2]; [in][vid2] overlay=main_w-overlay_w-20:main_h-overlay_h-20" "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov"
Finally, I drew a red boarder around the depthmap.
ffmpeg -i "/home/local/Desktop/DepthMap/De-Saturate_PicNPic.mov" -vf drawbox=x=1197:y=663:w=621:h=351:color=red@1 "/home/local/Desktop/DepthMap/De-Saturate_PicNPic_Final.mov"
That completes the process for creating depthmaps using FFMpeg and OpenCV. The results are far from ideal. This is mainly a proof of concept using existing footage. The efforts of creating depthmaps are still maturing. One effort by University College London is producing some striking results.