{"id":4003,"date":"2022-04-29T19:00:14","date_gmt":"2022-04-30T02:00:14","guid":{"rendered":"https:\/\/www.cloudacm.com\/?p=4003"},"modified":"2022-04-29T19:17:12","modified_gmt":"2022-04-30T02:17:12","slug":"out-of-the-depth-and-disparity","status":"publish","type":"post","link":"https:\/\/www.cloudacm.com\/?p=4003","title":{"rendered":"Out of the depth and disparity"},"content":{"rendered":"<p>This post will cover the concepts of image processing to generate and display image disparity or depth of field. This topic has been covered in earlier posts in regards to anaglyph imaging and OpenCV disparity mapping. The purpose of this post is to expand on this topic by demonstrating some features and limitations. Let&#8217;s start by leaving flatland.<\/p>\n<p>Flatland &amp; the 4th Dimension &#8211; Carl Sagan<br \/>\n<iframe loading=\"lazy\" title=\"Flatland &amp; the 4th Dimension - Carl Sagan\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/iiWKq57uAlk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>This image is of a stereo card from the Missouri History Museum that is dated between 1914 and 1918 which depicts a wasteland as a result from World War 1. It is available as a free media repository from Wikimedia Commons, <a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:%22Desolate_Waste_on_Chemin_des_Dames_Battlefield,_France.%22.jpg\">https:\/\/commons.wikimedia.org\/wiki\/File:%22Desolate_Waste_on_Chemin_des_Dames_Battlefield,_France.%22.jpg<\/a><\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-left.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-4013 size-medium aligncenter\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-left-291x300.png\" alt=\"\" width=\"291\" height=\"300\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-left-291x300.png 291w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-left-262x270.png 262w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-left.png 448w\" sizes=\"auto, (max-width: 291px) 100vw, 291px\" \/><\/a><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-right.png\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-4014 size-medium aligncenter\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-right-291x300.png\" alt=\"\" width=\"291\" height=\"300\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-right-291x300.png 291w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-right-262x270.png 262w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/resized_v18834-right.png 448w\" sizes=\"auto, (max-width: 291px) 100vw, 291px\" \/><\/a><\/p>\n<p>Stereo cards were a common technique during early photography to give the viewer the illusion of depth. They peaked in popularity between 1902 and 1935. The University of Washington contains a digital collection of these artifacts, <a href=\"https:\/\/content.lib.washington.edu\/stereoweb\/index.html\">https:\/\/content.lib.washington.edu\/stereoweb\/index.html<\/a><\/p>\n<p>I used a different method for processing images into depth with OpenCV from the example image above. First, the stereo card was split into 2 separate images and tonal qualities were applied which gave the best contrast. The following script requires that both images have matching dimensions.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">import numpy as np\r\nimport cv2 as cv\r\nfrom matplotlib import pyplot as plt\r\n\r\nimageLeft = cv.imread('resized_v18834-right.png', 0)\r\nimageRight = cv.imread('resized_v18834-left.png', 0)\r\n\r\n# Using StereoSGBM\r\n# Set disparity parameters. Note: disparity range is tuned according to\r\n#  specific parameters obtained through trial and error.\r\nwin_size = 2\r\nmin_disp = -4\r\nmax_disp = 9\r\n# num_disp = max_disp - min_disp  # Needs to be divisible by 16\r\nnum_disp = 32  # Needs to be divisible by 16\r\nstereo = cv.StereoSGBM_create(\r\n    minDisparity=min_disp,\r\n    numDisparities=num_disp,\r\n    blockSize=7,\r\n    uniquenessRatio=5,\r\n    speckleWindowSize=5,\r\n    speckleRange=5,\r\n    disp12MaxDiff=2,\r\n    P1=8 * 3 * win_size ** 2,\r\n    P2=32 * 3 * win_size ** 2,\r\n)\r\ndisparity_SGBM = stereo.compute(imageLeft, imageRight)\r\ncmap_reversed = plt.cm.get_cmap('gray_r')\r\nplt.imshow(disparity_SGBM, cmap_reversed)\r\nplt.colorbar()\r\n\r\nplt.savefig(\"ver11_v18834-depth.png\")<\/pre>\n<p>This created the following grayscale depth map image.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4016 size-full\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth.png\" alt=\"\" width=\"640\" height=\"480\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth.png 640w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth-300x225.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth-360x270.png 360w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/a><\/p>\n<p>I then manually edited the image in an attempt to reduce holes, pitting, and other noise while adding more contrast to the overall image, here is the result.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth_edited.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4018\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth_edited.png\" alt=\"\" width=\"334\" height=\"368\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth_edited.png 334w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth_edited-272x300.png 272w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/ver11_v18834-depth_edited-245x270.png 245w\" sizes=\"auto, (max-width: 334px) 100vw, 334px\" \/><\/a><\/p>\n<p>Some images that lack a secondary viewpoint can have a depth map layer manually created using GIMP. This technique is demonstrated in this video.<\/p>\n<p><iframe loading=\"lazy\" title=\"GIMP - Change the Perspective of the Image using Depth Map | Photoshop Alternative | #94\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/nmb-FPLDUtM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>I applied a similar technique with this image. The rgb image is on the left while the grayscale depth map image is on the right.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4020 size-full\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide.png\" alt=\"\" width=\"908\" height=\"604\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide.png 908w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide-300x200.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide-768x511.png 768w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Depth_SideBySide-406x270.png 406w\" sizes=\"auto, (max-width: 908px) 100vw, 908px\" \/><\/a><\/p>\n<p>Using GIMP, I generated a series of images by changing the amount of map and displace values. This group of images was then generated into a video with FFMPEG to give a sense of depth and motion.<\/p>\n<p><iframe loading=\"lazy\" title=\"IMG_1675_Motion\" src=\"https:\/\/player.vimeo.com\/video\/704572987?dnt=1&amp;app_id=122963\" width=\"604\" height=\"806\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture; clipboard-write\"><\/iframe><\/p>\n<p>The grayscale depth map image was further processed to create a false color image with the inferno scale using this script.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">import matplotlib.pyplot as plt\r\nimport numpy as np\r\nimport cv2\r\n\r\nimage = cv2.imread('IMG_1675_Depth_Blurred.png', 0)\r\ncolormap = plt.get_cmap('inferno')\r\nheatmap = (colormap(image) * 2**16).astype(np.uint16)[:,:,:3]\r\nheatmap = cv2.cvtColor(heatmap, cv2.COLOR_RGB2BGR)\r\ncv2.imwrite('IMG_1675_Color_Depth_Blurred.png',heatmap)<\/pre>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Color_Depth_Scaled.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-4023 size-full\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Color_Depth_Scaled.png\" alt=\"\" width=\"454\" height=\"605\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Color_Depth_Scaled.png 454w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Color_Depth_Scaled-225x300.png 225w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/IMG_1675_Color_Depth_Scaled-203x270.png 203w\" sizes=\"auto, (max-width: 454px) 100vw, 454px\" \/><\/a><\/p>\n<p>I also applied this technique to panoramic images to generate an VR perspective using Google Street View by importing the resulting images. Here is an example video of rgb, grayscale depth, and false color depth.<\/p>\n<p><iframe loading=\"lazy\" title=\"Depth_VR\" src=\"https:\/\/player.vimeo.com\/video\/704578463?dnt=1&amp;app_id=122963\" width=\"540\" height=\"960\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture; clipboard-write\"><\/iframe><\/p>\n<p>Here is the 360 degree false color depth map.<\/p>\n<p>[sphere 4028]<\/p>\n<p>Capturing images with a monocular camera presented challenges. There are a number of factors that will influence the final quality of the render, some of them being lens distortion, plane alignment, parallax bend, focus, and any image correction that might occur automatically. The following images were captured with an iPhone app call CrossCam, <a href=\"https:\/\/apps.apple.com\/us\/app\/crosscam\/id1436262905\">https:\/\/apps.apple.com\/us\/app\/crosscam\/id1436262905<\/a>. Unfortunately, as of this writing Apple has recently announced that they will be removing apps that have not received updates longer than 2 years, so I can&#8217;t say if Kevin Anderson&#8217;s work will still be available. Here&#8217;s a link to the story.<\/p>\n<p><a href=\"https:\/\/www.theverge.com\/2022\/4\/23\/23038870\/apple-app-store-widely-remove-outdated-apps-developers\">https:\/\/www.theverge.com\/2022\/4\/23\/23038870\/apple-app-store-widely-remove-outdated-apps-developers<\/a><\/p>\n<p>At any rate, the following image depth map was created which resulted in a large amount of noise.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-4031\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide.png\" alt=\"\" width=\"972\" height=\"405\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide.png 972w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide-300x125.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide-768x320.png 768w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2022\/04\/Woody_Depth_SideBySide-604x252.png 604w\" sizes=\"auto, (max-width: 972px) 100vw, 972px\" \/><\/a><\/p>\n<p>This paper details much of the requirements facing accurate depth map generation from stereo image sets.\u00a0 These same topics were pointed out in this post, <a href=\"https:\/\/stackoverflow.com\/questions\/36172913\/opencv-depth-map-from-uncalibrated-stereo-system\">https:\/\/stackoverflow.com\/questions\/36172913\/opencv-depth-map-from-uncalibrated-stereo-system<\/a><\/p>\n<p><a href=\"https:\/\/images.autodesk.com\/latin_am_main\/files\/stereoscopic_whitepaper_final08.pdf\">https:\/\/images.autodesk.com\/latin_am_main\/files\/stereoscopic_whitepaper_final08.pdf<\/a><\/p>\n<p>Taking all of these into account, it is entirely possible to use inexpensive ESP32-Cam modules to generate depth map images, provided they are aligned and set correctly.<\/p>\n<p>This process performs depth map estimation using MIDaS, details about MIDAS can be found here, <a href=\"https:\/\/pytorch.org\/hub\/intelisl_midas_v2\">https:\/\/pytorch.org\/hub\/intelisl_midas_v2<\/a><\/p>\n<p><iframe loading=\"lazy\" title=\"3d photo inpainting using AI in Google Colab\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/uRtor5E-jng?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>There is much more that can be covered on this topic which is beyond digestible in one sitting. Pun intended, this topic is deep.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post will cover the concepts of image processing to generate and display image disparity or depth of field. This topic has been covered in earlier posts in regards to anaglyph imaging and OpenCV disparity mapping. The purpose of this post is to expand on this topic by demonstrating some features and limitations. Let&#8217;s start by leaving flatland. Flatland &amp; the 4th Dimension &#8211; Carl Sagan This image is of a stereo card from the Missouri History Museum that is&#8230;<\/p>\n<p class=\"read-more\"><a class=\"btn btn-default\" href=\"https:\/\/www.cloudacm.com\/?p=4003\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,3],"tags":[],"class_list":["post-4003","post","type-post","status-publish","format-standard","hentry","category-computer-vision","category-rd"],"_links":{"self":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/4003","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4003"}],"version-history":[{"count":20,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/4003\/revisions"}],"predecessor-version":[{"id":4034,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/4003\/revisions\/4034"}],"wp:attachment":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4003"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4003"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4003"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}