{"id":3900,"date":"2021-12-21T14:00:38","date_gmt":"2021-12-21T22:00:38","guid":{"rendered":"https:\/\/www.cloudacm.com\/?p=3900"},"modified":"2021-12-13T07:12:51","modified_gmt":"2021-12-13T15:12:51","slug":"python-data-visualization","status":"publish","type":"post","link":"https:\/\/www.cloudacm.com\/?p=3900","title":{"rendered":"Python Data Visualization"},"content":{"rendered":"<p>In an earlier post, python was used to generate an image from datasets. That method lacked the full features needed to create a final image. This post will introduce Plotly as a preferred method. It offers more options and is targeted for use as a tool for dataset visualization.<\/p>\n<p>First we&#8217;ll need a dataset that is tangible. In this post the dataset will be generated from a field scan to create an image array. The device used to scan the field will be a photoresistor that will be pivoted by a pair of servos, each one on a x and y axis. This will be purely a theoretical scenario, so no actual code or hardware was developed for this concept.<\/p>\n<p>The photoresistor used will be a standard cadmium sulfide LDR. The sensor was fitted with a focal guide to allow only a narrow angular field of view. This was done by fitting heat shrink tubing around the sensor and allowing a long narrow opening for light to enter. The datasheet can be found online here, <a href=\"https:\/\/www.tme.eu\/Document\/0b7aec6d26675b47f9e54d893cd4521b\/PGM5506.pdf\">https:\/\/www.tme.eu\/Document\/0b7aec6d26675b47f9e54d893cd4521b\/PGM5506.pdf<\/a><\/p>\n<p>As mentioned, the photoresistor sensor will take stepped measurements of a field horizontally and continue that pattern for each step vertically . The step of each motion has been set to 1 degree and is centered on the photoresistor&#8217;s sensor surface. Likewise, the pivot on the y axis is also centered on the photoresistor&#8217;s sensor surface with each motion step set to 1 degree. The range of motion will be limited to 60 degrees on the x axis and 35 degrees on the y. This should result in an array that has a 60 x 35 pixel resolution.<\/p>\n<p>The speed that the servos can step will be limited by the response time of the photoresistor as defined in its datasheet. The larger value from either the rise or decay time for the sensor will determine dwell time for each measurement step or sample. Based on the datasheet, a dwell time of 50ms should be enough for the sensor to acquire its reading. Based on that limit, the entire time to scan the field will be x * y * 50 = time in ms, which is 1 minute and 45 seconds. So the object and photoresistor must be stationary for a reliable image to be created, which is why a still life was selected.<\/p>\n<p>Any microcontroller that can be programmed to step the pair of servos, quantify a value measured by the sensor, format the readings with delimitation and carriage returns, and store the reading to memory is suitable for use. The final results should look similar to this.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">135,135,135,....159,159,159\r\n136,136,136,....159,159,159\r\n...\r\n138,137,137,....160,160,161<\/pre>\n<p>The dataset source file is here, <a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/SCN_IMG_GRAY_Array1.csv\">SCN_IMG_GRAY_Array1<\/a><\/p>\n<p>Once the scan is completed, the dataset can be processed by Plotly to create the image. Here is the python code used with this command &#8220;python3 SCN_IMG_GRAY_Array1_Plotly.py&#8221;<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">import plotly.express as px\r\nfrom numpy import genfromtxt\r\n\r\ndata = genfromtxt('SCN_IMG_GRAY_Array1_Plotly.csv', delimiter=',')\r\n\r\nfig = px.imshow(data, color_continuous_scale=px.colors.sequential.gray)\r\n                 \r\nfig.update_layout(width=60, height=35, margin=dict(l=0, r=0, b=0, t=0), coloraxis_showscale=False)\r\nfig.update_xaxes(showticklabels=False).update_yaxes(showticklabels=False)\r\n# fig.show()\r\n\r\nfig.write_image(\"SCN_IMG_GRAY_Array1_Plotly.png\") \r\n\r\n# Command - python3 SCN_IMG_GRAY_Array1_Plotly.py\r\n# Source - https:\/\/plotly.com\/python\/heatmaps\/\r\n# Color Scales - https:\/\/plotly.com\/python\/builtin-colorscales\/\r\n\r\n# https:\/\/plotly.com\/python\/imshow\/\r\n# https:\/\/pypi.org\/project\/plotly-express\/\r\n# https:\/\/www.cloudacm.com\/?p=2674\r\n# https:\/\/plotly.com\/python\/plotly-express\/<\/pre>\n<p>&nbsp;<\/p>\n<p>Here is the resulting image file generated, which has been zoomed 10 times in order to allow for easier viewing.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_GRAY_Array1_Plotly.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-3904 size-full\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_GRAY_Array1_Plotly.png\" alt=\"\" width=\"600\" height=\"350\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_GRAY_Array1_Plotly.png 600w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_GRAY_Array1_Plotly-300x175.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_GRAY_Array1_Plotly-463x270.png 463w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p>It is remarkable that a simple photoresistor can create a grayscale image, but with the use of color filters, datasets can be merged to create a full color image. This has been demontrated here with some background on the process.<\/p>\n<p><iframe loading=\"lazy\" title=\"Color Photos from a Black and White Camera\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/a-ny3geJ-nk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>Each field scan is done using a separte color filter to only allow a given spectrum of color to reach the photoresistor. The resulting datasets are converted into grayscale images using the python process already discussed. This will result in 3 layers that are red, green, blue. Using Imagemagick, the 3 layers can be combined to create the color image. This is done using the following command.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">convert SCN_IMG_RED_Array1_Plotly.png SCN_IMG_GREEN_Array1_Plotly.png SCN_IMG_BLUE_Array1_Plotly.png -combine -set colorspace sRGB SCN_IMG_COLOR_Array1_Plotly.png<\/pre>\n<p>&nbsp;<\/p>\n<p>The following resized image is the result and it appears something is going on.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Array1_Plotly.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-3906\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Array1_Plotly.jpg\" alt=\"\" width=\"600\" height=\"350\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Array1_Plotly.jpg 600w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Array1_Plotly-300x175.jpg 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Array1_Plotly-463x270.jpg 463w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p>If we look at the datasheet, the light spectrum response of the photoresistor is not level across all of the color ranges. The following image is an overlay to give a better visual of what colors are affected.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Photoresistor-5516-Wavelength.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-3908\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Photoresistor-5516-Wavelength.png\" alt=\"\" width=\"487\" height=\"283\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Photoresistor-5516-Wavelength.png 487w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Photoresistor-5516-Wavelength-300x174.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Photoresistor-5516-Wavelength-465x270.png 465w\" sizes=\"auto, (max-width: 487px) 100vw, 487px\" \/><\/a><\/p>\n<p>Based on that color curve, the blue value is 60 percent and the red value is 35 percent of what green is. If we apply that logic to our datasets, we can adjust the values for blue and red to be level with the green values. After doing that adjustment and merging the images, we now have a more realistic color image.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Resized.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-3910\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Resized.jpg\" alt=\"\" width=\"600\" height=\"350\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Resized.jpg 600w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Resized-300x175.jpg 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/Scaled_SCN_IMG_COLOR_Resized-463x270.jpg 463w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p>This is fundamentally what all color cameras are, except on a much smaller scale. Instead of a single sensor that changes color filters and scans each scene, an array of millions of sensors that have color filters built in are used. Here is a view of a modern camera sensor.<\/p>\n<p><iframe loading=\"lazy\" title=\"Camera Sensor Under The Microscope 1000X\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/HM7LDpXqfTg?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>The acquiring and processing of the dataset to reproduce a scene into an image might have detracted or convoluted the intent of this post. For that reason, another example of python visualization will take a dataset created by a NRF240L0 sensor. The earlier post used a dataset to create a waterfall image of the RF scan. This same dataset was used to create the following image that is side by side with the original using this imagemagick command, &#8220;convert +append NRFDATA2A.png NRFDATA2B.png -resize 315x NRFDATA2C.png&#8221; The Plotly image is on the left, the unprocessed python method is on the right.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA2C.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-3913\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA2C.png\" alt=\"\" width=\"630\" height=\"388\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA2C.png 630w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA2C-300x185.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA2C-438x270.png 438w\" sizes=\"auto, (max-width: 630px) 100vw, 630px\" \/><\/a><\/p>\n<p>Plotly offers many more options on how to visualize the dataset, not just based on color scale, but also style. Here is another example of the same dataset.<\/p>\n<p><a href=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-3914\" src=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D.png\" alt=\"\" width=\"500\" height=\"500\" srcset=\"https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D.png 500w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D-300x300.png 300w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D-150x150.png 150w, https:\/\/www.cloudacm.com\/wp-content\/uploads\/2021\/12\/NRFDATA-3D-270x270.png 270w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/a><\/p>\n<p>This is fundamentally the process to generate wifi heat maps as demonstrated here.<\/p>\n<p><a href=\"https:\/\/github.com\/jantman\/python-wifi-survey-heatmap\">https:\/\/github.com\/jantman\/python-wifi-survey-heatmap<\/a><br \/>\n<a href=\"https:\/\/pypi.org\/project\/whm\/\">https:\/\/pypi.org\/project\/whm\/<\/a><\/p>\n<p>Data visualization can take cold analytics and make them tangible. The next post will introduce Processing as another visualization tool for taking sensor data and making it alive.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In an earlier post, python was used to generate an image from datasets. That method lacked the full features needed to create a final image. This post will introduce Plotly as a preferred method. It offers more options and is targeted for use as a tool for dataset visualization. First we&#8217;ll need a dataset that is tangible. In this post the dataset will be generated from a field scan to create an image array. The device used to scan the&#8230;<\/p>\n<p class=\"read-more\"><a class=\"btn btn-default\" href=\"https:\/\/www.cloudacm.com\/?p=3900\"> Read More<span class=\"screen-reader-text\">  Read More<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-3900","post","type-post","status-publish","format-standard","hentry","category-rd"],"_links":{"self":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/3900","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3900"}],"version-history":[{"count":10,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/3900\/revisions"}],"predecessor-version":[{"id":3902,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=\/wp\/v2\/posts\/3900\/revisions\/3902"}],"wp:attachment":[{"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3900"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3900"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cloudacm.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3900"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}