{"id":1524,"date":"2014-11-26T00:00:24","date_gmt":"2014-11-26T08:00:24","guid":{"rendered":"http:\/\/192.168.3.4\/?p=1524"},"modified":"2018-01-09T06:49:50","modified_gmt":"2018-01-09T14:49:50","slug":"raspberry-pi-image-processing","status":"publish","type":"post","link":"https:\/\/www.cloudacm.com\/?p=1524","title":{"rendered":"Automated web scraping with RPi"},"content":{"rendered":"

Introduction – time to let the machine do the work<\/strong><\/p>\n

In the last installment, I covered command functions to interact with the RPi’s GPIO header. \u00a0The examples were interactive and had little in the way of full automation. \u00a0In this discussion, I would like to focus on automating processes. \u00a0These are tasks that we will define and schedule to run at times we specify. \u00a0The purpose of this exercise is to establish a foundation for the RPi to run tasks, without the need of an operator. \u00a0This is useful for polling temperatures from a third party website, gathering photos from web cameras, or checking the status of a device connected to the GPIO header.<\/p>\n

I’ll cover some related subjects that were previously discussed in earlier posts. \u00a0These will be key in enabling automated functionality on the RPi. We will schedule\u00a0using Webmin, since it’s much simpler than the CLI task of setting CRON. \u00a0Most of our automation will revolve around CRON, so having a simple way to implement and debug will be helpful.<\/p>\n

Another aspect of this exercise will be image processing, so I’ll also cover this topic. \u00a0I won’t be going into great detail the uses of OpenCV, but I will cover the steps to install in on RPi. \u00a0Most of the image processing topic will be focused on\u00a0ImageMagick, ffmpeg, avconv, and mencoder.<\/p>\n

Purpose – a picture says a thousand words<\/strong><\/p>\n

Web scraping is a technique of\u00a0automatically downloading web content at set intervals. \u00a0The content I’ll\u00a0be downloading are images from traffic cameras, satellites, network cameras, and radar plotters. \u00a0The images are not compiled by the host providers, so I would like to add value to the data by rendering them into video.<\/p>\n

The steps to accomplish this will provide an understanding of how to automate web scraping for other purposes. \u00a0It will also provide an introduction to image processing and rendering. \u00a0Lastly, it provides\u00a0scalability\u00a0to manage automated tasks. \u00a0All of these skills will be useful\u00a0toward deploying RPi in advanced settings.<\/p>\n

Detail – how it’s done<\/strong><\/p>\n

I was initially interested in the project as a way to create video from\u00a0a web camera attached to the RPi. \u00a0I had no trouble<\/a> finding a way to setup motion for RPi<\/a>. \u00a0The only problem was it was the project was limited to my hardware and I felt it missed the point of web scraping. \u00a0So, with motion all setup and running I decided to move further on.<\/p>\n

These were the steps\u00a0I followed\u00a0in order to render\u00a0videos from the web scraped images.<\/p>\n

    \n
  1. Determine the resources to gather off the internet<\/li>\n
  2. The command “wget” would be used to gather and rename images from the internet.<\/li>\n
  3. Webmin was used to set CRON\u00a0schedules for the scrape\u00a0commands.<\/li>\n
  4. Create video from image repository<\/li>\n
  5. Install OpenCV on the RPi<\/li>\n<\/ol>\n

    I had the\u00a0sources that I wanted to scrape from, the trick was identifying\u00a0the image resources. \u00a0The easiest one was Unisys Weather<\/a>. \u00a0Getting the URL of the images was simple, all I had to do was load the page and right click the image properties to view it. \u00a0I had more of a challenge getting the image properties from Seattle’s DOT and Weatherspark\u00a0websites. \u00a0For those sites, I used the debug function on my browser to list resources as they loaded. \u00a0Once I was able to identify them, I tested to verify they worked. \u00a0I did this for my home internet camera as well.<\/p>\n

    Now that I have the resources picked out, I needed to set the parameters for the “wget” command. \u00a0Since the images would be processed into a video feed, I had to sequence the name. \u00a0Initially, I wanted to just number them, but I settled on a time stamp in the name for simplicity. \u00a0One added benefit of doing this is now I have a time I can reference later, if needed. \u00a0Here is the syntax I used:<\/p>\n