Create and Process 360 degree video for Youtube

Create and Process 360 degree video for Youtube

In this post I’m going to cover how to create 360 degree video and prepare it for uploading to Youtube.

I’d like to cover SimpleCV, specifically the functions of dewarping the video. The video from Tinkernut made reference to this process. You may find this link useful in the setup of SimpleCV, https://github.com/sightmachine/SimpleCV. He based his work off of Kathrine Scott’s work. She did an excellent write up on the process, http://www.kscottz.com/dewarped-panoramic-images-from-a-raspberrypi-camera-module/. The big caveat is that video is captured using a raspberry pi on board camera module. This is a problem for my intent because of the following reasons.

  1. The video source will be from action cameras, so I’ll be working with secondary device video files.
  2. I don’t intend to do my processing on a raspberry pi, most likely it will be a more power system.
  3. Tinkernut and Scott’s work is geared toward streaming video specific to the raspberry pi on board camera module.

I would really like to leverage SimpleCV to post process my video. However, it might prove to be too time consuming to dive into with my limited availability. Maybe another time.

Not to despair, I had worked with yet another ImageMagick script from Fred Weinhaus called “fisheye2pano”. Not long after the Kogeto lens arrived I looked into ways to dewarp the image. Turns out much of the leg work has been done. This site describes the IM script and provides examples, http://www.fmwconcepts.com/imagemagick/fisheye2pano/index.php.

img_1982

I have a snapshot that I’d like to dewarp. It’s an image file, not video. To use the IM methods for video I would most likely extract frames, process the images, then convert back to video. The SimpleCV method mentioned earlier does this as well.

The source image I’ll be using has the following attributes.

Width: 2448 pixels
Height: 2448 pixels
Focal Length: 4.3mm
ISO: 50
Aperture Value: 2.53 EV (f/2.4)
Exposure Time 1/812 sec.

After some initial testing, I’m able to see some distortion removed. However, there are some artifacts from the original image that are causing strange results. This is due to the alignment of the Kogeto lens over the camera. The spherical image isn’t dead center on the image. I’ll need to crop the image from the start. This is good, because it will decrease the outer dead space that is lost using the Kogeto lens.

Using GIMP, I’ve establish the working outer edge and crop to these dimensions.

Top 580
Bottom 1990
Left 510
Right 1922

You can see how much offset from center the image is. With this image, we can run through our convert processes and see that our results fair much better.

Fred outlines steps to take using native IM commands. I’m going to do the same thing and vary the arguments to improve the results. Here is a list of the commands that will be used.

1. Crop the image to eliminate the outer dead space.

convert IMG_1982.JPG -crop 1420x1420+510+582 IMG_1982_IM-Step1_Crop.JPG

2. Dewarp the circular image to a rectangular image.

convert -distort depolar 0 IMG_1982_IM-Step1_Crop.JPG IMG_1982_IM-Step2_Depolar.JPG

3. Depending on the orientation, the image may need to be flipped and / or rotated.

convert -rotate 90 IMG_1982_IM-Step2_Depolar.JPG IMG_1982_IM-Step3_Rotate.JPG

4. Half of the image will contain dead space, cropping the image again should remove this.

convert IMG_1982_IM-Step3_Rotate.JPG -crop 710x1420+0+0 IMG_1982_IM-Step4_Crop.JPG

5. The image will need to be scaled so the appearance is more realistic. The width is the only thing that needs to be increased, typically by a factor of 280 percent.

convert IMG_1982_IM-Step4_Crop.JPG -resize '100x280%' IMG_1982_IM-Step5_Scale.JPG

img_1982

To put it all together here is the command sequence.

convert IMG_1982.JPG -crop 1420x1420+510+582 tmp.jpg
convert -distort depolar 0 tmp.jpg tmp.jpg
convert -rotate 90 tmp.jpg tmp.jpg
convert tmp.jpg -crop 710x1420+0+0 tmp.jpg
convert tmp.jpg -resize '100x280%' Final.jpg
rm tmp.jpg

All of these commands take less than a second to complete for a single frame. Working with a series of image files, variables can be used to go through each of the image files and process them.

for i in *.JPG; do convert "$i" -crop 1420x1420+510+582 "Crop_Step1/${i%.*}.jpg"; done
for i in *.JPG; do convert -distort depolar 0 "Crop_Step1/${i%.*}.jpg" "Depolar_Step2/${i%.*}.jpg"; done
for i in *.JPG; do convert -rotate 90 "Depolar_Step2/${i%.*}.jpg" "Rotate_Step3/${i%.*}.jpg"; done
for i in *.JPG; do convert "Rotate_Step3/${i%.*}.jpg" -crop 710x1420+0+0 "Crop_Step4/${i%.*}.jpg"; done
for i in *.JPG; do convert "Crop_Step4/${i%.*}.jpg" -resize '100x280%' "Resize_Step5/${i%.*}.jpg"; done

To make it a sleek single liner, use this command.

for i in *.JPG; do convert "$i" -crop 1420x1420+510+582 "Crop_Step1/${i%.*}.jpg"; convert -distort depolar 0 "Crop_Step1/${i%.*}.jpg" "Depolar_Step2/${i%.*}.jpg"; convert -rotate 90 "Depolar_Step2/${i%.*}.jpg" "Rotate_Step3/${i%.*}.jpg"; convert "Rotate_Step3/${i%.*}.jpg" -crop 710x1420+0+0 "Crop_Step4/${i%.*}.jpg"; convert "Crop_Step4/${i%.*}.jpg" -resize '100x280%' "Resize_Step5/${i%.*}.jpg"; done

This is helpful and useful to have all the steps of processing in separate folders. It gives me the option to compile video from different stage sources. Now we’re ready to drop all our images into a video file in preparation for spacial metadata injection.

Some more prep work will need to be done before the file can be submitted to Youtube. The Kogeto lens is not a full x,y 360 degree lens, artificial dead space will need to be created, place the viewable area in the middle.

I used one of my 808-16 720p key-chain cameras. It doesn’t have a wide angle lens and this made it easier to place the Kogeto lens directly over the camera. I didn’t want to make any changes, all I wanted to do was a quick proof of concept shot. I’m able to use the 808-16 as a web cam by running Cheese. First, I connect the 808-16 to the computer using a USB cable. Then I power on the camera. Next, I run Cheese from the command line. I have the option with Cheese to snap a picture or capture video.

0001

The video is an 11 second clip at 1280 x 720 @ 25 fps. I’ll need to convert the video to image frames. I did that with this command.

avconv -i "/home/local/Desktop/360 Video and Metadata/808-16 Camera/2016-10-29-103647.webm" -r 25 -f image2 "/home/local/Desktop/360 Video and Metadata/808-16 Camera/ImageDump/%04d.jpg"

Now I’m ready to tweak my IM commands to apply to the lens alignment on the 808-16 and the dead spaces in the frames.

I’ll step through these commands line by line to be sure they work before running them against all my image frames.

for i in *.jpg; do convert "$i" -crop 592x592+320+62 "Crop_Step1/${i%.*}.jpg"; done
for i in *.jpg; do convert -distort depolar 0 "Crop_Step1/${i%.*}.jpg" "Depolar_Step2/${i%.*}.jpg"; done
for i in *.jpg; do convert -flip -rotate 180 "Depolar_Step2/${i%.*}.jpg" "Rotate_Step3/${i%.*}.jpg"; done
for i in *.jpg; do convert "Rotate_Step3/${i%.*}.jpg" -crop 592x592+0+296 "Crop_Step4/${i%.*}.jpg"; done
for i in *.jpg; do convert "Crop_Step4/${i%.*}.jpg" -resize '280x100%' "Resize_Step5/${i%.*}.jpg"; done

0001

That should do it, now for the one liner

cd "/home/local/Desktop/360 Video and Metadata/808-16 Camera/ImageDump"
for i in *.jpg; do convert "$i" -crop 592x592+320+62 "Crop_Step1/${i%.*}.jpg"; convert -distort depolar 0 "Crop_Step1/${i%.*}.jpg" "Depolar_Step2/${i%.*}.jpg"; convert -flip -rotate 180 "Depolar_Step2/${i%.*}.jpg" "Rotate_Step3/${i%.*}.jpg"; convert "Rotate_Step3/${i%.*}.jpg" -crop 592x592+0+296 "Crop_Step4/${i%.*}.jpg"; convert "Crop_Step4/${i%.*}.jpg" -resize '280x100%' "Resize_Step5/${i%.*}.jpg"; done

Sweet! Now we’re ready to convert the frame images back into video. I’m going to keep the original frame rate of 25. Here’s the command to do that.

cd "/home/local/Desktop/360 Video and Metadata/808-16 Camera/ImageDump/Resize_Step5"
mencoder mf://*.jpg -mf fps=25 -o YouTubeReady.mpg -ovc lavc -lavcopts vcodec=mpeg4

The rendered video is 1658 x 296 @ 25 fps. I took the sound track from the original video and copied it to the processed video using Openshot. The final video is 1280 x 720 at 25 fps. Not too bad. The IM processing was fast, in comparison to my tweaks and debugging.

If I were to take the idea of using this format seriously, I would invest in a better lens, preferably glass. Also, I’d use one of my 1080p cameras and remove it from its factory case. The lens and 1080p camera electronics would be housed in a water proof case with easy mounting and portability. Honestly, for a lens and camera that both cost under $50, this is most impressive. I might be back this way.

As far as image projection processing, I’ll be covering more of this in the posts to follow.

Enjoy!

Comments are closed.