/ Forums
New To The Forum? Click Here To Read The How To Guide. -- Developers Click Here.

Efficient 180° Fisheye to Equirectangular Video Conversion

thisisteekaythisisteekay Messages 6
mars 2016 modifié dans PC Development
Has anyone found an efficient solution to converting 180° fisheye video footage into Oculus-friendly projection like equirectangular?

I've been digging around like crazy and have a few solutions that sort of work, but nothing ideal:

I'd like to avoid converting to image sequences if possible. I can get my desired result on a single frame using PTGui (or the free alternative Hugin), but I can't seem to find any way to easily batch video.

Because I'm a Creative Cloud member, ideally I'd like to do something in Premiere Pro or After Effects, just because I know it would save me time in the future. I've tried some After Effects plugins, and Mettle almost worked. Mettle expects everything to be 360°, so to get the output I'm looking for, I had to do some tricky scaling of my source image, which I feel like degrades the quality of the final image. Also, I'm not too keen on paying a lot for something I have to hack my way through using.

I've looked into PixelBender. It's mostly unsupported now, but there is an inexpensive paid plugin that lets you use PixelBender (PBAccelerator) in newer versions of After Effects. But because it's unsupported, the PixelBender toolkit software doesn't work on Windows 8.1, and I don't know enough about the syntax to write my own pbk. There was a blog post that I came across that had a snippet of PixelBender code that would convert 180 Fisheye to Equirectangular projection: http://rcrmn.github.io/2015/07/24/after-effect-fisheye-projection-plugin/... but that guy wouldn't/couldn't release the actual final result.

It's frustrating, because the formula that would convert 180° fisheye mapping to equirectangular mapping doesn't look all that complex. I just can't find an efficient, inexpensive way to do it with video (and I don't know enough to be able to code my own).

So I'm kind of stuck. Has anyone come up with any efficient solutions working with 180° fisheye video sources?

Edit: The end goal here is to shoot stereoscopic 180° video. It's not so much the stereoscopy that's tripping me up as it is the 180° fisheye to equirectangular


  • scottsquiresscottsquires Messages 331
    Haven't tried my own fisheye but things to check out:

    http://www.andrewhazelden.com/blog/2012 ... ions-pack/

    You can sideload 180 x 180 material to Milk VR
    Add 180x180 to the .mp4 name


    You can get trial versions of AutoPano and Videostitch. I think they can both handle it.

    http://www.kscottz.com/fish-eye-lens-de ... -stiching/
  • TomCHallTomCHall Messages 17
    Virtual Boy (or Girl)
    If you are running a mac you can get this great premiere/FCPX plugin


    Has a ton of 360 equirectangular tools

    Right now you have to feed the fisheye to sphere plugin 1:1 images, so I nest them in a 1:1 sequence first and line up the edges of the image circle. I suggested to Tim that you can do it in future versions with the effect tab to save the step.

    best part of that plugin is you can run the dk2 at the same time and see the effect live.
  • Andrew Hazelden's stuff looks great, and I'll fallback on that if I have to. I plan to shoot at 60fps and I want to avoid converting to an image sequence if I don't have to.

    I tried VideoStitch Studio in the past, but it doesn't seem to honor my circular fisheye to equirectangular preference when I pass in the .pts file. I think it might only work when actually stitching multiple videos together to create 360 degree output, which is not really the case here.

    Also, I'm on a PC and not on a Mac, but that 360 VR Toolbox looks damn impressive!

    So, still nothing ideal as of yet.
  • mediavrmediavr Messages 222
    Hiro Protagonist
    I use Batch Builder and Batch Stitcher in PTGui Pro for image sequences for this. It is fast and accurate. PTGui Google groups forum has lots of posts on this.
  • TomCHallTomCHall Messages 17
    Virtual Boy (or Girl)
    Wish Dashwood would port to windows, but the suite heavily leans on the quartz compositor which is OS X only.
  • brad51brad51 Messages 2
    I'm not sure if you ever found a way to do this. I searched around and couldn't find anything. I did find a way that worked for me.
    I have a Nikon D610 with a 180 degree fisheye lens.
    My old technique:
    1. Convert the fisheye video into an image sequence
    2. Use PTGui to create a 360x180 equirectangular image set.
    3. Convert the image sequence back into video.

    My new technique:
    I manually created a Photoshop 'action' that takes the fisheye lens and converts it into a 360x180 equirectangular image. This action works on every video shot with the same lens.
    1. Open the fisheye video in photoshop.
    2. Apply the Photoshop action to the video to convert it to 360x180 equirectangular video.
    3. Render the video out from photoshop.

    Photoshop Action:
    To make the action I did the following...

    1. I converted a single frame of a fisheye video to a still image.
    2. I brought that frame into PTGui and made a 360x180 equirectangular version of it.
    3. I took the single frame of fisheye video into photoshop.
    4. I took the PTGui 360x180 equirectangular image into photoshop and placed it under the first layer.
    5. I set the top fisheye layer to 50% transparent.
    5. I scaled (if necessary) the equirectangular layer and positioned it to match directly under the top fisheye layer.
    6. Start Recording your action in the actions panel.
    7. Use Image>Canvas Size to expand the size of the canvas by about 120% (so that you will have enough room for the transform controls in the next step)
    8. Use Edit>Transform>Warp on the top fisheye layer. This took several minutes for me to transform the fisheye layer to match as closely as possible to the equirectangular layer underneath. I was able to get the image to match very closely in all areas except the extreme edges.
    9. Use Image>Canvas Size to Expand the size of the canvas to 4096x2048 (or whatever size you want to render your final video out)
    10. Stop your recording in the actions panel.

    I initially didn't think this would work. But when I tested this on my VR Gear, without extreme scrutiny, I couldn't tell the difference in the video created using the action and the perfect video created using PTGui. If you study the video carefully you will see that the edges are curved a little more in the photoshop actions version of the video. But this is not really noticeable at all. No real distortion anywhere to my eye! Pretty cool!
  • mediavrmediavr Messages 222
    Hiro Protagonist
    Autopano Video is fast but expensive and a bit complicated but you can go directly from an mp4 to a rendered frame sequence or mp4 or Cineform avi -- I get about 9 fps on my not very fast PC. I will write an account here of stereo 180 fisheye mp4 conversion with APV shortly. Another fast solution is Touch Designer but I havent tried that.
  • AymericB87AymericB87 Messages: 1
    Virtual Boy (or Girl)
    I am in the same situation. I am recording 4K videos with 2 DSLR Cameras with 180° fisheye lenses and am trying to convert them to equirectangular to play them in the Oculus DK2, but I'm not finding any easy way to do it. 360vrtoolbox would be perfect as I'm a CC user as well but I'm on a PC... so if anyone had another solution, that's be great!
  • scottoculusscottoculus Messages 32
    Check out PTGui  You'll have to batch it for video. Hugin is a free still open source stitcher.  All the apps are at least downloadable for testing (watermarkered)
  • BuckarooBuckaroo Messages 17
    dome2rect + VirtualDub have worked well for me. You still have to convert to images, though.
  • jhericojherico Messages 1,416
    Nexus 6
    I'm a little lost.  All of this commentary seems to be talking about pre-processing a video, but to my mind, the most efficient way to do this would be to feed the original video into OpenGL or DirectX and map it on to a mesh that provides the desired distortion.  That's how Oculus does the lens correction, and it's a great way of offloading the work onto the GPU where it belongs.  
  • scottoculusscottoculus Messages 32
    OpenGL can distort video but trying to combine 2 or more images in realtime has it's limits. It also means that the video can only played on that one app and not uploaded to any other video sites. preprocessing allwos you to adjust and refine the video stitch. You can then upload to any site that does VR and even with na player it's an earlier task to simply map it to a sphere or other shape.
  • Vern_SVern_S Messages 79
    Hiro Protagonist
    The Kodak SP360 4K cameras is a fisheye style camera and they have free software that may work. I have been able to load a 205 degree 2304 by 2304 MP4 exported from AE and it worked. Here is a link.

    PC SOFTWARE For KODAK PIXPRO SP360 4K Action Camera / YouTube™ 360° Support

    Let me know if it works.
  • Vern_SVern_S Messages 79
    Hiro Protagonist
    Try using the Sp360 4K software to unwrap. MP4 is the format excepted.

    PC SOFTWARE For KODAK PIXPRO SP360 4K Action Camera / YouTube™ 360° Support
  • scottoculusscottoculus Messages 32
    Be aware the Kodak software produces inferior quality to Kolor AVP and other stitchers. Lower res with minimal controls.
  • Vern_SVern_S Messages 79
    Hiro Protagonist
    Be aware the Kodak software produces inferior quality to Kolor AVP and other stitchers. Lower res with minimal controls.
    Not True. The new version of the software provided by PixPro including the stitch software now supports 4K  3840x1920 resolution export. The controls and adjustments are less then AVP but the price is a bit less, ie: Free vs ...

    They are continuing to refine the software and add features. 

    The 4K cameras by Kodak (JK Imaging) are quite good. I have the dual camera bundle and using it for immersive video in my VR apps.
  • Vern_SVern_S Messages 79
    Hiro Protagonist
    Just to add to my previous comment, I believe Kolor AVP/APG bundle the best option for Stitching 2 fisheye videos back to back with a max MP4 resolution of 4096x2048. Other non-standard formats have a larger resolution. Far superior stitching then PixPro Stitching software which will get the job done but you will have some seam issues. But again it's free.

    Just a reminder, there are two PixPro apps. One for controlling the camera from a computer and unfolding video and another for stitching two videos together.

    From what I understand of the needs of the OP what is needed is to take the 180° video and convert to Equirectangular. The PixPro SP360 4K camera saves a 1:1 aspect 2880x2880 MP4. The 180° video will need to be 1:1 as well. Exporting a 1:1 video out of Adobe Premiere or After Effects the max resolution for MP4 is 2304x2304. I have tested an exported MP4 at 2304x2304 and it works well in the PixPro software.

    In the PixPro SP360 4K app select playback and edit tab. Double click on the Disk icon to set the folder with the videos. Select the 1:1 video and select the YouTube icon to export. Select FOV orientation and resolution and export.

    One note, the SP360 4k camera is 235° fisheye or 205° if stabilization is on. You may need to adjust the 180° video to be slightly smaller to compensate. 

    Best of luck.

  • mediavrmediavr Messages 222
    Hiro Protagonist
    mars 2016 modifié
    My workflow for processing side by side 180 degree videos on modified Gopros (3+ Black with Dual Hero System for sync and 185 degree lenses) involves calibration of a frame pair of images with PTGui . I have calibrated one of the cameras with a panorama head  using PTGui. Then with the pair of images from both cameras I calibrate the second camera to the first pre-calibrated camera with a test shot with distant scenery with PTGui. So there are two lots of PTGui parameters for both cameras, fov, yaw, pitch and roll, a, b, and c distortion parameters and h and v sensor shift parameters.Output format is set as 360/180 equirectangular. Distant scenery is now rendered identically in equirectangular form if you output them now or check them in the PTGui preview window.

    These parameter values have to be entered into Autopano Video Pro for the rest of the AVP processing to stereo equirectangular 180 by 180 pair of videos. If you load both original videos into AVP it will try to stitch them as though it is a 2D 360 and give you a distorted picture whenever you try to "stitch" in AVP -- no matter if you enter ball park parameters manually.

    So forget about automatic stitching optimization with AVP and transfer your parameters directly from PTGui to the AVP parameter panel. This is the tricky bit: yaw, pitch and roll are transferable directly. H and V have to be transposed going from PTGui to AVP -- ie. H for PTGui is V for AVP etc.  I am not so sure about this H, V bit. (Maybe to do with whether my cameras are vertical in the rig). And then the fov has to be tweaked a degree or so from what PTGui says. You can see the effect of your changes directly in the AVP preview window so the fringes will disappear with distant scenery when you have  it right. You need to be careful with positioning your crop circles individually in both PTGui (in the initial calibration) and AVP. Then you have a template for your rig  in AVP you can reuse thereafter --if you have a rig with reproducible camera mounting - without going back to PTGui. And AVP stabilization and stereo previewing and fast output are a boon for 3d 180 processing.
  • RevisionfxRevisionfx Messages: 1
    @ thisisteekay

    You might be interested in RE:Lens by us (revisionfx.com) - (currently for AE and Premiere) - the plugin  ToLatLong supports fisheye to equirectangular -  equidistant, equisolid (like Canon, Sigma fisheye) and stereographic (like Rokinon/Samyang) lens mapping are supported.


  • nuclearsugarnuclearsugar Messages: 1
    Virtual Boy (or Girl)
    I've been thinking about this in the back of my head for a while and finally got around to tinkering... Turns out that Fisheye to Equirectangular conversion can be achieved directly within After Effects itself, without any plugins. The 'Polar Coordinates' effect is the secret sauce but you must precisely scale the footage to ensure the FOV is converted properly. I just wrote up a tutorial with all the necessary steps, tips, and caveats.

Connectez-vous ou Inscrivez-vous pour répondre.