by Stephan Gensch and Eva Lopez
“What kind of video footage can we best rehash for digital 3D reconstruction?” – that’s one of the main questions we’ve been exploring in the V4Design project. We aim to train innovative photogrammetry algorithms, i.e.: to automatically turn multi-perspective 2D video shots of an object (e.g. an iconic building) into a decent 3D version of that thing. Over the course of the project, we’ve evaluated all kinds of archived footage. Initial results were disappointing, but once we started looking for material in the right place, we discovered a very promising approach. In this article, broadcaster and content provider Deutsche Welle reflects the journey of identifying photogrammetry-suitable footage and how to make it available (more).
One of the basic requirements for good 3D reconstruction is video material that features so-called parallax camera movement. That means: an operator has moved her camera along the horizontal and/or vertical axes of a target object, most preferably using a dolly (think of nice, slow tracking shots of an old Cathedral). A shot sequencer and a computer vision algorithm working in tandem are then (hopefully) able to find all frames necessary to turn the flat material into a three-dimensional model fit to be used by a designer.
Our D4.3 deliverable represents the first iteration of 3D reconstruction and scientific report describing and showing the SoA methods for image sequence analysis, and 3D sparse and dense reconstruction.
No parallax in news – or is there?
Unfortunately, the kind of shots we were looking for are very uncommon in news broadcasting. DW focuses on to-the-point information and short video bits mostly meant to provide context. There are some nice establishing shots, and there’s also more sophisticated visual material in documentaries, but most of it has pans, and tilts, and zooms – not the camera movement we needed for our photogrammetry experiments. So where could we find material that would play nice with our system?
Just when we thought we’d hit a dead end, it suddenly came to us: What about exploiting drone footage? There should be substantial supply of that by now, right?
The perks of drone footage
It turns out we were on to something: Audiences love to see familiar scenes and landscapes in a different, exciting way – and shooting drone videos is getting more and more popular. A format tapping into the magic of the “flying eye” is DW’s travel magazine series Daily Drone. And while the list of episodes is getting longer and longer, other DW documentary teams have started using drones, too. So: Plenty of
material for the V4Design team to experiment with! And just the right one: A drone camera is moving constantly – and in all the right directions.
The V4Design pipeline
How do we make use of the drone footage now? The process is pretty straight forward:
V4Design partner CERTH has created a client to find and access specific drone videos via an API and third-party applications, including the API provided by in this case that of DW. The client passes the videos on to the V4Design analytics and processing pipeline. This pipeline creates so-called basic assets, i.e.: media entities containing visual content, descriptions and other metadata. The basic assets can be further analysed by image recognition algorithms, e.g. to cut out a specific object in a video or label the video content. The cut-out, nicely textured objects can then be used for 3D reconstruction.
And all this thanks to some baseline movement in the videos!
The technical workflow is now set: Users feed the V4Design pipeline with suitable footage, and all identified 3D assets are saved to an open, searchable online repository. Plugins ensure integration with Unity and Rhino.
In the last couple of months, the consortium has worked on the scalability of the platform, which already offers more than 1,000 assets, and completed v2 of the protype. At the moment, the consortium is working on interactive templates (e.g. for games), a remote tourism project (featuring landmarks along the Danube), and higher quality 3D assets – for an even more immersive V4Design experience.