What would happen if you could take any video and make it interactive? That’s the question posited by MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), who has discovered a technique to realistically move objects using touch controls from any static video shot, even ones taken from YouTube.
They’re calling this imaging technique Interactive Dynamic Video (IDV) and have released a stunning video showing how easy it is. In their first example they set up a wire frame figure and just bang on the surface next to it so it shakes around a tiny bit. Using this five second video and some algorithms that look at the minuscule vibrations of the object, they’re able to create simulations that users can interact with.
Amazing, isn’t it? Even though it’s somewhat amusing that with all these great brains behind the technique they still don’t know to shoot in portrait mode.
But the number of potential applications for this technology is staggering, especially for film and games. As they point out in the video, anytime you have CGI characters interacting in the real world it’s a pain, since you have to either physically manipulate real objects or simply create even more digital objects in order to look like the computer construct is really in the world. If they refined and worked on this technology all they would need to do is shoot a short video of a location to get the vibrations, drop the CGI character into it, and manipulate the background. They show this off at the 2:22 point and it’s remarkable how well it works, even with a MS Paint creation.
Besides entertainment there’s also the potential for engineering applications, like being able to easily conduct Structural Health Monitoring of bridges and such to make sure they’re still structurally sound.
The most amazing thing is that this could work with any static video shot. CSAIL proved this by taking a YouTube video featuring a test of a bridge in a wind tunnel and place an alien invader in it, making the bridge buckle in order to accommodate the digital creation.
As it is right now it’s still not quite perfect, but it’s obvious that this has the potential to completely change the game for indie filmmakers and video game developers. Just look at how much better Pokemon Go could be with it! You can read more about this technique at CSAIL’s project website.