The Israeli start-ups behind Google’s 3D technology

Petah Tikvah-based Mantis Vision and Raanana-based SagivTech are supplying core technologies for Google’s Project Tango, which is set to enable devices equipped with cameras, sensors and sophisticated algorithms to create a 3D model of the user’s environment and interact with it. Mantis Vision is key to Tango, producing the the main 3D engine used by the system.

Google has been working on Tango for some time, and this week announced details of the project. The company is preparing a Tango software development kit (SDK), to be released in June, to be paired with a tablet containing Nvidia’s Tegra K1 processor. The SDK and tablet will be available for $1,024. Google hopes that developers will develop cool games and applications that will bring 3D to “the masses.”

According to Nvidia, the tablet “incorporates cameras optimized for computer vision, advanced sensor fusion algorithms and integrated depth sensing tools, as well as the Nvidia Tegra K1 mobile processor. As a result, it can understand space and motion the way humans do, enabling interior spaces to be quickly mapped in three dimensions, allowing the creation of applications that blend real and virtual objects.”

Mantis Vision is central to the Tango project. The company’s 3D engine, called MV4D, will serve as the core 3D engine for Tango devices, which includes structured light-based depth sensing algorithms. That technology is what turns the 2D picture seen by a device’s camera into a 3D environment and recreates inside the device’s processor a very close approximation of what the camera sees and enables Tango-based software to manipulate and interact with the environment.

MV4D’s technology is based on an active triangulation system, meaning that it uses software to “observe” a scene from different points of view, taking into account the dimensions, lighting, shadows and other information via laser. The raw image is sent to a computer, where the 3D modeling takes place. A game creator could use the technology to build a 3D gaming environment, allowing individuals to upload images of themselves as avatars, which can then be programmed into the game.

Those are the kinds of things Google has in mind. According to the search giant, a Tango-based tablet could be used to build a 3D gaming environment in a living room, projecting a virtual gaming environment onto a room’s walls and ceiling and allowing users to play games as if they were inside the device, mimicking real 3D action. A Tango app for shoppers could take a photo of a dress or shirt and project it onto a wearer’s on-screen avatar, or allow users to see how furniture would look in their living room. Tango technology could make life much easier for the disabled, said Google, with an app for the visually impaired detecting depth and telling users how many feet away they are from a street corner or obstacle, for example.

SagivTech’s technology may come in handy once Tango devices are established in the market. The company, working with partners in the EU, is developing a technology to combine videos of a single event taken by different people from different points of view, turning them into a single 3D view of the entire event. The system, called SceneNet, “stitches together the videos at their edges, matching the scenes uploaded by the crowdsourced devices,” according to Dr. Chen Sagiv, the company’s CEO and chief coordinator of the SceneNet project. “It’s a complicated process, because you have to match the colors and compensate for the different lighting, the capabilities of devices and other factors that cause one video of even the same scene to look very different.”

SceneNet could be used to allow viewers to watch a rock concert from any angle or perspective they want, for example, focusing on the drummer or guitarist or watching the crowds, as if they were at the event itself, moving through the crowd or going to get a better look at the action onstage. The technology could be used in social networking apps that would allow users to collaborate on a YouTube video, uploading different points of view of an event and assembling them into a 3D recreation in the cloud, just as users of Google Docs can collaborate on a document stored online.

Google is not the only company working on 3D tech. Microsoft’s IllumiRoom uses a Kinect camera and a projector to lets users experience games and interactive videos on a much bigger surface than even a large screen TV, allowing a TV to “grow” out of its box and encompass the entire room and creating a 3D environment that blends screen and reality. The Kinect 3D scanner captures the layout of the room, including where the furniture is, pictures on the wall, colors and lighting, and adapts the games and videos being used by a gaming system, such as the Xbox One, to the contours of the room. IllumiRoom is still in development by Microsoft.

Besides Microsoft, Intel is also hard at work on its 3D tech, called Perceptual Computing. Intel came out with a Perceptual Computing SDK last year, utilizing a Creative 3D camera, but there has been no word on further progress by Intel in the field yet this year.

With Google set to come out with the first actual 3D-capable tablets and phones this year, it appears that the company is ahead of the pack in deploying the technology. Mantis had a lot to do with that. “3D represents a major paradigm shift for mobile. We haven’t seen a change this significant since the introduction of the camera-phone. MV4D allows developers to deliver 3D-enabled mobile devices and capabilities to the world,” said Amihai Loven, CEO of Mantis Vision. “This partnership with Google offers Mantis Vision the flexibility to expand quickly and strategically. It will fuel adoption and engagement directly with consumer audiences worldwide. Together, we are bringing 3D to the masses.”