When the coronavirus pandemic forced Israelis to celebrate Independence Day in quarantine without the usual parties and barbecues, President Reuven Rivlin decided to add holiday cheer by visiting homes around the country. Rivlin delivered a short speech and posed for selfies with thousands of Israelis in their own living rooms.
“I don’t have my camera with me, so you’re in charge, and don’t forget to smile,” Rivlin smiled before posing for a selfie with Israel’s enchanted citizens.
The thousands of photos of Rivlin uploaded by Israelis that day were made possible by Israeli startup TetaVi, which has developed new technology to create lifelike 3D video holograms that can be inserted into any film or photo using minimal equipment and at a fraction of the regular cost.
“It was easy for all involved, and only took about 30 or 40 minutes of Rivlin’s time,” says Gilad Talmon, CEO of TetaVi, which uses computer vision and deep-learning technology to produce quality content with fewer cameras. “You can really use this anywhere, to create any 3D content.”
Tailor-made for the TikTok generation, TetaVi is developing a next-generation system that will transform a regular smartphone into a broadcast-quality studio. It will enable users to create 3D holograms using a single device, creating a virtual portable immersive video studio without the need for multiple cameras or a green screen. The affordable 360° volumetric video-capture will allow everybody to create an immersive viewer experience with lifelike holograms for video-game quality realism.
The AI-created volumetric video, the enabling technology for immersive video experiences, allows consumers to see themselves inside a video game, on a concert stage or anywhere they want. 3D immersive video is a big revenue driver for video games and is ramping up in social media. The video game market is worth over $150 billion each year – more than triple the value of the global film industry. Creating 3D avatars with conventional technology requires a studio with up to 100 cameras, expensive green screens, miles of cable, and multiple high-powered servers.
As more governments, businesses and other organizations embrace immersive video experiences, including augmented and virtual reality, there is a growing need for high-quality, affordable 3D volumetric video that is easy to produce. The global market for volumetric video production and services is growing by more than 32% a year, and is expected to be worth $5.8 billion by 2025, according to MarketsandMarkets. In addition to the entertainment industry, its use is also expanding in areas like sports broadcasting, education and medicine.
TetaVi is soon closing an extension to its Series A funding round from investors including Adway Ventures, REDDSCapital and Jerusalem-based OurCrowd.
One problem is the limited capability of existing wireless networks. Sharing user-generated 3D content, made with the help of technology like TetaVi’s, could will encourage more people and places to embrace 5G, says Mikio Iwamura, director of the device innovation group at NTT MoDoCo, the Japanese telecom giant which is partnering with the Israeli company.
“XR devices and volumetric video content capabilities are the right contents to really pull out the potential of 5G,” Iwamura says.
But the broader use of volumetric video also presents challenges, including the need for expensive equipment and professional production. Usually, dozens of special cameras are needed to capture video from many viewpoints in order to produce 3D content. In addition, green studio backdrops are also often required when recording video to create a hologram, or 3D rendering of a person or object that can then be used in regular video, gaming applications or augmented and virtual reality.
“It’s something that can take hours to set up, or people need to go to a studio, which is not always possible,” Talmon says.
TetaVi simplifies and reduces the cost of this process by using between four and eight portable cameras, and does not require any special background to make holograms or avatars of people, like it did for Rivlin.
“We bring the same type of quality, but do it with a mobile system,” Talmon says. “We use deep learning to make up for the smaller number of viewpoints.”
Deep learning means training networks to use certain algorithms or computerized calculations to analyze large amounts of data, and learn to recognize patterns – similar to how the human brain works. In this case, the technology quickly combs through massive sets of image data, and uses it to instantaneously fill in areas not actually captured on video. Talmon declines to go into more detail, citing business confidentiality, but says it is this underlying technique that fuels the company’s technology.
“We are actually first and foremost a deep-learning company,” he says. “And because the data set is always expanding, the technology gets better and better all the time. I would say that it offers higher quality images than in cases where content is created by using hundreds of cameras.”
The company also offers a subscription software program for editing content, and any content produced can be integrated into gaming, broadcast and other applications.
TetaVi’s system is aimed at studios and professional content producers. Its clients range from wedding video producers to major operators like NTT DoCoMo.
Ultimately, the company wants to make the technology available to all consumers.
“We want anyone to be able to record 3D content from anywhere, that’s our long-term vision,” Talmon says. He sees user-generated 3D content, integrated into virtual reality and augmented reality applications, playing a big role on social media platforms in the near future. Social media companies are thinking along the same lines. Facebook, where more than 6,000 employees are dedicated to AR and VR, recently announced it would release smart glasses later this year, and will eventually add augmented-reality capabilities to them, allowing users to blend reality and the digital world.
“Eventually, our goal is to enable people to capture volumetric video for social media content, and to bring it to a level when you can just do this without any specific hardware, with just your phone,” Talmon says. “Our focus is on democratizing the capture ability.”
However, most mobile networks are not fast enough to quickly transmit the large quantities of data required for three-dimensional images. Hopefully, the faster 5G networks will change that.
“For now, the way to distribute this content is still missing,” Talmon says. “5G is the final block that will cement everything together.”
“We’re really excited,” Talmon says. “We believe that we will help consumers take their human connections, self-expression and creativity to the next dimension.”
For more information about TetaVi, click HERE.