Students build high-tech ‘guide dog’ for the blind

A 3D camera, a mini-PC, and an Android phone use audio to help the vision-impaired navigate around obstacles

Raw images captured by a Kinect camera, ready for processing in the Technion system (Photo credit: courtesy)
Raw images captured by a Kinect camera, ready for processing in the Technion system (Photo credit: courtesy)

A class project by three Technion University students could lead to a device to help the blind easily navigate their environment. Using a Kinect 3D camera, a computer and an Android phone for audio interaction, Tzahi Simkin, Gal Dalal and Danny Zilber developed a system that detects upcoming obstacles and uses audio messages and signals to alert users if they are about to bump into something and how to avoid the obstacle.

Many companies, including several in Israel, are working on technologies to help the blind, some of which entail surgery to implant sensors into a blind person’s brain, transmitting signals that are turned into images in the person’s mind. Simkin thought to use existing off-the-shelf technology to build a navigation system, use a mini-PC to process images taken by the Kinect and translate them into voice or signal commands for the user.

The Kinect camera is Microsoft’s motion-sensing input device that takes images, processes them and turns them into 3D immersive views. These can be displayed on a screen, as they are when the Kinect is used in tandem with Microsoft’s Xbox 360 and Xbox One video game consoles. The team adapted the Kinect to feed its images into a mini-PC, where the images are analyzed. The system determines if there are objects or obstacles in the path of the camera, how close they are, how far to the left or right the user would have to move to avoid them and more. That data is crunched and a sound interface issues orders to an Android device, using voice commands or a series of beeps to guide the user around obstacles.

“The camera sits on a belt and takes depth images of the surrounding area,” said Dalal. “The wireless device processes the information received from the camera and gives a voice indication to the user through the application. The application we developed helps blind people navigate inside a building, warning them of obstacles through voice indication and directing the user around them.” The system can be “trained” to identify objects, Dalal added. “Objects it can recognize include items such as keys or handbag. In other words, there is an element of recognition and learning.”

The system is still experimental, but could be perfected and brought to market at some point, Simkin said. “I’m not sure this team will be the one to do it, because we are in school and don’t necessarily have time to advance it,” he said in an interview. “But I could see this deployed in any number of ways, such as with a hat or helmet, with miniaturized components providing the guidance system.”

His system is not as good as a guide dog, admits Simkin. “Technology can help, but you still need the traditional solutions” in order to ensure that blind people are able to navigate successfully, he said.

But it could be the next best thing. “The idea came to me while I was driving, where, right before me, I saw a blind man having trouble crossing the road,” Simkin said. “I thought to myself that, if I could only describe to him, through technological means, a snapshot of the surrounding area, I would make it much easier for him and build his confidence, helping him get better oriented with his surroundings. I wanted to combine technological development with social assistance, and this is how this product was born.”

Koby Kohai, who heads the Control Robotics & Machine Learning Laboratory at the Technion’s Faculty of Electrical Engineering, guided the students throughout the project. “The project received a grade of 100 and has been submitted for a competition for outstanding faculty projects,” said Kohai. “The project was initiated by students, and I instructed and steered them towards technologies currently available on the market. The concept of the project was to test a technological concept that could, in the future, evolve into something more advanced. Every year we suggest ideas for project development to our undergraduate students, coming from industry or research of graduate students at the faculty. We do our best to provide students with a broad space with which to encourage their creativity and their ideas in their chosen projects.”

The students demonstrated the system at the recent Taglit Tech Challenge in Jerusalem, where it was reviewed by top executives of Israeli tech companies, and by Jewish Agency chairman Natan Sharansky, who praised it. “The system is a great example of the innovation being developed in Israeli universities,” Sharansky said, adding that “Israel is a technological superpower, as well as a center of Jewish inspiration, and a device like this that combines the two — the high-tech creativity combined with the commandment to help those in need — is a great example of Zionism in action.”

Click below for a clip of the students’ project in action:

read more: