// HACKER NEWS — CYBERSECURITY
Navigation App for People with Blindness and Low Vision
For people with blindness or who have low vision, moving around independently is a daily challenge. Harvard researchers have developed a new tool that could help such people navigate more quickly and safely, using nothing but a device they likely already have – a smartphone.
Engineers at the John A. Paulson School of Engineering and Applied Sciences (SEAS) have invented a new smartphone app called Mobilio that combines machine learning; algorithms that streamline input from multiple sensors; and personalized audio cues to provide a blind or low-vision person turn-by-turn directions, path guidance, and obstacle avoidance. The research was led by Ph.D. student Raymond Liu in collaboration with Patrick Slade, assistant professor of bioengineering and associate faculty member of the Kempner Institute. The research is published in Nature Biomedical Engineering.
This video shows the navigation algorithms and spatial audio feedback the smartphone app Mobilio performs during community path and obstacle course experiments.
To start, the researchers surveyed over 100 blind or low-vision people to understand their needs. Respondents highlighted three essential capabilities for navigation technologies: reliable turn-by-turn directions, continuous guidance along sidewalks and paths, and obstacle detection and avoidance.
Existing tools like canes, guide dogs, and electronic travel aids address parts of these problems but rarely all three together, and many such systems are expensive and inaccessible. While GPS map software has turn-by-turn navigation, it is not accurate enough for a person who is blind or low-vision to safely use.
By contrast, most people already have smartphones. The Harvard team saw an opportunity to deliver a navigation aid through a standalone app that could one day be downloaded from an app store and runs entirely on the phone.
Mobilio utilizes the phone’s built-in sensors, including the camera; GPS; motion- and orientation-sensitive inertial measurement unit; and (if available) the LiDAR sensor. A central technological advance is a custom computer-vision model that analyzes the live camera feed from the perspective of a pedestrian to find walkable paths in the user’s environment.
“Our system uses the sensors in a smartphone, GPS information, and the actual motion of the person to create what’s essentially a small autonomous vehicle plan on how to navigate them from where they are to where they want to go,” Slade said.
Bioengineering assistant professor Patrick Slade describes his lab's work creating smartphone applications for assisting people with blindness and low vision.
Since most publicly available street-scene datasets are captured from cars, many standard models misclassify pedestrian-relevant infrastructure, explained first author Liu. To address this issue, Liu trained a semantic segmentation model specifically on pedestrian-view images so it could reliably recognize sidewalks, crosswalks, roads, and other key surfaces during walking.