A smart phone app that helps blind people "see"

From The LA Times:

You can do a lot with a mobile phone these days besides make calls. You can play music, watch videos and even get driving directions.

Gary Kelly sometimes uses his smart phone as his eyes.

Kelly, 60, lost his vision at age 11 after a bout with glaucoma and detached retinas. Now he can see only very bright light, like those from a camera flashbulb.

In adapting to life without sight Kelly devised ways to identify things and get around the house. But some things just couldn't be done.

When Kelly wants to order a pizza, for example, he can't use a touch-screen phone — an increasingly common calling device — because he's unable to feel the virtual buttons. And when it comes time to pay the delivery person, it's tough to figure out what's in his wallet. While coins vary in size and texture, paper bills all feel the same.

Recently, Kelly found a solution in a smart phone application called LookTel.

The app, which he installed on a Windows Mobile phone from Microsoft Corp., helps Kelly make a call by audibly announcing which button on the keypad his finger is hovering over. A double tap dials that digit. LookTel also connects with Kelly's Outlook address book. (Similar functions come with Apple Inc.'s iPhone.)

But what's been a godsend for Kelly is a feature that identifies everyday objects on the fly.

LookTel uses the phone's camera and transmits an image of whatever is in front of it over the Internet to the company's servers. Computers compare the image to those in LookTel's catalog; when they find a match the item's name is sent back to the phone.

Hold up a green note with Alexander Hamilton's face on it, and less than a second later you hear: "Ten dollars."

"I use the LookTel for a couple of things that nothing else can do," Kelly said by phone from Albuquerque, N.M. He's not ready to cede many of his daily functions to a machine, but being able to figure out whether he needs to go to an ATM is significant.

As executive director of an Atlanta-based nonprofit called Blind Wisdom Inc., Kelly seeks to promote technology research for the visually impaired. Five years ago he connected with the folks at Ipplex, a Santa Monica firm that's working on an application to aid blind people.

Orang Dialameh, 46, is founder and chief executive of the 16-person company. Kelly, along with about 100 blind beta testers, has provided feedback on how to improve the service based on their experiences. (Kelly is compensated for his participation.)

Dialameh and an associate visited The Times recently to let me test a Windows Mobile touch-screen phone running LookTel. After activating the camera-recognition mode, I found that the device was fairly snappy at identifying boxes of cereal and graham crackers and announcing the brand names.

The system is similar to apps from Google Inc. and Amazon.com Inc. Google's "Goggles" lets users of Android phones snap pictures of products and then returns search results about them. Google acquired a similar technology for recognizing faces in photos from Neven Vision, a company Dialameh was involved with.

But LookTel identifies items without the user having to snap a static image, and it returns an audible response rather than text. (However, you do need to take a high-resolution picture for LookTel to read a page from a newspaper or book.) Essentially, it's pulling objects out of streaming video.

"We have the best technology in the world on this," Dialameh said, "even better than Google's."

But the system is still a work in progress. A box of tea bags that Ipplex provided for the demonstration was seemingly nonexistent in the catalog. And in some cases the app would take several seconds to speak the name of the crackers, and then repeat it several times.

Dialameh blamed the errors on the shoddy camera in his outdated phone. Newer hardware, with better exposure adjustments and higher-quality lenses, should eliminate the problems, he said.

Ipplex is accepting applications for more phone testers at LookTel.com. It will begin welcoming a larger group in December, when a new version of the software is ready, with apps for iPhone and Android platforms coming in January or February, Dialameh said.

The company will ship buyers a phone with the app already installed and the service activated. Customers can opt for an older device, free with a two-year service contract, or order a cutting-edge smart phone from Apple, Samsung or others. They can also download the app and install it on a phone they already own.

In addition to the hardware cost and wireless subscription (that can add up quickly if you have AT&T's tiered 3G data plan), LookTel will cost $9 to $10 a month, Dialameh said.

"This wasn't strictly a do-good situation," he said. LookTel could be "analogous to the hearing aid," he said, drawing parallels to an established multibillion-dollar industry.

The market for LookTel's service could grow immensely in the next decade. There are about 15 million blind and visually impaired Americans, according to a report by Research to Prevent Blindness.

Kelly, who has been blind for nearly five decades, points out that aging baby boomers could make that number double.

"Millions of people will be losing vision," Kelly said, "and they won't have the skills that I acquired over 49 years."