When will we be able to direct computers with our brains? Sooner than we think

From The NY Times magazine:

“Fingers!” Gerwin Schalk sputtered, waving his hands around in the air. “Fingers are made to pick up a hammer.” He prodded the table, mimicking the way we poke at computer keyboards. “It’s totally ridiculous,” he said.

I was visiting Schalk, a 40-year-old computer engineer, at his bunkerlike office in the Wads­worth Center, a public-health lab outside Albany that handles many of New York State’s rabies tests. It so happens that his lab is also pioneering a new way to control our computers — with thoughts instead of fingers. Schalk studies people at the Albany Medical Center who have become, not by choice, some of the world’s first cyborgs.

One volunteer was a young man in his 20s who suffers from a severe form of epilepsy. He had been outfitted with a temporary device, a postcard-­size patch of electrodes that sits on the brain’s cortex, known as an electrocorticographic (ECoG) implant. Surgeons use these implants to home in on the damaged tissue that causes seizures.

Schalk took advantage of the implant to see if the patient could control the actions in a video game called Galaga using only his thoughts. In the videotape of this experiment, you see a young man wearing a turban of bandages with wires running from his head to a computer in a cart. “Pew, pew,” the ship on the computer screen whines, as it decimates buglike creatures. The patient flicks the spaceship back and forth by imagining that he is moving his tongue. This creates a pulse in his brain that travels through the wires into a computer. Thus, a thought becomes a software command.

On the day I stopped by his office, Schalk hit a button on his computer, and Pink Floyd blasted from his speakers. He was running an experiment to see what happens to people’s brains when they listen to “Another Brick in the Wall, Part 1” (a question that has occurred to any stoner who ever contemplated human consciousness in the glow of stereo lights). Weeks before, Schalk played the Pink Floyd song for some of his epileptic volunteers and recorded the activity in the parts of the brain that process sound. Schalk showed me a volume meter on his computer screen — this was a brain, tracking the roar of a guitar solo. It worked just like any other volume meter, but in one experiment, Schalk found that the brain did something unexpected. When he interrupted the Pink Floyd song with moments of silence, the brain’s volume meter continued to tremble up and down, as if the song were still playing. This, Schalk said, showed that the brain creates a model of what it expects to hear — a shadow song that plunks out its tune in the player piano of our auditory system.

“Isn’t this crazy?” he shouted over the thunder of the bass. “We’re close to being able to reconstruct the actual music heard in the brain and play it. If we had several times more electrodes, I bet we could do it.”

But for Schalk — and many others in the field — the ultimate goal is not music. It’s language. Schalk dreams of letting people speak with their neurons, issuing silent commands to their machines. You could imagine the word “cat,” say, and it would pop up on your computer screen. The areas involved with imagined speech take up just a few centimeters in the brain. With better implants, Schalk said, he might be able to pick up a word that his volunteer beams at the computer. Even with today’s implants, he and his colleagues are getting closer. One epilepsy patient moved a ball across a computer screen simply by imagining either an “ooh” sound or an “aah” sound. It marked one more step toward telepathy with machines.

For years, computers have been creeping ever nearer to our neurons. Thousands of people have become cyborgs, of a sort, for medical reasons: cochlear implants augment hearing and deep-brain stimulators treat Parkinson’s. But within the next decade, we are likely to see a new kind of implant, designed for healthy people who want to merge with machines. With several competing technologies in development, scientists squabble over which device works best; no one wants theirs to end up looking like the Betamax of brain wear. Schalk is a champion of the ECoG implant because, unlike other devices, it does not pierce brain tissue; instead it can ride on top of the brain-blood barrier, sensing the activity of populations of neurons and passing their chatter to the outside world, like a radio signal. Schalk says this is the brain implant most likely to evolve into a consumer product that could send signals to a prosthetic hand, an iPhone, a computer or a car.

“The burr hole in the skull will be small,” Schalk told me enthusiastically, as if urging me to get one of the plugs. The first dedicated trials in human beings, he says, are only a few years away.

Schalk first began working with the ECoG implant in 2003, when a surgeon at Washington University in St. Louis invited him to visit the epilepsy ward; four patients had been taken off their medication and had portions of their skulls removed so they could be implanted with ECoG devices.

The implants — usually worn for about a week — allow surgeons to study the aberrant brain patterns of patients as they go into seizure and then cut out the damaged brain tissue. Schalk camped out in the Missouri hospital to wait for the periods when patients were between seizures, at which point he would try to transform the brain signals emitted by their thoughts into software commands. He was, in effect, designing a button that the mind could push.

“We had no clue what we were doing,” Schalk says.

On the first day of the experiments, he sat beside a young man who gamely waited to follow instructions. As researchers rolled a monitor up to the bed, Schalk told the patient, “Now you’re going to move this cursor by thinking.” For a few minutes, the guy floundered. And then, Schalk says: “Boom, the cursor hit the target. Everyone was ecstatic.”

Now Schalk can get all the human brains he wants within walking distance of his office. In 2007, he discovered that the Albany Medical Center houses an epilepsy center, and he set up shop in his hometown, working closely with Anthony Ritaccio, a professor of neurology and neurosurgery at Albany Medical College.

When I met Ritaccio in front of the hospital, he also talked about the problems with fingers. “We’re always interested in doing things faster,” he said. “I remember the transition to an electric typewriter. We’re addicted to speed. But obviously the way we communicate with computers is rather comical. The way we interact with this blazing fast machine is to poke at it with a finger.”

Schalk and Ritaccio’s research has been underwritten by a $2.2 million Department of Defense grant. The project is part of a $6.3 million Army initiative to invent devices for telepathic communication — for instance, a “telepathy helmet” that would allow soldiers to beam thoughts to one another. Schalk seemed untroubled by the military applications. He said the grant allows him to do research that could, one day, let us all — civilians included — merge with our machines.

Justin Williams, a biomedical engineer at the University of Wisconsin, has already transformed the ECoG implant into a microdevice that can be installed with a minimum of fuss. It has been tested in animals for long periods of time — the micro ECoG stays in place and doesn’t seem to negatively affect the immune system. Williams said he hopes to try it in humans soon. “Our goal is to make devices that would require only an outpatient procedure,” he says. “Even if we could make it an overnight stay in the hospital, that would be good.” The implant, in humans, would be about the size of a quarter and sit like a plug in the skull, with a tiny antenna for wireless hookup between machine and brain.

The first to receive the implants would most likely be patients with serious illnesses or disabilities. “We’re targeting neuroprosthetics, spinal-cord injury, A.L.S., motor disorders,” Williams said. “And there’s a lot of interest in epileptic patients.” Williams said he hopes that the tiny implants might afford surgeons more ways to monitor what happens inside the skull. “There are a lot of brain injuries where we don’t understand what’s going on,” he said. But Schalk anticipates a much larger audience for the implants. In fact, he believes that the tiny implant will be a kind of gateway drug for people who choose to become, essentially, cyborgs.

I told Schalk that I was dubious: it was hard to imagine anyone opting for an enhancement that could become a spy-cam in the skull. He exploded with impatience at my lack of imagination, hands cutting the air, as he leaned forward from his perch on a desk. “A small opening in the skull, you plop a device in, wireless transmission and you’re done,” he said. “It sounds crazy. But if you have something that works, people will do it in a nanosecond. Breast augmentation is totally invasive compared to this — and how many people do that every year?”

Dean Pomerleau, an engineer based at Carnegie Mellon University who leads Intel’s brain-computer initiative, might be one of the first to volunteer for a brain implant. (Intel, he stressed, is investing in research but is not developing any kind of “thought chip.”) Pomerleau, speaking as a representative of himself rather than his company, said he was so excited about the cyborging of the human brain that he has taken radical steps to live longer to see it happen. He follows a strict diet, limiting himself to two vegan meals a day; he has eaten the same superfood salad at every meal for the last 10 years. He said that the miracle he is most eager to witness is a kind of mind-meld with other people: a “two-way direct-brain interface” that would “revolutionize human experience.”

Pomerleau and his colleagues are already using machines to let people transmit their silent thoughts — but in an extremely rudimentary fashion. In order to send a telepathic thought in their lab, you have to climb into an fMRI machine, a multimillion-dollar scanning device that delivers detailed maps of the mind in action. How does it work? Try this: Tomato. Tomato. Tomato. Each time you read that word, a specific pattern of neurons fires in several different neighborhoods of your brain. You’re doing more than just passing your eyes over a bunch of letters; you’re also remembering the tomato’s ruby flesh and the way fingers dimple the skin and the earthy smell of its stem.

Tom Mitchell, a brain scientist and Pomerleau’s colleague at Carnegie Mellon, said it was possible to use this kind of activity to send a telepathic message. First, you lie in the scanner, thinking about the words you see on a screen — “airplane,” “truck,” “hammer,” “apple.” As you do this, a software program studies the patterns in your neurons. “I can give you different nouns, like hammer, screwdriver and tomato, and then use our software program to distinguish which of those nouns you are thinking about,” Mitchell said. Most intriguing: the pattern of “tomato” in your brain is similar to the tomato in my brain — and even to the tomate in the brain of a Portuguese speaker.

“Can our program distinguish if you are thinking about three tomatoes or eight tomatoes?” Mitchell asked. “The answer is, yes it can.”

Mitchell is the first to admit that the fMRI will never evolve into a wearable device or a reasonable brain-­computer interface. Nonetheless, the machine is helping to push the technology forward. “If you want to build a good device,” one that is noninvasive and portable, “it’d be really helpful for you to know what is the real neural activity going on,” he said.

Pomerleau said they are “trying to show the world what amazing things you can do if you could get good brain signals out” of the skull.

Schalk is grappling with his own version of this question. In 2008, he published a paper in The Journal of Neural Engineering, which proposed a kind of Esperanto for man to converse with machine. Human beings could learn to think in patterns easily recognized by computers, to create bursts of thought that acted as software code. “This paper is not science fiction,” he insisted.

At Dartmouth College, the future seems to have already arrived. Researchers there are in the process of creating an iPhone connected to an EEG (electroencephalography) headset. The beauty of an EEG is that no skull drilling is required. Sensors glued to the scalp record the patterns of neurons. The drawback is that the human skull blocks most brain waves before they rise to the scalp, so the EEG signal tends to be weak and muddy.

I encountered what they call a NeuralPhone in a basement lab littered with Doritos bags. Matt Mukerjee, the grad student who put it together as part of his senior thesis, stood to shake my hand. He was wearing what appeared to be an octopus on his head — an EEG headset made by Emotiv Systems. Its black plastic tentacles curved above his forehead and disappeared into his mop of hair.

“I’m going to call Einstein,” he said. In its demo mode, the phone gave you a choice of six people — including President Obama, Steve Jobs and Albert Einstein — and let you pick one, telepathically. The NeuralPhone was meant to demonstrate that one day we might mind-control the contact lists on our phones. In a preliminary study, Mukerjee’s iPhone responded to the user’s thoughts with an accuracy rate that approached 90 percent — but only under ideal conditions.

Mukerjee hunkered down in his chair and meditated, as he glared at the phone. After a long pause, a photo of Einstein expanded to fill the screen on his iPhone.

Now it was my turn. Mukerjee removed the headset and moistened the tips of its electrodes with contact-lens fluid, then arranged the EEG device on top of my hair. The electrodes poked into my scalp like wet fingers. I held the iPhone in front of me and beamed a blast of willpower at it. “Steve Jobs, Steve Jobs, Steve Jobs, Steve Jobs,” I shouted inside my mind. The phone picked George Bush.

“You have a small head,” Mukerjee said accusingly. He slid the headset around, trying to force the electrodes to grasp my scalp more tightly, which didn’t work. He decided to cheat a little. He rearranged the electrodes so that they detected the movement of muscles on my forehead.

This time, the results were magical. As soon as I chose Bill Gates, a photo of Gates splashed onto the phone’s screen. I was not aware of blinking or moving the muscles of my face. The phone seemed to have merged with my body, to be as much a part of me as a finger or toe. I found myself laughing. I couldn’t stop saying, “This is freaky.” And it was.