Memory, Encoded

Memory Encoded 1

What if you could safely keep your most important information in a database that could only be unlocked by your own memory? A web-based model developed by professors Brent Heeringa and Nate Kornell does just that.

The model relies on the proven strength of the human ability to correctly recall a series of images. “I like to think of our system as providing sophisticated mnemonic tools to the average person,” says Heeringa, chair and associate professor of computer science.

The system works this way: A user visits the website and is prompted to enter his or her email address and a string of numbers—a phone number, a credit card or a social security number. The user then views a slideshow of photographs and types in a description, in her own words, of each image in the series. So she might label a rainy cityscape “umbrellas,” a stadium as “football” and a brown and white puppy as “beagle.” The number of images labeled corresponds to the number of digits in the information she’s encoding.

To recall each number, the user is shown several images strung together in a row—new ones along with the one she labeled earlier. With each string of images, she is prompted to click on the one she remembers: the cityscape, the stadium, the puppy. If she selects all the correct images or all but one, the system reveals the number.

Study.EncodeDataThe beauty of the model is that there’s no way for hackers to crack the code. Even if they gain access to the information stored in the database, they’ll learn only that the images stored there each represent a different number or letter. “The only person that can decode it properly is you,” says Heeringa. “There’s no encryption here. The security is the exact same security that comes from memory.”

Years of memory research prove that visual cues are an effective tool to help recall information. Hundreds of years ago, monks used images to help them remember long passages of text, says Kornell, assistant professor of psychology. Award-winning memory experts use “memory palaces”—structures they “build” inside their minds that put stand-in objects and characters in different rooms—to accurately recall hundreds of numbers.

Those methods usually require a lot of concentration and training. But Heeringa and Kornell’s model—which builds on the ideas of Andrew Drucker, a theoretical computer scientist completing a postdoc at the University of Edinburgh—puts the images right in front of a user.

“We’re taking the limitations of human memory,” Heeringa says, “and we’re playing to its strengths and avoiding its weaknesses.”

It’s easy to envision a commercial application for the model. Imagine you’re at your home computer, about to make a purchase online, and you realize your credit card is in the kitchen, several rooms away. If you’ve purchased something from the retailer before and elected to save your card number, the website could prompt you with a series of images, including ones that only you would recognize. If you select the correct ones, you get your credit card number and save yourself a trip to the kitchen.

Preliminary tests of Heeringa and Kornell’s model, conducted with help from students participating in the college’s Summer Science Research Program, are encouraging. The researchers crowdsourced a pool of participants who were assigned randomly to one of three online experiments. They were asked to study 60, 80 or 100 images. Then they were asked to re-study them one, two or three times. The participants were tested after a week and again after a month. After a month’s time, the majority of participants in all the experiments correctly identified between 88 and 90 percent of the images.

Heeringa and Kornell have been working on the project for two years, and several students have made significant contributions. Brianne Mirecki ’14 built the first recognition model and helped run the first series of experiments. Carson Eisenach ’14 helped build the original working prototype. And Benno Stein ’15 helped build the system’s latest iteration.

There’s more to be done. The system must be tested in order to ensure it will be easy for people to use. And ease of use must be substantiated if the system has any chance of being shared with a wider audience.

“I know it’s going to work,” Heeringa says. “The question is, how well is it going to work?”

—Francesca Shanks

Try the model here: