The Sensory Substitution Experience

I was the first trained test subject on Zachary Capalbo's Kromophone sonification device. After just four hours of initial training, we took a trip to Harvard Square in Boston, MA. Using the sensory substitution device, which converts visual information from a camera into sound, I was able to navigate crowded streets, stairs, and life in the world, all while blindfolded.

You would think that this process would impair my normal auditory capabilities, but I was able to successfully stop before walking out into a busy road. There came a point when the sensation the device gave me in the form of unfamiliar sounds turned into a perception of shapes and colors, which was separate and distinct from normal auditory information, such as the sound of cars in city traffic. In a way you could think of it as, when learning a new word in a foreign language, there comes a point when you cease to translate the word in your head every time you hear it, and instead automatically understand its meaning.

At one point I came across a low, cylindrical, blue object that I could not identify. From the shape, it looked (or rather, sounded) like a fire hydrant, but I was also hearing that it was blue. I was right, it was both a fire hydrant, and blue, but having never seen or heard of a blue fire hydrant before, I concluded that it was not.

This seems to indicate that the device can be used to correctly identify objects, with sufficient practice and experience. Admittedly, it would require significant practice to 'see' with something near a normal proficiency level (it wasn't exactly amusing when I crashed into a squeaky clean window). But, I think this experience shows at least that the technology has potential for practical application.