See What I’m Saying: the extraordinary powers of our five senses
Lawrence D. Rosenblum, 2010
This book aims to bring to conscious level some of the amazing things our sensory systems do at a subconscious level, and also to highlight individuals with sensory superpowers. The author also includes fun experiments the reader can do at home.
Hearing:
Daniel Kish and Brian Bushway: blind mountain bikers who use echolocation to mountain bike and even lead tours. Kish was born blind, started clicking as a young child and was adept at echolocation by 7. He didn’t actually realize he was echolocating until adulthood, instead he just sensed his environment intuitively.
It used to be thought that blind people used changes in air pressure on their faces to navigate the environment, based on self-report of blind people, but in the 1940’s, Karl Dellenbach’s lab showed it was actually sound by removing sound and finding that both sighted people with blindfolds on, as well as blind people could not navigate at all.
Can everyone echolocate? Limit is 2 m or so, after that the differences in intensity and differences are too small for ear to resolve further. We actually use reflected sound all the time to navigate our environments.
To see how well you can hear sound interference patterns, make a shhh sound through your teeth, close your eyes, and place your palm in front of your face. Move your palm closer and further away, and you’ll hear the differences in distance as a wooshing sound. You can also stand close to a wall and sway close to and away from it while making the shhhh sound. Once you learn what that sounds like, back a few paces away from the wall and try to then walk towards it with your eyes closed while making the sound. Try to get as close as you can without colliding with it. Over the course of 10 trials or so, you’ll get much better at using the sound interference as a cue to locate the wall, and may even feel like you’re sensing the wall rather than hearing sounds reflected off it.
Jay Patterson is sound mixer for film and TV, and a big part of his job is making the reflected sound in a set congruent with what is being seen visually and consistent during a scene (e.g if the scene is a small bedroom but filmed in a large open set, the sound needs to be altered so audience stays immersed). Also, microphones aimed at different actors will have different reflected sound, and since our ears cannot pick up only one side of a room’s acoustics at a time, this will also sound funny to an audience, and needs to be corrected for. Patterson’s team typically hangs 4-8 sound absorbing blankets in various locations to better enable a congruent sound experience for the audience. Room tone recordings (of the silent set) are mixed with sounds in post-production as well.
If you close your eyes and clap in different rooms of your house, you’ll see that you can use reflected sound to learn about what kind of space you occupy.
Yasuhisa Toyota designs concert halls, and he observed that modern recording techniques allow the audience to hear each instrument (with its own microphone) so clearly that it has raised the bar for concert hall design. One experiment in which a sound was recorded in 4 separate indoor environments showed that subjects can identify which of four rooms the sound was recorded in 73% of the time.
Rosenblum’s lab has shown that even unpracticed, blindfolded individuals can determine the horizontal or vertical position of sound obstructing panels placed in front of a loudspeaker well enough determine if they would be able to fit between the panels or walk under the panels without turning or ducking. They can also determine above chance if the panel is a triangle, square, or disc shape! The subjects who did best typically had some experience with sound, like being a musician, suggesting that it is a very trainable skill.
Beep baseball has been played in the US for over 30 years and is played by blind individuals who hit a softball that makes a beeping sound continually throughout the game. They can do this because sound gets louder as it gets closer, and also clearer (you can hear more of the higher frequencies as the sound gets closer). Additionally, we can sense the relative amount of sound coming directly from an object vs reflected from other objects, changes in arrival time at our two ears, and frequency changes induced by Doppler shift. Importantly, our brains model all of these characteristics of sound to predict when an object will reach us. Interestingly, without practice, most people seem to have a built in buffer such that they anticipate that a consistently moving sound source will reach them before it actually will, perhaps part of a warning system. With practice, blind individuals overcome this overcompensation are become much more accurate at this than unpracticed individuals.
Try pouring a drink with your eyes closed. Even without practice, you might be able to pour a glass without overflowing based on changes in pitch of the sound of pouring. This is anticipatory acoustics.
Sacha Van Loo: blind from birth, works with the police to listen to wire tap recordings and determine what kind of environment the suspect is calling from, and how many people are present etc.
Lab research shows we can hear the gender of a person from their walk from using cues like pitch of footsteps, which changes depending on where the center of mass is (higher for men).
The Masters of Harmony: award winning a cappella group. How do they sound like one single voice? One trick is to sing just under the loudness at which they can hear their own distinct voice. Notes that are slightly out of pitch will create destructive interference, which dampens the sound intensity, which can induce singers to sing the off key notes louder to compensate. So singing below the audible individua threshold defeats this feedback loop.
Untrained singers are good at tempo and pitch regulation when key and range are left up to them, but they’re not good at matching pitch. We tend to also have a good memory for the pitch and key in which a piece of music was performed, and we can tell if it’s played in a different key. About 4% of the population is unable to do these things.
Emotional associations with major-minor tonality seem to be culturally learned, but emotional connotations of fast and slow tempo may have a more instinctual basis.
Smelling
Lab studies show that humans can train scent tracking, in this case a rope soaked in a fragrant oil. As subjects get better, they increase their sniffing speed. Interestingly, we seem to use the difference in scent between our two nostrils as a cue for which direction a smell emanates from. Particle imaging has showed that the two nostrils take in air from two distinct regions, and subjects are much worse at scent tracking with one nostril closed.
We smell more than we realize. Not as well as dogs though, they have 20x more olfactors receptors than we do. Although, since we’re not ground sniffers, we don’t have the complex filtering apparatus that dogs do to prevent infection, so more odor molecules get to our receptors. Also, unlike dogs, we pass odor-infused air over our receptors when we breathe out as well, not just in, and that exhalation plays a huge role in our taste system. Also, smell in humans activates brain areas more broadly than in other animals, which suggests more processing of the odor input. We can smell some stuff, like natural gas, better than dogs, we can detect a few drops in Olympic pool sized environment. We can also improve with practice. About 50% of the population can’t consciously detect androstenone (from human sweat), but it can improve with practice. This is associated not only with brain changes, but changes in the neurons in your nose, which had not been observed at the time of writing for any other sense.
We are excellent at odor discrimination for odors we can consciously smell. Additionally, exposure to a high concentration of a smell for a few minutes improves the ability of that person to discriminate between other similar smells. Repeated exposure to a complex mixture also improve the ability of that person to decipher the separate components (think of expert coffee and wine tasters). But most of us are pretty poor at explicitly naming smells, this can be improved with practice, or if we’re given choices of labels. We’re also quite poor at imagining smells.
We can use smells to identify people. Parents can pick out a child’s t-shirt from smell, unless they’re identical twins, so smell seems to have a genetic component. Our Major Histocompatability Complex genes (related to immune function) likely play a large role in our individual scent signatures. Familiarity seems to make us more likely to like smells (mothers prefer the smell of their own baby’s diaper).
Contrary to popular belief, odor isn’t a more useful tool for retrieving accurate memories than cues from sight and sound, but odor memories do seem to be more emotional and evocative. Scent can actually render a memory as more emotional, seen when you prime a subject with a memory then present odor, auditory, and visual cues. The vividness of a scent memory makes people inaccurately confident about details of the memory as well. Also seen in brain imaging, more activity in amygdala with perfume smell vs perfume bottle picture.
Odor preferences seem pretty malleable, they differ between cultures and subjects can be induced to form a positive or negative association with a novel odor depending on the rest of the lab environment. Some of it does seem innate, like putrid odors, so our odor preferences show constrained plasticity.
Some research shows that we can unconsciously smell each other’s fertility, mood, and genetic compatibility, but these are typically based on small sample sizes. While humans don’t seem to have releaser pheromones (pheromones that automatically trigger a behavior or physiological change in another individual), we may have modulator pheromones that can change things like out mood and heart rate, and influence our behavior.
Tasting
What you see influences what you taste. If 4 fruit drinks are presented, you may identify them all accurately, but if they’re colored in a non-intuitive way, accuracy drops significantly. This has been replicated with a variety of food and drinks. Most people even inaccurately identify tastes that aren’t there, like an orange flavor in an orange colored drink. Expert wine tasters given the same white wine twice (colored red the second time) reported red-wine-characteristic tastes when tasting the wine for the second time. The author went to a pitch-black experimental restaurant, and found that all the food tasted bland to him and his companions.
Sound also influences taste, playing back less crisp sounds to subjects eating potato chips makes them feel like the chip was less crisp.
Linda Blade is s sensory evaluator for food companies to detect things like warmed over meat flavor, oxidized fats, burnt elements etc. She now finds it hard to eat casually, she can’t eat warmed up leftovers at all. But the sensory evaluators were just people from the community with no special skills. They were just trained to concentrate on specific flavors and verbalize those experiences.
Tactile capture is part of our taste experience. Foods activate receptors near your tongue tip, and that taste tracks with the movement of the food along your tongue, despite us not having many taste receptors along the sides of our tongue. This is likely part of our perceptual system’s general strategy to perceive things as unitary and coherent.
Temperature also affects food taste, we taste optimally when food is between 71.6 and 98.6 F.
If you cool your tongue with an ice cube, then take a sip of warm water. You might experience a sweet taste. Temperature itself can induce cell activity on your tongue that the brain interprets as taste experience.
Spicy foods actually impair the ability to taste other tastes acutely, but after the sensation fades, taste is enhanced.
Much of what we experience as taste (perhaps 80%) is actually retronasal smell, which you can see when you try to discriminate between different foods with your nose plugged. Sensory evaluators taste food with their nose plugged to detect the component flavors like salty, sweet, sour, and bitter, then taste the food normally to see what is added by the aromatics (e.g chocolate flavor, lemon etc). Odor and flavor inputs converge on brain regions related to taste perception. Brain scanning shows that when a consistent flavor and odor are presented together, the amount of activity in taste regions is greater than the sum of activity from each individually. This might be why 2 flavors that are each below the sensory threshold can, together, produce a noticeable taste. This can work with up to 24 compounds under the detection threshold, a phenomenon expert chefs take advantage of. This is also true of a taste and a smell presented together, both individually under the detection threshold. But only if they are congruent (e.g cherry with cherry vs cherry with sage).
Steven Poe: In the master sommelier exam, you have 25 minutes to describe and recognize the region, grape, and year of 6 wines. There were only 167 master sommeliers in the world at the time of writing. Interestingly, these experts aren’t better than average at detecting components like tannins and alcohol, or at discriminating whether two wines are the same or different. But their brains do seem to process the information differently from novices. Master sommeliers show more activity in taste and smell converging areas in the brain, followed by more activity in higher cognitive areas serving memory, language and decision making. Experts have a rich conceptual understanding that allow them to attend to tastes associated with other tastes (meaningful constellations), allowing them to more easily recognize wines and detect subtle flavors because they’re actively looking for them.
Verbal overshadowing: your verbal description of a sensory experience actually interferes with simple recognition. This happens whenever your perceptual skills outstrip your verbal skills in a domain, like with recognizing faces. If regular wine drinkers are asked to recognize wine they had five minutes earlier, they can do it well, unless they’re asked to provide a detailed verbal description first! But this effect doesn’t happen to sommeliers, perhaps because they have developed a verbal ability about wines that is on par with their tasting ability.
Touching
John Bramblitt: became blind as an adult, took up painting for the first time after that. He uses puffy paint and paints from memory, a technique he has taught to both sighted and blind individuals. He asks students to close their eyes and touch an apple. Then open their eyes and paint what they felt, as an exercise. This produces very interesting results.
Cross-modal plasticity: When one sensory ability is lost, people often experience an enhancement of the other senses through co-opting of brain areas formerly devoted to that lost sense. This has been demonstrated with TMS, blind people find it really hard to understand objects from touch when the visual area is deactivated, not true of sighted individuals. Five days of blindfolding sighted individuals significantly improves touch ability and brain scans show different patterns of activation in visual area. TMS in these individuals over visual area erases new touch sensitivity, confirming that co-opted visual areas were likely responsible. However, these effects are gone after 24 hours.
Touch-based vision systems have been used successfully to restore some spatial awareness to blind people, usually something like a camera that translates visual areas into vibration patterns on the back or tongue (e.g brighter=higher frequency vibration). At first, it is experienced as consciously translating the vibrations, but later becomes just sensing the environment. It is quite low resolution though, like a very blurry camera image.
Everyday experience changes the touch representation in our brain substantially, musicians have a larger space devoted to the most necessary fingers than is average. Sensory organization in the brain can change with as little as ten minutes of doing a task.
Short term plasticity: nerve cells using connections more than they did before, often by unmasking dormant connections. This can turn into long term plasticity in which new connections are made. Plasticity can even occur in subjects who just mentally rehearse a task.
Rick Joy: deaf and blind man who uses the now rare Tadoma method (touching a speaker’s lips and vocal cords) to understand speech. He can carry on a very normal conversation this way. It used to be very common, but now there are portable devices that translate text into braille. Also, a lot of deaf blind people get cochlear implants now. Even unpracticed individuals can use the Tadoma method to understand speech better in a noisy environment. Your brain is ready to use speech info from all sources, even if you’ve never consciously attended to them before! Novice face touchers can discriminate faces and even simple facial expressions by touch.
Inverted faces are disproportionately harder to recognize than other inverted images, because we use more configuration information to recognize faces. The same thing applies to touched faces, suggesting we use similar methods in both modalities.
When you lift something, your muscle-tendon sense detect how the object resists motion in different directions, telling you a lot about the shape and weight of an object while only touching a small part of it. Our brains even use proxy touch well, for example your teeth, to figure out properties of an object.
Seeing
We’re very good at seeing faces. If a face of a known person is distorted even slightly, participants can usually tell. We detect faces twice as fast as other objects. Part of facial recognition is how faces move, you can recognize friend’s faces from 30 moving dots.
We use our own faces to recognize other people’s expressions. If participants bite down on a stick, they’re less accurate at judging the facial expressions of others. Even if your face doesn’t change, the muscles of your face show a changed electrical activity in similar muscles as the ones you saw change in a facial expression. Your own facial expressions also subtly affect your mood. There’s even some evidence that as you exercise facial muscles in certain expressions over your lifetime, the better you’ll be at expressing those expressions, which may be why participants can infer personality somewhat from looking at pictures of faces. Might also partially be a self fulfilling prophecy.
Attractive faces tend to garner both professional and personal success, and we tend to judge faces about twice as much as bodies when judging overall attractiveness. We tend to ascribe better personality traits to more attractive faces too. Participants can even rate the attractiveness of faces seen subliminally.
We inadvertently imitate what we see. We mirror the body language, facial expressions, and speech mannerisms of people we interact with. Imitation facilitates interpersonal interactions, people like you more if you mimic them. It also helps us understand other people’s body language, as shown when participants are restricted in their movement and they ability to read body language decreases. Your muscles produce motor-evoked potentials that mimic movements you see, even if you feel like you’re sitting perfectly still! This may be facilitated by mirror neurons. We’re also drawn to imitate someone’s gaze, a fact magicians use to misdirect your gaze to perform illusions.
Multisensory perception
We use lipreading as part of speech perception. What we see can even override what we hear, as in the McGurk effect. We also use non-lip parts of the face to help, like the shape of the jaw can tell us about the position of the back of the tongue. This is all done unconsciously. Even 5 month old babies are susceptible to this effect.
Conclusion
One pattern we see is that many skills we’ve long thought to be the purview of one sense is actually supported by many senses. For example, you can perceive speech by seeing, touching, and listening. We’re also often unaware of the separate sensory contributions to what we perceive. Some of our implicit processes may manifest as intuition. Regardless, it is clear that much of what we do is influenced by our unconscious perception, but as this book shows, we can make those processes conscious, and thus improve them purposefully.
Thought questions:
- The author states that scent-induced memories are often perceived as more vivid, because they are more emotional and evocative, but are not in fact more vivid. But is emotion part of what the common usage of vivid means, rather than accuracy?
- The author states that wine experts have a rich conceptual understanding that allow them to attend to tastes associated with other tastes, allowing them to more easily recognize wines and detect subtle flavors because they’re actively looking for them. This seems like it would also lead to more false positives for flavors than in novice testers. Do you think one necessarily goes with the other?
- The author presents a lot of studies as true that have not been replicated, which contributes to a nice narrative flow, but isn’t as conservative as many scientists prefer to be with data. What balance do you prefer? Do you think the balance should differ depending on if it is an academic text or a popular science book?
- One piece of evidence that is considered suggestive of human pheromone detection is the fact that axillary steroids like androstadienol, androstadienone, androstenone, androstenol, and androsterone are not active until puberty, and contribute to changes in body odor with puberty. Do you think this is good evidence that odor cues influence human behavior?