Frequency Lowering versus Speech Synthesis…
The ideal hearing aid is the goal of today’s blog. It’s a follow-on to the previous blog, in which we explained why most hearing aids are so inadequate. Here are the major sections:
– The Rocky Road of Frequency Lowering
– Examples of Frequency Lowered Speech
– The Ideal Hearing Aid
Let’s proceed to discover what may be the future ideal hearing aid.
The Rocky Road of Frequency Lowering
The previous blog ended with the “aha” idea to lower unhearable frequencies so that they become hearable. Of course, when I studied the situation I found that the idea of lowering audio frequencies to improve hearing is not at all new.
The first major review of the field in 1978 by Braida et al quotes many references on frequency lowering. They date back to 1929 (Harvey Fletcher), and work was going on even before that.
Recording and Playback
The earliest way to lower frequency was to record a voice, then play the recording back at a slower speed. That lowers the frequency in proportion to the playback speed. Unfortunately, it also slows the rate (words per minute), and that disrupts some of the cues we use to understand words. And of course, it’s not helpful for real-time hearing assistance.
Analog signal processing gave additional options. Researchers would use bandpass filters to isolate ranges of frequency. Then they would heterodyne, that is, mix the audio signal with an oscillator in a nonlinear device and filter the result to obtain frequency-shifted audio. The shifted audio would be added to the original signal, either overlapping it in frequency, or filling a gap created by more filtering.
The ultimate goal, to improve speech comprehension for people with hearing impairment, turned out to be elusive. This fussing with the frequencies caused unwanted side effects, such as:
- Clicks and chirps in the audio
- Unnatural-sounding voices due to dissonance in the harmonic frequencies
- Muddy or blurry diction.
The 1978 review concluded that frequency lowering work up to that date was not only spectacularly unsuccessful, it was also not usually conducted and reported carefully enough to allow understanding and analysis by others.
Today, semiconductor chips permit extensive digital processing in real time, built into a hearing aid. Thus many modern hearing aids incorporate frequency compression, frequency shifting or both as programmable options for the audiologist. At least six manufacturers offer such options in one or more hearing aids:
- Phonak (frequency compression, SoundRecover)
- ReSound (frequency compression, Sound Shaper)
- Siemens (frequency compression, FCo)
- Starkey (adaptive frequency lowering, Spectral iQ)
- Widex (frequency transposition, Audibility Extender)
- Oticon (frequency lowering, Speech Rescue)
Note that frequency compression hearing aids may cause music to sound out of tune because the harmonics no longer have the correct relative frequencies. If you’re a musician or a serious music listener, you might prefer an aid of another type. (Note added 12/13/21: Friend Michael tells me his ReSound aids from Costco are only intended for voice; they make music sound tinny.)
Digital Processing Limitations
There are many drawbacks to current devices that make them not nearly an ideal hearing aid:
- To protect the wearer, hearing aid adjustments are by default unavailable to the user. If the user wants any flexibility in use, as in different hearing situations, he must persuade a sometimes reluctant audiologist to unlock the controls.
- The available adjustments are so complex that some audiologists either don’t understand them, or are unwilling to take the time to find the best settings for the client.
- There’s no general agreement on how to characterize the aural condition of the patient, how to choose the right hearing aid for that individual, the right way to adjust it, and how to help the patient adapt to it. The patient has to try one model, then another, until he exhausts his patience or finds one that helps his particular hearing.
- Published studies on hearing aids have been conducted with very few participants (well under 100), and give inconsistent results.
- Different people have wildly varying reactions to the same hearing aid, even when it is adjusted by the same audiologist.
- Perhaps for some of these reasons, many elders resist or avoid the use of hearing aids. Dissatisfaction rates are as high as 47%, and the rate of hearing aid use is as low as 18%.
If you are a lucky user who has a hearing aid that suits you perfectly, congratulations! But not everyone shares your good fortune.
How To Find a Good Hearing Aid Today
Someday soon there may be an ideal hearing aid, as described later in this blog. It doesn’t exist at this moment. However, each of the hearing aids available does a good job for some patients. If you persist, you may find the one that is best for you.
Here’s how you might proceed:
- Prioritize your hearing goals: Understanding speech in a noisy place? Hearing the TV better? Hearing music more clearly?
- Which types of hearing aid would you find acceptable or preferable?
- If hearing aids totally fail to help you, are you so strongly motivated that you would consider cochlear implant surgery?
- Search your area and make a list of desirable audiologists. They should have thoughtful, positive reviews from patients. They should have a Doctor of Audiology (Au.D.) degree from a top graduate school. If they are on the staff of a top school or hospital, all the better. Study their websites, looking for competence in multiple brands of hearing aid plus an intelligent discussion of how they match the hearing aid to the patient.
- Interview your top candidates as if you were choosing a family doctor. Ask questions! Which brands of hearing aid do they use, and why? Which, if any, published protocols do they follow? How much training do their audio technicians have? What fraction of their patients buy a hearing aid and are still using it several years later?
- Do not buy unless the audiologist runs full tests without and with the hearing aid, at no cost to you, showing you how much improvement it provides.
- Ask: how are you protected if you buy an aid and you find it to be unsatisfactory after you use it?
What If That’s Not Enough?
Is this approach not progressive enough for you? Then you might want to find a clinical trial of hearing aids that’s recruiting participants close to where you live.
If you’re in need of a hearing aid, there’s no guarantee that the steps described above will satisfy you. Different brands use different proprietary signal processing techniques and there’s no easy way to find out which one may work best for you. What’s needed is an ideal hearing aid that can reliably address a wide range of hearing issues. So let’s figure out what that ideal looks like.
Examples of Frequency Lowered Speech
To understand the ideal hearing aid better, I experimented with shifting the frequency of normal speech. I will give examples that show what frequency shifting can, and cannot, do. They illustrate how frustratingly difficult it is to correct high frequency hearing deficits.
Librivox.org offers audio recordings by a variety of speakers and their recordings are in the public domain. I chose to use the first few lines of Clement Moore’s A Visit From St. Nicholas.
Santa Visits: Female Voice
Here is a sample as recorded by Annie Coleman Rothenberg, and its spectrogram:
Recall from the previous blog that seniors lose 20 dB of hearing sensitivity with each doubling of the frequency. Thus if we could reduce all the frequencies by one-half, all other things being equal, that would significantly improve speech understanding for these people. Jonas Wagner has made available a TimeStretch Player that does just this. It allows adjusting the pitch of a recording up or down by as much as 12 semitones (one octave in musical tones). And 12 semitones is a factor of two in frequency.
Here is Rothenberg’s reading when lowered by 12 semitones:
The original file apparently only extended to 11,000 Hz, so the spectrum is completely blank above 5,500.
You’ll note that Rothenberg’s voice now sounds male rather than female. In addition, it’s difficult though not impossible to make out the words she is speaking.
What about a less extreme frequency lowering? Here it is lowered by 9 semitones:
And here by 6 semitones:
The frequency halving of -12 semitones corresponds to adding 20 dB to the senior’s effective hearing. However, the sound quality is pretty shaky. The more desirable -6 semitone option adds 10 dB, which could help some people. Moreover, this option allows an older person to understand speech as well as he or she would have 12 years previously!
Santa Visits: Male Voice
What about a male voice, which is already pitched lower? Here is a sample as recorded by Peter Yearsley:
When we pitch a male voice down one octave it becomes difficult to understand:
A more acceptable shift is -9 semitones:
Or even -6 semitones:
Once again, the more desirable -6 semitone option adds 10 dB of high-frequency comprehension, which could help some people. And it allows an older person to become a dozen years younger in speech understanding.
Frequency Shifting Plus Mixing
I also experimented with dividing the vocal spectrum into high and low frequency components. I lowered the frequency of the higher ones, then added them at varying volume to the lower frequencies. However, I found that mixing two ranges, with one of them shifted, introduces artifacts into the combined sound, regardless of their relative volumes. I hear the artifacts as “chirps” but whatever you call them, they’re not a desirable feature.
The Ideal Hearing Aid
The hearing aid manufacturers will continue to study, enhance and market their devices. However, I suspect that there’s a quicker path to the ideal hearing aid.
Speech recognition has evolved to where it is widely available. You can speak to the computer you use, or to an app on your browser, and it will convert your words to text. And using an accessibility tool, you can have that text spoken by the voice of your choice. This is another form of frequency lowering: to recognize the words, then to articulate them with an artificial voice that matches the user’s hearing profile, probably at a lower frequency.
Consider: the ideal hearing aid could listen to the words around you, while re-speaking them with a pitch and a pronunciation that you can clearly hear. It would be as if a keen-hearing assistant with perfect diction were whispering in your ear to prompt you on things that you missed hearing. With an adjustable volume control, you could choose between having the artificial voice dominate, or speak in unison with the real person’s voice. For that matter, the user could have the words presented on a display, or even spoken in another language. To avoid an echo effect, the hearing aid must recognize and synthesize words very rapidly, within 50 milliseconds of the original sound.
I ran a test, speaking the Night Before Christmas text into my Macbook, set up for dictation (System Preferences – Keyboard – Dictation, Control-Control). It copied perfectly, so long as I cheated by saying “it was” instead of “’twas,” which was recognized as “close.” Then I commanded playback (System Preferences – Accessibility – Spoken Content, voice “Allison,” Option+Escape). Presto! My voice, converted from male to female, just like that! (Frequency raising, not lowering, in this case.)
Speech Recognition Limitations
Of course, nothing is easy. Some speech recognition systems work best when they are trained with a particular speaker’s voice. However, for a hearing aid, we would want a speaker-independent system.
Another limitation is that speech recognition makes more mistakes in the presence of noise. However, there are techniques available to reduce the effects of noise. In addition, algorithms are being developed to separate speech from noise under general conditions. And in the future, we can expect algorithms that learn (“A.I.“) to improve the accuracy of general-purpose speech recognition.
In case these approaches are not enough, the ideal hearing aid could offer additional options: if each ear has a microphone, the sounds can be combined with controlled time delay, to enhance words coming from a particular direction or individual. Geolocation by triangulation. Moreover, the capabilities of such a system could be expanded by adding user-selectable frequency shifting, plus a button to replay the last 30 seconds of speech. Ideally, if several people are talking, the hearing aid would synthesize different-sounding voices for each speaker.
At present, speech recognition is not yet able to obsolete all other forms of hearing aid. However, in the foreseeable future it’s possible that speech recognition plus voice synthesis will emerge as a general-purpose ideal hearing aid.
If you or a loved one are dealing with hearing loss, there is hope! Today, with persistence and luck you may find a hearing device that works well for you. But someday soon, a more general solution will come available that may be truly an ideal hearing aid.
– Pig from Barbara Barbosa, wildcat from David Selber on pexels.com
– World Has Ears from liftarn on openclipart.org
– ‘Twas the Night Before Christmas readings from librivox.org
– Spectrograms from Edward Ball, Michael Ruiz and Boris Smus on academo.org
– Pitch shifts from TimeStretch Player by Jonas Wagner
– mp3 edits using Audacity.app
Hey Art ! Have enjoyed reading your 2 posts re Hearing Aids and have to admit Kaye and I have joined the club. I researched providers before jumping in and will cut to the chase. Our local COSTCO beat the big name guys (Maricale Ear, Beltone, Livingston, etc,) hands down. Excellent testing facility staffed with pro audiologists, offers top name brands (we settled on ReSound) at reasonable pricing and follow-up service which is outstanding.
As for the sound quality, I couldn’t be more pleased. I use my i12 pro with the ReSound app to control and tweek my hearing needs. The Blue Tooth sync enables me to listen to stations of stations choosing which is great when pre-occupied with various activities. The only negative I have about wearing hearing aids is MANDATED WEARING OF FACEMAKS ! Mask ties and the component that lies behind your ear are not compatible friends. When pulling off the mask, the ear piece tends to come off too resulting in upsetting consternation.
Besides that inconvenience, they make life easier in hearing clearly !
Thanks for the great posting for us elderly and we really appreciated your annual Christmas letter. I thought the 2021 version was your Best yet.
Hello Joe, it’s great to hear from you again, and good to know that our letter made it through the uncertain mail service.
Costco is certainly a good source, and if you are pleased with your aids then you have won the comfort lottery! I’m glad that you can control the parameters yourself, that will certainly improve your experience. I was concerned that aids using frequency compression such as ReSound might deliver distortion and sour notes when listening to music. If you have not had this problem, perhaps it’s not as big a problem as I feared.
I recommend for your consideration a face mask that doesn’t attach to the ears at all. Nola has one with has elastic bands that simply go around the head, one above the ears and one on the neck. The mask is somewhat stiff, it may or may not be an N95 type. When we take a long trip such as to Hawaii, options like this are useful because ears get really sore holding a mask on during a long flight.
Warm holiday wishes to you!
Just as my reply was being transmitted I noticed that something was spelled wrong. Maybe I can hear better than I can type. You have my permission and my encouragement to correct my typo before posting.
Jay, you don’t have a problem with your hearing. Your problem is that you are just too funny! Have a great holiday! – Art
What say?? Eh?
I was able to hear “the Night Before Christmas” without a problem. I’m good.
Thank you for that. Most of the info flies right over my head … but, some of the funniest stuff I’ve ever heard and it gets me thinking, which is important.
So, how are Nola’s “How to Speak In 3 Semitones Lower” lessons coming along? And then I was thinking, “If you could get Nola to speak in 12 semitones lower it would be like sleeping with Al Jolson.
I wish you’d focus on the this problem. Inventing a microwave oven which would re-heat my coffee to the exact moment till just before it explodes in the microwave. I’m sure you’ll need some info on how much milk I put in my coffee (3 ounces in a 16 ounce paper cup.) Do you think you’ve got a handle on that?
More to follow on your post … but I’ve got to run off to the swimming pool at school and do some homework for school.
Miss you a ton. Maybe Nola’s kids will move out this way… “Mom and Dad, we’re moving to California.”
Don’t forget Christmas newsletter. How’s that coming along?
Hi Steve, by the time I got home to approve your comment, I was glad to read that you received our newsletter.
Concerning your coffee, perhaps you could get a microwave oven with a temperature sensor. You would set the sensor for your perfect coffee temperature and it would turn the oven off when your cup reaches that temperature. After you experiment with a few cups of coffee, cleaning up the explosions, you would learn exactly which setting makes you happiest. Then you would simply need to make sure to always use the same cup with exactly the same amount of coffee and milk.
Hope your holidays are the merriest!
Art – A beautiful two-part discussion of hearing aid technology. Although I have significant loss in upper ranges, I’m finding that just boosting those ranges is still adequate. So I don’t know what percentage of your readers will find the need for more radical interventions. Another intervention that ameliorates hearing problems is captions. I now have them on for all home video, and they help fill in any glitches in my hearing — especially with today’s muddy, overwrought soundtracks.
Most of all, your analysis shows your mind is as acute as ever, which is an added delight. Happy Holidays!
Thanks for your comments and for your good wishes, Bob! I wish you a wonderful 2022 with continued hearing (and captioning) success… Art