Last Updated on December 13, 2021 by

Frequency Lowering versus Speech Synthesis…

The ideal hearing aid is the goal of today’s blog. It’s a follow-on to the previous blog, in which we explained why most hearing aids are so inadequate. Here are the major sections:
– The Rocky Road of Frequency Lowering
– Examples of Frequency Lowered Speech
– The Ideal Hearing Aid

Let’s proceed to discover what may be the future ideal hearing aid.

The Rocky Road of Frequency Lowering

The previous blog ended with the “aha” idea to lower unhearable frequencies so that they become hearable. Of course, when I studied the situation I found that the idea of lowering audio frequencies to improve hearing is not at all new.

The first major review of the field in 1978 by Braida et al quotes many references on frequency lowering. They date back to 1929 (Harvey Fletcher), and work was going on even before that.

            Recording and Playback

The earliest way to lower frequency was to record a voice, then play the recording back at a slower speed. That lowers the frequency in proportion to the playback speed. Unfortunately, it also slows the rate (words per minute), and that disrupts some of the cues we use to understand words. And of course, it’s not helpful for real-time hearing assistance.

            Analog Processing

Analog signal processing gave additional options. Researchers would use bandpass filters to isolate ranges of frequency. Then they would heterodyne, that is, mix the audio signal with an oscillator in a nonlinear device and filter the result to obtain frequency-shifted audio. The shifted audio would be added to the original signal, either overlapping it in frequency, or filling a gap created by more filtering.

The ultimate goal, to improve speech comprehension for people with hearing impairment, turned out to be elusive. This fussing with the frequencies caused unwanted side effects, such as:

• Clicks and chirps in the audio
• Unnatural-sounding voices due to dissonance in the harmonic frequencies
• Muddy or blurry diction.

The 1978 review concluded that frequency lowering work up to that date was not only spectacularly unsuccessful, it was also not usually conducted and reported carefully enough to allow understanding and analysis by others.

            Digital Processing

Today, semiconductor chips permit extensive digital processing in real time, built into a hearing aid. Thus many modern hearing aids incorporate frequency compression, frequency shifting or both as programmable options for the audiologist. At least six manufacturers offer such options in one or more hearing aids:

• Phonak (frequency compression, SoundRecover)
• ReSound (frequency compression, Sound Shaper)
• Siemens (frequency compression, FCo)
• Starkey (adaptive frequency lowering, Spectral iQ)
• Widex (frequency transposition, Audibility Extender)
• Oticon (frequency lowering, Speech Rescue)

Note that frequency compression hearing aids may cause music to sound out of tune because the harmonics no longer have the correct relative frequencies. If you’re a musician or a serious music listener, you might prefer an aid of another type. (Note added 12/13/21: Friend Michael tells me his ReSound aids from Costco are only intended for voice; they make music sound tinny.)

           Digital Processing Limitations

There are many drawbacks to current devices that make them not nearly an ideal hearing aid:

• To protect the wearer, hearing aid adjustments are by default unavailable to the user. If the user wants any flexibility in use, as in different hearing situations, he must persuade a sometimes reluctant audiologist to unlock the controls.
• The available adjustments are so complex that some audiologists either don’t understand them, or are unwilling to take the time to find the best settings for the client.
• There’s no general agreement on how to characterize the aural condition of the patient, how to choose the right hearing aid for that individual, the right way to adjust it, and how to help the patient adapt to it. The patient has to try one model, then another, until he exhausts his patience or finds one that helps his particular hearing.
• Published studies on hearing aids have been conducted with very few participants (well under 100), and give inconsistent results.
• Different people have wildly varying reactions to the same hearing aid, even when it is adjusted by the same audiologist.
• Perhaps for some of these reasons, many elders resist or avoid the use of hearing aids. Dissatisfaction rates are as high as 47%, and the rate of hearing aid use is as low as 18%.

If you are a lucky user who has a hearing aid that suits you perfectly, congratulations! But not everyone shares your good fortune.

            How To Find a Good Hearing Aid Today

Someday soon there may be an ideal hearing aid, as described later in this blog. It doesn’t exist at this moment. However, each of the hearing aids available does a good job for some patients. If you persist, you may find the one that is best for you.

Here’s how you might proceed:

• Prioritize your hearing goals: Understanding speech in a noisy place? Hearing the TV better? Hearing music more clearly?
• Which types of hearing aid would you find acceptable or preferable?
• If hearing aids totally fail to help you, are you so strongly motivated that you would consider cochlear implant surgery?
• Search your area and make a list of desirable audiologists. They should have thoughtful, positive reviews from patients. They should have a Doctor of Audiology (Au.D.) degree from a top graduate school. If they are on the staff of a top school or hospital, all the better. Study their websites, looking for competence in multiple brands of hearing aid plus an intelligent discussion of how they match the hearing aid to the patient.
• Interview your top candidates as if you were choosing a family doctor. Ask questions! Which brands of hearing aid do they use, and why? Which, if any, published protocols do they follow? How much training do their audio technicians have? What fraction of their patients buy a hearing aid and are still using it several years later?
• Do not buy unless the audiologist runs full tests without and with the hearing aid, at no cost to you, showing you how much improvement it provides.
• Ask: how are you protected if you buy an aid and you find it to be unsatisfactory after you use it?

            What If That’s Not Enough?

Is this approach not progressive enough for you? Then you might want to find a clinical trial of hearing aids that’s recruiting participants close to where you live.

If you’re in need of a hearing aid, there’s no guarantee that the steps described above will satisfy you. Different brands use different proprietary signal processing techniques and there’s no easy way to find out which one may work best for you. What’s needed is an ideal hearing aid that can reliably address a wide range of hearing issues. So let’s figure out what that ideal looks like.

Examples of Frequency Lowered Speech

To understand the ideal hearing aid better, I experimented with shifting the frequency of normal speech. I will give examples that show what frequency shifting can, and cannot, do. They illustrate how frustratingly difficult it is to correct high frequency hearing deficits.

Librivox.org offers audio recordings by a variety of speakers and their recordings are in the public domain. I chose to use the first few lines of Clement Moore’s A Visit From St. Nicholas.

            Santa Visits: Female Voice

Here is a sample as recorded by Annie Coleman Rothenberg, and its spectrogram:

Recall from the previous blog that seniors lose 20 dB of hearing sensitivity with each doubling of the frequency. Thus if we could reduce all the frequencies by one-half, all other things being equal, that would significantly improve speech understanding for these people. Jonas Wagner has made available a TimeStretch Player that does just this. It allows adjusting the pitch of a recording up or down by as much as 12 semitones (one octave in musical tones). And 12 semitones is a factor of two in frequency.

Here is Rothenberg’s reading when lowered by 12 semitones:

The original file apparently only extended to 11,000 Hz, so the spectrum is completely blank above 5,500.

You’ll note that Rothenberg’s voice now sounds male rather than female. In addition, it’s difficult though not impossible to make out the words she is speaking.

What about a less extreme frequency lowering? Here it is lowered by 9 semitones:

And here by 6 semitones:

The frequency halving of -12 semitones corresponds to adding 20 dB to the senior’s effective hearing. However, the sound quality is pretty shaky. The more desirable -6 semitone option adds 10 dB, which could help some people. Moreover, this option allows an older person to understand speech as well as he or she would have 12 years previously!

            Santa Visits: Male Voice

What about a male voice, which is already pitched lower? Here is a sample as recorded by Peter Yearsley:

When we pitch a male voice down one octave it becomes difficult to understand:

 

A more acceptable shift is -9 semitones:

Or even -6 semitones:

Once again, the more desirable -6 semitone option adds 10 dB of high-frequency comprehension, which could help some people. And it allows an older person to become a dozen years younger in speech understanding.

            Frequency Shifting Plus Mixing

I also experimented with dividing the vocal spectrum into high and low frequency components. I lowered the frequency of the higher ones, then added them at varying volume to the lower frequencies. However, I found that mixing two ranges, with one of them shifted, introduces artifacts into the combined sound, regardless of their relative volumes. I hear the artifacts as “chirps” but whatever you call them, they’re not a desirable feature.

The Ideal Hearing Aid

The hearing aid manufacturers will continue to study, enhance and market their devices. However, I suspect that there’s a quicker path to the ideal hearing aid.

Speech recognition has evolved to where it is widely available. You can speak to the computer you use, or to an app on your browser, and it will convert your words to text. And using an accessibility tool, you can have that text spoken by the voice of your choice. This is another form of frequency lowering: to recognize the words, then to articulate them with an artificial voice that matches the user’s hearing profile, probably at a lower frequency.

Consider: the ideal hearing aid could listen to the words around you, while re-speaking them with a pitch and a pronunciation that you can clearly hear. It would be as if a keen-hearing assistant with perfect diction were whispering in your ear to prompt you on things that you missed hearing. With an adjustable volume control, you could choose between having the artificial voice dominate, or speak in unison with the real person’s voice. For that matter, the user could have the words presented on a display, or even spoken in another language. To avoid an echo effect, the hearing aid must recognize and synthesize words very rapidly, within 50 milliseconds of the original sound.

I ran a test, speaking the Night Before Christmas text into my Macbook, set up for dictation (System Preferences – Keyboard – Dictation, Control-Control). It copied perfectly, so long as I cheated by saying “it was” instead of “’twas,” which was recognized as “close.” Then I commanded playback (System Preferences – Accessibility – Spoken Content, voice “Allison,” Option+Escape). Presto! My voice, converted from male to female, just like that! (Frequency raising, not lowering, in this case.)

            Speech Recognition Limitations

Of course, nothing is easy. Some speech recognition systems work best when they are trained with a particular speaker’s voice. However, for a hearing aid, we would want a speaker-independent system.

Another limitation is that speech recognition makes more mistakes in the presence of noise. However, there are techniques available to reduce the effects of noise. In addition, algorithms are being developed to separate speech from noise under general conditions. And in the future, we can expect algorithms that learn (“A.I.“) to improve the accuracy of general-purpose speech recognition.

In case these approaches are not enough, the ideal hearing aid could offer additional options: if each ear has a microphone, the sounds can be combined with controlled time delay, to enhance words coming from a particular direction or individual. Geolocation by triangulation. Moreover, the capabilities of such a system could be expanded by adding user-selectable frequency shifting, plus a button to replay the last 30 seconds of speech. Ideally, if several people are talking, the hearing aid would synthesize different-sounding voices for each speaker.

At present, speech recognition is not yet able to obsolete all other forms of hearing aid. However, in the foreseeable future it’s possible that speech recognition plus voice synthesis will emerge as a general-purpose ideal hearing aid.

If you or a loved one are dealing with hearing loss, there is hope! Today, with persistence and luck you may find a hearing device that works well for you. But someday soon, a more general solution will come available that may be truly an ideal hearing aid.

Credits:
– Pig from Barbara Barbosa, wildcat from David Selber on pexels.com
– World Has Ears from liftarn on openclipart.org
– ‘Twas the Night Before Christmas readings from librivox.org
– Spectrograms from Edward Ball, Michael Ruiz and Boris Smus on academo.org
– Pitch shifts from TimeStretch Player by Jonas Wagner
– mp3 edits using Audacity.app