Last Updated on April 17, 2024 by Art

(Sleep #3 of 3) Test Driving an Option to Improve Sleep…

SleepScore app is the focus of today’s blog. The previous two blogs described sleeping sensing devices and apps, then how healthy sleep relates to risk of dementia. This, the third of three, relates my personal experience with this particular app.

SleepScore App Description & Review

After considering the other options, in February 2023 I started tracking my sleep using the SleepScore app. Unlike other apps, the free version of SleepScore is actually useful: unless your sleep is suffering and you’re desperate for someone to guide you, the free app gives plenty of useful data about your sleep.

            Data in the App

I determined that I would keep track of factors that I could control, and then collect the sleep data that resulted. Obviously, I made sure to have a good sleeping environment: the right temperature, no noise, comfortable bedding, and adequate time to sleep.

            Input Variables

With those factors OK, there were still elements under my control, which we might call “input variables”:

• Steps walked (lower body exercise)
• Time playing the piano (mental and upper body exercise)
• Alcohol consumption (in standard drinks)
• Caffeine consumption (not relevant for me, since I avoid caffeine except in my breakfast coffee)
• Water consumption (in 12-ounce bottles)
• Sound before sleeping: type and duration (I tried a dozen different types, each 3.5 to 10 minutes long)
• Melatonin before sleeping (I tried only a very small dose, 0.5 mg)
• Time of going to bed
• Time of getting out of bed

When you go to sleep, the SleepScore app asks you to fill out a questionnaire:

• How sleepy do you feel? (Not at All to Extremely)
• How many cups of caffeine did you drink? (0 to 8+)
• How many alcoholic drinks did you consume? (0 to 8+)
• How many minutes did you exercise? (0 to 70+)
• How stressed do you feel? (Not at All to Extremely)

As many reviewers point out, these are stupid questions. After all, the effect of caffeine, alcohol and exercise depend on many factors. Was that coffee or tea or Coca Cola that you drank? Did you drink coffee right before bedtime, or early in the day? How much alcohol was in those drinks, and were they right before bedtime? And so forth. So I never answer their questions. I will keep my own data, thank you, in a form that I consider relevant and repeatable for me.

            Output Variables

When you get up in the morning, you turn off the app, which then gives you all kinds of data:

• Start time
• End time
• Average number of breaths per minute
• Time to fall asleep
• Total time in light sleep
• Total time in deep (slow-wave) sleep
• Total time in REM sleep
• Total time spent awake
• Number of times the sleeper woke up during the night

We could call these the “output variables” since they are not directly under our control. In addition, there’s a graph showing sleep state as a function of time, throughout the night.

            SleepScore App Outputs

Here’s a sample of the screens that the app displays, as saved on my iPhone:

As you see, the app awards points in each category. My data shows that to get maximum score in each category you need:

• Sleep duration >7 hours
• Time to fall asleep <16 minutes
• Light sleep time >4.5 hours
• REM sleep time between 1.5 and 2 hours: in other words, the app wants me to get enough but not too much REM sleep

In the category of deep sleep, the highest I have ever scored is 19 out of a possible 20, with 1.37 hours of deep sleep. If there is a maximum that the app doesn’t want me to exceed, I have not yet found it.

The wake time score is a mystery: I have scored the maximum of 10 for a variety of wake times between 0.6 and 2 hours; however, sometimes wake times between 1.5 and 2 hours score a 9 or even an 8. The app may be computing the ratio of wake time to total sleep, or some other factor, when it comes up with a score.

            Trust but Verify?

The companies who sell these hundreds of sleep apps are in business to make money. In the case of SleepScore, the parent company is ResMed, whose main business is selling CPAP machines for sleep apnea sufferers. Therefore, one of their goals may be for the SleepScore app to identify apnea patients who will then use ResMed products. And because they have many competitors, proprietary issues greatly limit what they say about their app.

However, this company presents a sincere statement on their “Why Trust Us?” page. More importantly, unlike many app vendors, they provide genuinely useful data on a totally free basis. That persuades me to believe that, despite the necessary business motivations, they genuinely want to help people have healthier sleep. So with the help of this verification, I’m willing to place some trust in their data.

            Accuracy of Sleep Staging

This figure from ResMed compares the SleepScore App (Sonar) with SleepScore Max (RF) and Polysomnography (PSG).

This chart shows lengths of time during which either of the sensors (sonar, RF) disagree with polysomnography (PSG). These range from 10% to 35% of the total time, with the largest disagreement occurring when measuring light sleep. However, note something interesting: the total time spent in each sleep state (that is, adding together the colored segments of each bar) shows better agreement with PSG.

Here’s what I believe is happening: when sleep state changes, PSG detects it immediately. However, there’s a time delay before the sonar and RF sensors accumulate enough data to register the change. For example, when the user enters deep sleep, there’s a delay before the sensors detect the change; similarly, when the user leaves deep sleep, there’s a similar time delay in the sensor data. The net result is that although the sensors disagree with PSG large portions of the time, they are somewhat more accurate in logging the total time spent in each sleep state.

            SleepScore App Shortcomings

The SleepScore App does have some shortcomings:

• The company does not reveal how their app extracts sleep state information from the sonar signals (proprietary algorithm, no doubt).
• Nor do they explain how they award points in the different sleep categories.
• Nor do they explain whether they adjust the app’s point scores according to the age and gender of the user.
• The app has no provision for scoring naps, or for combining nap data with nighttime data.
• Background noise can render this app useless. In August 2023 while I was on a cruise ship, the rumble of ship sounds made the app think I was largely awake, and it scored my sleep very low, between 50 and 70 points. And in October while I was in Maui, with a room air conditioner running through the night, the app could not detect my presence part of the time, leading to another instance of low sleep scores. Perhaps the company’s RF device (SleepScore Max) would not be disrupted in situations like these.

Art’s SleepScore App Analysis

Here’s a report on how I analyzed the data, and what I concluded. I don’t expect the results to apply to anyone else, but at least they illustrate what can and cannot be extracted from this app.

I began by eliminating all the data on the cruise ship and in Maui since it was disrupted by background noise. This left me with 239 nights of data, which I think should be enough to conclude something.

            Let’s Try AI!

When I totaled them up, each of my 239 nights of data included 31 input variables and 20 output variables. So I figured, could an artificial intelligence tool give me predictions or insights that I can’t extract on my own?

I consulted some review articles and settled on an AI tool called Polymer Search as an interactive tool that is optimized to analyze tables of data. So I opened a trial account with them.

Unfortunately, it was a waste of time. The responses the tool generated were simplistic and useless. It advised me to average some scores, or analyze their distribution. It also picked out a couple of random columns of data and suggested that I compare them. In the context of my task, I scored the tool -1 out of a possible 10 and deleted my account.

            Manual Analysis

Evedently, there’s no free lunch, so I stuck with manual analysis.

            Correlations

I started by computing correlations between the 11 principal input variables and 10 output variables. Nine of the outputs are scores provided by the app: SleepScore, sleep duration, time to fall asleep, light sleep, deep sleep, REM sleep, wake time, mind score and body score. The tenth is sleep consolidation, which I computed from the other data. None of the correlations were above 0.26. I classified the correlations as moderate (absolute value of at least 0.2) and mild (0.15).

The moderate correlations suggested that I should go to bed later, sleep later, spend more time in bed, and avoid melatonin (which increased my wake time, thus reducing my sleep consolidation). The mild correlations suggested more piano, less walking and less water. Irrelevant factors were amount of alcohol, drive time, and sound before sleep. So I gained some useful information for improving my sleep.

            Standard Deviations

I then computed the standard deviation (amount of fluctuation) in the output variables. I looked at individual days and also results averaged over multiple days. This accounts for the possibility that sleep performance fluctuates night to night, but if you are short on some portion of your sleep on one night, you’ll tend to make it up on the following nights. Here’s a plot of the data that shows how the variability decreases with multi-night averaging:

This is what I learned from this chart and its underlying data:

• Deep sleep time and the REM sleep time fluctuate strongly, typically 40% to 50%, from night to night. However, the fluctuations drop by one-half when you average them over five-night periods.
• SleepScore and sleep consolidation fluctuate much less, only about 10% night to night. The SleepScore fluctuations drop to one-half when averaged over 10 nights, the consolidation drops when averaged over 17 days. Thus both of these measures are relatively stable from night to night.

I also looked at correlations between the data on one night and the data one to four nights later. Here’s what I learned:

• A higher SleepScore is accompanied by higher SleepScore on nearby days, especially two nights before and after.
• High deep sleep score tends to bring a high score two nights later.
• A higher REM score does not affect the REM score on nearby days.
• Higher sleep consolidation is accompanied by higher score on all other nearby nights.

What did I learn? Mainly, not to be too concerned about one or two nights with poor deep sleep or REM scores, because they are quickly compensated on subsequent nights.

            Best Nights

I also looked at my “best nights” to see what I could learn from their data. Specifically, looked at the scores for SleepScore, Sleep Consolidation, Body Score (which relates to deep sleep) and Mind Score (which relates to REM sleep). I looked at the nights in the top 20% of these scores (which were respectively, 90, 78.7%, 74 and 96) to see how the input variables differed from all the other nights. And I repeated this analysis for the top 50% and the top 80%.

My conclusions were that to attain a “best night” I should go to bed by 11:40 or 11:45 pm, and sleep until 9 am. A bit of alcohol may help. I don’t need more than about 3000 steps, nor more than about 15 minutes of piano. So this didn’t add much to my advice to myself.

            General Conclusions

My bottom line from all this is that for the best sleep, I should:

• Get to bed before midnight
• Allow at least 9 hours in bed
• Play 15 minutes of piano regularly
• Walk daily, but not excessively
• Drink a bottle or two of water daily
• Avoid melatonin
• And not worry much about night-to-night variations.

These conclusions apply to me, and I see no reason that they should apply to you, since sleep is a very individual thing.

Note added 4/17/24: As I continue to monitor my sleep, I learned two more things:

• If I start sleep by lying on my right side, I fall asleep more quickly.
• Sometimes I wake during the night with a slight urge to pee, not strong enough to make me get out of bed. If I choose to get up and pee, I often gain an additional increment of deep sleep after I return to the bed.

I also read two interesting articles to offer for your consideration:

– An article by journalist Michael Jorrin (“Doc Gumshoe”) on sleep apnea: Follow-up on Sleep

– An article suggesting a slow deep breathing exercise to treat insomnia: Self-Regulation of Breathing as an Adjunctive Treatment of Insomnia

Is the SleepScore app broadly useful? The answer for me is “yes,” even in its free version. I did not test the app’s advice for people who hate their sleep, since I’m not one of those. However, for someone who wants to improve an already satisfactory sleep experience, I think this app is worth your consideration.

Image Credits:
– SleepScore cartoon from SleepScore app video
– Other images from data at SleepScore.com, its app, and Art’s analysis