Last year I had a minor medical problem, more of a bother, really. It’s one for which a few supplements have been shown to help. My doctor said I could try “Supplement X, Y, or Z.” I didn’t do anything at the time I talked to him because the problem hadn’t been happening right then.
But then the problem started bugging me. I had it for about 10 days in a row. I thought, “Hey, this would be a great time to try Supplement Y.” I wrote down in my calendar that I had had the problem for those 10 days, and wrote down the day I started the supplement. And the problem improved massively for the next 14 days.
Yay, it worked, right?
No. I didn’t have enough evidence to conclude that. What I could have been seeing was regression to the mean.
When problems vary in intensity, people tend to try interventions during a “bad” spell. They call the doctor; ask their neighbor; or start a supplement, diet, or exercise regime.
That’s exactly what I did. I started a supplement when my problem was bad. But if a variable problem has been worse than average for a period of time, what usually happens next? It will likely get better. It’s variable. It is so easy to attribute the good spells to whatever intervention we might be trying. We want things to work, and we are wired to grab at such correlations.
I had 10 days of data before I started the supplement, but it was 10 days of **extreme** data. And 10 days is pretty short in any case. (I made up the data on the graph for the first few weeks as a demonstration. It represents the low level of the problem that mostly passed under my radar.)
Regression to the mean is why people frequently will say of an intervention that it “worked for a while then stopped.” When we have that particular experience, the intervention likely didn’t help at all. We were only experiencing the effects of regression to the mean. Things got better because they were very unlikely to get worse. But we are humans, and we assign causation at the drop of a hat.
Health problems that are especially vulnerable to this false attribution include chronic conditions like pain from arthritis or headaches, allergies from unknown triggers, fatigue, and mild to moderate gastrointestinal problems. Any painful or bothersome condition that comes and goes but rarely reaches an acute level. (Because that’s when we go to the doctor for a more aggressive intervention.)
If I want to improve my chances of really knowing whether a supplement helps for a chronic, variable condition, I should take data for as long as possible before starting the supplement. (Almost none of us do this; more on taking data below.) I should start it at a random time. After I start the supplement, I should take it for a preassigned number of days. If there are studies supporting the use of the supplement, I should record data for at least as long as the studies did. I should decide the number of days beforehand and stick to it. (This removes the chance of stopping at a fortuitous time for the data.) I should record my data religiously. After the period is over, I should go off it again for the same amount of time and keep taking data. This is called a reversal design, and give you two chances to look for a difference due to the intervention: when you start it and when you stop it.
Obviously, I am not recommending you ever go off a doctor-recommended prescription medication as an experiment. The reversal design should only be implemented when it is safe to do so. In my situation, my doctor had encouraged me to experiment with the supplements.
Implementing a better structure would have given me a better chance of knowing whether the supplement had an effect. But even then, it would not tell me for sure. Besides regression to the mean, which still could be in play, there are a dozen other reasons and biases that could make it appear that the supplement was effective.
Regression to the mean is both a statistical event and a cognitive fallacy. The statistical event is as described above. When an entry in a time series is at an extreme value, the most likely thing to happen next is for it to fall back toward the mean, or average. This is not the same as the “law of averages,” which is itself a fallacy. Regression to the mean has to do with values over a span of time, and what happens after an extreme value.
If images are helpful to you, check out this article about regression to the mean that includes graphs and a couple of really clear scenarios. Also, it will let you know that some businesspeople and marketers know about regression to the mean and use it to sell products or services.
Regression to the Mean and Our Dogs
The above is not a made-up story. It did happen to me. I’m sharing it because we do the same thing with our dogs. It’s a cautionary tale about how damn easy it is for us to assign causation when it’s not really there.
It happens all the time in life. How many times on a dog group have you seen someone write, “Yes, Acme Supplement is great, but after a while, it did stop working.” Or “I used Smoke’em Powder and it worked great at first. It was worth it for that period, even though it stopped working as my dog’s condition deteriorated.”
Again, the way regression to the mean works is that we tend to consider interventions—diets, exercises, supplements—when the condition is at its worst. For example, we know our dog has some arthritis that is pretty well controlled most of the time with a prescription drug. But the arthritis is getting worse. Instead of going back to the vet, we may try a supplement that our neighbor told us about. And when do we tend to try it? When our dog is acting like she is in more pain than usual. When we are having extreme values in the time series.
So we start her on Acme Supplement. Lo and behold, over the next few days, her pain appears to lessen. We automatically attribute it to the supplement. That is how our brains work, noticing correlations and leaping to assigning causation. Between the regression to the mean fallacy and the owner placebo effect (since our dogs may not actually be feeling better—we just wish they were), we feel certain we have solved the issue. Our dog feels better and it’s because of Acme!
These two biases together keep unproven supplement companies in business and whole product lines lucrative. All we have to do is try a certain product when the condition is at its worst. Then when the condition naturally improves, we are certain it is due to the product.
An interesting twist is that even after some time has passed and the product doesn’t seem to be working “anymore,” we don’t doubt its original efficacy. We rarely go back and say, hmmm, maybe it never worked after all! What we say, and tell others, is that it worked, but then stopped working. We often keep recommending it!
You will read versions of this over and over. You’ll see it in dog health, in suggestions for dealing with fearful dogs, in dog training of all sorts, in human health, and in other practices.
Owners of dogs with canine cognitive dysfunction are particularly vulnerable to regression to the mean with regard to interventions. Not only do the dog’s symptoms change from day to day, but the owners know it is a terminal condition and the worsening of the dog’s condition can’t be stopped. If the dog has a few good days in a row, it serves as a beacon of hope. That’s human nature.
Take Data and Be Skeptical of Assumptions
It’s normal to start to pay attention to a problem when it reaches a certain threshold. So it’s rare that we are going along, for instance, keeping a journal about how often we have headaches if we are not often troubled by them. But that’s exactly the type of data we need. We can’t go back in time when our dog develops a problem, but we can start keeping track immediately and also record anything we remember from the past.
Blanche Axton, a champion data recorder about her dogs’ health, wrote a super helpful article about this: “The Importance of Tracking Changes in Dog Behavior.” She is my inspiration.
There are lots of tracking tools available for those of us who like to use technology. When my dog Zani started exhibiting extremely fearful behaviors for which I couldn’t identify a trigger, I started taking data. I used a Mac journaling app called Day One. I created a template that included the behaviors I tracked. With the press of a button, I could pull up a blank template to fill out at the end of the day. This made it as easy as possible to write every day. I could add other notes and often included photos.
My system was primitive compared to what Blanche describes in her article. If you need to start tracking a health or behavioral problem with your dog, I suggest you take a look at hers, my simple lists below, and create something in between in a format that works for you.
Some of the things I tracked for Zani’s mysterious behavior change and included on the template were:
- slinking around
- unhappy facial expression, ear set
- staying at back door
- whining at front door
- avoiding bedroom
- avoiding kitchen when I am sitting at table
- coming to me repeatedly for help, then leaving
- refusing food
- getting in my lap trembling
- tail up
- enjoying training
- came in the bathroom while I showered
- did agility enthusiastically
- slept in bedroom
- hanging out with the rest of the dogs
- lying in dog bed on couch (instead of hiding in crate)
These items were mostly hard to quantify. For most of them, I just answered yes/no for the day, then sometimes elaborated with notes. If you are tracking medical symptoms, you’ll probably need fewer categories than I needed for Zani’s mental breakdown, but you may need more detail about circumstances. Keep in mind Blanche’s system of noting things like times of day and other environmental factors and how much that helped. Perhaps if I had done that, I would have eventually found out what was freaking out my little dog.
Another thing to do is to be skeptical. I knew about regression to the mean, but I forgot and still thought the best time to start a supplement was during a symptom flareup! I think I’ve got the lesson a little better firmed in my mind now, but at the same time I need to remember that there is part of my brain that will grab at correlations whether I want it to or not.
Read the Kahneman book. It is full of examples about our cognitive biases and gives solid instructions about how to fight back against them.
And finally, regression to the mean can keep us messing around with things that aren’t really working. It’s a good reason to take data and be skeptical of our own “intuitive” responses that say something is working. We might be doing this while neglecting a proven intervention that can really help.
My Regression to the Mean Experience
In my case, it is unlikely that Supplement Y worked. After the first dramatic two weeks when my problem disappeared, it came back at about its normal levels (the mean). But then later when I finished the bottle and stopped the supplement, there was no uptick in the severity of the problem. It just continued in its on-again, off-again manner. I would need more carefully planned and gathered data to give me a better idea of whether Supplement Y helped me with my problem. But at this moment, despite those glorious few days when it was at a very low level, it probably didn’t help.
Copyright 2020 Eileen Anderson