When Is Citing a Research Study Not Enough?

Answer: Almost always. One study is usually flimsy evidence. What we need to consider is the bulk of the research. I’ll explain.

Most of the online requests for studies I see are from people who want to support their points of view in online arguments. Others are investigating a health or behavior condition that has to do with their own dog. Some need references for a position paper on dog training or another aspect of care. There are also people who are delving deep into an issue for reasons of education or scholarship. But usually these people don’t need that much help.

Requests are almost always couched as follows:

“Is there a study that shows XYZ?”

This is human. We believe something, either from a perspective of faith or a review of the evidence. We want to bolster our belief with stronger evidence. But thinking we can do this with one study is based on a misunderstanding of how science and research work. In order to find strong evidence, we need to view any study in the context of the other research related to that topic.

There are plenty of contradictory studies in the canon. You can often find one that supports your position even if it’s wrong. It’s only over time that the best evidence floats to the top. And it takes an expert to assess that evidence.

The most recent study is not necessarily definitive. In fact, recent studies should be treated with healthy skepticism. Even when they are building on previous research, there has not been time to replicate or contradict their findings.

All this leaves us with some problems and challenges.

What’s Better Than One Study?

Wouldn’t it be cool if there were a way to get an expert’s view of a study or a set of studies? To get an educated opinion about them? Well, there is a way. Experts tend to write books and articles. Here are three types of publications that will help the reader get a broad sense of a topic. Citing one of these publications is usually superior to picking out a single study.

A sampling of learning theory research books

  • Textbooks, depending on the level, cover a broad view of a field of study or topic. Good ones provide the standard research citations for every subtopic they discuss. They are almost always more appropriate for “winning an argument” than a single study. That’s because the author will cover all views and note which have the most supporting evidence. See Example 1 below.
  • Scholarly compilations are based on a large topic within a field of study. Usually, world experts are asked to contribute an article or chapter on one aspect of the topic. For example, the red book in the picture above is Operant Learning: Areas of Research and Application and has chapters by Azrin, Sidman, and other heavy hitters. Some of it has been superseded as time passed but it is still a great reference for the classic research.
  • Review articles summarize the research on a certain topic up to the current date. An example is James McGaugh’s article on memory consolidation: “Memory: A Century of Consolidation.” If you take a look at that on Google Scholar, you’ll see that it has been cited several thousand times by other authors.

These three types of publications provide the views of experts. They can tell us which studies have stood the test of time, been replicated, or been expanded on. They can tell us when the research took a wrong turn. They can tell us what new research to take a look at, and they do it without the sensationalist headlines we often get in blog posts.

Here are a couple of examples of what I learned on two different topics using textbooks (Example 1) or a personal review (Example 2). Oh yes, and a third example where my research had big holes in it.

Example 1: Punishment Intensity

Last year I wrote a post called Don’t Be Callous: How Punishment Can Go Wrong. In it, I talked about the pitfalls of using punishment. On the one hand, starting with too low an intensity allows the animal to habituate. On the other, starting with a high-level intensity risks fallout. There is nothing controversial about this finding. You can find information about it in any learning theory textbook.

A picture of picking cherries with a line through it. Cherry picking one's research is a bad idea.

A commenter claimed I had cherry-picked the studies I cited. But I hadn’t. I had cracked four learning theory textbooks. All of them covered the topic of punishment intensity. And they cited the same group of studies.

Textbooks are giant literature reviews created by experts in the field. They are generally way more helpful than a study or two.

Example 2: Dogs and Music

I keep track of studies on the purported effects of music on dogs. I am actually fairly qualified to assess some aspects of that literature, as I have master’s degrees in both music and in engineering science with an emphasis on acoustics. I keep a list of dogs and music studies.

A brown dog appears to read a learning theory textbook for research

It’s usually safe to quote Chance

This is a new field so you won’t find extensive coverage in textbooks. The research is still in what we might call an oscillating phase, with conflicting, back-and-forth results. Yet there is a burgeoning market of music products for dogs, most of which claim that research has “proven” that music is beneficial to dogs.

That’s a stretch. And it pays to know something about the literature before taking such claims at face value. For instance, you can buy recordings of music that is specially altered for dogs. A certain brand claims that their music has been clinically proven to relax dogs and allay their fears. The product’s website cites a study. One study.

But what about the bulk of the research? Is there more than that one study? There sure is. And they leave out of the marketing materials the fact that their specific product has been tested twice in subsequent research studies. Guess what? In both the studies the product has been no more beneficial than regular “classical” music. Instead of mentioning that, they just continue to cite the older article that shows benefits to dogs from classical music.

If we trace the current threads of research on dogs and music, we will see that a current hot topic is habituation. There are some studies that have shown that dogs habituate to music that is played regularly. Think about that one for a minute. Those tracks you play during every thunderstorm (if they ever did contribute to your dog’s relaxation) may have become so much background noise to your dog.

The lesson I have learned here is to always, always check the sources myself. Whether  deliberately or through an oversight, product marketers, writers, and private individuals often cite studies that don’t actually support their claims. In some cases, they cite studies whose results are the opposite of their claims. One company referred me to a study that found their product to perform no better than a placebo!

Example 3: Research Blooper

Yellow sign that says "OOPS!"

In 2013 I wrote a blog post about errorless learning. I performed my standard research procedures and came up with Herb Terrace’s work starting in the early 60s with pigeons. My post was critical of applying his methods to dog training. The pigeons were food deprived and their training necessitated hundreds, even thousands of reps. Plus I disliked the absoluteness of the term “errorless” since even Terrace’s pigeons made errors.

I published my post and a friend whose parents trained with B.F. Skinner gently showed me Skinner’s work and his suggestions about setting up antecedents for errorless learning. Turns out my post on errorless learning had many errors! Several decades before Terrace, there was an important discussion regarding the role of errors.  The topic was important in Skinner’s work. Skinner disagreed with Thorndike, who claimed that errors were necessary for learning. I could get behind Skinner’s claims, which centered on skills and planning used by the teacher/trainer to make the learning process as smooth, efficient, and stress-free for the learner as possible.

In my defense, most textbooks and scholarly discussions about errorless learning center on Terrace’s work, not Skinner’s. Terrace’s own references and credits to Skinner are skimpy. I’m just lucky I had a friend who could direct me to the right place. I published a second post on errorless learning with updated information and corrections. I left the first one published (with cautions for the reader and links to the second article) as an example of how easy it is to miss a research elephant in the room.

Who else has a personal “Oops” story? Did you get taken in by a popular article on a study that turned out to miss the point of the study? Did you go as far as I did and publish an article that didn’t cover the research well?  (Not sure I can get any takers on this but it’s worth a try!)

My Learning Theory Go-To Resources

Here’s a list of the textbooks I use most often when researching a learning theory topic. Enjoy!

  • Chance, P. (2013). Learning and behavior. Nelson Education.
  • Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis. Pearson.
  • Domjan, M. (2014). The principles of learning and behavior. Nelson Education.
  • Domjan, M. (2000). The essentials of conditioning and learning. Wadsworth/Thomson Learning.
  • Goodwin, C. J. (2016). Research in psychology methods and design. John Wiley & Sons.
  • Honig, W. K. (1966). Operant behavior: areas of research and application. Appleton-Century-Crofts.
  • Keller, F. S., & Schoenfeld, W. N. (1950). Principles of psychology: A systematic text in the science of behavior (Vol. 2). Appleton Century Crofts.
  • Klein, S. B. (2011). Learning: Principles and applications. Sage Publications.
  • Schwartz, B. (1989). Psychology of learning and behavior. WW Norton & Co.
  • Shettleworth, S. J. (2010). Cognition, evolution, and behavior. Oxford University Press.

Copyright 2017 Eileen Anderson

Share Button
This entry was posted in Behavior analysis, Critical Thinking, Research. Bookmark the permalink.

16 Responses to When Is Citing a Research Study Not Enough?

  1. Sonya says:

    I can’t believe I accepted a website’s claim at face value while researching the pros and cons of prosthetics versus three legs in dogs. “The old saying that a 3-legged dog does fine has been disproven through advanced research at Colorado State University among other research facilities. 4-legged animals do better and suffer less compensatory problems than their 3-legged friends.”

    I tried to get hold of the research mentioned on the site after choosing the prosthetic option. It’s actually not a finished study or available! Bit naughty to make such a claim on the website me thinks. Masive oops moment for me. Don’t trust any site that is selling or marketing anything… even medical sites. I felt so naive for my level of trust! Its truly a minefield information out there with vested interests using words like “science says” and “research proves” Thank goodness for the Eileens who quietly write guidelines to help us improve our critical thinking skills and navigate this messy terrain.

    • Eileen Anderson says:

      Yes that is naughty! And actually appalling… Thank you for sharing the story. I think it has happened to most of us.

      Thank you for your kind words!

  2. Jackie Earnshaw, CPDT-KA says:

    Thank you for this.

  3. Rebecca says:

    No comment, except to say love the photo of Summer ❤️

  4. Hey Eileen!
    I’m curious if in your reading on music and dogs, you came across the study that found audio books to be more relaxing to shelter dogs than music. (http://www.appliedanimalbehaviour.com/article/S0168-1591(15)00312-3/abstract)
    And will you be doing a post eventually on music and dogs?? ;-P
    For masking aversive sounds, I usually recommend white noise machines or fans since music has pauses and quieter moments, etc., but I sometimes think owners might be more likely to remember to turn on the radio or put on a CD than to buy a white noise machine or set up a fan in January….

    • Eileen Anderson says:

      Yes, I sure have! Thanks for mentioning it. It’s another one that doesn’t tend to be promoted by the dog music producers. I’ve implemented it as well, using the website LibriVox. I play books in Russian so they don’t distract me.

      Radio/TV is tricky when one has a dog who is sensitive to beeps and whistles. Have to stay away from all sports, hospital shows, and anything with a lot of tech. (That doesn’t leave much….)

      I have all sorts of things in the works regarding dogs and sound and music. Watch here: https://soundsensitivedogs.com Also I have a magazine article coming out in August or so next year. It will cover the best approaches for different types of sensitivity. I will keep you posted. Thanks for the comment!

      • Chris from Boise says:

        Thank you for introducing us to LibriVox. I had never heard of it. Wow – now I just need another 40 hours in the day…! And Russian literature – in Russian – as white noise: you are a genius!

        Also, I’ll be watching for your soundsensitivedogs.com site to develop. Thank you in advance for sharing your knowledge and experience. As Habi becomes more deaf, she is much less troubled by noise. Obi, our outgoing, boisterous, stable-tempered sweetie, on the other hand, is becoming more noise-sensitive at seven years, which we understand is (alas) not uncommon.

        And as always, thanks for including the blooper. We love that you’re human like the rest of us, and we learn from your mistakes.

        • Eileen Anderson says:

          Thanks, Chris! And credit in turn to Jay Koes for introducing me to LibriVox. Although not fluent in anything other than English, I’ve taken a lot of language courses over the years, and with most Germanic and Romance languages there’s just enough I can almost understand that it’s distracting. Not so with Russian! I hope things go well for both your dogs. Cricket was scared of some high noises and thunder, so her going deaf was a blessing in that one way. Thank you for your kind words!

  5. Pat says:

    The thing about textbooks is that, no offense meant, they are often knocked out in a somewhat perfunctory fashion and in most sciences it is common for them to be composed largely in mimicry of each other, rather than truly independantly. This goes for what subjects are covered in the books, what ‘the state of the field’ is represented as being, *and* what examples/evidence/studies are given. (I was a research biologist for many years, and also taught and thus had the interesting experience of evaluating textbooks and comparing them to each other to choose what to teach from, while knowing the research in the field well).

    Unfortunately the best, or arguably the *only*, way to really have an intelligent opinion about what studies do or do not say, or what a scientific field does or does not know at a given moment in time, is to read most or all of the original studies in their entirety for yourself, using all one’s critical thinking skills rather than merely accepting what the authors say it all means. One big thing to keep in mind when reading journal papers, for instance, is that the methods used in a study — right down to the last detail — very very often mean that the experiment or survey is not actually testing quite the same hypothesis as the authors claim.

    (For instance, food preference studies are now notorious for often giving quite different results depending on whether you offer each food alone, or paired (and paired with which other food exactly), or as a smorgasbord… as well as the physical setup in which the food is offered. I haven’t looked at any studies of this sort for fifteen plus years now so maybe people have finally caught on, but back when I was following things, this resulted in papers claiming wildly different food preference schemes for the same animals, NOT because of real variability but because the studies were not testing “food preference” in general. Everyone had hold of a different part of the elephant, so to speak.)

    • Eileen Anderson says:

      Hi Pat,

      I agree with all you say here. I would go a step further and say that not only should we assess the studies ourselves, which includes being proficient in experimental design and statistics, but really, only credentialed people in a particular field should be venturing opinions and writing about findings in that field. That would include that it is far from ideal that I, with my degrees in music and engineering, write about learning theory and venture into related sciences.

      Quality of textbooks and the whole textbook selling system, with the high prices and planned artificial obsolescence, are big problems. If you want to make any recommendations or cautions about books relevant to trainers, feel free. I have two (older edition) learning theory textbooks that I do check for comparison but I didn’t include in my own list in this post and I don’t generally cite them. Even I can tell that they are weak and a bit strange. One even has some wrong information in it.

      In this day and age where anyone at all can write, publish, and viralize an opinion, my goal is to try at least to get people away from the “I found one study to support my point” syndrome. I appreciate your cautions about textbooks and research assessment. You are spot on.

      • Pat says:

        I dunno, have known way too many “credentialed people in field” who do not think very clearly about some aspects of their subject; and some of the best critical thinkers I’ve known have been horseshoers, librarians, etc. Music and engineering people can be just as qualified (in some ways, maybe more so, because approaching the subject with fewer biases and less personal stake in it) to think and write about learning theory etc. Yes, it takes a bunch of esoteric math to evaluate the adequacy of the statistical methods (frankly though, many academic biologists are very poorly versed in stats!). And while many experimental-design issues are evident to anyone with common sense, admittedly experience in the field helps.

        However in my opinion the biggest issue, which anyone willing to use their brain CAN sort out for themselves (sometimes better than those doing the research can), is the question of what hypothesis did the study REALLY test, as opposed to what the authors say. And, are the results just as consistant with some other interpretation.

        For instance, suppose I get a selection of pet owners to follow a careful protocol for rewarding , one group using a particular toy and one group using a particular food. And then I find that the dogs in the food group showed statistically-significant higher rates of performing the behavior than the dogs did in the group using toys. I then assert this means that food is more effective than toys in training dogs for .

        Well, that might be actually true; but it might instead be true that the dogs found the particular food and its method of delivery appealing whereas maybe most of them did not find the particular toy (or its method of use by the dog owners) to be especially rewarding. Or it might mean that the particular training protocol I had my subjects follow was better-suited for food than for toy use. Really, since there are gazillions of kinds of foods, and gazillions of kinds of toys, and people vary quite a lot in their proficiency in using toys, and dogs evidently vary quite a lot in their individual preferences (innate, learned, whatever)… you really cannot make such a gigantic leap from this specific study to some grand sweeping generalization. But you *know* how such a study would get cited by most people who read it.

        Honestly, I am glad that dogs and their behavior are now being studied in a scientific context… but people gotta realize that almost nothing is actually convincingly *known*, and some percent of that is probably wrong anyhow. I know it seems commonsense and desirable to try to base dog training on a scientific foundation but in my opinion there just aren’t enough solid bricks around, at present. Also, :studies show” or “the laws of behavior say…” tends to get thrown around an awful lot as an excuse *not* to confront real-world evidence in our own lives, which is sort of anti-scientific. Science is constructing hypotheses and then testing them in as unbiased and open-minded a way as possible so as to see where we’re wrong and hopefully converge gradually towards truth… and that’s something we can all do in our daily lives, it doesn;t take universities and research grants 🙂

        Sorry for the soapbox, it’s just soemthing that bugs me and I enjoyed your post a lot, just thought it could go much further 🙂

        • Eileen Anderson says:

          No problem! I am very pleased with your comments, and it’s a great soapbox.

          “…the question of what hypothesis did the study REALLY test” Yes, yes, yes. But even common sense can’t always help, there. We need both common sense and a fair amount of book learnin’, there. But you’re right that it doesn’t necessarily have to be in the same field to be helpful. Sometimes people on the outside can see the Emperor’s new clothes effect. But we all have biases, and they get in the way. (Understatement.)

          I like your food and toy example. A real article being discussed has bugged me for a while–the one about verbal cues for dogs vs. gestures. I need to read the study first, though, to make sure. I think their outcome is probably right (gestures are easier for dogs), but there appear to be some problems with what was being compared.

          I like your point about doing science in our daily lives, and I wish people would write more about that. There will always be enormous limitations on personal experience–that’s why we have studies–but learning to be good observers and think our way beyond biases is a great thing.

  6. jay says:

    First ,thank you for your writing in general. I’m sure you made your list as short as possible but if you had which one text (maybe 2) would pick as a starting point? I’ve read no learning theroy texts at all.

    • Eileen Anderson says:

      Hi Jay,
      I would probably recommend the Paul Chance book “Learning and Behavior” (you can pick up a previous edition for a fraction of the cost of the newest one) for a first book. He is very readable, uses relevant examples, and is all around approachable. But the book also has great references if you want to delve deeper. I could be prejudiced because the Chance book was my first. But I do think it’s a good one. Thank you for your kind words!

Comments are closed.