[In operant learning], extinction means withholding the consequences that reinforce a behavior. –Paul Chance, Learning and Behavior, Fifth Edition, 2003
This post is Part 2 (a year later!) of But Isn’t it Punishment to Withhold the Treat?
In that post I discussed the common error of arguing that withholding a treat from a dog in a training session (or other time) comprises punishment. On the contrary, when nothing is contingently added or taken away but behavior decreases, the process at work is extinction, not punishment.
But that is not to say that extinction is automatically better. In Dr. Susan Friedman’s Humane Hierarchy for behavioral intervention (see graphic below), extinction by itself is at the same level as negative punishment and negative reinforcement. They are roughly at the same level of (un)desirability, and the level of unpleasantness of any particular technique would depend on the circumstance and individual animal. Dr. Friedman makes a point to say that these three are not ranked in any particular order of overall undesirability.
Extinction is often overlooked when considering or analyzing methods. People often mix it up with negative punishment. It’s a bit of an oddball learning process since it applies to both operant learning and respondent conditioning. In operant learning it is sometimes jokingly called the “fifth quadrant.” The important thing to me is that its unpleasant effects can vary wildly, from practically nil to complete misery.
Extinction can be very, very frustrating. Here you are with these behaviors that you have been performing for such a long time that they are habitual, and all of a sudden they don’t work anymore! But there are ways to use extinction in combination with other processes to make it much less hard on the learner. And in fact, if we look at the Humane Hierarchy a little more closely we will see that extinction is actually lurking in another of the levels, partnering with something much nicer.
Let’s explore this by way of a thought experiment.
Extinction Scenario #1. You get your car out of the shop after a tune-up. You buy a pint of your very favorite ice cream, or other perishable treat. You realize you need one more thing from the store, so you lock the car and go back in. When you come back out, you try to unlock your car with a remote. It doesn’t work! You press the remote again and again. You press it harder. You aim it differently. No go. Then you try unlocking the door with the key. That doesn’t work either! You jiggle and jiggle the key, and try the different doors. Nothing works. You bang on the car doors. You can’t get into the car using the methods that you have always used. You are starting to cuss now. Your ice cream is melting. You finally yell at the car and it opens!
You drive back to the repair shop and ask the guy what the heck he did to your car. He said your car doors now work by voice control. Apparently he thought that sending you off to find that out on your own would be the best way to teach you.
Two questions. 1) Was that learning process fun? 2) What are your feelings towards the mechanic?
That is a description of the process of extinction. A behavior that has previously been reinforced is no longer reinforced. In this case it was actually two behaviors: opening the car with the remote and opening it with the key. Both used to be reinforced by your gaining entry to the car. Both stopped working with no warning. Stinky!
Three characteristics of extinction are the extinction burst, an increased variability of behavior, and aggression. We got all three. When your normal methods for opening the car door didn’t work, there was a big burst of behavior from you as you tried stuff. You unconsciously started adding variety in how you performed the behaviors. And you started doing everything a little harder and banging on stuff. None of that was fun for you.
Now let’s try a different version of the scenario.
Scenario #2 When you first go to the mechanic, he tells you about a new option to have your car respond to voice commands, including that if you opt for the upgrade, in some cases the old methods will not work. You decide that it sounds good.* Your mechanic takes 10 minutes to go over the voice commands that you will use with your car, including that you practice unlocking it with your voice.
When you stop off to go to the store and return to your car, if you are like 99% of the human race, that huge reinforcement history for using your remote or keys during your whole driving career kicks in and you initially try to use one of these to open your car. But the practice of the new behavior is fresh in your mind, so as soon as the remote doesn’t work, you remember to give the voice command. Your car unlocks!
But old habits die hard. You will probably be hitting that remote or trying your keys for quite some time, each time you approach your car. The old behaviors will diminish slowly as their reinforcement histories fade into the past and the practice of the new successful behavior overshadows them. However, there will be comparatively little frustration. You are never in the dark about what behavior will actually work. You’ll probably perform the old behavior once, go “oops!” and immediately use your voice without wasting much time.
Not so bad!
What About Dog Training?
Here are the dog training corollaries to Scenario #1 and #2 above.
Let’s say you want to address the following behavior problem: When you get out your dog’s leash, your dog gets excited and runs around getting all aroused, barking and jumping on things.
Scenario #1 You have never trained your dog to do anything, but you’ve had enough of the overexcitement. So you decide you aren’t going out that door until your dog sits calmly for you to put the leash on. So you take your dog into the front room and pick up the leash. Dog runs around. You just stand there. Dog jumps on you and on the furniture. Runs around and barks. This goes on for about 5, maybe 10 minutes. Finally your dog wears out and sits down and looks at you. You take one step towards him, holding the leash out to attach it. He gets all worked up again and you have to wait out another few minutes of excited activity. This happens over and over.
From your dog’s point of view, the rules have changed. All that previous barking and running around have been reinforced by getting to go outside. Many people frankly don’t have the stamina to outwait a dog in this situation, and will finally break down and take the dog out anyway, which worsens the problem (by finally reinforcing the behavior they’ve made it more persistent). If you do succeed and the dog calms down in 20 minutes on that first day, it may take a bit less the next day. But since this is completely new to your dog and you are asking so much of him when he is already wildly excited, it will take a while, and be a frustrating process for him
Scenario #2 You have trained your dog to sit in all sorts of situations and for all sorts of reinforcers. He sits for his supper. He sits to go outside. He sits to greet people. He sits at the agility start line. He can hold a sit stay while you run around and play tug with another dog. So when you decide to teach him to sit calmly to put the leash on, you first practice some sits for treats in a random room of your house. Then you do the same in the room where you keep the leash. Then you pick up the dog’s leash and look at him expectantly. If he starts running around you wait. When he makes contact again you give him the expectant look. He will likely sit pretty soon. Treat!! He may jump up again when you approach him, but he is already learning.
This fits a pattern he is familiar with: sit and something good happens. You can use treats to reinforce those sits in this new situation so he doesn’t have to wait so long for the ultimate reinforcement, going out. You practice in small steps until you can put the dog’s leash on while he sits calmly. Depending on the dog and what you have trained, you may be able to take him straight out the door calmly that first day, or you may practice a few more days just putting the leash on and off before you go out the door.
Defining the Difference
Take a look at Dr. Friedman’s diagram again. See the area just below “Extinction, Negative Reinforcement, and Negative Punishment”? It is called “Differential reinforcement of alternative behaviors.” Guess what? That one corresponds exactly with both of the Scenarios #2 above. Dr. Friedman’s definition is, “Differential reinforcement is any procedure that combines extinction and reinforcement to change the frequency of a target behavior.”
Instead of being gobsmacked by the normal behavior not working anymore, the learners, dog and human, are given a big fat clue about what is going to work to get what they want. That clue is the positive reinforcement of an alternative behavior.
Extinction is part of all differential reinforcement training methods. Those methods are on a more humane rung of the hierarchy because the animal is given immediate opportunities for positive reinforcement. This can be done either by reinforcing successive approximations (shaping), or by separate practice of the desired behavior before it is evoked in the situation where the undesired behavior is likely.
So when someone says to you, “Neener neener neener, you use punishment when you withhold a treat,” say, “No, that’s extinction.” Then if they say, “Neener neener neener, you use extinction and that’s mean,” say “I use it in combination with differential positive reinforcement.” And make sure you do!
Be the mechanic who shows his client ahead of time what is going to work, instead of the one who sends him off with no clue.
* The car thing is a deliberately ridiculous scenario. Obviously, to cause a car’s keys and remote not to work would be horribly dangerous, and hardly anyone would consent to that even if it allowed one access to a new feature like voice commands.
Related Posts and Pages
But Isn’t It Punishment if You Withhold the Treat? (Extinction Part 1)
I never got to the issue of “ignoring” in these extinction posts. So I guess there is going to be a Part 3.
© Eileen Anderson 2014 eileenanddogs.com
Huge thanks again. You have the most wonderful knack for wording that really helps me understand. I particularly like this lesson! I just feel like a lot of diverse information came together for me. Thanks!!
Wow, thanks Lyndsey! I am so glad. I just wasn’t too sure about this one.
And hey, you read fast!
Fair Pairs = if you are reducing a behavior, then you must simultaneously be increasing another behavior. A behavior that the being is already fluent in, and now you expect to perform in a different context. Great examples!
Fair pairs, yes! Thanks Randi!
Awesome 🙂
Wouldn’t it also work in the Antecedent arrangement in that scenario of the leash if you took the leash to another room, sat down on the floor and then proceeded to put the leash on. I use this a lot with board and train dogs who are so excited about going out with the leash on. I don’t do it near a door, I arrange the environment so there are little to no triggers. Eventually, I can put the leash on at the door with no issues.
Good idea, and a good example of antecedent arrangement. I’m guessing it works particularly well with board and train dogs since they don’t have a huge reinforcement history of hectic behavior near the door? Not saying that your own dogs do; I’m just making a distinction for readers. Even though antecedent arrangement is better on the Humane Hierarchy, once hyper behaviors are established near the front door, most people are going to need to teach calm in that location in some kind of differential reinforcement scenario.
Still, love your suggestion. Will be very helpful for some folks.
I’ll follow Jamie, but disagree with you, Eileen. When dogs have a huge reinforcement history of hectic behavior in any situation, I suggest the basic learning of an alternate behavior is both easier and less stressful (for both you and the dog) when first performed in a passive scenario lacking triggers. Logically, you do not have to contend with the undesired behavior during initial learning, and the general learning process is always much easier in a quiet library then while cheering your team on at a noisy game.
This applies not only to unruly behavior, but also and especially to very fearful and scared dogs, where their high anxiety in some situations reduces learning efficiency.
Also, without dipping into the other types of differential reinforcement, I’ll make a distinction between the Incompatible Behavior in your example #2 (he can’t both sit and jump at the same time), and that involving Alternative Behaviors (DRA). That while both are useful, the incompatible version (DRI) is often more efficient, when it is possible.
And at that point, I suggest any extinction aspect is only academic, since it’s not primarily withholding the consequences that prevents the undesired behavior, but the reinforced conflicting behavior. And that it’s not antecedent arrangement as you’re only passively manipulating the environment and doing that to enhance the likelihood of learning efficiency rather than a specific behavior (using your definition on antecedent arrangement).
Of course, we have a little difference in orientation, as you’re noting Dr. Friedman’s work, and I learned years ago based on Dr. Karen Overall.
Good idea, and a good example of antecedent arrangement. I’m guessing it works particularly well with board and train dogs since they don’t have a huge reinforcement history of hectic behavior near the door? Not saying that your own dogs do; I’m just making a distinction for readers. Even though antecedent arrangement is better on the Humane Hierarchy, once hyper behaviors are established near the front door, most people are going to need to teach calm in that location in some kind of differential reinforcement scenario. In other words, they might get the leash on calmly, but that won’t help everybody get out the door calmly.
Still, love your suggestion. Will be very helpful for some folks.
Thank you so much Eileen. Every time I read your blogs I learn something new that really helps me improve how I interact with my puppy and the dogs I work with (I’m a dog walker). You have a fantastic way of explaining things. When will we be able to buy a book written by you? You have a knack of giving answers to questions I didn’t even know I had, where as many dog trainers blogs just leave me with more questions than answers.
Jill, you are very kind! That’s such a great compliment! In fact I am working on a book, and will make an announcement on my blog in the next couple of months. Nice of you to ask!
Well done Eileen! This is one of my criticisms of doctrinal use of the humane hierarchy; you end up arguing with yourself (or at least some folks do) about if you should use extinction since it is so far down the list of optimal choices, but the fact is, you cannot use R+ without the use of extinction. The key is to set up your learner to understand the process. Nice use of analogy!
Thanks! Yes indeed, we can’t go very far without extinction being experienced. To me it’s all about how well we pave the road to the new behavior…
Love this, car key example is an awesome way to help create empathy!
Entry request: Something you’ve sort of touched on with stimulus control but maybe a bit different – directionals. Some sessions I can get my dog to nail her lefts and rights 10-20 times in a row, and I’m like “YES! She gets it!” but then the very next day she’ll totally flub it and keep giving me the wrong direction. Any quick tips on how to fix?
Same goes with her “Away” command, actually. We’re practicing going around an object from the left or right, depending on which arm you throw up. She tends to prefer one side and has a hard time switching to the other.
rOsey, this is something I am not too good at. But a couple of standard suggestions. First, have you tried the step in cue recognition (this is pre-stimulus control) where you throw out nonsense words? Set up to do the directionals, then say, “Massachussetts,” or “vertigo,” or “plop.” The “correct” response to those (unless I have hit on one of your cues) is for her to do nothing. You can either reinforce that directly, or reinforce by then giving an easy cue that you can reinforce (not anything she is likely to get wrong!). If you have never done that, it is a revelation. You’ll find out exactly how much your dog has been guessing. If she is really good at verbals, you can start testing words that are closer and closer in sound to the actual cues, but I never get that far with my dogs.
The other thing is whether you have you worked on eliminating your body language. Would she still know the cues if she couldn’t see you? Dogs are SO good at this. I actually have a post (and embarrassing movie) about it: A Little Heavy on the Body English. Hope some of this helps!
Actually “punishment” doesn’t need to *stink* either. All it HAS to do to be ‘punishment’ is decrease the preceding behaviour.
This can take the form of rewarding an incompatible behaviour. We reinforce sitting to greet, at the same time punishing jumping up.
See both Thorndike (new comment in “isn’t it punishment . . . “) but also Tolman’s “Expectancy-Value Theory” of behaviour.
https://wikispaces.psu.edu/display/PSYCH484/Expectancy+Theory+Overview
There is a lot missing from your assessment that I am not willing to go over here. Citing Thorndike’s Law of Effect and giving a link on expectancy theory (a link which doesn’t mention punishment or extinction) do not support your assertion that reinforcing a behavior_necessarily_ means that you are punishing others.
The information in this blog is based on the scientific discipline of behavior analysis as it it practiced, studied, and reported in 2014. It is to the best of my understanding through study of the work of current experts and I will happily correct errors that I might make.
The aspects of learning theory I presented in this post are at the level of Behavior Analysis 101. They are basic and can be found in any learning theory textbook under differential reinforcement and extinction. These topics are even covered in the behavior analysis section of some Psych 101 textbooks.
An idiosyncratic claim that goes in the face of basic behavior science needs strong support. You have not provided that. I would highly recommend you take Dr. Susan Friedman’s professional course if you haven’t already. It is utterly clarifying on current behavior science.
Thanks for the comments.
Jenny:
The ABA definition of differential reinforcement is:
DIFFERENTIAL REINFORCEMENT :
Any procedure that combines extinction and reinforcement to change the frequency of a target behavior.
———-
Expectancy theory is a theory of motivation. A mathematical formula of what takes us from inactivity to activity.
Motivational force = expectancy x instrumentality x valance.
One of the core elements of expectancy is the belief that increased performance results in certain outcomes. It’s not a replacement of operant conditioning. If you expect to gain nothing (appetitive or avoiding an aversive) then you have no motivation to do anything. Expecting nothing sets a variable to zero and your motivation becomes zero. Expect to gain nothing, no motivation to do anything.
It’s part of psych. It’s old (1960’s) and very much part of what dog trainers have been doing for a long time. “How do I get my dog motivated to do that behaviour quickly and enthusiastically?”
If you’re measuring motivation, use theories of motivation.
If you’re asking, “How do I teach someone to learn,” then look at learning theory (or other types of learning – which are generally impractical due to their difficultly to put into practice with any consistent reliability.)
It’s not a matter of “preferring” one over another. They are a measure of different things.