UNITED STATES OF AMERICA
+ + + + +
DEPARTMENT OF EDUCATION
+ + + + +
THE USE OF SCIENTIFICALLY BASED
RESEARCH IN EDUCATION
+ + + + +
WORKING GROUP CONFERENCE
+ + + + +
FEBRUARY 6, 2002
+ + + + +
The conference was held in the Barnard Auditorium at the United States
Department of Education, 400 Maryland Avenue, S.W., Washington, D.C., at 9:00
Susan B. Neuman
Welcome and Introduction ................................................................................... 3
What is Scientifically Based evidence?
What is Its Logic?
Valerie Reyna ........................................................................................ 5
The Logic and the Basic Principles of
Scientific Based Research
Michael Feuer ...................................................................................... 25
Lisa Towne .......................................................................................... 34
Research .............................................................................................................. 46
Math Education and Achievement ..................................................................... 65
Implications for Scientific Based Evidence
Approach in Reading
Eunice Greer ........................................................................................ 78
Safe and Drug-Free Schools ............................................................................... 92
Comprehensive School Reform ....................................................................... 103
2 9:05 a.m.
SUSAN NEUMAN: Good morning. My name is Susan 3
4 Neuman. I'm Assistant Secretary for Elementary and Secondary Education. It's 5 just thrilling to have all of you here today.
6 One of our goals today -- we have a very practical goal actually. 7 We're no longer debating whether scientifically based research and scientifically 8 based evidence is important, we know it now is important and we know it is critical. 9 As many of you know, we have counted one hundred and eleven times that the 10 phrase "scientifically based research" is in our new law.
11 What our goal today is, is a very practical one. What we want to 12 do is begin to explore the logic of scientifically based evidence or research and to 13 really to begin to understand both its definition as well as its intent. 14 The second goal is something that is very particular to our office, 15 the Office of Elementary and Secondary Education, and that is, how do we begin to 16 put this into practice? How do we begin to suggest guidance?
17 What you are going to hear today is not only some wonderful
18 papers on what is scientifically based evidence, what is it in its logic, it's 19 characteristics, what it is and what it isn't. But, then, after a break, what we hope to 20 do is really focus on what does this mean for safe and drug-free schools, reading, 21 math, comprehensive school reform?
22 What we want to do eventually is move this debate throughout
23 all of our programs so that we begin to really look at the scientific basis underlying 24 what we say and what we do for schools in districts across the country. 25 What I want to do today is I want us to keep very much on pace.
26 You'll see that there is opportunity to ask lots of questions. We ask you that the
1 questions you raise, please focus on the implications of this issue, not whether or 2 not scientifically based evidence is a good thing or not.
I'm going to keep people very closely -- Valerie reminded me 3
4 that I was already late. What we are going to do is we are going to keep people 5 moving in a very fast pace and then give time for your questions. Then have a little 6 break, move it on to implications and then, finally, have a panel where you really 7 are able to address even more questions. We are delighted to have you all today. 8 What I'd like to now do is introduce Valerie Reyna. Valerie is
9 the Deputy OERI, Office of Educational Research and Improvement. Her topic is 10 what is scientifically based evidence, what is its logic?
11 VALERIE REYNA: Thank you very much. If you could go
12 ahead and put my first slide up that would be great.
13 Welcome, it is a please to have the opportunity to talk to you and 14 I gather that our well-organized organizer is going to keep the question and answer 15 period to the end after all the speakers.
16 My usual style as a teacher is to have questions during the talk, 17 so that's kind of constraining for me but I will try to contain myself. 18 MS. NEUMAN: You will be good!
19 MS. REYNA: Absolutely! But if there is something that is
20 burning that's informational, if there's something that doesn't make sense at all, it 21 wouldn't be a good idea not to communicate. So, please do raise you hand for that. 22 At the end, of course, I will be delighted to entertain questions. In fact, a kind of 23 give and take session is what I am really looking forward to, so that I can learn from 24 you too.
25 Yes, that's who I am. We can go to the next slide.
26 I am going to talk briefly about: why scientific research,
1 although I don't think in the very short time that I have available that I could really 2 give you a coherent argument that supports and defends the notion of scientific
research, but I can touch on a few ideas very, very lightly. 3
4 One of them is: why scientific research? I think to think about
5 that it's useful to think about what is the alternative to scientific research? If you 6 didn't base practice on scientific research, what do you base it on? 7 Those alternatives include (this is not an exhaustive list, of
8 course), it includes such things as tradition -- this is the way we've always done it, 9 for example, superstition, there are -- you know, you throw the salt over your left 10 shoulder and the reading scores go up! No, actually, there are things that are not 11 based in fact that in fact become lore that if we really knew the scientific basis of it 12 we would discover that those things in fact are just superstition. They are 13 unfounded beliefs.
14 Then, there's anecdote. A fairly well-known obstetrician
15 physician asked me once, "What's wrong with anecdotal evidence?" I think it is 16 really a good question. Anecdote is a series of stories that you tell about things that 17 have happened to you in your life. They can be very entertaining anecdotes. 18 The reason why we can't base practice on mere anecdote,
19 however, and this is, of course, well known in medicine, is that individual cases 20 may be exceptions. That may be the only case of that type.
21 In fact, anecdotes are often more entertaining when they are
22 unique. But that is a weak basis to generalize to many, many people. 23 We know on the basis of experience that anecdotes have turned
24 out to be false and misleading. Sometimes they are very representative, sometimes 25 they're not. The problem is we don't know when.
26 Next slide. There's an analogy to medicine that I have obviously
1 drawn on already.
2 The first example, of course, is the classic one of when they used
to bleed people. People would get sick. You know, I think it was when George 3
4 Washington was bled that contributed to his death.
5 Why was it that good, well-intentioned physicians, because I
6 think they probably were well-intentioned, I don't think they were trying to hurt the 7 president, why is it that they didn't notice that it wasn't working? It wasn't just with 8 this one patient, it was with many patients. Yet, somehow, personal experience 9 was not sufficient to dissuade them from this practice.
10 Well, in fact, clinical trials are very recent in medicine. It was 11 only in the 1940s that the randomized experiment where you know you had 2 12 groups, and you randomly assigned and all of that became routine and a standard, 13 the gold standard in medicine. That is very recent in historical terms. Prior to that, 14 we relied on those things I talked about in the first slide, like tradition and bleeding 15 people.
16 One of the reasons why clinical trials are not sufficient has to do 17 with the psychology of human thinking. I won't go into it in any depth, but I'm 18 actually a cognitive psychologist and there's been research done about when you 19 ask people to report about things they have directly observed and directly witnessed 20 and the biases that can creep into that type of reporting. These are normal human 21 biases that are generally adaptive, but they have predictable pitfalls. So, if you rely 22 on your memory for past events, we know that that memory will be biased, and so 23 on. Drawing simply on your personal experience alone is not a solid foundation for 24 generalization.
25 Clinical trials in fact are the only way to really be sure about
26 what works in medicine. The logic of it -- and the other speakers are going to go
1 into far more depth than I really have the time to do, the logic of it is basically the 2 following: You have a group of people that you want to make a conclusion about.
You want to say this intervention -- whatever it is, if it's a new reading technique, 3
4 or whatever -- works for this group or not.
5 So, what you do is you take members of that population and you
6 flip a coin essentially as to whether they are going to be in the group that actually 7 gets the intervention or gets some kind of comparison, like what you would have 8 done had you not done this new thing. Standard treatment, that's a common 9 control.
10 The idea is that if you do this enough times and you get big
11 enough groups, you've got two groups, the fact that you're flipping a coin ensures 12 that these two groups, if you have enough people in them, are going to be 13 comparable in every way except the intervention you're interested in. 14 Why is that? Because there was nothing that put one person in
15 one group as opposed to the other. It was all by chance alone that you ended up in 16 the reading intervention group as opposed to the control group. And, so, all the 17 ways in which people do in fact differ, and people do differ, should be represented 18 in both groups. They should be comparable in every way, except the one thing that 19 you made different in their lives, therefore, we can isolate the effect of the outcome 20 and trace it to that intervention uniquely.
21 This is the only design that allows you to do that, to make a
22 causal inference. Everything else is subject to a whole bunch of other possible 23 interpretations.
24 Now if you have too small a sample, obviously the logic
25 doesn;t follow. Because you can have all the smart people in one group, the not 26 so smart people in the other
1 if you only have a few. If you do this enough times, you get a big enough group, 2 they will be representative. That has been proven mathematically by things like --
well, we won't get into that! 3
4 The bottom line here is these same rules about what works and
5 how to make inferences about what works, they are exactly the same for 6 educational practice as they would be for medical practice. Same rules, exactly the 7 same logic, whether you are talking about a treatment for cancer or whether you're 8 talking about an intervention to help children learn. The same logic applies. In fact 9 that's something I've said in talks for a period of time and the National Academy of 10 Sciences report, which I know Mike and Lisa are going to talk about, in fact makes 11 a similar claim. The rules of the game are the same.
12 I have the word "brain surgery" up there. The reason I have the
13 word "brain surgery" up there is that I think, you know, when we talk about 14 medicine and things like brain surgery and cancer, it is very, very important to get 15 it right. We all recognize that and most of us buy into that. You know, that you've 16 got to have randomized clinical trials because we want to be able to benefit for 17 these treatments for cancer.
18 But when we teach students we really are engaging in a kind of
19 brain surgery. We are effecting them one way or the other. Sometimes what we do 20 helps, sometimes what we do, in fact, inadvertently, harms. We really don't know 21 until we do a randomized clinical trial whether what we are doing is benefiting that 22 student or not. We really don't know. It may be well intentioned, but that's not 23 sufficient as we can see from the example from bleeding. So, it is brain surgery 24 essentially and it deserves the same kind of respect for the nature of the 25 consequences, in my opinion.
26 Next slide. So, I just told you that the randomized clinical trial,
1 this randomized experiment where you can assign people to two groups and chance 2 alone determines which one they end up in so that they are comparable in every
way except for that key thing you want to look at in terms of cause and effect, I said 3
4 that is the best form of evidence, and it is. It is the best form of evidence. 5 However, do we have a lot of that type of evidence in this field 6 that you can draw on? Now, we've exhorted you through legislation and a number 7 of other things, you must use this, but is there a lot of gold standard level evidence 8 out there about all the things we do on a daily basis in the classroom? 9 No, there isn't. There is some. There's some evidence out there. 10 A lot of the evidence, however, is lower on the hierarchy of the strength of 11 evidence. I am going to just touch on this briefly. Again, the other speakers are 12 going to talk about it in more detail. When did I start?
13 MS. NEUMAN: Like ten of.
14 MS. REYNA: Okay. So, there is a lower level of evidence that
15 we can describe as quasi experimental or large data bases that essentially have lots 16 of characteristics of students in them that you can correlate with one another and 17 you can correlate with outcomes.
18 The idea here is that nobody has been randomly assigned. In the 19 real world randomness is a very rare thing. It's a very artificial thing. In the real 20 world there's lots -- everything's correlated with everything else. 21 Think about the example of socio-economic status. Correlated
22 with everything, you know, your neighborhood, your number of books in the home, 23 all of these things are associated in real life.
24 But when you look at the pattern of associations, you can go in
25 through statistical magic, that's basically it, and you can artificially create a sort of 26 comparison or control by sort of equating people on things. If you look at enough
1 different combinations of people and enough different characteristics you can 2 statistically attempt to control, to capture basically the logic of that gold standard,
the randomized experimental trial. That's always the logic, that's always the goal. 3
4 But here you attempt to do that by statistics. It's not as good. It's 5 a lower level on a hierarchy of evidence, because there could always be something 6 you are not controlling for that in fact is causing your outcome. That's always 7 possible.
8 However, it is second best. It is not nothing. So, for example, 9 you at least know that something is maybe probably true, that there's a large 10 number of what's called in public health epidemiological studies, and there would 11 be an analogy in education to those large studies, lots of attributes, the obvious 12 things controlled for. You know, you could at least say, well, it's probably true. 13 That's certainly better than we have no idea, much better than no evidence, well, 14 what do you think? It's not the top level of evidence, but at least it is evidence. 15 Another thing that is a good source of extrapolation to practice 16 is evidence based theory, and the evidence based theory is the crucial part. 17 Theories whose predictions have been confirmed and disconfirmed -- you know, 18 there's been an opportunity to disconfirm them as well, they've been tested -- that 19 are explanatory, that go into the mechanisms of how people learn, how they learn, 20 what's the process going on.
21 If you know something about how people learn and how an
22 intervention was effected, than you have some clue as to whether you can 23 generalize it to your classroom, because you know the mechanism. You know 24 what's relevant and what's irrelevant to the causal course of that intervention. 25 Is the shoe size of the student relevant? Probably not. Why is 26 that? Because we have an inclusive theory of how learning happens and it doesn't