No Child Left Behind (NCLB) requires that federally funded programs and practices must be grounded in "scientifically based" research (SBR). [1] NCLB references SBR over 111 times and it impacts all education practices.[2] School leaders who depend on federal funding must know the nature of the research that guides their programs and practices. You must understand SBR to understand NCLB.
So what does SBR mean? For guidance we turn to Doris Redfield, Vice President for Research and Director of the Regional Educational Laboratory.[3] Doris sheds light on the matter by discussing:
(transcript table of contents):
1. The balance of burden between providers and educators
2. Where to go to learn more about SBR
3. AEL’s services
4. The six hallmarks of SBR
read as a PDF
|
listen with REALONE (Qs 1-3)
|
listen with REALONE (Q 4)
|
The Doyle Report: SBR is about informed purchasing. This requires information gathering and that entails transaction costs. The law does not specify the format of the information exchange, so will educators be forced to pull the information from providers? Should producers be proactive in showing that they are NCLB compliant?
Doris Redfield: I think it's a two-way street. I think the producer should provide the available evidence, and if the educator does not see the evidence that they are looking for then they should ask about it. So, I think the burden for providing information falls on the producer, and the burden for seeking additional information or judging the quality of that information falls to the educator.
2. Where to go to learn more about SBR [back to contents]
The Doyle Report: If educators must judge the quality of SBR information where can they go, as a quick-fix, to learn more about the subject (other then, of course, the following material)?
Doris Redfield: I think there are a number of good web sites. Right now the U.S. Department of Education has a lot of useful information on their site, and that is at www.ed.gov. There is a-lot of information about things like random assignment and the No Child Left Behind Act and its research requirements on that site, particularly if you look in the section on the Institute for Educational Sciences. Other really good sources are the National Educational Laboratories and Research Centers [that are available at] www.relnetwork.org. [This] will tie you into the regional laboratories […]. AEL is one of those regional laboratories. They work with schools to conduct or apply scientifically-based research.
If a school district is interested in conducting some of its own research, then, again, labs and other research enterprises can help, such as universities.
The bottom-line is that it is good […] that both publishers and educators inform themselves about scientific research; but the truth is that they went to school […] to be experts in other things, like publishing and teaching. So when it comes to actually designing and conducting scientific research they might be well served by consulting with someone who is experienced in conducting such research.
The Doyle Report: The costs in time and effort for educators to learn about SBR and for providers to conduct SBR are high. Can they go to a group like AEL to do the work and analysis in a more efficient way?
Doris Redfield: In fact, AEL has started an institute called the Institute for Advancement of Research in Education, and that institute specializes in scientifically-based research applied in education settings. AEL has a track record of over 35 years of working with schools and state departments of educations and other education policy makers in a number of enterprises that involve the evaluation of education programs using principles of scientific research. We can be reached at 1- and by visiting www.ael.org.
The Doyle Report: Understanding an issue is often about asking the right question. So, what are the top things to know about SBR and what questions should school leaders, policy makers and education consumers ask about it?
Doris Redfield: Something I should say a up-front is that, while I know a lot about No Child Left Behind, and I'm very familiar with the legislation, when I talk about scientifically based research, I really talk about it in the “Catholic” sense of the word. It's hard to do that now without talking about No Child Left Behind because, of course, it has made the principles of scientifically-based research very prominent.
The National Research Council has published a document entitled Scientific Research in Education.[4] The reason that publication is so important is because the panel that wrote it is an independent panel. The National Research Council is an arm of the National Academy of Sciences, established by Congress as an independent body. It pulled together renowned researchers from around the country, from different areas of research, (such as the hard sciences and the social sciences, and so forth) and what they outlined are the six hallmarks of scientific research.
[1] One is that it poses significant questions – meaningful questions – questions that are going to make a difference. In education, it seems to me, one of the most important questions is if this product will actually help the teachers teach better or students learn better.
[2] Another thing that scientific research does is link to relevant theory. What this means for me, in a product arena, is that when someone develops a product or some materials that will be used in schools, […] that they did not just draw it out of a hat - that [they simply think that] this seems like a great idea and that it should work. [Rather, that] they have looked at the research that has gone-on before [and that it] helps them build a case for why, what it is that they developed, should work. Now, they may put a new twist on it or add a new element to it, but, basically, the product should be a logical building block of what is already known about solid and proven approaches to learning.
[3] Another thing that scientific research does is use tools that are valid for the question that they are trying to answer. One of the things that is fairly common right now is to measure by using a norm-reference achievement test. […] Well, that could be problematic because in 49 states, now, they have standards based assessment systems which means schools are held accountable for teaching kids the state's standard. So if you use a measure that is not aligned […] with that state's particular standard, you may get a false reading when you go to measure the results. So the measures that you use should be aligned with the treatment, the intervention.
[back to contents]
[4] Another thing that scientifically based research does is rule-out alternative explanations. In other words, let's say that I develop some instructional materials, and I use those with a treatment group or an experimental group of students, and the other group does not get that same set of materials; are there other things that could explain the changes that occurred or did not occur? Could it be that one of the groups had more prior experience in this particular topic? Could it be that something happened during the intervention period (like, well, think about 9-11 or something like that) that could have influenced the student's ability to concentrate on the intervention?
There are certain research designs that really are very good at ruling-out counter explanations and those are: experimental-designs and quasi-experimental designs. That is why No Child Left Behind states a preference for those two designs, because of their ability to rule-out counter explanations.
Now, is it always important to rule-out counter explanations? That is a good question, but [ruling-out] is important if you are trying to make a statement about cause and effect. If you're making a statement that a particular product causes or contributes to causing a certain outcome, then an experimental-design or a quasi-experimental design is really the design of choice. If on the other hand, you are doing some exploratory work and you are trying to find out if a relationship even exist before you test cause and effect, you might use a correlation technique. The unique thing about experiments and quasi-experiments is that they have control groups, and control groups are very important to ruling-out counter explanations.
The difference between an experiment and a quasi-experiment is that, in an experiment, the people […] are randomly selected from a population to which the results are going to be generalized, and/or the people participating in the experiment are randomly assigned to […] control groups. Whereas in quasi-experiments, randomization is not feasible. For example, you may just take two classes […] [without any] random assignment, which often is the case in the real world of schools.
[5] The other thing scientifically based research studies do is produce findings that can be replicated. What that means is that the researcher or the publisher needs to describe what they did to demonstrate their claims so that someone else can do it. It needs to be clear and carefully and precisely defined.
[6] The final thing that scientific research does is submit itself to review by external third parties. […] Publishers [will often ask] if this means that a third party needs to conduct [their] research. Not necessarily, but a third party should definitely review the research.
So what does all this mean for educators, superintendents, policy makers and so forth? It means that they need to be asking themselves these questions when a publisher or developer of materials makes a claim. They should ask if the claim is related to [their] goals. Is it […] [based on research] or does this seem to be haphazard in its approach? Did it ask the right questions and did it use measures that are important to answering those questions? Am I convinced that they really did rule-out alternative explanations? Did they use a control group? Did they have randomization and can I tell from the information they provided me how the study may be replicated, and has it been submitted to an unbiased review process?
[back to contents]
The Doyle Report: That brings us to the end of another Spotlight Interview. I thank Doris for all her work and time. I also thank you, the listener, for coming to The Doyle Report to learn about scientifically based research. As always, we encourage comments and input, so drop us a line or give us a call.
Until next week, enjoy browsing The Doyle Report.
David A DeSchryver
Issue 3.21
05/20/2003
comments encouraged:
[1] As defined in Title IX of the No Child Left Behind Act the term scientifically based research is defined as: (source: TITLE IX — GENERAL PROVISIONS, PART A SEC. 9101. DEFINITIONS).
(A) means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs; and
(B) includes research that —
(i) employs systematic, empirical methods that draw on observation or experiment;
(ii) involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;
(iii) relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators;
(iv) is evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random-assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls;
(v) ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and
(vi) has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review.