Quasi-experimental Designs

By Carolyn Howard,2014-05-05 21:29
10 views 0
Quasi-experimental Designs

    Quasi-experimental Designs Cozby Ch. 11

    Intro to Experimental Psychology

    Psych 185

    Dr. Isonio

    Golden West College


    ; To date, we have discussed:

     True experimental designs (Chapters 8 and 10)

; What remains is today’s topic:

     Quasi-experimental designs

     These are ―intermediate‖ designs—they lack the randomization and full control of

    extraneous variables that true experiments have but can nevertheless yield

    reasonably reliable conclusions

     For completeness we will also mention a couple of non-experimental designs (what

    Campbell and Stanley call ―pre-experiments‖) that are fraught with confounds

    Internal Validity

    ; Remember: Internal validity refers to the extent to which we are able to clearly attribute the observed differences on the dependent variable to the effects of the independent variable. ; Thus, true experimental designs have high internal validity and quasi-experimental designs have some intermediate level of internal validity

    Threats to Internal Validity

    ; Factors which, if uncontrolled can confound (i.e., ―mess up‖) a


    ; The classic list comes from Campbell and Stanley Threats to Internal Validity

    ; History

    ; Maturation

    ; Repeated Testing

    ; Instrument Decay

    ; Statistical Regression

    ; Selection

    ; Mortality

Threats to Internal Validity

    ; History

     Outside events (beyond the context of the study itself) influence the

    participants during the study, possible affecting their behavior This threat is relevant in any type of repeated measures design

    ; Pretest ? IV ? Posttest

    Threats . . . History

    ; Example of History operating as an internal validity threat:

    ; Study to assess how attitudes toward welfare program are affected by having a personal discussion with recipients

     Pretest measure of attitudes on Monday; discussion occur Tuesday through

    Thursday; posttest measure of attitudes on Friday

     BUT, a special TV documentary reporting on widespread welfare fraud was aired

    on Wednesday evening.

     Results show no difference between pre- and posttest attitude assessments; due to

    true lack of difference, or the confounding effect of the TV show???

Threats to Internal Validity

    ; Maturation

     Any change in the participants (beyond the direct effect of the IV)

    during the course of the study. The changes can be permanent (e.g.,

    physical growth) or temporary (e.g., fatigue)

     Like History, this threat arises in pre-post designs

    Threats . . . Maturation

    ; Example of maturation operating as a threat to internal validity

    ; Study to assess effectiveness of training program to increase capacity of STM; pretest given at noon; program held from 1to 5pm; posttest given at 530pm

    ; Results indicate no difference between pre- and posttest memory scores; due to ineffective program, or . . . fatigue ????

Threats to Internal Validity

    ; Repeated Testing

     This threat occurs when participants are measured multiple times. The

    threat is that they might be somehow changed by the measurement process

    itself. This could then affect their behavior.

    One common outcome of repeated testing: becoming “Test Wise”

    ; Experience with tests affect measurements in sometimes dramatic ways--

     Classic IQ effect: benefit of being test wise is estimated to be about 35 points Personality teststest wise persons appear to be better adjusted

    Threats . . . Repeated testing ; Example of repeated testing operating as an internal validity threat: ; Does a special program teaching mental math increase the speed of doing everyday calculations?

     Pretest (10 arithmetic problems) is given; one-hour of mental math instruction

    follows; then, the posttest is administered (same problems in order to see

    ―improvement‖; results indicate dramatic improvement in calculation speed

     Do you sell the program to school districts OR is the ―improvement‖ a reflection of

    the fact that the participants had seen the problems previously, only an hour

    earlier ???

    Threats to Internal Validity

    ; Instrument decay

     This threat operates whenever a change in the measurement

    instrument itself

     Instrument can be a device (e.g., batteries run down) a human (e.g.,

    changes standards over time or becomes fatigued)

    Threats . . . Instrument decay ; A good example of how instrument decay can threaten internal validity is with the evaluations of writing samples English professors designate a whole day to read and evaluate the

    samples; late in the day as they get tired, they begin to miss or errors

    (or they begin to grade more stringently or . . . )

     The analogy to springs in a scale becoming stretched out over time

    fits well

Threats to Internal Validity

    ; Statistical Regression (regression toward the mean) refers to the natural tendency for extreme values to move closer to average upon subsequent measurement

    ; ―Extremes moderate over time‖

     Any superlative performance by an individual (e.g., in sports) is likely

    to be followed by a more ordinary one from that same individual Threats . . . Statistical Regression ; Example:

     A new therapy for severe depression is evaluated. Twenty severely

    depressed persons go through a two-week ―just smile‖ program; after

    the program, their mood is re-assessed. Results show a marked

    improvementaverage depression level is only mild to moderate. Do you conclude the program is indeed wonderful, or might it just be

    that extremely depressed people will naturally be (at least slightly) less

    depressed two weeks later ????

    Threats to Internal Validity

    ; Selection operates whenever groups used in a comparison are not equivalent at the outset of a study

    ; We have talked about this threat previously

    ; The confound is that we don’t know whether group differences seen on the dependent variable are due to the effects of the independent variable or to pre-existing differences Threats . . . selection

    ; EXAMPLE: Two psychology professors argue over the merits of providing students with class notes. Professor A thinks doing so will send the wrong message to studentsyou have the notes, so there is

    no need to attend class. The other believes that students can understand that the notes are an aid to help them get more out of class lecture/discussion. They decide to do a studyprofessor A makes

    notes available to his students for the remaining 8 weeks of the semester; professor does not do so for the same time period. They compare performance on the final exam and find that student in professor B’s class get better exam scores.

    ; What do you conclude and why?

    Threats to Internal Validity

    ; Mortality refers to the effects of the loss of participants over time in a longitudinal study.

    ; Those remaining in the study, then, may not be representative of the original group nor of the population.

; Participant mortality can result from a variety of factorsliteral

    mortality, moving out the area, losing contact, no longer committed to the project, etc.

    Threats . . . mortality

    ; An example of mortality operating as an internal validity threat: a school district is evaluating the effects of mandatory school uniforms on discipline problems (expectation is that uniforms will cause a reduction in problems). However, during the course of the study, the greatest trouble-makers leave the district. Results indicate a significantly lower level of problems at the end of the first year of uniforms. What do you conclude? Now to the designs

    ; They differ in format (structure, design)

    ; Because of their design they handle (or fail to handle) potential threats to internal validity in different ways

    ; See page 1 of the ―Non-experimental, quasi-experimental and

    true experimental designs‖ Handout

    True experimental Designs

    ; (We discussed these previously)

    ; Posttest only control group design and pretest-posttest control group design (bottom half of page) were covered in chapter 8; factorial designs were introduced in chapter 10

Non-experimental designs

    ; These designs are inherently flawed and fraught with internal validity threats:

     One shot cases study

     One-group pretest-posttest design

     Non-equivalent control group design

    Non-experimental designs

     One shot case study

One-group pretest-posttest design

Non-equivalent control group design

Quasi-experimental Designs Quasi-experimental Designs

    ; Non-equivalent control group pretest-posttest

    ; Interrupted time series

    ; Control series

    ; Retrospective

    ; Reversal

    ; Multiple-baseline across participants

    ; Multiple-baseline across target behaviors

    ; Equivalent time samples

    ; Equivalent materials samples

    ; Proxy pretest

    ; Double pre-test

    Nonequivalent Control Group Pretest-Posttest Nonequivalent Control Group Pretest-Posttest

    Design Design

    ; This design salvages the non-equivalent control group posttest only design

    ; This is a strong quasi-experimental design

    ; A good choice when it is not possible to randomly assign to conditions. ; Provides good control for history, maturation, testing, instrumentation, and mortality as threats to internal validity

    ; Example—see handout and ―Notel‖ discussion in Cozby (p. 202)

    Interrupted Time Series Design Interrupted Time Series Design

    ; A series of measures both before and after the implementation of the treatment condition are compared.

    ; See biofeedback example in Handout and Connecticut traffic fatalities (Cozby, Figure 11.2, pg. 203)

    ; Weakness is definitely failure to control for history effects Control Series Design Control Series Design

    ; Like the interrupted time series, multiple pre- and post measures are taken and compared. However, a control group is also tracked, allowing for a ―between groups‖ comparison

    ; ExampleHandout: DARE program; Cozby Figure 11.3

    Retrospective Study Retrospective Study

    Existing, archival data are compiled to test the effects of some past treatment (IV).

    Most serious threat to validity = history

Example: Effect of establishing Outreach Office at GWC

    Reversal Designs Reversal Designs

    ; Comparison of multiple periods, designated by letters (A, B, C) in which treatment is alternatively either absent or present.

    ; The design attempts to establish the link between the administration of the treatment and a specific behavior change

    ; Example: Handouttruth brushing in developmentally disabled men

    ; ABA; ABAB

    Multiple Baseline Designs Multiple Baseline Designs

    ; Multiple baseline designs are similar to plain reversal designs, except that the treatment is introduced at different times for different subjects or targeting different behaviors for a single subject

    ; Across Subjects see Handout: token system to reinforce tooth-brushing in subjects A, B, C

    ; Across Behaviors -- see Handout: token system applied to three self-care behaviors across three phases

    Equivalent Time Series Designs Equivalent Time Series Designs

    ; Useful when the treatment effect is short lasting and no control group is readily available

    ; Example: see Handout: effect of music on industrial production

Equivalent Materials Samples Design Equivalent Materials Samples Design

    ; Various sets of materials deemed ―equivalent‖ are used in a repeated measures format, altering levels of treatment because exposure to a set of materials is likely to cause some ―carry over‖

    ; Example: see Handoutdistributed vs. massed practice with different sets of to-be-remembered words

    Proxy Pretest Design Proxy Pretest Design

    ; This design is similar to the standard pre-post design, but in this case the ―pretest‖ measure is collected at the end of the study. Actually, a

    proxy variable is used to establish (after the fact) where the groups would have been on the pretest

    ; Two variations:

     Recollection proxy pretest design

     Archived proxy pretest design

    Double Pretest Design Double Pretest Design

    ; This design builds on the moderately strong nonequivalent control group pretest-posttest design by adding a pretest for both groups.

    ; This allows for an assessment of how the two groups might be changing prior to the start of the program

Report this document

For any questions or suggestions please email