Edited by Peter Kiefer
May 28, 2004
The court system is at the nexus of many perspectives. What is legal, what is moral, what is practical, what is effective, and what is political? Many times these differing views wash over onto each other, yet we are still forced to separate them and deal with each individually.
The court decides to institute a drug court and contracts with Professor Williams, a local university professor, to conduct a longitudinal evaluation of the pilot. The professor joins the drug court team in the pilot‟s development phase before the first participant is ever ordered into treatment. Prof. Williams analyzes the metropolitan area along ethnic and socioeconomic lines, and then convinces the team (including the drug court judge) to divide the city, by zip code, into two cohort zones. Defendants from the drug court zones
will be sentenced to drug court; defendants from the control group zones will be
processed according to standard drug offense procedures.
Once the drug court pilot starts, the professor begins measuring, analyzing, and compiling data for his eventual evaluation. His plan is to track participants and control group members through treatment and either graduation or termination, then follow up for three years afterwards.
The drug court and the evaluation move along with few glitches for about a year until two defense attorneys (within three months of each other) appeal their clients‟ drug sentences. Attorney Dennis McKay‟s client is sentenced to the control group; attorney Dora
Kemper‟s client is sentenced to drug court. Both sentences are based on the defendant‟s residence at the time of arrest. Both attorneys claim their clients‟ rights under the equal protection clause of the state constitution were violated. Arguing before the state Supreme Court, McKay and Kemper claim their clients received inherently different (and 1therefore unequal) treatment based on nothing more than geography. Both attorneys
also take their cases to the court of public opinion, writing letters to the editor of the local newspaper and accepting joint invitations to appear on several talk radio shows.
While the decision from the State Supreme Court is pending, I hold another of my infamous backyard barbeques where my obnoxious neighbor (you may remember him from our discussion on the death penalty) once again corners me about how unfair courts are in this country. “Excuse me” he starts, “I could have sworn you told me that courts were the last bastion for the protection of individual rights. You didn‟t let on that the
1 My thanks to Ms. Darian Taylor, Maricopa Superior Court‟s Capital Case Staff Counsel, in evaluating the legal
aspects of Kemper and McKay‟s legal challenge. Justice Brandeis once commented on this theme, “[to] stay experimentation in things social and economic is a grave responsibility. Denial of the right to experiment may be fraught with serious consequences to the nation. It is one of the happy incidents of the federal system that a single courageous state may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.” See New State Ice Co. v. Liebmann, 285 U.S. 262 (1932).
courts are also test tubes for experimentation with drug addicts saving them from punishments they so richly deserve.”
I have asked Jim Murchison, Court Administrator of the Marion County trial courts, in Salem, Oregon; the Honorable Gary Carlson, Judge of the Taylor County Circuit Court, in Medford, Wisconsin; and Gwendolyn Lyford, Court Administrator in Anchorage, Alaska, to comment on the scenario.
What has been your experience with evaluators in pilot programs?
Jim Murchison noted that he has seen numerous evaluation reports of many programs over the years. These reports are usually (but not always) intended to meet federal funding requirements. While there have been exceptions, his impressions have not been favorable. “Evaluators seem to write glowing reports, praising the program, often writing whatever the program managers want written. (After all, there is a self–interest for the
professional evaluators.)” His experience has been that evaluations seldom provide a
truly independent, in–depth analysis of a program. An evaluation may be highly glowing, or merely glowing, but it is seldom negative, and truly based upon an honest, objective, and analytically sound assessment.”
Judge Carlson shared Jim‟s skepticism and commented that a good evaluator must have a thorough understanding of how the court system really works. He noted that often evaluators come from a non–law-related field in academia without significant experience in the courts. “We‟ve probably all had the experience of somebody coming into the courts, looking around for a while and them making a suggestion that is immediately recognized as impossible or illegal: „Yes, Mr. Evaluator, it is true that if we kept all
defendants in jail pending trial it would speed up the process because there wouldn‟t be any worry about continuances from non–appearances, but, you see, there is this little
thing called „the Constitution....‟” The judge did offer a positive note by adding that
outsiders can make excellent proposals due to their fresh look at a situation in which the rest of us are wearing blinders.
Is separating defendants into different groups for experimentation inherently unfair? Judge Carlson put forth a legal perspective. He thought separating defendants for purposes of experimentation is not inherently unfair unless we use inappropriate criteria
to make the separation. Inappropriate criteria is generally recognized as violating discrimination laws, such as race, creed, color, sex, national origin, etc. “For example, „all Catholics, line up over there. You‟re going into the rehabilitative module; all others, line up over there for the incarceration module.‟”
He then looked at the scenario from a moral perspective. “The problem with using
criminal defendants as subjects for this type of zoological research (isn‟t that a pejorative way to phrase it?) is that it smacks of unfair play. A defendant is entitled to be treated with individual consideration while trying to emphasize the societal goals of the criminal
justice system: rehabilitation, deterrence, and punishment. It seems „unfair‟ to deny to a particular defendant a rehabilitative program to which the defendant might respond positively just to justify the existence of a control group to test a theory.”
Next, he took a very practical perspective drawing an analogy with the medical profession‟s dilemma when new medications go through clinical trials. “Perhaps a page need be drawn from the medical field‟s ethical requirement that if the experimental
program indicates that the medication is significantly effective and withholding the value of the medication to the control group would significantly damage the control group, stop the trial and make the new drug available to both as soon as possible.”
Finally, he joined Justice Brandeis in seeing the political perspective, noting that our nation was structured to provide for just this kind of experimentation. “We just use different jurisdictions as the control groups. County A has the resources for Program 1; county B doesn‟t have the resources for Program 1 so it isn‟t offered. The study thus compares County A with County B. Perfectly fine. Legal. Okey dokey. No problem because no inherent discrimination is occurring.”
Wendy Lyford recently had a university researcher invite her court to participate in a project examining the effectiveness of batterers‟ intervention programs in criminal
domestic violence cases. The researcher wanted to replicate a study conducted in the Bronx. His proposal was to have the court order defendants to attend or not attend a program based on random criteria (i.e. case number or the day of the week the defendant was sentenced). Random separation was not reasonable for her court. “Our state now
mandates a 36-week program that costs almost $1000 to complete. Our defense attorneys report that BIP [batterer intervention programs] participation is the sentencing component that their clients most often want excluded from a plea agreement. More importantly, in DV cases the victim impact of a sentencing decision is particularly critical factor.” No form of random selection for sentencing options would be reasonable in this situation.
Jim looked at the scenario from the perspective of effectiveness. “Regardless of how one designates eligibility for any program, the success or value of a new approach to any problem should be reviewed. If there is legitimate interest in outcomes, there must be some way to measure it. There are various methods for creating a control group, or for even identifying randomly selected participants vs. non–participants. Statisticians can
develop the methodology and assure that it provides a statistically valid database for analysis. Without such ability to compare outcomes, any effort to critically grade the project is seriously impaired (read: doomed).”
Can courts experimenting with sentences be fair under any circumstances? While there probably is no such thing as a truly “victimless” crime, Wendy saw programs
such as drug court as a better place to experiment. Experimentation is never appropriate when victims are routinely involved such as in domestic violence cases. She based her argument, not just on fairness, but also the perspective of political realities. “One of the
bigger challenges of our fledgling therapeutic courts has been recruitment, because many criminal defendants correctly perceive that the requirements of drug/alcohol court
participation is more onerous than a „traditional‟ criminal sentence.” Having a
conservative legislature, Wendy said the Alaskan courts “. . . have generated good support for therapeutic programs by emphasizing good results and cost savings and the
challenges an individual faces in successfully completing a therapeutic court program.”
Jim concluded that experimentation can be fair and from a practical standpoint is absolutely necessary. “A court must measure whether or not a program works, to decide whether or not to continue to use it. If there was no experimentation, there never would have been the first anger management class for domestic violence offenders; there never would have been a traffic school for offending drivers; arbitration and mediation would not have become an integral part of court systems throughout the country; jury selection would not be conducted individually in one court and in groups in another; jurors would not be submitting questions during trial in those jurisdictions; and the list goes on and on. If we had no experimentation, only those procedures steeped in tradition would be available: a dangerous and disturbing proposal in an ever-changing world.”
Judge Carlson disagreed, saying that strict sociological experiments cannot be fair. “Certainly as a judge I would not feel comfortable assigning defendants randomly with a number and then saying all even-numbered defendants will go to jail while all odd-numbered defendants will go into a rehabilitative program.”
After that warning, however, he said experiments with different sentences could be based on more individualized factors concerning the defendant or the crime. He suggested calling it “trial and error” rather than “clinical trials.” “I might try out a certain program
or option if it appears it might be appropriate for Defendant A while not appropriate for Defendant B. If Defendant A shows some success with the program, I am more inclined to try it again.”
Understanding that academia has a different agenda and value set than judicial administrators, are there ground rules we can suggest in dealing with evaluators? Jim proposed that the court administrator and the academician evaluator must establish their relationship very clearly and early on. “Roles, expectations, and interaction should be fully understood from the start. The evaluation process should be spelled out in advance, the data necessary to be collected must be defined, and how it will be collected must be established. The goals of the program, and how those goals will be measured, should be defined. The desire for a legitimate and critical outcome-based evaluation must be clearly agreed upon.” He stressed the need for full and open communication, and an agreement on the need for a thorough and true analysis and evaluation, regardless of the conclusions. “. . . the evaluation must be as valid and statistically correct as possible.”
Wendy pointed out that first and foremost, a judge must always have the authority to exclude a defendant from a program. In addition, the court must retain the right to terminate the program, change the selection criteria or change other basic “ground rules” at any time if the court determines the program conflicts with the court‟s basic mission.
As Judge Carlson pointed out, the scientific model does not lend itself to the criminal justice system due to the constitutional constraints of due process and equal protection. He offered the following guidelines in working with consultants or evaluators:
1. The evaluator‟s knowledge of the court system must be explored;
2. The specific parameters or limitations of the experimental program being
considered must be fully disclosed to the court personnel involved in the program;
3. The legal or policy limitations under which the court is required to operate must
be fully disclosed to the evaluators.
4. Apparent deficiencies in knowledge or expertise by either the evaluator or court
staff should be examined and explained;
5. Intended differences between groups being proposed for the studies must be
evaluated from the standpoint of legal sufficiency, particularly regarding how
participants will be included or excluded from any group;
6. A full disclosure of the program being conducted must be explained to the
participants with the option for an “opt out” by any particular defendant;
7. Specific measurable criteria should be established by which to measure the
8. The program should have not only a beginning and an end date, but also interim
assessment reviews to consider preliminary results;
9. Interim assessment reviews indicating the program is having a significant
deleterious or disadvantageous effect on the experimental group should result in
termination of the program.
10. Interim assessment reviews indicating the program is having a significant
advantageous effect on the experimental group should result in termination of the
program so as to make the program available to all possible participants.
I want to thank Jim, Wendy, and Judge Carlson for their experiences and insights regarding experimenting with defendants and evaluating court programs. Diagnosing social ills, creating new programs, then determining the effectiveness of those very programs is a necessary and yet complex task. There is disagreement on the moral fairness of experimentation and evaluation and some differences on how politics can affect a program; legally the practice appears on good footing; from a practical and effectiveness standpoint, it is necessary. Jim and the judge have given us much to think about as we strive to administer justice in the future. If you have an ethical issue you would like to have discussed, or you would like to comment on this scenario, please contact me at firstname.lastname@example.org.