A Preliminary Investigation of Empirical Methodologies
Applicable to Criminal Justice Reform
Song Yinghui and He Ting
Beginning in 2004, with funding assistance from the Ford Foundation and others, we launched an empirical legal research project based on several studies in different regions of China. During the process of implementing the project, we made a preliminary investigation of methods for conducting empirical legal research in China. Below we will integrate the empirical research we have conducted with a discussion of several questions related to study-based empirical research. I. The Fundamental Stages of Study-Based Empirical Research
The key difference between study-based empirical research and other empirical research lies in designing certain legal systems or operating methods, and testing them in order to observe their effects. The aim of the empirical research is not only to discover problems existing in judicial practice, but also to conduct studies which, within certain parameters, designs specific solutions and verifies the effects and feasibility of those solutions. Study-based empirical research is more comprehensive and its conclusions more persuasive, while its standards for empirical methodology are correspondingly higher. This is expressed first in terms of the time frame; study-based empirical research is a complete process involving distinct, successive stages, rather than being limited to merely an observation of judicial practices. Based on our experience, conducting comprehensive study-based empirical research in China should include at least the following fundamental stages:
1. Setting up the Project
Setting up the project is the start-up stage of a study-based empirical research project. The primary task in this stage is to define the focus of the empirical research, which is also the object of the study. In theory, any legal system can become a focus for empirical research, but not every legal system is suitable for conducting study-based empirical research, especially given that empirical legal research in China is still in its early stages. Defining the focus of the study-based empirical research requires insight into judicial practices in China and sensitivity to the direction of China’s judicial
reforms. An analysis should be made of whether the project is both necessary and feasible. Necessary means an urgent need for improvement in the current phase of the legal system. The development of China’s legal system is at present still in its early stages, and there are many problems needing
improvement. But these problems can be classified by their severity and urgency. Some problems have an overall impact and urgently require resolution, while the urgency of others is relatively low. To define the focus of the research, one must first choose those legal systems that urgently require improvement. For example, we launched an empirical research project focusing on awaiting trial on bail, because awaiting trial on bail is a key measure in China’s system of compulsory measures, and improving it will effectively lower the rate of pre-trial detention in China and spur the reform of pre-trial procedures in criminal litigation, thereby meeting our definition of necessary.
Feasible refers to the feasibility of conducting empirical research at the present time in China. Conducting study-based empirical research requires the cooperation of judges, prosecutors, and police, but not all projects are able to garner both their attention and their desire to participate in the study. Feasibility must therefore be considered from the standpoint of judges, prosecutors, and police. In
addition, for feasibility one must take into consideration the overall direction and development of China’s judicial reforms, selecting to the extent possible legal systems that are, or will soon be, part of the reform plan. This will ensure that the results of the empirical research will be utilized as quickly as possible in legislation and in the judiciary.
2. Pre-Study Preparation
The pre-study preparation stage is for completing all preparatory work for the study about to begin, focusing on the object of the empirical research. There are four main tasks in this stage, which include:
1) Gain a full understanding of all relevant circumstances surrounding the project. Understanding the specific operational and developmental environment of the study in different regions of China is a precondition for successful implementation. Because of China’s large size, there are vast differences between its eastern and western regions; it is therefore necessary to fully understand the circumstances in different areas. For our empirical research on victim-offender mediation, we selected Shanghai, Zhejiang, Jiangsu, and Shandong in the eastern region; Hebei and Hunan in the central region; and Sichuan and Gansu in the western region. We also focused in on certain cities within these provinces and municipalities in order to understand how victim-offender mediation specifically operated there. Our methodology included questionnaires, interviews, case reviews, on-site observation, and other methods.
2) Determine the location(s) of the study. Studies are conducted within certain geographic areas; therefore the choice of locations for the study is extremely important. The locations selected should be representative. As an example, for a juvenile justice project, one should consider the representativeness of the juvenile criminal situation and social and economic development of a certain location. In addition, the study locations cannot be too large and there cannot be too many; otherwise it may results in loss of control over the study. Nor can the study locations be too small or too few, which may result in the impact of the study being too small or its results lacking persuasiveness. Similarly, due to China’s specific circumstances, representative locations should be chosen for the
study. It is best to choose two or more study locations representing different regions. For example, we chose two locations within the eastern Jiangsu Province for our empirical research on victim-offender mediation, and one location in the central Hebei Province. The choice of study location should be decided upon after also considering and synthesizing an entire set of factors such as cultural considerations, cases, personnel, and other elements.
3) Work out the study plan. The study plan consists of the specific methods used to conduct the study, including procedures, operating rules, and data collection methods. The study plan should be formulated after the study location has been determined, and it should be designed based on the specifics of the object and locations of the study. After the preliminary study plan is worked out, one should solicit the opinions of judges, prosecutors, and police cooperating with the study in order to increase the operability of the study plan.
4) Train those participating in the study. Those participating in the study, other than the researchers, include judges, prosecutors, police, and others; it is extremely important to the smooth implementation of the study to train the latter to promote their full understanding of the intent and specific methodologies of the study. During our empirical research on victim-offender mediation, we trained participants using both centralized discussions and separate interactions. Prior to the study, we
conducted discussion sessions on victim-offender mediation, bringing all of the study participants from all study locations together in one place to train them, using lectures and discussion. In addition, during the implementation of the study, we communicated, negotiated, and resolved any questions arising at the study locations, as needed.
3. Study Implementation
The implementation of the study is the key stage of the entire empirical research project. The main tasks during the study implementation stage include the following:
1) Test in actual cases the new systems, processes, or methods designed for the study. The new systems, processes, or methods are theoretical hypotheses for improving the various problems existing in actual judicial practice, which assert that such new systems, processes, or methods are able to resolve problems in actual practice. The implementation of the study is the testing of these hypotheses using actual cases. Here the application of a new system, process, or method becomes an independent variable, and the results in the actual case become the dependent variable(s). The object of the study is to examine whether the relationship between the independent variable and the dependent variable(s) coincides with the hypothesis. For example, in judicial practice, it is difficult to resolve the conflict between an offender and a victim using the criminal litigation process; it is difficult to guarantee the victim’s rights, and the over-use of the short-term free sentence results in abuses and other problems. We hypothesized that the use of victim-offender mediation can, to a great degree, resolve the problems described above, and we tested this hypothesis by using victim-offender mediation in actual cases.
2) Collect relevant data during the process of handling of actual cases. Empirical research ultimately relies upon data to shed light on problems. Various types of data must therefore be collected in the process of testing the study’s new systems, processes, or methods in actual cases. This
point will be expounded upon below.
3) Make a record of occurrences during the study that are relevant to the study, especially any factors that might restrict the progress of the study, in order to comprehensively evaluate the applicability of the study’s new systems, processes, or methods to actual judicial practice. A key
element in the persuasiveness of the study’s conclusions is whether the study environment remains the same as a typical judicial environment, but the implementation of a study often gives rise to changes that favor the study and cause the judicial environment to benefit the implementation of the study. Under such circumstances, the study’s conclusions must take into account various factors possibly
impacting the new systems, processes, or methods in a typical judicial environment. These factors must also be fully recorded during the process of the study. For example, during our empirical research on awaiting trial on bail, our researchers were deeply involved in the study and took on some of the work normally done in a typical judicial environment by prosecutors or police, including social background research and monitoring and supervision during the period of awaiting trial on bail. This human resources factor had a rather small impact on the study, but could possibly have had a rather large, negative impact in a typical judicial environment. In order to comprehensively evaluate the applicability to judicial practice of the process and system that we designed for awaiting trial on bail, we had to record the volume of the prosecution and police work that was done by our researchers, including specific numbers of persons, times, and tasks.
Another important question during the implementation stage is the study’s time frame. In theory,
the longer the time frame of the study, the easier it is to avoid the impact of various random factors, and the more objective the results. But, considering the need for the research results to become legislation, the time frame of the study should not be overly long. Based on our experience, it is suitable for an empirical research study in China to take over one year; only by continuing for over one year can the study extend beyond the administrative year of the entities handling cases. An effort also needs to be made to reduce the influence of random factors like caseload and case type.
4. Post-Study Follow-up
Some of a study’s results can be realized in the process of handling cases during the study’s
implementation, such as changes in the pre-trial detention rate or the use rate of the short-term free sentence. Some longer-lasting impacts, however, cannot be directly observed during the implementation of the study, and the cases studied can only to be analyzed a certain period after being closed. For example, an understanding of the return of offenders to society and whether they re-offend after victim-offender mediation, or of the restoration of relationships between offenders and victims, requires the passage of a period of time. Post-study follow-up is the stage in which an understanding is obtained of study results some time after the cases within the scope of the study are closed. The importance of the post-study follow-up cannot be underestimated, in view of the entire study-based empirical research process. The conclusions of a study-based empirical research project must be based on information obtained during the post-study follow-up. The conclusions of a project that lacks post-study follow-up will not be persuasive.
The post-study follow-up of different studies will need to focus on different areas, so the follow-up should be designed based on the research objective and the specific focus of the study. The follow-up outline and data-collection forms should be formulated prior to beginning the follow-up. For example, during our research on victim-offender mediation, the data that we needed to collect during the follow-up included the offender’s current situation and whether re-offense had occurred,
whether mediation had mitigated the harmfulness of the crime, the attitudes of both the offender and
1the victim toward victim-offender mediation, and the recognition of the restoration of the
2relationship between the offender and the victim (see Table I). Different methods can be used to
collect the different types of information required for the follow-up. For example, for the offender’s
current situation and whether re-offense had occurred, we would question both of the offender as well as the people around him or her and people in the neighborhood. For the attitudes of the offender and the victim toward victim-offender mediation, we would primarily question the parties themselves. In situations in which there is a large caseload requiring follow-up, telephone follow-up could be used for some of the cases, while in-person questioning could be used for the more representative, key cases.
Table I？Information Collected in Victim-Offender Mediation Follow-up
Case No. and Group (very important)
Original Case No. ___-___-____ Study Group No. Control Group No.
1 During the implementation of the study, we had already collected information on the offenders’ and victims’ attitudes
toward mediation. During the follow-up stage, after a period of time, we again noted the offenders’ and victims’ attitudes
toward mediation, to determine whether there had been a change in attitude on the part of the offenders and victims, and also to examine the long-term impact of victim-offender mediation. 2 Only for cases in which the two parties were acquainted prior to the occurrence of the crime.
Offender Name Current address
? Sentence completed;including unprosecuted & withdrawn cases；
Sentence ? Still on probation or sentence not completed
? Still detained
School or ? In School ? Employed ? Fully unemployed
Employment Status ? Unemployed but assisting with household work
? Has re-offended;incl. currently under litigation； ? Has not re-offended Re-Offense Status ? Has not re-offended but subject to other penalty such as public order
Attitude Toward Description: ? Satisfied ? Basically Victim-Offender satisfied ? Not satisfied Mediation
Victim Name Current address
Attitude Toward Description: ? Satisfied ? Basically Victim-Offender satisfied ? Not satisfied Mediation
Mediation’s ? Crime has no impact Description of impact:
Mitigation of ? Still some impact
Crime’s Impact ? Still significant impact
Restoration of Relationship between Offender and Victim (when acquainted prior to crime)
? Both parties feel it is restored ? Offender feels it is restored ? Victim feels it is restored
? Both parties feel it is not restored
Because of the highly mobile population in China at present, one of the thorniest problems in our study-based empirical research projects has been how to make contact with parties to cases and engage in successful follow-up some time after the cases have been closed. We often encounter this kind of situation: at the time of follow-up, some parties no longer live in the city in which the study is taking place and there is no way to contact them. In theory, it is unavoidable and acceptable to lose a portion of the information during follow-up, but if a large number of the parties cannot be contacted, the study’s conclusion will lose its persuasiveness. Under these circumstances, we believe that the following work can be done in an effort to maintain contact with the parties: 1) while the case is still active, obtain all of the parties’ contact information, including that of their close relatives and friends;
2) after the case is closed, periodically keep in touch with the parties until the follow-up begins and, if appropriate, begin follow-up early. Based on our experience, the first follow-up is best begun between six months and one year after the case is closed.
5. Evaluation of Study Results
After the conclusion of the post-study follow-up, an overall evaluation must be made of the results of the study based on the data collected during the implementation period and during the follow-up. An evaluation of the results of the study actually consists of using a set of data obtained during the handling of specific cases to test whether the actual effects of the systems, processes, or
methods designed in the study were the same as predicted in the project plan and whether they can resolve problems in judicial practice. An evaluation of the study’s results needs to address each
objective hypothesized during the design of the study. For each objective there needs to be a specific target measure; these specific target measures decide which data should be used in the evaluation and how they should be collected. As Table II shows, while designing the victim-offender mediation study we hypothesized that victim-offender mediation would be able to achieve each target listed in the target column, including promoting the criminals’ return to society and the restoration of the social
relationships ruptured by the crime. The evaluation of the study’s results therefore needed to be based
upon the specific target measures, such as the re-offense rate and the rate of entering school or finding employment.
Table II: Evaluation of Results of Victim-Offender Mediation Study
Objective Target Measure Data Collection Method
Promote Re-offense rate Follow-up (question and survey for information
offenders’ return from parties to case and people around them)
to society School attendance & Follow-up (question parties to case and people
employment rate around them)
Promote Victim satisfaction rate Follow-up (question parties to case)
restoration of Offender satisfaction rate
social Rate of restored Follow-up (question parties to case)
relationships relationships between
ruptured by the previously acquainted
4Neighborhood positive ) Follow-up (interviews in neighborhood
Reduce use of Proportion of un-litigated or Collect data during study
the short-term withdrawn cases & use of
free sentence sentences, suspended
sentences & fines
Increase pre-trial Proportion of un-litigated or Collect data during study
division of cases withdrawn cases in the
Save judicial Decreased use of the Select and analyze certain cases
resources short-term free sentence
Increased pre-trial division
The ideal conclusion of the evaluation would be that each specific target measure indicates that the study achieved each of its predicted objectives, which would demonstrate that the new systems,
3 To calculate this target measure, we first selected cases that had a relatively large impact on the neighborhood and then conducted the comparison within this scope.
4 For the neighborhood follow-up, we determined a specific number of interviewees based on actual conditions in the neighborhood and, if more than half the persons interviewed gave positive evaluations, the entire neighborhood was deemed to have made a positive evaluation.
processes, or methods designed for the study are worth disseminating. In reality, however, there may be a gap between each separate target measure and the ideal. There are many factors leading to this situation, including problems in the study design, as well as the impact of some objective factors, and perhaps even that he new systems, processes, or methods designed for the study are not matched to actual circumstances in China. The evaluation of the study results needs to objectively analyze such circumstances; only then can the conclusions of the empirical research project be objective and persuasive.
6. Dissemination of Research Results
Only when the results of a successful study-based empirical research project are well disseminated can its value be more fully realized. Methods for disseminating results include publicizing through the media, holding discussion sessions, and publishing the research results. The most important and effective dissemination method is the systemization of the research results; that is, the new systems, processes, or methods designed for the study are established in a systematic fashion and applied on a broader scale. This includes systemization at both the national and local levels. At the national level it primarily means that legislative suggestions based on the research results are absorbed into state laws, judicial interpretations, or judicial policies and then applied across the country. Systemization at the local level primarily involves the research results being absorbed into local judicial practices, including judges, prosecutors and police who participated in the study continuing, after the end of the study, to use the study’s new systems, processes, or methods. It also
includes judges, prosecutors and police from other localities learning from the research results and using them in their own practices. Systemization at the local level is a necessary preparation stage for systemization at the national level. After systemization at the local level reaches a certain scale, it can frequently be applied at the national level. Our empirical research always has, as an important objective, perfecting the law and improving the judiciary. For juvenile cases, reforms involving expanding the use of awaiting trial on bail and non-litigation have already been disseminated to a certain degree. Our victim-offender mediation study is also working hard to spur reforms in the law and in judicial interpretation, for the benefit of a harmonious society.
II. Dividing the Study Group and the Control Group
The study group is comprised of all cases on which the new systems, processes, or methods designed for the study are tested. The control group is comprised of cases comparative to those of the study group, on which the new systems, processes, or methods designed for the study are not tested, but which are included in the observational scope of the study, and for which data must be collected. The study group and control group must be divided because the relationship between the independent variable (the application of the new systems, processes, or methods) and the dependent variable (the results of the cases) cannot be proven only by testing the new systems, processes, or methods on the study group cases. A conclusion can only be made after using the control group of cases as a reference against which to conduct a comparison. In other words, when the other circumstances of the cases in the study group and in the control group remain essentially the same, any effects on the study group cases that are different from those of the control group cases can then be attributed to the new systems, processes, or methods tested on the study group. For example, during the victim-offender mediation research project, the rate of study group cases not litigated or withdrawn in the pre-trial phase was significantly higher than that of the control group, which demonstrated that the victim-offender
mediation tested on the study group was the primary factor leading to an increase in cases not litigated or withdrawn in the pre-trial phase. It further demonstrated that victim-offender mediation had a significant, positive impact on an increase in the pre-trial division of cases. Study and control groups must therefore be created for a study-based empirical research project.
The division of a study group from a control group actually pervades the research project from beginning to end. When formulating the study, one must determine a method for separating the study group from the control group. During the implementation stage, the study is tested on the study group cases and relevant data are collected. Though the study is not conducted on the control group cases, the relevant data must still be collected. During the follow-up stage of the study, follow-up must be done on both the study group cases and control group cases. When evaluating the study’s results, a
final conclusion can only be made after comparing the circumstances of the study group with those of the control group
The rational division of a study group and a control group is an important guarantee of the objectivity and persuasiveness of the research project’s conclusions. We discovered, however, a very
thorny problem in attempting such a division between the study group and the control group in a study-based empirical research project. The inappropriate division of the study group and control group can lead to confusion between the two groups, impacting the ultimate evaluation of the results of the study. Based on our experience with study-based empirical research projects, the following two principles should be followed when defining the study group and the control group: 1) relative independence, which means that the study group and the control group should be independent of one another, and should not influence each other or become mixed together. Relative independence is required because, if the study group and the control group influence each other or become mixed together, it is difficult to clearly observe the effects of the new systems, processes, or methods by comparing the study group with the control group, which will then severely impact the evaluation of the study’s results; 2) similarity, which means that the composition and basic circumstances of the study group should be similar to those of the control group. The principle of similarity must be thoroughly applied when dividing the study group and the control group because, if there is a relatively large gap between study group cases and control group cases, then a difference in results between study group cases and control group cases could be caused by the innate differences between the cases and perhaps not largely attributable to the new systems, processes, or methods tested. This could severely impact the objectivity and persuasiveness of the study’s conclusions. We should point out that the principles of relative independence and similarity may contradict each other in the process of dividing a study group and a control group. Finding similar cases for the study and control groups may impact their relative independence from each other, and vice versa. This means that we must balance these two principles when creating a study group and a control group.
In our study-based empirical research, we use different methods to divide study groups and control groups.
One method is based on the different entities handling cases, putting the cases handled by certain entities into the study group, and cases handled by certain other entities into the control group. During our research on awaiting trial on bail, for example, we chose six police stations, A, B, C, D, E, and F which belonged to District H in City Q, to conduct our study. Among them, A, B and C were selected for the study group and our new method for awaiting trial on bail was to be tested on the cases from these police station, according to the study protocol; D, E and F were selected for the control group
and would continue handling awaiting trial on bail as before. In order to guarantee the relative independence between the study and control groups, we selected police stations based on certain standards. A and D were urban police stations , B and E were suburban police stations, and C and F were rural police stations. The overall caseload and types of cases for each pair of police stations were essentially similar. In this way, the study group cases handled by police stations A, B and C, were essentially the same as the control group cases handled by police stations D, E and F in terms of caseload and types of cases. The two groups were therefore fundamentally similar. This method of dividing the study and control groups maintained relative independence between the two groups quite well, but the groups were not comparable enough in terms of similarity. Although we had been painstaking in our choices of police stations in order to achieve similarity between the study and control groups, other objective factors threatened to spoil the similarity because there were totally
5separate entities handling the cases. In addition, using such separate entities handling cases as the
basis for dividing the study and control groups can typically only be applied where the jurisdictions of such entities are relatively small and they are adjacent or located very close to one another. If the jurisdictions of the entities handling the cases are as large as a county, a district, or even a city, or the distance between the jurisdictions is relatively great, one needs to reconsider whether this method of dividing the study and control groups can guarantee the principle of similarity described above.
Another method we use to create study and control groups is to select, from within the same case-handling entity and based on a certain standard, cases for both the study group and the control group. During our research on victim-offender mediation, we chose for the study eight local procuratorates in two provinces. In each procuratorate, we divided the study group and control group cases based on whether the case number was odd or even. All of those among the odd-numbered cases that met our standards, and on which victim-offender mediation could be tested, became the study group. All of those among the even-numbered cases that met our standards, and on which victim-offender mediation could be tested, became the control group. Because of the rather long time frame of this study and because the odd and even numbered cases were randomly selected, the caseload and types of cases in the study group and the control group could be considered essentially the same. Also, because there was one entity handling the cases, and the cases had occurred in the same area, there typically would not be influences from other objective factors, and one could basically guarantee the similarity between the study group and the control group. This method, however, may be lacking in terms of guaranteeing the relative independence between the study group and the control group. Because the cases are handled by the same entity, even if the specific people handling the cases in the study group and the control group are different, it is difficult to guarantee that the study group cases and control group cases will not influence each other. In particular, it is difficult to avoid having the new systems, processes, or methods tested on the study group permeate the handling of the control group cases. In view of guaranteeing a certain number of cases for the study, one should avoid selecting entities with overly-small caseloads when utilizing this method of creating study and control groups, since the cases within the one entity will be divided into two groups.
5 For example, we discovered that the gap between the different police stations’ leadership and case workers’ knowledge
and understanding of the concept of awaiting trial on bail, directly and rather significantly impacted the number of those police stations’ utilizing awaiting trial on bail. This gap in knowledge and understanding between study group police
stations and control group police stations actually spoiled, to a certain degree, the similarity between the study group and the control group.
The two methods that we have tested for dividing study groups and control groups, as described above, are actually premised on the cases in the study group and the cases in the control group occurring within the same time period. There is another method that can be used if this premise is set aside. For example, cases handled during the year prior to the launch of the study can be put in the control group, and cases handled during the period of the study can be put in the study group. This method can also guarantee the relative independence of the study group and the control group, but may be lacking in terms of similarity, because the time difference of one year may cause result in the influence of a number of objective factors on the similarity of the study group and the control group. III. Collection and Use of Data
In a certain sense, data are the “soul” of study-based empirical research. No matter how a study is
specifically carried out or what its results are, it must use data to enhance its explanations. The veracity and completeness of the data gathered by the researchers, and whether or not the data are used scientifically, will directly impact the objectivity, accuracy, and persuasiveness of the research conclusions. From this point of view, the collection and use of data is another issue that pervades the entire empirical research process and needs to be conscientiously addressed by researchers. The collection and use of data has actually been another thorny problem encountered during our
6study-based empirical research. As Figure I shows, there are six stages of data collection and use
during our study-based empirical research process. The first three steps are to determine the scope of the data collection, design and create data collection forms and a database, and collect and record data. These steps comprise the data collection stage. The next three steps are to screen the data (removing invalid data), analyze the data, and evaluate the study’s results based on the outcome of the data
analysis. These steps comprise the data use stage. Since the data analysis and evaluation steps have already been addressed above, below we will primarily expound upon the first four steps.
Figure I: Flow Chart of Collection and Use of Data from
Study-Based Empirical Research
Design and create data collection Determine scope of Collect and record
forms and a database data collection data
Evaluate study results based on data Analyze data Screen data (remove invalid data)
1. Determine the Scope of the Data Collection
During a study, researchers encounter a great deal of data and information, some of which is very relevant to the study and must be collected, and some of which is not strongly relevant and does not need to be collected. The main task in defining the objectives of the data collection is to determine
6 Many types of data are collected during the entire study-based empirical research process, including data collected via questionnaires and reviewing case files during the pre-study preparation stage, as well as data collected from the cases under the scope of the study during the implementation and post-study follow-up stages. The data collection and use described here primary refers to the latter.