Direct and Indirect Measurement Result
Measures of assessment refer to the methods or processes or tools used to evaluate students’ performance in context of intended student learning outcomes.
While course grades should not be used as measures of student learning, student
work that is completed in a class can be used as part of the assessment process. That is to say, if the instructor creates a mechanism for verifying that the grades share the same meaning among all those who assign them for a given assignment or course, then the same piece of student work that the instructor assigns for a grade also can be used as part of the assessment procedures.
This process is “norming.” Usually it involves the creation of a scoring rubric, and the criteria or performance standards are described in the rubric. Then the piece of student work may be used for assessment as well as be assigned a grade.
(Adapted from http://www.hpcnet.org/assessment )
Assessment measures are often categorized as direct or indirect. Direct measures of
assessment are those in which the products of student work are evaluated in light of the learning outcomes for the program. Evidence from coursework such as projects or specialized tests of knowledge or skill is examples of direct measures. In all cases, direct measures involve the evaluation of demonstrations of student learning.
The following are examples of direct measurement method of assessment:
; Course-embedded assessment
; Standardized tests
; Locally-developed tests
; Portfolio evaluation
Indirect measures of assessment are those in which students judge their own ability
to achieve the learning outcomes. Indirect measures are not based directly on student academic work but rather on what students perceive about their own learning. For example, alumni may also be asked the extent to which the program prepared them to achieve learning outcomes, or people in contact with the students, such as employers, may be asked to judge the effectiveness of program graduates. In all cases, the assessment is based on perception rather than direct demonstration.
The following are examples of indirect measurement method of assessment:
; Student self-efficiency surveys
; Student attitudinal change surveys
; Exit interviews
; Alumni surveys
; Employer surveys
Thorough program assessment combines both direct and indirect measurement
methods accompanied by challenging yet realistic standards or criteria for success.
(Adapted from http://academic-affairs.csusb.edu/progs/assessment/forumf98.htm and
(a.) Direct Measurement Methods: Examples
In course-embedded assessment, student work in designated courses is collected and assessed in relation to the program learning outcomes, not just for the course grade. The products of student work need to be considered in light of the learning outcomes. Products may include final exams, research reports, projects, papers, and so on. The assessment may be conducted at specific points (e.g., introductory course and upper level course) in a program.
Benefits include the fact that assessment is conducted as part of the normal workload of students and faculty, although additional work may be needed to incorporate program assessment into the course.
Disadvantages include the potential for a faculty member to feel that her or his work in a particular course is being overseen, even if it is not. Also, rubrics may need to be chosen or developed that are associated with the particular learning outcomes, increasing the preparation time.
The Educational Testing Service and other companies offer standardized tests for various types of learning outcomes, such as critical thinking or mathematical problem solving. Scores on tests such as the GRE or the Major Field Achievement Test (MFAT) may be used as evidence of student learning.
Benefits include the reliability and validity of an assessment instrument that is commercially developed, eliminating the arduous process of developing an instrument in-house; simplicity in administration and evaluation of test
results; and the potential for cross-institutional comparisons of results.
Disadvantages include the generic nature of standardized tests and their potential lack of fit with a particular program; a possible lack of motivation by students to take the test or do well on it; and the debatable question of whether a standardized test gives a true measure of student learning. Also, ETS and other services charge substantial fees for these tests, which is an added administrative cost or possibly a cost to the students.
(Text and links taken from
The Web provides an easy way to locate off-the-shelf tests. The Buros Institute and ERIC have combined their efforts to put searchable databases of tests, references of test reviews, and test publishers online. They are located at http://ericae.net/testcol.htm. When searching for tests or reviews, one can
enter a word, (e.g. “biology”) and get back a number of names and tests or a list of reviews of tests. Furthermore, ERIC has teamed up with Amazon.com to provide an online assessment bookstore for additional resources.
(Taken from http://www.geneva.edu/academics/assessment/oaguide.pdf )
Locally developed tests
Faculty in a program may decide to develop a test that is reflective of the program’s mission and learning outcomes. Multiple evaluators usually grade the test. Locally developed tests are less costly than a standardized test, but require work by the program’s faculty in test development and scoring.
Benefits include the ability to tailor a test to a specific program.
Disadvantages include the challenge of developing a test with proven reliability and validity, the potential need to develop rubrics and train multiple test evaluators in the use of these rubrics, and the need to develop a new test periodically.
A portfolio is a compilation of student work that, in total, demonstrates a student’s achievement of various learning outcomes. Portfolios can be created for a variety of purposes aside from program assessment, such as fostering reflection by students on their education, providing documentation for a student’s job search, or certifying a student’s competency. Portfolios
created over the span of a student’s academic career, compared to those
consisting of a student’s work only at the end, provide the basis for a
Portfolios may combine multiple types of evidence and are not necessarily limited to classroom work. For example, portfolios may contain research papers, presentations, videos, audio recordings, work done through employment, or journal entries discussing co-curricular activities or programs. Once the material is collected, it falls upon an individual or group to establish a system by which to evaluate the contents of the portfolio in terms of a program’s learning outcomes.
In program assessment, a cross section of students may be sampled to evaluate student learning outcomes. A key question in portfolios arises in the collection of evidence. For program assessment, the department itself may have to assemble the student portfolios; in this case, issues must be considered about how the students are to be informed of the fact that their work is being assessed for programmatic reasons. Some faculties ask students to sign consent forms to copy work products and to use student work products in accreditation reports.
Benefits of portfolios include the ability to document student development over time, and the potential benefit to the students of seeing their own development and in collecting material that may support their career goals. Thus, program assessment becomes an integral part of the learning process.
Disadvantages include a labor-intensive process in the evaluation of evidence in student portfolios. Also, there is an expense in storing and organizing the evidence.
(Text taken from
(b.) Indirect Measurement Methods: Examples
Surveys, the primary indirect assessment measure, are a systematic means of collecting data from a group of people in order to describe some aspects, characteristics, or perceptions of the population in question.
(Text taken from:
Student self-efficacy surveys
Students have a sense of their own competence. Student self-efficacy involves the rating by students of their perception of their own achievement in particular learning outcomes. Research shows a significant, although
imperfect, correlation between actual and perceived competence. What can be problematic are gender and demographic differences in the accuracy of self-efficacy. For example, certain groups of students may rate their quantitative skills at a level below that indicated by standardized tests. Also, unless the answers are anonymous, students will be likely to overrate their abilities. The same is true if students perceive they can be penalized by their answers.
Self-efficacy as an assessment tool is relatively simple. For example, a Researcher/assessment expert at Clemson University has designed a test that asks students to rate the perceived importance and self-efficacy of leadership skills, communication skills, interpersonal skills, analytical skills, decision-making skills, and technological skills, the global economy, ethics, and business practices.
Benefits include the inexpensive nature of the tool. A relatively simple survey can be constructed which simply asks students to rate their competence in different areas. Also, pre- and post-test assessment can be conducted to examine changes both in self-efficacy and perceived importance of a topical area. Another benefit is that all learning outcomes can be assessed simultaneously, in one test.
Disadvantages include an imperfect relationship between self-efficacy and actual competence; student self-reporting may not always be congruent with their actual level of achievement.
Student attitudinal change surveys
If learning outcomes include elements of appreciation or understanding of particular issues of concern, student attitudinal change can be measured as part of the assessment program. For example, informed appreciation for the arts may be assessed using an attitudinal survey. Another example may be students’ empathy toward disadvantaged groups, which can be measured in an attitudinal survey. A further example would be attitudes toward learning or toward the profession before and after completion of the program. Both standardized tests and locally designed surveys can be used for this purpose, although the responses are very sensitive to the wording of the questions.
Benefits include the simplicity of administering the system.
Disadvantages include the challenge of determining student attitudes in a reliable manner.
Rather than assess students’ attitudes, self-efficacy, or satisfaction through
the use of surveys, students may be interviewed directly in individual or focus-group settings. Such interviews allow a more thorough, free-form exploration of the issues through the use of follow-up questions that depend on students’ responses. To encourage this open exchange in a controlled
setting, a mix of both structured and open questions is suggested.
Benefits include the depth and richness of information that can be obtained through interviews.
Disadvantages include the time- and labor-intensive nature of conducting such interviews and in analyzing the information obtained from interviews for comparison across multiple interviews. Also, student anonymity needs to be protected in this tool, and stray comments about individual faculty must not become part of the assessment data.
The perspective that students have on their education may change significantly after time away from school. Some learning outcomes lend themselves more naturally to questions posed some time after graduation. For example, an outcome involving preparation for professional practice can best be assessed after the student has graduated and been employed in the job market.
Benefits include the real-world perspective that can be obtained from alumni.
Disadvantages include the difficulty of finding and reaching alumni, the possibly self-selective nature of those who choose to respond, and the relatively narrow scope of learning outcomes that can be assessed in this manner.
It is possible that some of the students' knowledge and skills are evident to the employers who rely on these characteristics. Thus, some accrediting bodies either require or encourage programs to perform an assessment through the major employers of their students. These may range from information as basic as hiring data, to site supervisor evaluations, to detailed surveys of the characteristics that the employers perceive in program graduates.
Benefits of this tool include the real-world perspective that employers might be able to provide.
Disadvantages include the potentially limited ability of employers to assess their employees’ characteristics in terms of specific learning outcomes, or the
inability of employers to assess graduates only from a particular school. Also, this tool depends on surveying employers with sufficient numbers of graduates. In large corporations, it may even be difficult to find the right person to contact for this information. In addition, former students may object to having their employers surveyed in this way.
(Text Adapted from: