DOC

Quality assessment framework for the external review reports

By Victoria Watson,2014-05-17 13:31
8 views 0
Report's feasibility: Were the evaluation objectives identified? Was the design of the evaluation realistic, practical and adequate to respond to those

Quality assessment framework for the external review

    reports

    Canadian Partnerships Program Review. Evaluation Unit, December 2009.

This is the quality assessment framework that the EU applies to the external

    review reports.

The report is given an overall rating of acceptable or unacceptable. A report is

    deemed unacceptable if one of the following conditions hold:

    ? If it does not adhere to the terms or reference (utility); or

    ? if it is deemed unacceptable on both accuracy and feasibility; or

    ? if it is deemed unacceptable on propriety issues.

    See below for a fuller explanation of these terms.

     Rating Description

I. Report’s utility: Does the report adhere to the terms of reference that were

    designed to support the evaluation’s intended uses by its intended users?

    High Satisfactorily addresses all of the review objectives

    Medium Satisfactorily addresses most of the review objectives

    - EU reviewers coincided in their assessment that

    findings on process (the mechanics of how CP

    programs) could have been better balanced with

    findings on outcomes. However, we also recognized

    that:

     (1) the CP program is different than other PI program

    reviews (with a greater importance assigned to

    program responsiveness and flexibility vs. research

    questions/problematiques); and

    (2) CP`s response to the Canadian ID community and

    its proximity to those actors puts the program in a

    particularly visible position.

    In light of these two considerations, the process

    discussion on how things work is an important one

    for the Centre.

    Satisfactorily addresses few or none of the review objectives Unacceptable

II. Report’s feasibility: Were the evaluation objectives identified? Was the

    design of the evaluation realistic, practical and adequate to respond to those

    evaluation questions?

    High The report describes a design that responds to all of the

    evaluation objectives.

    Medium The report describes a design that responds to most of the

    evaluation objectives.

    The report describes a design that responds to only a few or Unacceptable

    none of the evaluation objectives.

III. Reports accuracy: Did the evaluation use appropriate tools and methods?

    Did the application of the tools and methods generate rigorous, valid and credible

    evidence that is presented in the report? Does the evidence substantiate the

    conclusions/ recommendations?

    High Always uses appropriate tools and methods, and provides

    evidence to support its findings, conclusions and

    recommendations

    Medium Mostly uses appropriate tools and methods, and provides

    evidence to support its findings, conclusions and

    recommendations

    - All reviewers of the report coincided that there were

    instances where the External Review Team needed to

    better clarify findings and underpin them with evidence

    from interviews, secondary documents, etc.

    - The reviewers also felt that the distribution of key

    informant interviews across Canada could have been

    stronger. At the same time, however, we are cognizant

    of the difficulties encountered by the reviewers in (a)

    locating CP partners, a number of whom have moved on

    and (b) securing interviews, particularly in the university

    community between the months of June and August.

    Uses few or no appropriate tools and methods, and rarely Unacceptable

    provides evidence to support its findings, conclusions and

    recommendations

III. Report’s propriety: Did the content of the evaluation report raise ethical

    concerns (pertaining to the right of human subjects; respect for human dignity;

    the completion of a fair evaluation; disclosure of conflicts of interests, etc.)? Acceptable The report raised no serious ethical concerns. Unacceptable The reports raised one or more serious ethical concerns.

    Overall, this report is deemed: Acceptable / Unacceptable.

Background:

The Evaluation Unit assesses the quality of all evaluation reports commissioned

    by the Centre. We use a form that is based on internationally-accepted criteria

    for evaluation quality: utility, feasibility, accuracy and propriety.

Utility: The framework for evaluation at IDRC is utility: evaluation should have a

    clear use and should respond to the needs of the user, whether the user is

    management, a program or a partner organization. IDRC’s approach to evaluation prioritizes equally the use of rigorous methods and the utility of the

evaluation process and findings. The intended uses of the evaluation and the

    questions to be answered guide the selection of the evaluative purpose

    (formative, summative, developmental), the appropriate type of data (quantitative,

    qualitative, mixed), design (naturalistic, experimental), and focus of the

    evaluation (processes, outcomes, impacts, cost-benefit, etc.)

Feasibility: A positive assessment of feasibility means that the methods and

    approaches are well matched to the questions and issues the evaluation set out

    to examine. Issues around resources, timing, perspectives represented, and

    information sources consulted can affect feasibility.

Accurate: Evaluation reports are deemed accurate when they present

    conclusions and recommendations that are supported by evidence that has been

    derived through the application of appropriate and solid methods.

Propriety: As seen in the questions in the chart, propriety issues could entail the

    right of human subjects; respect for human dignity; the completion of a fair

    evaluation; and disclosure of conflicts of interests. A “serious” propriety concern

    is one that undermines the credibility of the evaluation (e.g., an undisclosed

    conflict of interest).

Report this document

For any questions or suggestions please email
cust-service@docsford.com