Quality assessment framework for the external review
Canadian Partnerships Program Review. Evaluation Unit, December 2009.
This is the quality assessment framework that the EU applies to the external
The report is given an overall rating of acceptable or unacceptable. A report is
deemed unacceptable if one of the following conditions hold:
? If it does not adhere to the terms or reference (utility); or
? if it is deemed unacceptable on both accuracy and feasibility; or
? if it is deemed unacceptable on propriety issues.
See below for a fuller explanation of these terms.
I. Report’s utility: Does the report adhere to the terms of reference that were
designed to support the evaluation’s intended uses by its intended users?
High Satisfactorily addresses all of the review objectives
Medium Satisfactorily addresses most of the review objectives
- EU reviewers coincided in their assessment that
findings on process (the mechanics of how CP
programs) could have been better balanced with
findings on outcomes. However, we also recognized
(1) the CP program is different than other PI program
reviews (with a greater importance assigned to
program responsiveness and flexibility vs. research
(2) CP`s response to the Canadian ID community and
its proximity to those actors puts the program in a
particularly visible position.
In light of these two considerations, the process
discussion on how things work is an important one
for the Centre.
Satisfactorily addresses few or none of the review objectives Unacceptable
II. Report’s feasibility: Were the evaluation objectives identified? Was the
design of the evaluation realistic, practical and adequate to respond to those
High The report describes a design that responds to all of the
Medium The report describes a design that responds to most of the
The report describes a design that responds to only a few or Unacceptable
none of the evaluation objectives.
III. Reports accuracy: Did the evaluation use appropriate tools and methods?
Did the application of the tools and methods generate rigorous, valid and credible
evidence that is presented in the report? Does the evidence substantiate the
High Always uses appropriate tools and methods, and provides
evidence to support its findings, conclusions and
Medium Mostly uses appropriate tools and methods, and provides
evidence to support its findings, conclusions and
- All reviewers of the report coincided that there were
instances where the External Review Team needed to
better clarify findings and underpin them with evidence
from interviews, secondary documents, etc.
- The reviewers also felt that the distribution of key
informant interviews across Canada could have been
stronger. At the same time, however, we are cognizant
of the difficulties encountered by the reviewers in (a)
locating CP partners, a number of whom have moved on
and (b) securing interviews, particularly in the university
community between the months of June and August.
Uses few or no appropriate tools and methods, and rarely Unacceptable
provides evidence to support its findings, conclusions and
III. Report’s propriety: Did the content of the evaluation report raise ethical
concerns (pertaining to the right of human subjects; respect for human dignity;
the completion of a fair evaluation; disclosure of conflicts of interests, etc.)? Acceptable The report raised no serious ethical concerns. Unacceptable The reports raised one or more serious ethical concerns.
Overall, this report is deemed: Acceptable / Unacceptable.
The Evaluation Unit assesses the quality of all evaluation reports commissioned
by the Centre. We use a form that is based on internationally-accepted criteria
for evaluation quality: utility, feasibility, accuracy and propriety.
Utility: The framework for evaluation at IDRC is utility: evaluation should have a
clear use and should respond to the needs of the user, whether the user is
management, a program or a partner organization. IDRC’s approach to evaluation prioritizes equally the use of rigorous methods and the utility of the
evaluation process and findings. The intended uses of the evaluation and the
questions to be answered guide the selection of the evaluative purpose
(formative, summative, developmental), the appropriate type of data (quantitative,
qualitative, mixed), design (naturalistic, experimental), and focus of the
evaluation (processes, outcomes, impacts, cost-benefit, etc.)
Feasibility: A positive assessment of feasibility means that the methods and
approaches are well matched to the questions and issues the evaluation set out
to examine. Issues around resources, timing, perspectives represented, and
information sources consulted can affect feasibility.
Accurate: Evaluation reports are deemed accurate when they present
conclusions and recommendations that are supported by evidence that has been
derived through the application of appropriate and solid methods.
Propriety: As seen in the questions in the chart, propriety issues could entail the
right of human subjects; respect for human dignity; the completion of a fair
evaluation; and disclosure of conflicts of interests. A “serious” propriety concern
is one that undermines the credibility of the evaluation (e.g., an undisclosed
conflict of interest).