Working Paper: Review of 1 Year of IDRC Experience
Implementing Stage 1 of the rPCR Process (August 2005 – August
Prepared by Laura Haylock, Evaluation Unit
August 30, 2006
Table of Contents:
Section A. Review of Transcripts:...………….…………………4
Section B. Review of Feedback Forms………………………...7
Section C. Notes from Ottawa Review Meeting………………9
This review examines the implementation of the Rolling Project Completion report (hereafter rPCRs) after one year of full implementation by trained Research Officers (ROs). It is composed of three main sections: a review of rPCR transcripts, a review of the feedback forms completed by all ROs, and notes from Ottawa Review Meeting with ROs. The review of transcripts looked at stthrPCRS scanning from July 1 2005-July 27 2006. The feedback forms were thcompleted by ROs between August 15-20 2006 and the Ottawa Review thmeeting took place on August 16 2006. It is noteworthy that this review is a
working document. It is the intention of the Evaluation Unit to capture feedback from Program Managers and Team Leaders as well as Directors of Program Areas and Regional Directors. When this information is available, it will be compiled with this information and presented in a larger report.
Within the three sections this assessment, patterns of positive feedback emerged. These trends were generally centred on the interview process and learning. As far as the interviews are concerned, respondents found the process to be dynamic and useful, with thought provoking questions. Others commented on the importance of keeping the interviews flexible, within both the specifics of the questions asked and the method used. Additionally, respondents commented on different aspects of learning within the rPCRs. Many noted that the rPCRs are effective mechanisms to capture both lessons and nuances of the project.
Based on the feedback and recommendations given within the different sections of this review, the Evaluation Unit will complete the following tasks.
; prepare a short list of guidelines for the interviewee, which will outline
the main objectives of the rPCRs and the role of the interviewer and
; develop a short highlight on Suggestions for Self-Study on the rPCR
process for new staff.
; in order to facilitate the issue of balancing listening, probing and typing,
the Office of the Vice-president purchase additional tape recorders.
; review of the rPCR questionnaire.
Other challenges surfaced during this review and require further investigation. First, it was suggested that the Centre look into hiring a transcription service. Some ROs indicated that their solution to balancing probing, listening and typing is to tape record the interview. At the Review lunch, some ROS felt that this technique results a better quality rPCR transcript; however, it was noted that this is an extremely time consuming transcription process. Additionally, it was recommended that the Centre place equal emphasis on the learning and accountability functions of the rPCRs and operationalize this by using the rPCRs in other aspects and activities of the Centre besides the ALF.
Section A. Review of rPCR Transcripts
This short analysis explores the responses given to Question 7 of the Stage 1 Interview of the rPCR process---as this is a new process, do you have any
comments on the interview itself?--- The timeframe examined spans from
This analysis is dependent on the accuracy of iRIMS search function. The
rPCRs have been cross-referenced within two key searches on iRIMs, one based on total rPCRs and the other based on the different stages of the rPCR process. It is noteworthy that out of the total 27 rPCRs examined, there
were 18 responses given to Question 7.
Overall, the majority of interviewees who responded to Question 7 provided positive feedback without mentioning any suggestions or challenges. That said, a number of interviewees provided positive feedback, but noted either
challenges or suggestions. One respondent provided suggestions only.
1Total number of rPCRs completed: 27
Positive Responses: (12)
; Questions (4)
; Usefulness of the multiple Interview Process (7)
; Time for reflection, sharing and learning (1)
“The level of interview fits the level of development of activities, it is timely. I think the
questions are absolutely pertinent at this stage. I feel there is a building up from the first interview. There are aspects of the issues identified in the first interview that
remains, there are aspects of issues by the time of the second interview that had evolved and progressed.” (Project Number: 102470, Interview Date: 06/21/05)
“The questions are very relevant. It is an interesting process.” (Project Number: 102508, Interview Date: 04/07/2005)
“I think it was very well done-I think the questions are very good they provide a guide to
1 Categorized by Program Area
think about key issues of the project and it was not too time consuming-last time (during
the test phase) it took longer but we are now becoming more efficient” (Project Number 102741, Interview Date: 09/30/05)
“I like the idea of the interviews. It makes you actually think about things. It is helpful.
It reminds us about why we’re doing things (i.e the ethics review; we are doing it because they thought they wanted to set up the structures to so it for other projects so we should follow up).”
(Project number: 102892, Interview Date: 30/05/06)
“C’est processus nouveau pour moi. Les questions sont intéressantes. Généralement, j’aime bien recevoir les questions avant, mais après avoir fait l’interview je me suis rendu compte que c’est bien de ne pas les avoir reçu. ? (Project Number: 102933,
06/07/2005) Interview Date:
? C’est une interview très dynamique. L’interviewer m’a permis d’approfondir mes pensées. ? (Project Number : 103109, Interview Date: not listed)
“It was fine. No Problems. I have participated in other rPCR interviews so that helps. It
is a very useful tool. The interviewer’s (Nasreen) collaboration was good. This is very useful for monitoring projects.” (Project Number: 103211, Interview Date: 01/25/06)
“Fine I dreaded it but it was useful.”
“This process has provided me with the opportunity to comment on Equinet”
“Without a doubt useful.”
“The interview was very good. The interview questions are appropriate for the first 2stage interview.” (Project Number: 103253, Interview date: 03/09/2006)
“I think it is quite good to talk about issues. It also tends to refresh issues in your mind as you talk about them. The same time as you do the interview, you start thinking about other issues and it is not the same as writing it down. This is a good process—I
don’t mind the interview at all.” (Project Number: 103322, Interview Date: 11/28/2005)
“I enjoyed the interview and think the process is a big step forward for the PCR problem. I shall look forward to seeing it written up. It is helpful to send the transcript
afterwards to add to.” (Project number: 103430, Interview Date: 03/20/2006)
“I think this is an excellent tool to share information with colleagues in the Centre and strengthen the IDRC principle of being a learning institution.” (Project number 103554,
Interview Date: 06/26/06)
“I think it is early to have a good comment on this [do you mean the stage 1 interview?] No, I mean the entire rPCR new system. We are just starting with this process. But let us try and see. It is a good thing to have 3 steps and not wait until the end.
Sometimes the PO leaves, or a PO leaves the reporting until the very end and it is hard
2 Multiple people were interviewed.
to remember what happened. With this new process, we have something to build on. We have documentation to read. It is not as heavy as long reports and this gives you
something to build on. Also iRIMS it is interesting- it allows anyone to see what is
going on and get the whole story of the project.” (Project Number: 103679, Interview date: 04/12/2006)
Positive Responses with Suggestions for Improvement: (3)
; Faster/excellent process/idea (2)
; Thought provoking/reflective process (1)
; Scale-up/Process should be more institutionalized (1)
; Face-to-face interviews (1)
; Questions ahead of time (1)
“This is a very important process, if anything this process should be expanded to be more than just three projects. All of our projects should go through this process, and we should look at the cost of doing this. This is something the
Centre will have to look at. Or maybe we could have a scaled down version. I really think doing r-PCRs at the end of the project is necessary, but it is important to keep engaged at the beginning. We want to keep transaction costs low. This is
a very valuable process.” (Project Number: 102965, Date of Interview: 09/01/2005)
“I think it is a great idea. The only thing is I don’t feel it is absolutely necessary to avoid face-to-face interviews. I feel that interview bias can be perceived through
voice intonation as much as through body language.” (Project Number: 103553, Date of Interview: not listed)
“Having the questions ahead of time is useful. It makes it easier to launch into discussions and is good to look over the questions to think ahead. Being
interviewed comes up with a different type of story than if I had been asked to write about it. In writing I would have been more structured. It is useful to be able to talk freely. It is difficult to have a call late at night, but nonetheless the interview
was useful and allowed me to talk to a real person.” (Project Number: 103175, Date of Interview 11/19/2005)
Positive Responses with Challenges: (2)
; Interview went well (2)
; Telephone line disconnection (1)
; Interviewer balancing typing and questioning (1)
“It is good, although the line kept getting cut off, but these elements are beyond control.” (Project Number: 103025, Interview Date: 03/27/2006)
“There was an awkward moment when you were reading back what you captured and couldn’t understand your typed shorthand, but other than that the process went smoothly. I thought it went a bit fast, hurried but went well otherwise” (Project number: 102963, Interview Date: 09/01/05).
; Additional administrative/financial questions needed
“I would only add that is does seem a good idea that some guidelines be developed to probe the financial status of a new prospective institution, because it
is beyond the remit of the PO to do this systematically and thoroughly. Maybe RCs collectively could develop a few simple key questions.” (Project Number 103025, Interview Date: 03/27/06).
Section B. Review of Feedback Forms Completed by all
This section examines the responses to the Feedback forms given to Research Officers (ROs) before the Ottawa Review Meeting (see Section C). Research Officers were asked to reply to the following three questions:
; On a scale of 1-10 (with 10 being the highest score), please rank the
overall usefulness of the rPCR process. Please provide any
recommendations for the improvement of this process.
; On a scale of 1-10 (with 10 being the highest score), please rank
your overall experience with the rPCRs. Why?
; Please provide any changes/suggestions for the improvement of
the interview questions of the rPCRs.
ROs gave a ranking of 8.5 in response to the overall usefulness of the
rPCR process and a ranking of 8.2 to the overall experience. General
suggestions for improvements included a quick reference guide, guidelines for the interviewees, and an overall review of the questionnaire.
Total number of forms completed: 13
1) Average Score of the overall usefulness of the rPCR
process: 8.5 (12 responses)
; generally well managed (3)
; process useful and works well (5)
; captures important lessons (2)
; documents discussion and activities early on before information is lost
; works best when POs are fully engaged
; POs are not providing revisions in a timely fashion
; one-sided, presents IDRC’s views not the partners
; not being exploited to the full potential
; interviews are time consuming (2)
; majority of work falls to RO/TL/DPA.
; more focus needs to be on the learning process (2)
; roles need to be clarified (2)
; process for CAP projects needs to be revised
; interviews better face-to-face
; POs should feel equally ‘responsible’ for preparing for and making
appropriate time for the interviews
2) Average Score of overall experience with the rPCRs.: 8.2
; access to documents and information good
; good communication process
; interviews useful in capturing lessons (3)
; useful way to learn nuances of projects (2)
; dynamic process
; flexibility of the interview (2)
; good time for reflection
; confusing at the beginning
; transcription time consuming & post-interview time consuming (3)
; late rPCRs may not capture the same lessons
; translating and editing-time consuming
; not confident in r-PCR scheduler
; useful to share questions a head of time
; scheduling should be automated
; like to use Skype
; updates and clearer modus operandi about EPIK needed (2)
3) Changes and suggestions for the improvement of the
interview questions of the rPCRs.
; Fine, adopted different questions to make them more relevant and
; some questions seem repetitive and need refining
; discrepancies in how questions are interpreted-for example does
Question 2 inquire talk about why the project was selected for the
rPCR process or why it was selected by IDRC
; Question 3a and 3b-confusion as to whether this refers to the phase
previous to the formal signature of legal documents or to the internal
; compendium or quick reference guide for anyone who wishes to
explore the answer or the best way to explore a problem using the
; guidelines and training should be provided to interviewees ; write up should not be changed significantly but reviewed for accuracy ; executive assistants should be involved in the evaluation process ; question 5 should be placed as a final question