DOC

MODULE 3

By Dean Myers,2014-05-07 10:54
7 views 0
MODULE 3

    MODULE 3

    RESEARCH METHODS FOR

    NEEDS ASSESSMENT AND MONITORING AND EVALUATION

1. INTRODUCTION

    In the development of your telecenter you will have to deal with several project

    development stages: project design, implementation, and evaluation.

    Where do you start? The first step is to answer the question “Where are we

    now?” It involves the collection and analysis of information on the current situation in the

    community where a telecenter is being considered or already exists. This step is also

    known as a needs assessment.

     The four steps of project design:

    Step 1: Step 2: Step 3: Step 4:

    Analyze the Develop a Create a Develop a

    situation framework strategy M+E plan

    The next step developing a framework answers the question “Where do we

    want to go?” In this step you will develop a clear purpose and goals for your telecenter

    and define what you hope to achieve.

    Next, you will need to create a strategy for your telecenter answering the

    questions “How do we get there?” This step will help to identify where in a community

    the telecenter should be located; what kind of hardware and software are needed; what

    kind of training should be offered; what kind of and how many staff to hire; what services

    to offer; how much to charge for the services; how to get volunteers from the community

    involved in managing the telecenter, etc.

    Finally, you must develop a Monitoring and Evaluation system (M+E) that will be

    able to answer the question “How will we know when we get there and how are we

    doing?” This step will help to assess how well the telecenter is functioning and whether it

    is meeting the needs of the community.

1.1. What is this module about?

    In this module we will focus on the first and last steps in the project cycle

    described above. Keep in mind that stages 2 and 3 are mostly covered in Module 5

    (Strategies for Sustainability). First, we will answer the following questions regarding

    needs assessments and M+E:

? What exactly are needs assessment and M+E systems?

    ? Why are they important?

     41

    ? Why is participation so crucial in the processes? ? How do they fit into the larger context of project development and implementation?

    We will then identify and describe some of the tools that are useful for conducting

    needs assessments and M+E. Various resources (books and websites) for further

    exploration of these topics are provided at the end of the module.

1.2. Participation, Needs Assessment, Monitoring and Evaluation

Participation

    Of all the lessons learned over the past five decades of development practice

    perhaps the most important one is the need for participation of the proposed

    beneficiaries in any development effort. Beneficiary communities are in the best position

    to know what their needs are, what their resources are, and what direction they want to

    go. When these communities are involved in the various phases of a project from

    needs assessment, to project design, to project implementation, to M+E that project is

    far more likely to address their felt needs and they are far more likely to feel “ownership”

    of the project. On the other hand, if people from outside of the community control the

    entire process, the project is unlikely to address the genuine needs, or attract the

    attention, of the beneficiaries. As a result it will lack success and sustainability.

    As you consider establishing and maintaining a telecenter in your community,

    consider this need for community involvement and ownership.

Needs Assessment

    A needs assessment (or, “Where are we now?”) should always be the first step

    when designing a project. Taking the time to involve representatives of all potential

    telecenter users in deciding on its services will result in a telecenter that is used and

    supported by the community. The community should be equal partners in the needs

    assessment process and therefore assume equal responsibility for analyzing, planning,

    providing information, interpreting findings, and carrying out action strategies. It is very

    useful to form a steering committee that is made up of community representatives who

    can help you to facilitate the needs assessment process. Careful planning will help

    ensure that the right information is collected and in the most efficient way.

    There are two main ways to approach a needs assessment a strength-based

    approach, and a problem solving approach. Both, of course, involve stakeholders in the

    processes (those people who will be affected and who will influence the telecenter:

    community members; community organizations; schools; government officials; telecenter

    managers; telecenter volunteers; donors, etc.).

    Strength-based approaches or “appreciative inquiry” emphasizes the positive attributes of the community and looks at what has worked well in the past. It motivates

    people and focuses on repeating successes. Information gathering begins by

    discovering what is working at the present time, what resources are already available in

    the community, and what assets exist for bringing about positive change. By thinking of

    and exploring what is good and successful, communities develop commitment,

    confidence, and energy for new efforts.

    The problem-solving approach begins with communities analyzing their conditions and identifying problems that they want to change. In addition to uncovering

     42

and defining problems, the community attempts to diagnose the causes of these

    problems and to explore potential solutions. The approach you use whether strength-

    based or problem-focused will help determine the kinds of questions you ask when

    collecting information.

    The three most common ways of collecting information are document reviews,

    interviews, and observations. Most practitioners recommend at least two or three

    sources for identifying needs. This is known as “triangulation” and helps to increase the quality of your research. Documents may include government development plans,

    sector-specific documents from the various government ministries, project or annual

    reports from various non-government or donor organizations, and other articles or

    reports. These documents can provide extensive information on various issues, needs,

    or problems in a country or region.

    Interviews, observation, and various other participatory tools help to narrow the

    scope of issues that are important to the community. Simply spending time with

    community members and observing can provide a wealth of relevant information. For

    example, observing visitors at health clinics may suggest what groups do not get health

    services. Interviews with visitors may also reveal the range of health problems in the

    community, and for future reference, suggest information and communication resources

    that might support preventive medicine. It is important to know in advance what you want

    to learn and to keep records of what you observe. Note, also, that needs of communities

    will likely change over time and therefore needs assessments should be an ongoing

    process.

    The tools for conducting needs assessments will be described in detail in section

    2 of this module.

Monitoring and Evaluation (M+E)

    Monitoring is an ongoing process that answers the question “How are we doing?”

    It tells us whether we are achieving the objectives of the project and can help us to

    improve the design or implementation of the project. Evaluation is generally done at a

    particular point in time perhaps in the middle or at the end of a project and can be

    done with the assistance of an external evaluator. It generally answers in greater depth

    the question, “What differences can we see as a result of our efforts?” It addresses the

    overall “value” and long-term impact of the project. M+E are tools they are a means to

    an end and not ends in themselves.

    The advantages of involving stakeholders in the various steps of the M+E

    process (planning, data collection, analysis, and using the results) are:

? providing more relevant and better quality information;

    ?the results more likely to be used by stakeholders; ?greater ownership of the project by stakeholders; and

    ?participants develop M+E skills.

    Not all stakeholders will necessarily be involved with the entire M+E process and

    various stakeholders may be involved in different ways. It may be helpful to establish an

    M+E team from among the stakeholders. Selection of team members may depend on

    their interests, skills, and availability, and can include members of the telecenter steering

    committee mentioned previously. The M+E team plans and implements M+E activities.

     43

    One of the most important ways for the community to be involved in M&E is determining appropriate indicators for measuring success. Indicators are markers that

    show progress and help to measure change. They are tools that help to answer M+E

    questions. There are both quantitative and qualitative indicators. Quantitative or numeric

    indicators show “how much” or “how many”, while qualitative indicators show how or why

    people think, feel, or behave in a particular way. For example, the users of the telecenter

    may decide on indicators that look at changes over time in:

? the number of people using the telecenter;

    ? the willingness of the community to volunteer to run the telecenter;

    ?the willingness of the community to pay for services.

    If your telecenter is being used to disseminate information about agriculture then indicators for success may involve observing changes in agriculture practice, production,

    or marketing.

    If one of the goals is to help increase literacy, then users of the telecenter may decide that an appropriate indicator is whether literacy among users is increasing or not.

    These are, of course, just a few examples. Final selection should be based on an

    indicator’s relevance to answering your M+E questions; its relevance to the activities of

    the telecenter; and the amount of expertise and effort needed to collect data. Ideally,

    final indicators should be relevant, specific, measurable, and observable.

    A more detailed step-by-step description and the necessary tools for an M+E system will be provided in section 3 of this module.

2. NEEDS ASSESSMENT

    The main purpose of a telecenter is to meet the needs of a community and, more directly, its information and communication needs. In order to serve a community, it is

    crucial to know their exact needs. And none is better able to assess these needs than

    the community itself. A needs assessment in the respective community must therefore

    be an initial step in the development and design of a telecenter.

    The more participatory the needs assessment is conducted, the better it will able to give a complete picture of the needs of the community.

    The following are suggested steps for a telecenter needs assessment:

     44

1. Identifying the users of the needs assessment: In the case of a telecenter, the

    users are both those who will act on the assessment such as the manager, the

    volunteers, or a telecenter committee, as well as the users of the telecenter services.

2. Identifying uses: A needs assessment for a telecenter is most likely to be used for

    planning of telecenter services, which, in turn, relate to decisions about staffing, facilities

    and other aspects of the telecenter's operations.

3. Describing the context: What is the physical and social environment of the

    telecenter? Has the telecenter been operating for a long time? Is this an initial

    assessment or are you trying to verify the appropriateness of the telecenters services?

4. Identifying needs:

    Descriptions of the circumstances/problems of the stakeholders. Suggestion of possible

    solutions to their needs and analysis of the likely effectiveness, feasibility, and utilization

    of these solutions.

     5. Meeting needs:

    Recommendations for actions based on the needs, problems, and solutions identified.

     6. Communicating results and recommendations: Communicating the results to the

    stakeholders is an important element of the assessment.

    ______________________________________________________________________

    Several techniques are suitable for a participatory needs assessment. The

    methods you use will depend on the particular situation within your community, your

    resources and the time frame.

    The most important methods that will be described in some detail here are:

    ? Interviews

    ? Focus Groups

    ? Surveys

2.1. Interviews

    Interviews are a widely used method for needs assessments, M + E, and in

    formal evaluations. Typically interviews take place in a face-to-face situation with one

    interviewer and one interviewee. There are several different types of interviews to be

    considered.

    Open-ended or informal interviews: These interviews are free flowing and are conducted in a very conversational and informal manner. Open-ended interviews are

    useful for exploring ideas and hypotheses (assumptions). An example could be when an

    investigator (e.g. you as the telecenter manager) wants to learn about what is important

    to a particular community in the context of needs that might be provided by the

    telecenter. In this form open-ended interviews could be called “listening surveys.”

     45

    You could start an informal, open-ended interview by asking questions such as: “Please tell us about the community and its recent problems, and how you feel about

    them.” You would listen carefully to the responses and follow-up with questions

    requesting clarifications, explanations or more details

Semi-structured interviews: These interviews are typically more structured than open-

    ended interviews. They generally consist of a series of open-ended questions asked in a

    pre-determined order. If an interviewee starts to cover a new area as a response to a

    question, then the interviewer keeps the flow going by asking relevant questions on his

    list of topics. Each question is followed with additional probes until the answer is

    explored in some depth.

    A set of topics for an interview is called the “Interview Guide.”

Example of an Interview Guide

1) Introduction by the interviewer

    ? Who you are and what you are doing and why

    ? Request to tape interview or to take notes

    ? Assurance on confidentiality and anonymity

    ? Purpose and length of interview

    ? Reason for choice of interviewee

    2) Questions about the interviewee’s prior knowledge about the telecenter

    3) Question about why or why not the interviewee is using/not using the services of the telecenter

    4) Question about previous experiences with telecenters, computers etc.

    ? Worries?

    ? Fears?

    5) Expectations/anticipations about the telecenter

    6) Interests/activities that might affect the usage of the telecenter

    7) Issues that might cause problems

    ? Money

    ? Telecenter access

    ? Illiteracy

    8) Any other issues related to the telecenter and the interviewees involvement with it

    9) Demographic data such as age factors in the community, education, family size

    10) Willingness to be interviewed again?

    11) Thanks

Structured interviews: This type of interview is the opposite of the free-flowing,

    informal interview. Structured interviews allow little room for additional questions or

    probes. These interviews can be useful when the existing data base is already

    substantial and what is needed is a quick quantification of narrowly defined topics. (We

    will discuss this topic in more detail when we turn to surveys in section 2.3 below.)

    However, for an initial needs assessment, focus groups and semi-structured interviews

    are recommended.

2.2. Focus groups

    A focus group is a type of semi-structured interview carried out in a group setting. The person running the group session - the facilitator - can be the telecenter manager, a

    team member or an outside consultant. The main role of the facilitator is to "guide" the

     46

discussion. A focus group usually consists of 8-12 participants and a facilitator. Typically,

    the focus group runs 1-2 hours.

    Benefits of conducting focus groups

    ? Additional insights are provided through the interaction of ideas and suggestions of the

    participants

    ? Focus groups can be used to gather information about people's beliefs, and to collect

    kinds of detailed data that are difficult to obtain through structured surveys.

    ? Focus groups involve more people more quickly than individual interviews.

    ? Focus groups do not require much training for the personnel conducting them.

    Focus groups can be conducted at any point in a planning process. However, focus groups are particularly important in the evaluation process: as part of a needs

    assessment, during a program (monitoring), at the end of the program, or months after

    the completion of a program to gather perceptions on the outcome of that program.

    For managing a telecenter, the uses of focus groups include:

    ? Obtaining general background information about a topic of interest (for example,

    the economic and social situation of a community, the community's information

    and communication needs, and other concerns that might be relevant to

    telecenter operations);

    ? Stimulating new ideas and creative concepts for telecenter services or the

    solution of existing problems (for example, low level of use of telecenter services);

    ? Anticipating the potential for problems with a new telecenter program, service or

    product.

    Steps in conducting focus group interviews

    Follow this step-to-step guide to ensure reliable results:

    Step 1: Select the research team

    Conducting focus groups requires a small team with at least a facilitator to guide the discussion and a note taker to write down (or record) participants' important

    comments. The facilitator should be a native speaker who can put the people at ease.

    The team should have substantive knowledge of the topic under discussion.

    Step 2: Select the participants

    First, identify the groups and institutions that should be represented in the focus group (such as community members, NGOs, government officials, partners, etc.). The

    selection of the participants will be determined by the objectives of the assessment.

    Separate focus groups can be held for each type of group (women, men, elders, political

    leaders, farm laborers, etc.) especially if the possibility of intimidation exists). Second,

    identify the most suitable people in each group. It is advisable to have a variety of people

    in the selection process to minimize the biases of individual preferences.

     47

    Each focus group should consist of 8-12 people to allow the smooth flow of conversation. Participants should be relatively homogenous; for example, they might be

    from similar socio-economic and cultural backgrounds.

    Step 3: Decide on timing and location

    Discussions usually last one to two hours and should be conducted in a convenient location with some degree of privacy. Focus groups in a small village arouse

    curiosity and can result in uninvited guests. Open places are not good spots for

    discussions.

    Step 4: Prepare n discussion/interview guide

    In order to structure the focus group discussion and to replicate discussions with different groups, it is necessary to prepare an interview guide.

    Whenever possible, the interview guide should be developed in collaboration with all telecenter staff and possibly with some members of the community. The guide sets

    the agenda for the discussion. It should grow directly from the questions that you have

    regarding the role and operations of the telecenter. These questions should be clearly

    identified in the first steps of the needs assessment where you indicate the users and

    the uses of the assessment.

    When formulating questions for the interview guide, you should consider two principles:

? Questions should be ordered from the more general to the more specific (for

    example, ask about the general situation within a community before you ask about a

    particular kind of problems or a problem-solving approach).

? Questions of greater importance should be placed early, near the top of the guide,

    and be asked earlier during the group session while those of a lesser significance

    should be placed near the end.

    If you find there is a conflict between these two principles, start with general

    questions, and move to specific questions and then back to a set of more general

    questions. The number of questions should be fewer than 10, ideally around five or six.

Types of questions

    Questions should be phrased carefully. Certain types of questions impede group discussions. Avoid using questions that can be easily answered with "yes" or "no"

    because they do not help stimulate discussion. Be careful of why? questions that might

    put participants on the defensive and cause them to take “politically correct” sides on

    controversial discussions.

    The questions should be unstructured and open-ended because this allows respondents to answer from a variety of dimensions, and to tell their story in their own

    words. These types of questions allow the participants to add details that can result in

    unanticipated findings.

    Some examples of general open-ended questions include:

     48

What do you think about the healthcare situation of your community?

What are the main difficulties in living in this community?

    How have you lives changed during the last few years?

Where do you get new information about government programs?

Or more specific: What do you think about the new telecenter coming to the community?

How did you feel using the telecenter services?

What do you like best about the telecenter?

Step 5: Conduct the interview:

    As a facilitator you should attempt to build rapport in the group. Often participants do not know what to expect from focus group discussions. It is therefore helpful to have

    the facilitator outline the purpose and format of the discussion at the beginning of the

    session, and set the group at ease. Participants should be told that the discussion is

    informal, everyone is expected to participate, and divergent views are welcome.

    It is always a good idea to have group members introduce themselves and tell a little about themselves. This method can help "break the ice."

    To keep the discussion flowing, the facilitator:

? Should practice the discipline of listening to others in group situations;

? Should memorize the questioning route;

? Should try to listen and think at the same time;

? And should be able to manage time, in particular, noting when a topic has been

    exhausted and further discussion will yield little new information and in that case

    pose a new question or end the session.

? Should control the discussion. In most groups a few individuals dominate the

    discussion. To balance out participation:

    o Direct questions to individuals who are reluctant to talk

    o Give non verbal clues (look in another direction or stop taking notes when an

    individual talks for an extended period)

    o Intervene, politely summarize the point, then refocus the discussion

    o Take advantage of a pause and say, “Thank you for that interesting idea,

    perhaps we can discuss it separately. Meanwhile with your consent, I would

    like to move to another item.”

     49

Step 6: Collect data

    Tape recorder

    Tape recorders are invaluable for focus group discussions; however, they are prone to pick up background noises. Set up microphones and the recorder prior to the

    focus group session, and make it visible to participants. As the facilitator, you should

    encourage participants to speak one at a time to avoid garbling the tape. It is a good

    idea to ask members of the group to identify themselves before they speak.

Note taking

    The facilitator can attempt to make notes, or an assistant facilitator can try to capture exact phrases and statements made by participants. The consideration here is

    that the note taking should not interfere with the discussion. Notes should be complete

    and useable in the event the tape recorder stops working. It is advisable, regardless of

    the method of data collection, that the moderator makes field notes after each session to

    facilitate data analysis.

    Generally, after each group discussion, the team should briefly summarize the information provided by the focus group's participants, the team’s impressions, and the

    implication for the assessment..

Step 7: Analyze the focus group data

    Analyzing the results is the most difficult part of a focus group. In particular, it is difficult to remain objective and refrain from making judgements when evaluating focus

    group results. Therefore, if possible have someone available to do the analysis before

    carrying out a series of focus groups. Either locate a local person with experience in

    analysing focus group results, arrange for someone of your team to be trained, or hire an

    outside consultant.

    However, if you have to do the analysis yourself, the first step is to transcribe the sessions. This will provide a complete record of the discussion and will facilitate analysis

    of the data. The next step is to analyse the content of the discussion. Start with reading

    each transcript and highlight sections that correspond with the interview guide questions.

    Analyse each question separately. After reviewing all the responses to a question or

    topic, write a summary statement that describes the discussion.

    In analyzing the results, you and your team should consider:

    ? Words: Weigh the meaning of words participants used. Can a variety of

    words and phrases categorize similar responses?

    ? Framework: Consider the circumstances in which a comment was made

    (context of previous discussions, tone and intensity of the comment)

    ? Internal agreement: Figure out whether shifts in opinion during the discussion

    were caused by group pressure.

    ? Precision of responses: Decide which responses were based on personal

    experience and give them greater weight than those based on vague

    impersonal impressions.

     50

Report this document

For any questions or suggestions please email
cust-service@docsford.com