DOC

Compendium on Impact Assessment of ICT-for-Development Projects

By Helen Robinson,2014-10-17 11:02
15 views 0
Compendium on Impact Assessment of ICT-for-Development Projects

    Compendium on Impact

    Assessment of ICT-for-

    Development Projects

    2009

    Richard Heeks & Alemayehu Molla

Creative Commons Licence

     to share (copy, distribute and transmit this work); and to adapt this work under the You are free

    following conditions:

     Attribution: you must attribute the work by identifying both the sponsor/licensor (IDRC) and

    authors (but not in any way that suggests that they endorse you or your use of the work). Share Alike: if you alter, transform, or build upon this work, you may distribute the resulting work

    only under the same or similar license to this one.

    For any reuse or distribution, you must make clear to others the license terms of this work: http://creativecommons.org/licenses/by-sa/2.5/ca/

    Any of the above conditions can be waived if you get permission from the licensor or authors. The authors' moral rights are retained in this licence.

    Table of Contents

Introduction to the Compendium .................................................................................. iii

COMPENDIUM OVERVIEW ........................................................................................... 1

    1. AN OVERVIEW OF IMPACT ASSESSMENT FOR ICT4D ..................................................... 1

    1A. Guiding Model The ICT4D Value Chain .............................................................. 2

    1B. Classifying the Overall Impact of an ICT4D Project ............................................... 4

    2. AN OVERVIEW OF ICT4D PROJECT IMPACT ASSESSMENT FRAMEWORKS ...................... 5

    2A. Comparing IA Frameworks By Method ................................................................... 7

IMPACT ASSESSMENT FRAMEWORKS ...................................................................... 9

    1. COST-BENEFIT ANALYSIS (CBA)................................................................................... 9 2. PROJECT GOALS .......................................................................................................... 18 3. COMMUNICATIONS-FOR-DEVELOPMENT ...................................................................... 23 4. CAPABILITIES (SEN) FRAMEWORK ............................................................................... 32 5. LIVELIHOODS FRAMEWORK ......................................................................................... 40 6. INFORMATION ECONOMICS .......................................................................................... 47 7. INFORMATION NEEDS/MAPPING .................................................................................. 54 8. CULTURAL-INSTITUTIONAL FRAMEWORK .................................................................... 64 9A. ENTERPRISE (VARIABLES) ......................................................................................... 73 9B. ENTERPRISE (RELATIONS) ......................................................................................... 85 9C. ENTERPRISE (VALUE CHAIN) ..................................................................................... 90 10. GENDER .................................................................................................................... 95 11. TELECENTRES .......................................................................................................... 104

ICT4D IMPACT ASSESSMENT BIBLIOGRAPHY .................................................... 114

    1. GENERIC ICT4D IMPACT ASSESSMENT DOCUMENTS ................................................. 117

    2. DISCIPLINE-SPECIFIC ICT4D IMPACT ASSESSMENT DOCUMENTS............................... 121

    3. ISSUE-SPECIFIC ICT4D IMPACT ASSESSMENT DOCUMENTS ....................................... 133

    4. APPLICATION-SPECIFIC ICT4D IMPACT ASSESSMENT DOCUMENTS ........................... 141

    5. METHOD-SPECIFIC ICT4D IMPACT ASSESSMENT DOCUMENTS .................................. 151

    6. SECTOR-SPECIFIC ICT4D IMPACT ASSESSMENT DOCUMENTS .................................... 154

     ii

    Introduction to the Compendium

    Billions of US dollars are invested each year by the public, NGO and private sectors in information-and-communication-technologies-for-development (ICT4D) projects such as telecentres, village phone schemes, e-health and e-education projects, e-government kiosks, etc.

    Yet we have very little sense of the effect of that investment. Put simply, there is far too little impact assessment of ICT4D projects.

    In part that reflects a lack of political will and motivation. But in part it also reflects a lack of knowledge about how to undertake impact assessment of ICT4D.

    This Compendium aims to address that lack of knowledge. It presents a set of frameworks that can be used by ICT4D practitioners, policy-makers and consultants to understand the impact of informatics initiatives in developing countries.

The Compendium is arranged into three parts:

     Overview explains the basis for understanding impact assessment of ICT4D

    projects, and the different assessment frameworks that can be used. Frameworks summarises a series of impact assessment frameworks, each one

    drawing from a different perspective.

     Bibliography a tabular summary of real-world examples of ICT4D impact

    assessment.

     iii

    Compendium Overview

1. An Overview of Impact Assessment for ICT4D

    As with any investigative process, two questions drive ICT4D impact assessment: What do we not know, that we need to know?

     How are we going to find that out?

    Specifically, impact assessment of ICT4D projects can be based around six questions (see Figure 1):

     Why: what is the rationale for impact assessment?

     For whom: who is the intended audience for the impact assessment?

     What: what is to be measured?

     How 1: how are the selected indicators to be measured?

     When: at what point in the ICT4D project lifecycle are indicators to be measured? How 2: how are impact assessment results to be reported, disseminated and used?

     For When? Whom?

     How? How? Pilot & Why? What? (1) (2) Action

    Figure 1: ICT4D Project Impact Assessment Planning Overview

In more detail:

     Why this can include both the externally-stated rationale, and the internal purpose for

    the organisation(s) driving the impact assessment. In most cases, the external rationale

    will be one or more of: a) retrospective achievement post-hoc assessment of what has

    been achieved from investments to date; b) prospective priorities pre-hoc assessment of

    future development project investments; c) accountability enabling agencies to be held

    to account for their ICT4D spending.

     For whom typical audiences are a) ICT4D investment decision-makers; b) ICT4D

    policy decision-makers; c) ICT4D project decision-makers; d) ICT4D project

    users/beneficiaries; e) other ICT4D stakeholders.

     What a mixture of the indicators the key audience will best consume, the indicators it is

    most feasible to measure, and the indicators the assessment team is most familiar with.

    This may also include identifying the conceptual framework guiding the impact

    assessment; the focus of this Compendium.

     How 1 alongside the specific measurement issues, a key element here will be the extent

    of participation of project users in measurement (and in more upstream processes such as

    selection of indicators).

     When the classic impact assessment failure has been to assess ICT4D pilots rather than

    fully-scaled-up projects; and to assess too early in the project's history. How 2 probably the most important and the most overlooked element in the whole

    process, with some impact assessments being conducted but having little impact.

    Includes questions on whether indicators are reported "as is", or communicated via causal

    models, case sketches, stories, etc.

     1

1A. Guiding Model The ICT4D Value Chain

    The basis for understanding the assessment of ICT4D projects is the ICT4D value chain, shown in Figure 2.

     Exogenous Factors

     Precursors Inputs Intermediates / Outputs Outcomes Development -Data systems -Money -New -Financial & Deliverables Impacts -Legal -Labour -Telecentres Communication other quantitative -Public goals Strategy Implementation Adoption Use -Institutional -Technology -Libraries Patterns benefits (e.g. MDGs) -Human -Values and -Shared -New -Qualitative -Technological Motivations telephony Information & benefits -Leadership -Political support -Other public Decisions -Disbenefits Sustainability -New Actions & -Drivers/Demand -Targets access systems Transactions Scalability

    IMPACT UPTAKE READINESS AVAILABILITY

    Figure 2: The ICT4D Value Chain

    This builds on a standard inputprocessoutput model to create a sequence of linked ICT4D resources and processes. It is divided into four main targets for assessment:

     Readiness: "e-readiness" assessment typically measures the systemic prerequisites for any ICT4D initiative e.g. presence of ICT

    infrastructure, ICT skills, ICT policies, and so on. One could also assess the strategy that turns these precursors into project specific inputs,

    and the presence/absence of those inputs.

     Availability: implementation of the ICT4D project turns the inputs into a set of tangible ICT deliverables; one can assess the presence and

    availability of these intermediate resources.

     Uptake: assessment typically measures the extent to which the project's ICT deliverables are being used by its target population. Broader

    assessment could look at the sustainability of this use over time, and at the potential or actuality of scaling-up.

     2

Impact: as the name suggests, only this focus actually assesses the impact of the project and we can divide it into three sub-elements:

    o Outputs: the micro-level behavioural changes associated with the ICT4D project.

    o Outcomes: the specific costs and benefits associated with the ICT4D project.

    o Development Impacts: the contribution of the ICT4D project to broader development goals.

    To some extent and particularly in relation to outputs, outcomes, and development impacts as you move from left to right along the value chain, assessment becomes more difficult, more costly but also more valuable. That move also represents something of a chronology. Thus, as indicated in Figure 3, interest in assessing different aspects of the ICT4D value chain has changed over time, with the strong diffusion of ICT4D projects now creating most particular interest in assessment of impacts, as opposed to uptake, availability or readiness. In this Compendium, the main focus is on assessment of impacts rather than other value chain stages.

     Level of ICT4D

     Activity Impact

    - Efficiency

    - Effectiveness - Equity

     Uptake

     - Demand

    - Usage - Use Divide

     Availability Readiness - Supply - Awareness - Maturity Stage - Infrastructure - Digital Divide

     Time

    Figure 3: Changing Focus of ICT4D Assessment Over Time

     3

1B. Classifying the Overall Impact of an ICT4D Project

    We can classify the overall impact of an ICT4D project into one of the five following categories:

     Total failure: the initiative was never implemented, was implemented but

    immediately abandoned, or was implemented but achieved none of its goals. Largely unsuccessful: some goals were attained but most stakeholder groups did

    not attain their major goals and/or experienced significant undesirable outcomes. Partial success/partial failure: some major goals for the initiative were attained

    but some were not and/or there were some significant undesirable outcomes Largely successful: most stakeholder groups attained their major goals and did

    not experience significant undesirable outcomes.

     Total success: all stakeholder groups attained their major goals and did not

    experience significant undesirable outcomes.

    Major goals are the main objectives a group wanted to achieve with the ICT4D project (which might typically relate to outputs and/or outcomes and/or development impacts); undesirable outcomes are unexpected outcomes that a group did not want to happen but which did happen.

     4

    2. An Overview of ICT4D Project Impact Assessment Frameworks

    Section A provided an overview of ICT4D impact assessment but gave no specific guidance on how to undertake such an assessment. The main role of this Compendium is to provide such guidance: not so much in terms of specific data-gathering methods, but in terms of "frameworks": ways of understanding ICT4D projects and organising knowledge about them.

    We can classify impact assessment frameworks into six categories (summarised in Figure 4):

     Generic: general frameworks usable in assessment of any development project. Discipline-Specific: assessment drawing from a particular academic discipline. Issue-Specific: assessment focused on a particular development goal or issue. Application-Specific: assessment focused on one particular ICT4D technology. Method-Specific: assessment centred on a particular approach to data-gathering.

    (None of these is included in the current Compendium of frameworks, but

    examples of literature are included in the Bibliography).

     Sector-Specific: assessment centred on an individual development sector. (None

    of these is included in the current Compendium of frameworks, but examples of

    literature are included in the Bibliography).

    Generic

    Frameworks

    Discipline-Specific Sector-Specific Frameworks Frameworks

     Impact Assessment of ICT4D Projects

     Method-Specific Issue-Specific

    Frameworks Frameworks

    Application-

    Specific Frameworks

    Figure 4: ICT4D Project Impact Assessment Frameworks Overview

     5

The Compendium offers a synopsis of frameworks within four of the six categories,

    as summarised in Table 1.

Type Sub-Type Focus Compendium

    No.

    GENERIC Cost-Benefit Analysis 1

    Project Goals 2

DISCIPLINE-SPECIFIC Communication Communications-for-3

    Studies Development

    Development Studies Capabilities/Sen 4

    Livelihoods Framework 5

    Information Science Information Economics 6

    Information Needs/Mapping 7

    Sociology Cultural-Institutional 8

ISSUE-SPECIFIC Enterprise (Growth) 9a (Variables)

    9b (Relations)

    9c (Value Chain

    Gender 10

APPLICATION- Telecentres 11

    SPECIFIC

    Table 1: ICT4D Impact Assessment Frameworks in Compendium

    For each of the frameworks, the Compendium entry covers: Summary: a one-paragraph overview of the framework.

     The Framework: an explanation of the origins and content of the particular

    approach, explaining how it would organise ICT4D impact assessment data and

    knowledge.

     SW Analysis: a summary of the perceived strengths and weaknesses of this

    approach to impact assessment.

     Methodological Summary: an overview of the nature and requirements of data-

    gathering using this framework.

     Method Recommendations: good practice notes on applying the framework. References: literature sources referred to in the entry. Bibliography: additional key literature sources, where found. Variants: variations on the main framework that may be used in ICT4D impact

    assessment.

     Examples of Use: summarised examples of applying the framework to ICT4D

    project assessment.

     6

2A. Comparing IA Frameworks By Method

    Table 2 summarises the various Compendium entries in terms of the nature and requirements of their data-gathering methods. These are:

     Primary/Secondary?: whether primary data from the field is required or impact

    assessment can make use of existing secondary data sources.

     Data-Gathering Methods?: what methods (interviews, focus groups, observation,

    document analysis, etc.) are used? In practice, almost all frameworks can use

    multiple methods.

     Participatory?: to what extent can the framework be used in a participatory

    manner that involves ICT4D project recipients beyond a role just as data subjects. Quasi-Experimental?: can the framework be applied in a controlled, experimental

    manner, e.g. comparing impacts on one group that was vs. one group that was not

    involved in the project?

     Quantitative/Qualitative?: are the data gathering and analysis methods mainly

    quantitative, mainly qualitative, or some mixture?

     Multi-Disciplinarity?: does the framework allow for a mixing of different

    disciplinary perspectives?

     Timing?: does impact assessment using this framework have to be cross-sectional

    in timing, or longitudinal, or can it be either?

     Level?: does impact assessment using this framework mainly focus at the micro

    (individual) or meso (e.g. community) or macro (e.g. national) level? Audience/Discipline?: does the disciplinary foundation of the framework create a

    particular likely audience for impact assessment results?

     Resource Requirements?: typically, how costly is ICT4D impact assessment

    using this framework in human and financial terms?

     Generalisability From One Project?: to what extent can you generalise about the

    impact of ICT4D from the assessment of one project using this framework? Comparability Across Projects?: if you are using this framework to assess impact

    of several ICT4D projects, to what extent can you compare the results between

    projects?

    Table 2 can be used in various ways. Just picking a few examples: If you are committed to participatory methods, you can select a framework that

    allows such an approach.

     If your impact assessment team is multi-disciplinary, you can select a framework

    that is appropriate to this mixture.

     If your resources are constrained, you can avoid the high-requirement frameworks. If you are undertaking a multi-project assessment, you can select a framework that

    provides at least some degree of comparability.

     7

Report this document

For any questions or suggestions please email
cust-service@docsford.com