DOC

Workshop 6 Telling the story - using case studies

By Herbert Reynolds,2014-05-07 16:00
7 views 0
Workshop 6 Telling the story - using case studies

    Workshop 6: Telling the story - using case studies

    Presentation 3:

    Conference Paper on

    Practical experiences gained from carrying out case studies in

    relation to different policy themes and/or overall context settings”

    Dr. Thomas Stumm

     (Managing Director of EureConsult S.A., Echternach/Luxembourg)

1. Introduction

The present paper should not be understood as a contribution to further elaborate or fine-tune

    methodological aspects in relation to the realisation of case studies. It is a personal reflection

    upon my direct involvement in a number of Community-level and project-level evaluations

    over the past 12 years, which involved in most cases also the realisation of case studies.

Although I am very aware of the fact that most of these previous case studies do not fully

    correspond to the present-time expectations of DG REGIO, they still do allow identifying

    aspects which “worked well” and thus could be considered in future case study work.

     1

2. Evaluating Structural Funds interventions in individual Member States -

    telling the story from a country-specific perspective

At several occasions in the past, I took part in Community-level evaluations as country expert

    for Germany and Austria which all entailed the realisation of case studies on specific

    Structural Funds mainstream programmes (Thematic evaluation of the partnership principle in

    the 1994-1999 programming period; ex-post evaluation of Objective 1 interventions 1994-

    1999) or on novel approaches generating employment at regional and local levels (i.e.

    Thematic evaluation of Territorial Employment Pacts).

Although the subsequent field work carried out for the German and Austrian case studies in

    the above-mentioned evaluations has been each time very demanding, the practical

    experiences gained with them were very rewarding.

When to apply the case study method in a single-country perspective?

A general look back at all these experiences confirms to me in the present time that in-depth

    case studies have great potentials for unveiling a number of issues which are normally

    not perceivable at a first glance by any horizontal evaluation:

    ? The precise but often also diverse nature of impacts resulting from a policy

    intervention in a given region (e.g. “What has been achieved in terms of direct/indirect

    effects and intended/unintended effects?).

    ? The way in which a policy intervention is implemented and working in practice in the

    particular region being examined (e.g. “How has a policy intervention achieved the

    impact?).

    ? The influence of contextual factors or overall developments in order to locate the

    effect of a particular intervention or the way how a policy intervention has been

    realised within these (“Why” is the “what” and “how” of a policy intervention like it

    is?).

    Regional-level in-depth case studies can on the one hand - be used after a more general assessment of a funding scheme has been carried out for further deepening specific

    aspects alongside the basic issues mentioned above. This was more or less the approach

     2

adopted by the ex-post evaluation of Objective 1 Interventions in Germany 1994-1999, which

    formed part of the wider Community-level ex-post evaluation of Objective 1 interventions.

On the other hand, case studies can be used as a baseline information source for specific

    topics or novel interventions on which an aggregated analysis can hardly be realised at

    the outset. This was more or less the approach adopted by the thematic evaluations of the

    Territorial Employment Pacts and of the Structural Funds partnership principle in the 1994-

    1999 programming period.

    The latter thematic evaluation is an early but still good example, as it assessed within individual countries the main patterns of the country-specific partnership tradition and the

    specific features characterising the partnership arrangements set up for various types of

    Structural Funds programmes to arrive at Community-wide conclusions on the issue at stake.

    Here, the case study approach was a very suitable way for successfully revealing the complex

    inter-relations between the established domestic national and regional-level administrative

    structures and processes and their influence on the decision-making, management and

    operational implementation of Structural Funds programmes (see box below).

In case of Austria, for example, the case studies on the three programmes (Objective 5b

    Niederösterreich, EMPLOYMENT, INTERREG IIA Austria-Hungary) generally confirmed that the long standing tradition of a strong social partnership (Sozialpartnerschaft) within the political

    culture of the country clearly facilitated the establishment of generally rather "inclusive" partnership

    arrangements for the first generation of Structural Funds programmes. This could in particular be

    observed in case of the ESF-interventions (mainstream interventions & Community Initiatives), but also for the different regional-level programmes analysed more in-depth. For the latter interventions, however, the regional case studies were particularly successful in revealing that quite considerable

    differences exist in the partnership arrangements adopted (esp. the “Sonderweg” adopted by the

    Land Government of Burgenland, which had not allocated voting rights to social & economic

    partners in the Monitoring Committee in the context of its Objective 1 & INTERREG IIA Austria-

    Hungary programmes).

For Germany, the 6 case studies could reveal that the Structural Funds partnership arrangements

    generally reflected the country's federal nature. However, the role played by the various federal

    ministries within programmes implemented in Western Germany differed strongly from the role

    played in the Eastern German context. They could also reveal that beyond a tradition of good formal

    co-operation among the key actors, also the wider range of informal relations, the overall policy

    approach pursued by the national or Land governments, as well as the motivation and the character

    of persons involved were further decisive pre-conditions for developing a strategic partnership culture at the level of the respective Structural Funds programmes considered.

     3

Deciding the presentation format of multi-site case studies realised for a specific country

    As under each of the above-mentioned EU-wide evaluations, multi-site case studies on the same Structural Funds-related topic (i.e. partnership) or on the same intervention scheme

    (i.e. Objective 1, territorial employment pacts) were carried out in the perspective of a

    single Member State, it has proved very useful presenting the overall outcome of the case

    study analysis in a wider format (e.g. in form of a “country report”).

Bearing in mind that each case study is holistic, information rich and builds up a very detailed

    in-depth understanding of complex real-life interactions and processes paying special

    attention to context and setting, future multi-site case study exercises should therefore

    consider the main advantages which are associated to such a presentation format:

    ? A multi-site case study exercise realised for an individual country should start with a

    general context analysis preceding the actual case study part, so as to “set the

    scene for the following case studies. In such a general section, most of the

    information about country-specific factors applying to all cases (institutional, socio-

    economic macro trends etc.) and/or overall patterns characterising the funding

    scheme(s) to be analysed (i.e. short review of all programmes implemented in a

    country) should be assessed.

    ? On ground of this context analysis, the subsequent in-depth analysis of the cases

    under review can therefore put stronger emphasis on the specific core issues to be

    addressed and on the particularities of a given case, while referring back to general

    country-specific factors only in so far as any significant deviations from the general

    pattern are observed. If such deviations exist, the case study analysis should however

    also provide an explanation on “why this has happened” and inter-relate those findings

    - where possible with the specific core evaluation questions to be examined.

    ? A multi-site case study exercise on one country should finally conclude with a

    section in which all the overall case study findings are again related to the initial

    working hypotheses and to the general patterns of the country context. Through

    such a synthesis effort, which can also involve some kind of group-building or

    taxonomy (identification of types), the field teams of an evaluation can already at an

     4

    early stage contribute to the subsequent aggregation and synthesis work to be carried

    out by the core team across all country reports.

Preparing & launching case study work in a single-country perspective on ground of a sound

    common understanding

For each of these evaluations, a general framework for carrying out the case study work

    was prepared by each core teams of the main contractor. It set out the baseline requirements

    for gathering data on the cases and for ensuring a coherent reporting about the cases. Beyond

    an identification of the core issues to be addressed by the respective field teams, these

    frameworks also included an “interview guide with a set of pre-defined questions and a

    standard reporting template for presenting the case studies. This allowed the field teams (or

    national experts) to start quickly with their case study work and ensured also that

    subsequently a sound cross-country comparison among the case studies could be elaborated at

    a more aggregated level by the core teams.

    An important issue on which I would like to draw our attention to is the realisation of a sound initial briefing of the field teams carrying out the case study work at a later stage.

    As good and well-elaborated the general framework for carrying out case study work might

    be, it can not replace the valuable outcomes of a direct exchange of views between the core

    team and the field experts carrying out later the actual case studies. Due to my positive

    experiences made with such briefings in the three evaluations mentioned above, I strongly

    recommend that they should become a mandatory first step in any evaluation involving

    extensive case study work.

    Such a briefing can be organised in form of a one-day “kick-off workshop” bringing together all the parties involved in an evaluation. Such a workshop should however pursue various objectives at the time for ensuring that a common understanding is actually achieved:

    ? The field teams/local experts need to get a clear understanding about the wider

    purpose and strategic objectives of the overall evaluation. Although such aspects

    are normally laid down in the terms of reference of any evaluation, it might be that

    they are formulated in such an abstract manner that their basic relevance for the field

    work is not perceived at a first view. I have therefore always felt it useful that a clearly

     5

    understandable “message” was communicated by a representative of the client (i.e. the

    Commission in case of EU-wide evaluations; national or regional authorities in case of

    comprehensive country-specific evaluations) to the involved field teams for

    strengthening their commitment to the overall evaluation exercise.

? The field teams/local experts must be made fully aware of the central purpose of

    the case study within the wider evaluation. The initial hypotheses to be tested and

    the issues which need to be explored by one or more case studies must be made clear

    by the members of a core team. Especially in case of more complex EU-wide

    evaluations, this should in my opinion also involve an open discussion with the field

    teams for gathering their impression on the overall approach. Due to their tacit

    knowledge about country- or region-specific settings, such discussions can reveal

    some additional issues which might not yet have been sufficiently covered in the

    general framework of an evaluation.

? Finally, the field teams/local experts must become fully familiar with all aspects

    relating to the practical realisation of case studies. This is again a task of the core

    team, which should provide practical and easy-to-understand explanations of all

    aspects related to data gathering, overall presentation and the narrative reporting style.

    An interesting approach could be to make such issues more explicit alongside a

    previously realised “pilot case study” which has been disseminated to the field teams;

    but this option is however strongly conditioned by the overall budget and time-

    sequencing of an evaluation.

Realising case studies in practise, however with a certain distance and “flexibility

My experiences gained from all these country-specific case studies mentioned above has also

    made clear to me that field evaluators should realise a case study-based in-depth analysis

    with a certain degree of distance and flexibility.

? Evaluators should first of all try to take some distance with respect to the often

    inevitably occurring time-pressure in the practical field work process. Due to my

    personal experience, complex evaluations do most often not operate according to what

    has been initially planned and the fixed deadlines for delivering mandatory outcomes

     6

tend to create extreme time-pressure at certain stages of the process. Sufficient time is

    however needed for preparing and elaborating a high quality case study. Therefore and

    as far as a situation of time pressure should occur in the field work phase, I can only

    recommend that evaluators stick to their professional standards. This is particularly

    important during the information & data gathering process, as it generates the crucial

    input on ground of which the story is subsequently told. The evaluator should first

    acquire sound background knowledge about the case to be examined (on ground of

    desk research) and only afterwards carry out direct stakeholder interviews. If the

    inverse is done, it is then very likely that the right questions on the “what”, “how” and

    “why” will probably never be asked and that the real point is not made by the final case study elaborated.

    ? Field evaluators should also be flexible in the actual preparation and realisation

    of stakeholder interviews. An interview guide” with a set of pre-defined general

    questions to be addressed is certainly a helpful tool (i.e. function as a “reminder” or a

    “not-to-forget about-list”). However, before carrying out stakeholder interviews, these

    questions should be carefully reviewed and/or complemented by the evaluator on

    ground of the better in-depth insight into the case he/she has gained from previous

    desk-based research. Derived from my personal experience, it makes much more sense

    to eliminate those questions from the pre-defined catalogue for which sufficient

    information is already available and to focus an interview on exploring more in-depth

    those issues which are still unclear. Beyond carrying out single-person interviews, it

    has also been very rewarding for the information gathering process if various

    stakeholders with different interests/viewpoints are brought together (if possible) and

    if a group discussion among them is initiated by the evaluator alongside a limited

    number of “open” and/or “closed questions.

    ? When it comes to drawing up the report on one or more case studies on ground of

    the previously gathered data/information, evaluators should try taking some

    distance with respect to an initially defined reporting template. Having more

    recently also been involved in the synthesis / aggregation work of large-scale

    evaluations, I do acknowledge that this is to some extent a rather delicate and

    ambivalent issue. On the one hand, there is no doubt that a field evaluator has to

    address and explore further the initially identified issues and core evaluation questions

     7

while putting together his/her report on a case study. Especially in a Community-wide

    evaluation covering various countries, a too heterogeneous presentation of various

    case-study reports can endanger the subsequent comparison and aggregation work to

    be carried out by a core team. On the other hand, however, a standard reporting

    template with a list of core evaluation questions and related issues to be addressed is

    only a guidance tool which might require a further adaptation or completion.

    Evaluators should therefore take a certain distance with respect to such a tool (or even

    better: have an initially well-agreed degree of flexibility), as this can be very

    rewarding for the overall quality of the final case study to be delivered. The field

    evaluator should “intuitively co-determine” the importance given to the pre-defined

    items on ground his/her actual knowledge about the given case and with respect to

    their actual relevance within the story to be told about this case. Two rather simple

    examples shall illustrate this: If the standard template requires exploring the degree of

    involvement of regional authorities in the delivery of a national policy, but it appears

    that in the specific country no regional level exists, it is obvious that no information on

    this aspect can be provided. If something particularly important is discovered by a

    specific case study on which no assumptions were made in the reporting template

    provided (i.e. due to singular specificity of the issue), then a new section has to be

    created for telling the story about this particular phenomenon.

    ? Finally, some kind of “intuitive flexibility should also be adopted by an

    evaluator while elaborating an informative and analytical but still appealing

    narrative which tells the story about a specific case. In my previous field activities,

    I have always liked as far as possible within a pre-defined reporting structure - to focus on “critical incidents” for telling an interesting story. For a given case study, I pragmatically considered a critical incident as being something which made a

    significant contribution - either positively or negatively - to an activity, process or

    phenomenon. Identifying critical incidents did certainly not mean to forget about the

    rest; they only helped me to put more emphasis on certain aspects which were

    particularly interesting or specific in the light of the overall evaluation exercise.

    Building the narrative presentation of the case study around such critical incidents is in

    so far rewarding as it produces a certain “drive” which helps keeping a future reader connected to the story until its very end.

     8

3. Evaluating territorial co-operation programmes & projects - telling the

    story from a multi-country perspective

The use of the case study method is from my point of view particularly recommended

    for evaluating European Territorial Co-operation (ETC) as supported by the Structural

    Funds. This is mainly due to the fact that the complexity of real-life interactions within cross-

    border, transnational or inter-regional co-operation processes is considerably higher if

    compared to those taking place in interventions carried out in a single-country perspective.

Before the start, become aware of the particular challenges associated to an ETC-case study

    exercise!

According to my practical experience, however, case studies on ETC are also a very

    demanding and particularly challenging task for each evaluator involved. There are

    several reasons for this, which can also be considered as an initial but non-exhaustive list of

    baseline requirements for such an exercise:

Firstly, because an ETC-case study has to tell the story from a multi-country perspective

    which requires that the evaluator is familiar with more than only one country-specific context

    (basic political-administrative settings, different cultures and ways of “doing” and “thinking”).

    Especially in the context of cross-border co-operation, evaluators need to have a good

    understanding about the general legal framework conditions under which co-operation is

    taking place (in order to better understand why co-operation is taking or not taking place) and

    about the problems caused by persisting differences of the national legal/regulatory

    frameworks which meet at a common border (in order to better understand how such

    differences affect the daily life in border areas and how they can be overcome by means of co-

    operation).

Secondly, the evaluator (or the evaluation team) needs to have well-developed and

    appropriate language capacities. They are especially required in the context of cross-border

    co-operation, but to some lesser extent in the field of transnational or interregional co-

    operation as here most often English is used as a working (or vehicular) language within

    multi-country partnerships.

     9

Thirdly, the evaluator needs to have a well developed understanding about the previous

    history of co-operation (i.e. long-term development and core activities carried out in the past),

    as this might significantly influence upon the present time situation. The required amount of

    tacit ex-ante knowledge tends to be higher especially in many western European cross-border

    areas where a rather long-standing co-operation does exist which often pre-dates the

    launching of INTERREG (in 1989/1990). In Central and Eastern European cross-border areas

    but also in case of transnational co-operation, where the history of co-operation is often much

    more recent and closely related to the implementation of EU-funding schemes (INTERREG,

    PHARE / TACIS-CBC etc.), this might be less important and mostly require the examination

    of such previous EU-interventions.

Fourthly, the activities for preparing and realising such an exercise are much more complex

    as a greater number of key stakeholders originating from different countries and from

    different levels of co-operation (strategic level & project level actors; project lead partners &

    other project partners; directly involved actors or indirectly concerned actors) needs to be

    identified/contacted and directly interviewed in order to be able to tell a story on co-operation

    which is comprehensive and balanced. This is often forgotten or at least under-estimated, as it

    is assumed that co-operation is taking place in an apparently homogenous context building

    upon fully shared views and objectives. In reality, however, the actual perception of problems

    and the way of tackling them is largely conditioned by culturally different ways of thinking

    and doing and the real action is also here most often driven by complex interest- and actor

    constellations (e.g. by one or more partners particularly concerned by an issue who are

    “pushing” the project or certain actions of it; by one or more very active partners who make follow a number of other more inactive project partners).

At what levels can the case study method be used in the context of ETC-evaluations?

    For evaluating European Territorial Co-operation, the case study method can in principle be used at three different levels:

    ? A comprehensive evaluation of Community-level funding schemes such as the

    “old” INTERREG Community Initiative or the new Objective European Territorial

    Co-operation”.

     10

Report this document

For any questions or suggestions please email
cust-service@docsford.com