Workshop 6: Telling the story - using case studies
Conference Paper on
“Practical experiences gained from carrying out case studies in
relation to different policy themes and/or overall context settings”
Dr. Thomas Stumm
(Managing Director of EureConsult S.A., Echternach/Luxembourg)
The present paper should not be understood as a contribution to further elaborate or fine-tune
methodological aspects in relation to the realisation of case studies. It is a personal reflection
upon my direct involvement in a number of Community-level and project-level evaluations
over the past 12 years, which involved in most cases also the realisation of case studies.
Although I am very aware of the fact that most of these previous case studies do not fully
correspond to the present-time expectations of DG REGIO, they still do allow identifying
aspects which “worked well” and thus could be considered in future case study work.
2. Evaluating Structural Funds interventions in individual Member States -
telling the story from a country-specific perspective
At several occasions in the past, I took part in Community-level evaluations as country expert
for Germany and Austria which all entailed the realisation of case studies on specific
Structural Funds mainstream programmes (Thematic evaluation of the partnership principle in
the 1994-1999 programming period; ex-post evaluation of Objective 1 interventions 1994-
1999) or on novel approaches generating employment at regional and local levels (i.e.
Thematic evaluation of Territorial Employment Pacts).
Although the subsequent field work carried out for the German and Austrian case studies in
the above-mentioned evaluations has been each time very demanding, the practical
experiences gained with them were very rewarding.
When to apply the case study method in a single-country perspective?
A general look back at all these experiences confirms to me in the present time that in-depth
case studies have great potentials for unveiling a number of issues which are normally
not perceivable at a first glance by any horizontal evaluation:
? The precise but often also diverse nature of impacts resulting from a policy
intervention in a given region (e.g. “What has been achieved in terms of direct/indirect
effects and intended/unintended effects?).
? The way in which a policy intervention is implemented and working in practice in the
particular region being examined (e.g. “How has a policy intervention achieved the
? The influence of contextual factors or overall developments in order to locate the
effect of a particular intervention or the way how a policy intervention has been
realised within these (“Why” is the “what” and “how” of a policy intervention like it
Regional-level in-depth case studies can – on the one hand - be used after a more general assessment of a funding scheme has been carried out for further deepening specific
aspects alongside the basic issues mentioned above. This was more or less the approach
adopted by the ex-post evaluation of Objective 1 Interventions in Germany 1994-1999, which
formed part of the wider Community-level ex-post evaluation of Objective 1 interventions.
On the other hand, case studies can be used as a baseline information source for specific
topics or novel interventions on which an aggregated analysis can hardly be realised at
the outset. This was more or less the approach adopted by the thematic evaluations of the
“Territorial Employment Pacts” and of the Structural Funds partnership principle in the 1994-
1999 programming period.
The latter thematic evaluation is an early but still good example, as it assessed within individual countries the main patterns of the country-specific partnership tradition and the
specific features characterising the partnership arrangements set up for various types of
Structural Funds programmes to arrive at Community-wide conclusions on the issue at stake.
Here, the case study approach was a very suitable way for successfully revealing the complex
inter-relations between the established domestic national and regional-level administrative
structures and processes and their influence on the decision-making, management and
operational implementation of Structural Funds programmes (see box below).
In case of Austria, for example, the case studies on the three programmes (Objective 5b
Niederösterreich, EMPLOYMENT, INTERREG IIA Austria-Hungary) generally confirmed that the long standing tradition of a strong social partnership (Sozialpartnerschaft) within the political
culture of the country clearly facilitated the establishment of generally rather "inclusive" partnership
arrangements for the first generation of Structural Funds programmes. This could in particular be
observed in case of the ESF-interventions (mainstream interventions & Community Initiatives), but also for the different regional-level programmes analysed more in-depth. For the latter interventions, however, the regional case studies were particularly successful in revealing that quite considerable
differences exist in the partnership arrangements adopted (esp. the “Sonderweg” adopted by the
Land Government of Burgenland, which had not allocated voting rights to social & economic
partners in the Monitoring Committee in the context of its Objective 1 & INTERREG IIA Austria-
For Germany, the 6 case studies could reveal that the Structural Funds partnership arrangements
generally reflected the country's federal nature. However, the role played by the various federal
ministries within programmes implemented in Western Germany differed strongly from the role
played in the Eastern German context. They could also reveal that beyond a tradition of good formal
co-operation among the key actors, also the wider range of informal relations, the overall policy
approach pursued by the national or Land governments, as well as the motivation and the character
of persons involved were further decisive pre-conditions for developing a strategic partnership culture at the level of the respective Structural Funds programmes considered.
Deciding the presentation format of multi-site case studies realised for a specific country
As under each of the above-mentioned EU-wide evaluations, multi-site case studies on the same Structural Funds-related topic (i.e. partnership) or on the same intervention scheme
(i.e. Objective 1, territorial employment pacts) were carried out in the perspective of a
single Member State, it has proved very useful presenting the overall outcome of the case
study analysis in a wider format (e.g. in form of a “country report”).
Bearing in mind that each case study is holistic, information rich and builds up a very detailed
in-depth understanding of complex real-life interactions and processes paying special
attention to context and setting, future multi-site case study exercises should therefore
consider the main advantages which are associated to such a presentation format:
? A multi-site case study exercise realised for an individual country should start with a
general context analysis preceding the actual case study part, so as to “set the
scene” for the following case studies. In such a general section, most of the
information about country-specific factors applying to all cases (institutional, socio-
economic macro trends etc.) and/or overall patterns characterising the funding
scheme(s) to be analysed (i.e. short review of all programmes implemented in a
country) should be assessed.
? On ground of this context analysis, the subsequent in-depth analysis of the cases
under review can therefore put stronger emphasis on the specific core issues to be
addressed and on the particularities of a given case, while referring back to general
country-specific factors only in so far as any significant deviations from the general
pattern are observed. If such deviations exist, the case study analysis should however
also provide an explanation on “why this has happened” and inter-relate those findings
- where possible – with the specific core evaluation questions to be examined.
? A multi-site case study exercise on one country should finally conclude with a
section in which all the overall case study findings are again related to the initial
working hypotheses and to the general patterns of the country context. Through
such a synthesis effort, which can also involve some kind of group-building or
taxonomy (identification of types), the field teams of an evaluation can already at an
early stage contribute to the subsequent aggregation and synthesis work to be carried
out by the core team across all country reports.
Preparing & launching case study work in a single-country perspective on ground of a sound
For each of these evaluations, a general framework for carrying out the case study work
was prepared by each core teams of the main contractor. It set out the baseline requirements
for gathering data on the cases and for ensuring a coherent reporting about the cases. Beyond
an identification of the core issues to be addressed by the respective field teams, these
frameworks also included an “interview guide” with a set of pre-defined questions and a
standard reporting template for presenting the case studies. This allowed the field teams (or
national experts) to start quickly with their case study work and ensured also that
subsequently a sound cross-country comparison among the case studies could be elaborated at
a more aggregated level by the core teams.
An important issue on which I would like to draw our attention to is the realisation of a sound initial briefing of the field teams carrying out the case study work at a later stage.
As good and well-elaborated the general framework for carrying out case study work might
be, it can not replace the valuable outcomes of a direct exchange of views between the core
team and the field experts carrying out later the actual case studies. Due to my positive
experiences made with such briefings in the three evaluations mentioned above, I strongly
recommend that they should become a mandatory first step in any evaluation involving
extensive case study work.
Such a briefing can be organised in form of a one-day “kick-off workshop” bringing together all the parties involved in an evaluation. Such a workshop should however pursue various objectives at the time for ensuring that a common understanding is actually achieved:
? The field teams/local experts need to get a clear understanding about the wider
purpose and strategic objectives of the overall evaluation. Although such aspects
are normally laid down in the “terms of reference” of any evaluation, it might be that
they are formulated in such an abstract manner that their basic relevance for the field
work is not perceived at a first view. I have therefore always felt it useful that a clearly
understandable “message” was communicated by a representative of the client (i.e. the
Commission in case of EU-wide evaluations; national or regional authorities in case of
comprehensive country-specific evaluations) to the involved field teams for
strengthening their commitment to the overall evaluation exercise.
? The field teams/local experts must be made fully aware of the central purpose of
the case study within the wider evaluation. The initial hypotheses to be tested and
the issues which need to be explored by one or more case studies must be made clear
by the members of a core team. Especially in case of more complex EU-wide
evaluations, this should in my opinion also involve an open discussion with the field
teams for gathering their impression on the overall approach. Due to their tacit
knowledge about country- or region-specific settings, such discussions can reveal
some additional issues which might not yet have been sufficiently covered in the
general framework of an evaluation.
? Finally, the field teams/local experts must become fully familiar with all aspects
relating to the practical realisation of case studies. This is again a task of the core
team, which should provide practical and easy-to-understand explanations of all
aspects related to data gathering, overall presentation and the narrative reporting style.
An interesting approach could be to make such issues more explicit alongside a
previously realised “pilot case study” which has been disseminated to the field teams;
but this option is however strongly conditioned by the overall budget and time-
sequencing of an evaluation.
Realising case studies in practise, however with a certain “distance” and “flexibility”
My experiences gained from all these country-specific case studies mentioned above has also
made clear to me that field evaluators should realise a case study-based in-depth analysis
with a certain degree of distance and flexibility.
? Evaluators should first of all try to take some distance with respect to the often
inevitably occurring time-pressure in the practical field work process. Due to my
personal experience, complex evaluations do most often not operate according to what
has been initially planned and the fixed deadlines for delivering mandatory outcomes
tend to create extreme time-pressure at certain stages of the process. Sufficient time is
however needed for preparing and elaborating a high quality case study. Therefore and
as far as a situation of time pressure should occur in the field work phase, I can only
recommend that evaluators stick to their professional standards. This is particularly
important during the information & data gathering process, as it generates the crucial
input on ground of which the story is subsequently told. The evaluator should first
acquire sound background knowledge about the case to be examined (on ground of
desk research) and only afterwards carry out direct stakeholder interviews. If the
inverse is done, it is then very likely that the right questions on the “what”, “how” and
“why” will probably never be asked and that the real point is not made by the final case study elaborated.
? Field evaluators should also be flexible in the actual preparation and realisation
of stakeholder interviews. An “interview guide” with a set of pre-defined general
questions to be addressed is certainly a helpful tool (i.e. function as a “reminder” or a
“not-to-forget about-list”). However, before carrying out stakeholder interviews, these
questions should be carefully reviewed and/or complemented by the evaluator on
ground of the better in-depth insight into the case he/she has gained from previous
desk-based research. Derived from my personal experience, it makes much more sense
to eliminate those questions from the pre-defined catalogue for which sufficient
information is already available and to focus an interview on exploring more in-depth
those issues which are still unclear. Beyond carrying out single-person interviews, it
has also been very rewarding for the information gathering process if various
stakeholders with different interests/viewpoints are brought together (if possible) and
if a group discussion among them is initiated by the evaluator alongside a limited
number of “open” and/or “closed” questions.
? When it comes to drawing up the report on one or more case studies on ground of
the previously gathered data/information, evaluators should try taking some
distance with respect to an initially defined reporting template. Having more
recently also been involved in the synthesis / aggregation work of large-scale
evaluations, I do acknowledge that this is to some extent a rather delicate and
ambivalent issue. On the one hand, there is no doubt that a field evaluator has to
address and explore further the initially identified issues and core evaluation questions
while putting together his/her report on a case study. Especially in a Community-wide
evaluation covering various countries, a too heterogeneous presentation of various
case-study reports can endanger the subsequent comparison and aggregation work to
be carried out by a core team. On the other hand, however, a standard reporting
template with a list of core evaluation questions and related issues to be addressed is
only a guidance tool which might require a further adaptation or completion.
Evaluators should therefore take a certain distance with respect to such a tool (or even
better: have an initially well-agreed degree of flexibility), as this can be very
rewarding for the overall quality of the final case study to be delivered. The field
evaluator should “intuitively co-determine” the importance given to the pre-defined
items on ground his/her actual knowledge about the given case and with respect to
their actual relevance within the story to be told about this case. Two rather simple
examples shall illustrate this: If the standard template requires exploring the degree of
involvement of regional authorities in the delivery of a national policy, but it appears
that in the specific country no regional level exists, it is obvious that no information on
this aspect can be provided. If something particularly important is discovered by a
specific case study on which no assumptions were made in the reporting template
provided (i.e. due to singular specificity of the issue), then a new section has to be
created for telling the story about this particular phenomenon.
? Finally, some kind of “intuitive flexibility” should also be adopted by an
evaluator while elaborating an informative and analytical but still appealing
narrative which tells the story about a specific case. In my previous field activities,
I have always liked – as far as possible within a pre-defined reporting structure - to focus on “critical incidents” for telling an interesting story. For a given case study, I pragmatically considered a critical incident as being something which made a
significant contribution - either positively or negatively - to an activity, process or
phenomenon. Identifying critical incidents did certainly not mean to forget about the
rest; they only helped me to put more emphasis on certain aspects which were
particularly interesting or specific in the light of the overall evaluation exercise.
Building the narrative presentation of the case study around such critical incidents is in
so far rewarding as it produces a certain “drive” which helps keeping a future reader connected to the story until its very end.
3. Evaluating territorial co-operation programmes & projects - telling the
story from a multi-country perspective
The use of the case study method is from my point of view particularly recommended
for evaluating European Territorial Co-operation (ETC) as supported by the Structural
Funds. This is mainly due to the fact that the complexity of real-life interactions within cross-
border, transnational or inter-regional co-operation processes is considerably higher if
compared to those taking place in interventions carried out in a single-country perspective.
Before the start, become aware of the particular challenges associated to an ETC-case study
According to my practical experience, however, case studies on ETC are also a very
demanding and particularly challenging task for each evaluator involved. There are
several reasons for this, which can also be considered as an initial but non-exhaustive list of
baseline requirements for such an exercise:
Firstly, because an ETC-case study has to tell the story from a multi-country perspective
which requires that the evaluator is familiar with more than only one country-specific context
(basic political-administrative settings, different cultures and ways of “doing” and “thinking”).
Especially in the context of cross-border co-operation, evaluators need to have a good
understanding about the general legal framework conditions under which co-operation is
taking place (in order to better understand why co-operation is taking or not taking place) and
about the problems caused by persisting differences of the national legal/regulatory
frameworks which meet at a common border (in order to better understand how such
differences affect the daily life in border areas and how they can be overcome by means of co-
Secondly, the evaluator (or the evaluation team) needs to have well-developed and
appropriate language capacities. They are especially required in the context of cross-border
co-operation, but to some lesser extent in the field of transnational or interregional co-
operation as here most often English is used as a working (or vehicular) language within
Thirdly, the evaluator needs to have a well developed understanding about the previous
history of co-operation (i.e. long-term development and core activities carried out in the past),
as this might significantly influence upon the present time situation. The required amount of
tacit ex-ante knowledge tends to be higher especially in many western European cross-border
areas where a rather long-standing co-operation does exist which often pre-dates the
launching of INTERREG (in 1989/1990). In Central and Eastern European cross-border areas
but also in case of transnational co-operation, where the history of co-operation is often much
more recent and closely related to the implementation of EU-funding schemes (INTERREG,
PHARE / TACIS-CBC etc.), this might be less important and mostly require the examination
of such previous EU-interventions.
Fourthly, the activities for preparing and realising such an exercise are much more complex
as a greater number of key stakeholders originating from different countries and from
different levels of co-operation (strategic level & project level actors; project lead partners &
other project partners; directly involved actors or indirectly concerned actors) needs to be
identified/contacted and directly interviewed in order to be able to tell a story on co-operation
which is comprehensive and balanced. This is often forgotten or at least under-estimated, as it
is assumed that co-operation is taking place in an apparently homogenous context building
upon fully shared views and objectives. In reality, however, the actual perception of problems
and the way of tackling them is largely conditioned by culturally different ways of thinking
and doing and the real action is also here most often driven by complex interest- and actor
constellations (e.g. by one or more partners particularly concerned by an issue who are
“pushing” the project or certain actions of it; by one or more very active partners who make follow a number of other more inactive project partners).
At what levels can the case study method be used in the context of ETC-evaluations?
For evaluating European Territorial Co-operation, the case study method can in principle be used at three different levels:
? A comprehensive evaluation of Community-level funding schemes such as the
“old” INTERREG Community Initiative or the new Objective “European Territorial