Data collection methods
Handout for the Qualitative Research Module – Prepared by Anna Voce, March 2005. rdDrawn from: Patton (2002) Qualitative Research and Evaluation Methods. 3 Edition.
Sage Publications. USA.
The primary methods of data collection in qualitative research are:
; Observation (including document review)
; Focus Group Discussion
Data collection method is distinguished from data collection technique. Data collection
refers to the systematic approach to data collection. refers to the art methodTechnique
of asking, listening, and interpreting.
2. Using multiple sources of information – triangulation
Usually multiple sources of information are used, because no single source of information can be trusted to provide a comprehensive perspective on the issue being studied. By using a combination of observation, interviews and document review, the researcher is able to use different data sources to validate and cross-check findings. Each type and source of data has strength and weaknesses. Using a combination of data types –
triangulation – increases validity as the strengths of one approach can compensate for the weaknesses of another approach.
The limitations of observations include: the possibility that the observer may affect the situation being observed in unknown ways; participants may behave atypically if they know they are being observed; the selective perception of the observer may distort the data; observations focus only on external behaviour; observational data are often constrained by the limited sample of activities observed. Observers need other data sources to find out the extent to which observed activities are typical or atypical.
The limitations of interview data include: possibly distorted answers as a result of personal bias, emotional status, recall error, reactivity of the interviewee to the interviewer, and self-serving responses. Observations provide a check for what is reported in interviews. Interviews on the other hand, permit the observer to go beyond external behaviour to explore thoughts and feelings.
Documents and records also have limitations. They may be incomplete or inaccurate. Document analysis however, provides a behind-the-scenes look, at the programme that may not be directly observable and about which the interviewer might not ask appropriate questions without the leads provided through documents.
By using a variety of sources, the researcher can build on the strengths of each type of data collection while minimising the weaknesses of the single approach.
Scientific inquiry using observational methods requires disciplined training and rigorous preparation … Training to become a skilled observer includes:
; Learning to pay attention, seeing what there is to see, and hearing what there is to
; Practice in writing descriptively
; Acquiring discipline in recording field notes
; Knowing how to separate detail from trivia to achieve the former without being
overwhelmed by the latter
; Using rigorous methods to validate and triangulate observations
; Reporting the strengths and limitations of one‟s own perspective, which requires
both self-knowledge and disclosure
… Preparation (for observation) has mental, physical, intellectual, and psychological dimensions … Evaluators and researchers (must) move their observations from the level of ordinary looking to the rigour of systematic seeing.
The purpose of direct observation is to:
; Describe the setting, in order to understand and capture the context ; Provide a first-hand account of the setting (as opposed to having it described
; Capture things that would normally be taken for granted by someone who is
routinely exposed to the setting
; Observe what people may be unwilling to talk about in the interview ; Confirm the perceptions reported by interviewees
; Provide the researcher with first-hand knowledge of the setting during the analysis
and interpretation stage. The impressions and the feelings of the observer become
part of the data; the observer takes in information and forms impressions that go
beyond what can be fully recorded in even the most detailed field notes.
The descriptions should be factual, accurate, and thorough, without being cluttered by minute detail and trivia. The descriptions must enable the reader to enter into and understand the situation described.
3.2. Observational methods
There are six dimensions that can describe the variations in observational methods. (Please see Patton (2002) Exhibit 6.1. Dimensions showing fieldwork variations. From rdPatton (2002) Qualitative Research and Evaluation Methods. 3 Edition. Sage
3.2.1. Variations in observer involvement – the first and most fundamental
distinction that differentiates observational strategies concerns the extent to
which the observer will be a participant in the setting being studied, from full
participant in the setting, to complete onlooker. The extent of participation can
change over time, as the study progresses. E.g. A research can start the study
as onlooker and then become a full participant, or vice versa. When conducting
full participant observations, this will most likely be implemented concurrently
with other data collection methods (i.e. document analysis, interviews, direct
participation and observation, and introspection). As an onlooker, the process of
observation can be separated from interviewing.
3.2.2. Variations in perspective i.e. emic vs. etic perspectives – A participant
observer shares as intimately as possible in the life and activities of the setting
under study in order to develop an insider‟s view of what is happening (emic
perspective), using the language and categories for classification as identified by
people in the setting. This means that the participant observer not only sees
what is happening but feels what it is like to be part of the setting or programme.
When a researcher must stand far enough away from, or outside of, a particular
setting in order to describe it and how it may differ from other settings, an etic
(outsider) perspective is used to describe what has been observed.
3.2.3. Solo vs. Team observations, participatory vs. collaborative approaches –
Degrees of collaboration in observational methods vary along a continuum. At
one end is the solo observer of a team of observers – what characterises this end
of the continuum is that researchers completely control the inquiry. At the other
end are collaborations with people in the setting being observed, in a co-
researcher role, helping to design the research, collecting data, and analyse the
data. Along the middle of the continuum are various degrees of partial and
periodic (as opposed to continuous) collaboration.
3.2.4. Overt vs. covert observations – A traditional concern about the validity and
reliability of observational data has been the effects of the observer on what is
observed. People may behave quite differently when they are being observed
versus how they behave naturally when they don‟t think they are being observed.
Thus, the argument goes, covert observations are more likely to capture what is
really happening than are overt observations where people in the setting are
aware that they are being studied. Researchers have debated the ethics and
morality of conducting covert observations/research, without the explicit and fully
informed permission of the person to be observed.
3.2.5. Variations in the duration of observations – the duration of observations will
depend on the time and resources available, and on the information needs and
what decisions will result from the information. On one end of the continuum are
short-term studies that involve observations of a single segment of a programme,
sometimes for only one hour or two. On the other end of the continuum,
observations are conducted over months or years. Observations should last long
enough to answer the research questions being asked and to fulfil the purpose of
3.2.6. Variations in the observational focus – The scope can be broad,
encompassing virtually all aspects of the setting, or it can be narrow, involving
only a small part of what is happening. The focus of the observation is provided
by the study design and the nature of the questions being asked.
3.3. What to observe - Sensitising concepts
Part of the value of observational studies is the opportunity to look where no-one lese has looked before and „to see what there is to see‟. However, it is not possible to observe everything. The observation must be focused. The focus is provided by the study design and the nature of the research questions being asked. In the filed, the
observer must organise the observations so that observing becomes and remains manageable. Observers often use „sensitising concepts‟ to guide their observations.
A „sensitising concept‟ is a starting point, an initial guide to the research, and may include
loosely operationalised concepts such as: victim; stress; stigma; group process; leadership; power. The six questions of „what, why, when, how, where, and who‟ constitute a fundamental „sensitising framework‟ based on the central elements of good
The notion of „sensitising concepts‟ reminds us that observers do not enter the field with a completely blank slate. Highly experienced observers have internalised a „sensitising framework‟, to the point where they would not need to list these concepts in a formal
written design. Less experienced researchers, will usually benefit from preparing a formal checklist of major „sensitising concepts‟ in the formal design and then using these concepts to help to organise the observations, and guide the fieldwork, at least initially.
3.4. What to observe – sources of data
The following may be included in a „sensitising framework‟, depending on the research questions:
3.4.1. The physical setting: describe the physical environment, in sufficient detail to
permit the reader to visualise the setting (e.g. in programme evaluation). The
physical environment of a setting can be important to what happens in that
environment. A common mistake among observers is to take the physical
environment for granted. E.g. a researcher may report that a programme took
place in a school. The researcher may have a mental image of a school that
matches what was observed, but schools vary considerably in size, appearance,
and neighbourhood setting. Even more so, the interiors of schools vary
considerably. The same may be said of criminal justice settings, health settings,
and any other human service activity. Nota Bene: use descriptive adjectives
rather that interpretive adjectives. Vivid descriptive adjectives provide sufficient
information that the reader does not have to speculate at what is meant. For
example: simply reporting „a crowded room‟ requires interpretation. Contrast
with this: “the meeting room had a three-person couch across one side, six
chairs along the adjoining walls next to the couch, which included the door. With
20 people in the room, all standing, there was very little space between people.
Several participants were overheard to say, “this room is really overcrowded”.
Such descriptive writing requires attention to detail and discipline to avoid vague,
3.4.2. The human, social environment: In describing the social environment, the
observer looks for ways in which people organise themselves into groups and
sub-groups. Patterns and frequency of interactions, the direction of
communication patterns, and changes in these patterns tell us about the social
environment. How people group together can be illuminative and important.
Decision making patterns can be important. An observer‟s description of a social
environment may not be the same as the perceptions of that environment
expressed by the participants. Nor is it likely that all participants will perceive the
social climate in the same way. At all times it is important that the observer
record participants‟ comments in quotation marks, indicating the source – who
said what? – so at to keep perceptions of participants separate from the
observer‟s own descriptions and evaluations.
3.4.3. Historical information: Historical information is an important part of describing the context in which the research is taking place. The kind of questions that will be asked include: How was the programme created and funded? Who were the original people targeted for programme services, and how have the target populations changed over time? What have been staffing patterns over time? To what extent have goals and intended outcomes changed over time? If the programme is embedded in a larger organisational context, what is the history of that organisation in relation to the programme? How has the larger political and economic environment changed over time, and how have these changes affected programme development?
3.4.4. Planned programme activities: Build observations around activities that have
a kind of structure to them – a beginning, a middle point and a closure point e.g. a class session, a counselling session, a meal time, a meeting, a home visit, etc. The following descriptive questions guide the researcher through the full sequence of observation: Who is involved? What is being said and done by all participants (e.g. staff and clients)? How do they go about what they do? Where do the activities occur? When do things happen? What are the variations in how participants engage in planned activities? How do behaviours and feelings change of the course of the activity? How is the activity ended? How do participants react to the ending of the activity? Each unit of activity is observed and treated as a self-contained even for the purpose of managing the field notes. During analysis one looks across these discrete units of activity to find patterns and themes.
3.4.5. Informal interactions and unplanned activities: During periods of informal
interactions and unplanned activities it may be difficult to organise observations. At these times the researcher needs to remain open to opportunity sampling. The observer watches, listens and looks for opportunities to deepen observations, recording what people do, the nature of informal interactions (e.g. what sub-groups are evident) and what people are saying to each other. The latter is particularly important because during informal interactions and unplanned activities, people have the greatest opportunity to exchange views and talk with each other about what they are really feeling. The researcher may have an opportunity to converse with participants, employing informal, conversational, interviewing. How something is said should be recorded along with what is said. It is these spontaneous interactions that often provide the most significant learning and insight. Thus it may be seen that observation is often combined with informal interviewing.
3.4.6. “Native” language of the programme: The field notes and reports should
include the exact language used by participants, as this provides an indication of the meaning that participants attach to something and illuminates their experiences. Observers must learn the language of participants in the setting they are observing in order to faithfully represent participants on their own terms and be true to their worldview.
3.4.7. Nonverbal communication: While recording the language of participants, the
observer should also attend to nonverbal forms of communication. But because nonverbal behaviours are often misinterpreted, especially cross-culturally, whenever possible and appropriate, having observed what appear to be significant non-verbal behaviours, some effort should be made to follow-up with
those involved to find out directly from them what the non-verbal behaviours really meant.
3.4.8. Unobtrusive observations: Being observed can make people self-conscious
and generate anxiety, and regardless of how sensitively observations are made, the possibility always exists that people will behave differently under conditions where an observation is taking place than they would do if the observer were not present. Rather than resort to „covert‟ observations (which are associated with ethical concerns) choose „unobtrusive measures‟. Unobtrusive measures are those made without the knowledge of the people being observed and without affecting what is observed. For example: In an evaluation of the National Museum of Natural History at the Smithsonian Institute (Wolf and Tymitz 1978), the researchers looked for „wear spots‟ as indicators of use of particular exhibits.
Dusty equipment and files may indicate things that are not used. Worn books to indicate that they have been used.
3.4.9. Documents: Records, documents, artefacts and archives constitute a
particularly rich source of information about many organisations and programmes. In contemporary society, all kinds of entities have a trail of paper, a kind of spoor that can be mined as part of fieldwork. E.g. Families keep photographs, letters, sentimental objects; people who commit suicide leave behind suicide notes; gangs inscribe public places with graffiti; organisations produce records; service providers keep client files. Indeed, and often intriguing form of analysis involves comparing official statements found in public documents (annual reports, policy statements) with private memos and what the observer actually hears or sees. At the very beginning of the study, access to important documents and records should be negotiated. The ideal situation would be to have access to all routine records, correspondence, financial and budget records, organisational rules, regulations, memoranda, and any other official and unofficial documents generated by or for the programme. Document review can be valuable in guiding the researcher in what needs to be pursued further in direct observation and interviewing. Confidentiality must be respected, as with all information to which the researcher has access. The extent to which actual references to, and quotations from, records and documents are included in the final reports depends on whether the documents are considered part of a public record and therefore ale to be publicised without breach of confidentiality. In some cases, with permission and proper safeguards to protect confidentiality, some information from private documents can be quoted directly and cited.
3.4.10. Observing what does not happen: If programme goals,
implementation design, and/or proposals suggest that certain things ought to happen or are expected to happen, and they don‟t, then it is appropriate for the
observer to note these things that do not happen. E.g. if a community in which water is scarce shows no evidence of conflict over water rights, a researcher could be expected to report on the absence of community conflict. If a school programme is intended to provide children with opportunities to explore the community, and no such explorations occur, it is appropriate for the observer to note this. If clinic supervisors are supposed to conduct monthly supervisory visits to the clinics, and these do not happen, then it is entirely appropriate for the observer to record this.
3.4.11. Observing oneself: The principle of reflexivity reminds the qualitative researcher to observe oneself, to be attentive to and conscious of the cultural, political, linguistic, ideological origins of his/her own perspective and voice as well
as, and often in contrast to – the perspectives and voices of those s/he is
observing. Reflexivity calls for critical self-reflection and self-knowledge, and a
willingness to consider how one is affected by, and how one influences, what is
observed. The observer must ultimately deal with issues of authenticity, reactivity,
and how the observational process may have affected what was observed as well
as how the background and predisposition of the observer may have constrained
what was observed and understood.
We interview people to find out from them those things we cannot directly observe. We cannot observe feelings, thoughts, and intentions. We cannot observe behaviours that took place at some previous point in time. We cannot observe how people have organised the world and the meanings they attach to what goes on in the world. The purpose on interviewing is to allow us to enter into the other person‟s perspective.
Qualitative interviewing begins with the assumption that the perspective of the other is meaningful, knowable, and able to be made explicit. We interview to find out what is in and on someone else‟s mind, to gather their stories.
The quality of the information obtained during an interview is largely dependant on the interviewer. It is important to develop disciplined an rigorous interview techniques. It is also important to have a deep and genuine interest in what people have to say about their world.
4.1. Types on interviews
There are three basic approaches to collecting qualitative data through open-ended interviews. Each approach has strengths and weaknesses, and each serves a somewhat different purpose. Each approach involves a different type of preparation, conceptualisation and instrumentation. The three approaches differ in the extent to which interview questions are determined and standardised before the interview occurs.
4.1.1. The informal, conversational interview: relies entirely on the spontaneous
generation of questions in the natural flow of an interaction, often as part of
participant observer fieldwork. It is also called „unstructured interviewing‟. The
conversational interview offers maximum flexibility to pursue information in
whatever direction appears to be most appropriate, depending on what emerges
from observing a particular setting or from talking with one or more individuals in
that setting. Most of the questions will flow from the immediate context. Data
gathered from informal conversational interview will be different for each person
interviewed. The same person may be interviewed on several occasions, with
questions specific to the interaction or event at hand. Previous responses can be
revisited and deepened. This approach works particularly well where the
researcher can stay in the setting for some period of time so as not to be
dependant on a single interview opportunity. Being unstructured does not mean
that conversational interviews are unfocused. Sensitising concepts and the overall
purpose of the interview will guide the questions that are asked. The strength of
the informal conversational method is that it offers flexibility, spontaneity, and
responsiveness to individual differences and situational changes. Questions can
be personalised to deepen communication with the person being interviewed. A
weakness in the informal conversational interview is that it may require a greater
amount of time to collect systematic information because it may take several
conversations with different people before a similar set of questions has been
posed to each participant in the setting. Because the approach depends on the
conversational skills of the interviewer to a greater extent than do formal standardised formats, this go-with-the-flow style of interviewing may be susceptible to interviewer effects, leading questions, and biases, especially with novice researchers. Data obtained from informal conversational interviews may be difficult to pull together and analyse. Because different questions will generate different responses, the researcher has to spend a great deal of time sifting through response to find patterns that have emerged at different points in different interviews with different people. By contrast, interviews that are more systematised and standardised facilitate analysis but provide less flexibility and are less sensitive to individual and situational differences.
4.1.2. The general interview guide approach: involves outlining a set of issues that
are to be explored with each respondent before the interview starts. The guide serves as a basic checklist during the interview to make sure that all relevant topics are covered. The same basic line of questioning is pursued with each respondent. The interview guide provides topics within which the interviewer is free to explore, probe, and ask questions that will elucidate and illuminate that particular subject. The interviewer remains free to build a conversation within a particular subject area, to word questions spontaneously, and to establish a conversational style but with the focus on a particular subject that has been predetermined. The advantage of an interview guide is that the interviewer has carefully decided how to use the limited time available in an interview situation. The guide helps make interviewing a number of different people systematic and comprehensive, by delimiting in advance the issues to be explored. A guide is essential in conducting focus group discussions. Interview guides can be developed in more or less detail, depending on the extent to which the interviewer is able to specify important issues in advance and the extent to which it is important to ask questions in the same order to all respondents.
4.1.3. The standardised open-ended interview: consists of a set of questions
carefully worded and arranged with the intention of taking each respondent through the same sequence and asking each respondent the same questions with essentially the same words. The standardised open-ended interview is used when it is important to minimise variation in the questions posed to interviewees. In a fully structured interview instrument, the question would be completely specified, as would the probes (designed to get deeper information) as well as the transition questions (designed to introduce the next topic). An Ethics Committee may request this kind of detail, especially if the topic of the research is controversial or intrusive. In team research, standardised interviews ensure consistency across interviewers. In multi-site studies, structured interviews provide comparability across sites. Structured questions may also compensate for variability in researcher skills. Training in the interview approach will minimise variability due to differences in experience and skill. Structured open-ended interviews may also be used to collect data before, during and after a programme of intervention. This allows for answers to be compared across time periods.
4.1.4. Combining approaches: these interview strategies are not mutually exclusive. A combined strategy offers the interviewer flexibility in probing and in determining when it is appropriate to explore certain subjects in greater depth, or even to pose questions about new areas of inquiry that were not originally anticipated in the interview instrument‟s development. A common combination strategy involves using standardised interview formats at the beginning of the interview and then leaving the interviewer free to pursue any subjects of interest during the latter parts of the interview. Another combination would include using the
informal conversational interview early in an evaluation report, followed midway
through by an interview guide, and then closing with a standardised open-ended
interview to get systematic information from a sample of participants when
concluding the study.
4.2. Types of questions
Six kinds of questions can be asked of people. On any given topic it is possible to ask any of these questions. Distinguishing types of questions forces the interviewer to be clear about what is being asked and helps the interviewee respond appropriately.
4.2.1. Experience and behaviour questions: Questions about what a person does,
has done, has experienced, and what activities would have been observable if the
researcher were present. “If I followed you through a typical day, what would I
see you doing?”
4.2.2. Opinion and value questions: Answers to these questions tell us about what
people about a certain action or experience, as opposed to their actions and think
behaviours. These questions help us to understand people‟s goals, intentions,
desires, and expectations. “What do you believe?” What do you think about …?”
What would you like to see happen?” What is your opinion of …?”
4.2.3. Feeling questions: feeling questions aim at eliciting emotions – feeling
responses to of people to their experiences and thoughts. “How do you feel
about that?” Opinions and feelings are often confused. It is critical the
interviewer understand the difference between the two in order to know that they
have the kind of answer they want for the question they are asking. Sometimes
an interviewer will ask “How do you feel about that?” and the response will be “I
think it is the best we can do under the circumstances.” The question about
feelings has not really been answered. The confusion sometimes occurs because
the interviewer gives the wrong cues and does not asking the question correctly.
When asking feeling questions, the interviewer has to ask and listen for feeling
4.2.4. Knowledge questions: knowledge questions inquire about the respondent‟s
4.2.5. Sensory questions: sensory questions ask about what is seen, heard, touched,
tasted, and smelled. Sensory questions attempt to have the interviewees
describe the stimuli they experience.
4.2.6. Background/demographic questions: age, education, occupation – are
standard background questions. Answers to these questions help the interviewer
locate the respondent in relation to other people. Qualitative inquiry is
particularly appropriate in finding out how people perceive and talk about their
backgrounds. E.g. Tiger Woods has African, Thai, Chinese, American Indian and
European ancestry and calls himself a “Cablinasian”.
4.3. Asking questions
4.3.1. The time frame of questions: can be in the present, past or future tense.
4.3.2. Sequencing questions: begin an interview with non-controversial present
behaviours, activities and experiences. Such questions ask for relatively
straightforward descriptions. They require minimal recall and interpretation, and
are generally easy to answer. Then opinions and feelings may be solicited,
building on and probing for interpretations of the experience. Opinions and
feelings are likely to be more grounded and meaningful once an experience has
been relived. Knowledge and skill questions can be quite threatening if asked too
abruptly. They are best asked when rapport and trust have been established.
Questions about the present are easier than questions about the past. Then ask
about the future. Background and demographic questions, depending how
personal they are, may make the respondent feel uncomfortable. If asked at the
beginning of the interview they may condition the respondent to give short
answers. It may be best to save these questions until the end.
4.3.3. Wording questions: How a question is worded affects how the interviewee
responds. Asking questions is an art. They should be genuinely open-ended,
neutral, singular and clear. An genuinely open-ended questions minimises the
possibility of imposing predetermined responses. “How do you feel about … ?” is
open-ended whereas “How satisfied are you with …?” already presupposes an
answer about satisfaction. Avoid questions that will lead to Yes/No answers. E.g.
“Are you satisfied with …?” Neutral questions refer to the non-judgementalism
adopted by the interviewer. Singular questions ensure that not more than one
idea is contained in any given question. “How well do you know and like the staff
in this clinic?” contains two ideas. The clarity of questions is enhanced by asking
simple, understandable, unambiguous questions, using language and terminology
that is familiar to the respondent.
4.3.4. Why questions: Take care about asking why questions. They imply causal
relationships, which may be complex to unravel, an may make respondents feel
defensive. Think of other ways of asking what it is you want to know.
4.3.5. The final or closing question: provide th interviewee with the opportunity to
have the final say. E.g. “That covers all I wanted to ask. Anything you would
like to add?”
5. Focus group discussion
Focus group is a complementary technique to individual interviews. It is a discussion on a specific topic, with a small group of people (6-10) with similar backgrounds who participate in the discussion for 1 – 2 hours. In a given study, a series of different focus groups will be conducted to get a variety of perspectives and increase confidence in whatever patterns emerge. In a focus group, participants get to hear each other‟s responses and make additional comments beyond their original responses as they hear what other people have to say. Participants do not need to agree with each other, and no consensus needs to be reached. Nor is it necessary for people to disagree. The objective is to get high quality data in a social context where people can consider their own views in the context of the views of others. The interviewer may rather be called the moderator – highlighting the specific function of the interviewer, that of moderating or guiding the discussion. The focus group is not a collection of simultaneous individual interviews, but rather a discussion where the conversation flows because of the function of the moderator.