THE USE OF QUALITATIVE INTERVIEWS IN EVALUATION
Meg Sewell
WHAT IS QUALITATIVE INTERVIEWING?
At the most basic level, interviews are conversations (Kvale, 1996). Kvale defines qualitative research interviews as "attempts to understand the world from the subjects' point of view, to unfold the meaning of peoples' experiences, to uncover their lived world prior to scientific explanations." Interviews for research or evaluation purposes differ in some important ways from other familiar kinds of interviews or conversations. Unlike conversations in daily life, which are usually reciprocal exchanges, professional interviews involve an interviewer who is in charge of structuring and directing the questioning. In some professional interview situations, such as job interviews or legal interrogations, the power of the questioner is much greater than the power of the one being questioned. Therapeutic or clinical interviews are another special kind of professional interview, in which the purpose is to increase understanding and produce change in the person being interviewed. While interviews for research or evaluation purposes may also promote understanding and change, the emphasis is on intellectual understanding rather than on producing personal change (Kvale, 1996).
In qualitative program evaluation, open-ended responses to questions provide the evaluator with quotations, which are the main source of raw data. Patton (1987) notes that quotations "reveal the respondents' levels of emotion, the way in which they have organized the world, their thoughts about what is happening, their experiences, and their basic perceptions. The task for the qualitative evaluator is to provide a framework within which people can respond in a way that represents accurately and thoroughly their point of view about the program."
In practice, open-ended, qualitative interview questions are often combined with more closed-ended, structured interview formats. Qualitative interviews may be used as an exploratory step before designing more quantitative, structured questionnaires to help determine the appropriate questions and categories. Conversely, interviews may be used after results of more standardized measures are analyzed to gain insight into interesting or unexpected findings. While quantitative results are sometimes dismissed on political or methodological grounds by those who disagree with the findings, it can be harder to dismiss the actual words of participants which convey their powerful emotions. Patton (1990) gives an example of a school board which dismissed a survey showing teacher dissatisfaction as just the complaints of lazy teachers who did not want to be held accountable in their work. However, when confronted with actual quotations from teachers, reflecting both commitment to their jobs and deep concerns about problems in this particular system, the Board was more willing to hear and respond to their concerns.
QUALITATIVE INTERVIEWING IS MOST USEFUL FOR:
* Evaluating programs that are aimed at individualized outcomes
* Capturing and describing program processes
* Exploring individual differences between participants' experiences and outcomes
* Evaluating programs that are seen as dynamic or evolving
* Understanding the meaning of a program to its participants
* Documenting variations in program implementation at different sites
QUALITATIVE INTERVIEWING IS NOT AS USEFUL FOR:
* Evaluating programs that emphasize common outcomes for all participants
* Measuring specific, predetermined effects of a program on participants
* In impact evaluations, deciding whether your intervention caused changes or effects in participants (since determining causality requires more controlled conditions)
USING QUALITATIVE INTERVIEWING WITH THE STATE STRENGTHENING EVALUATION GUIDE
If you are using the Five-Tiered Approach to Program Evaluation outlined in the State Strengthening Evaluation Guide (Callor, Betts, Carter & Marczak, 1997), qualitative interviewing can be used at several levels:
TIER 1 - Program Definition
A key task in Tier 1 of the planning and evaluation process is to assess and document the need for a particular program in the community. At this stage, qualitative interviews with a few key stakeholders can help document the need. These key people might include potential program participants, parents, and community leaders such as the principal of the neighborhood school. The open-ended and personal nature of qualititative interviews can help ensure that you understand how (or even whether) community members perceive a problem and what they would see as key elements in a program designed to resolve it. Interviews might also alert you to potential barriers to participation. For example, parents may be unable to attend parent meetings unless childcare is provided.
TIER 2 - Accountability
Although qualitative interviews are less likely to play a major role in Tier 2 activities, they may still have a place in assessing implementation of the program at this stage. A few informal, conversational interviews with participants might quickly verify that the program is reaching its intended audience or that key services are being provided in a timely way.
TIER 3 - Understanding and Refining
Interviewing may be especially useful for the Tier 3 activities of understanding and refining the program. Qualitative interviews with staff and participants lend themselves well to the tasks of gathering program satisfaction data, process data, and identifying "lessons learned" that can be used to refine and modify the program.
TIER 4 - Progress Toward Outcomes
In Tier 4, qualitative interviews may be most useful in combination with more objective or quantitative measures and indicators. While the quantitative measures indicate in a literal sense whether outcome objectives are being met, qualitative interviews speak more to how the participants feel about what is happening, and its meaning in their lives. An after-school child care program might be quite successful in its stated objective of enrolling a specified number of children who would otherwise be unsupervised at home in the afternoon. However, it might still be important to know that some parents had concerns because staff seemed to tolerate excessive levels of teasing or bullying behavior by some children, so that their own children did not like to attend.
TIER 5 - Program Impact
Selective use of quotations from participants, staff, and community members in reports, news releases, and publications can be very effective in "getting the word out." When a program, especially a new or innovative program model, has been successful, positive endorsements from those close to the program are good for public relations. Even when feedback was not all positive or significant problems arose along the way, the insights recorded from interviews can be very helpful to those who might wish to replicate the program or model elsewhere. Qualitative interviews might also be useful in sorting out why the program is having greater impact on some participants than others. For example, interviews might reveal that youths who participate voluntarily benefit most from your program, while those who are forced to participate resist making changes and show less impact.
ADVANTAGES OF USING QUALITATIVE INTERVIEWING
* Allows the participant to describe what is meaningful or important to him or her using his or her own words rather than being restricted to predetermined categories; thus participants may feel more relaxed and candid
* Provides high credibility and face validity; results "ring true" to participants and make intuitive sense to lay audiences
* Allows evaluator to probe for more details and ensure that participants are interpreting questions the way they were intended
* Interviewers have the flexibility to use their knowledge, expertise, and interpersonal skills to explore interesting or unexpected ideas or themes raised by participants
* Sometimes no existing standardized questionnaires or outcome measures are available that are appropriate for what your program is trying to accomplish
DISADVANTAGES OF USING QUALITATIVE INTERVIEWING
* May be experienced as more intrusive than quantitative approaches; participants may say more than they intended to say, and later regret having done so
* May be more reactive to personalities, moods, and interpersonal dynamics between the interviewer and the interviewee than methods such as surveys
* Training interviewers and conducting interviews can be expensive and time-consuming, because qualititative interviewing requires considerable skill and experience
* Analyzing and interpreting qualitative interviews is much more time-consuming than analyzing and interpreting quantitative interviews
* More subjective than quantitative interviews because the evaluator/researcher decides which quotes or specific examples to report
HOW TO USE QUALITATIVE INTERVIEWING
Types of Qualitative Interviews
Patton (1990) identifies three basic types of qualitative interviewing for research or evaluation: the informal conversational interview, the interview guide approach, and the standardized open-ended interview. Although these types vary in the format and structure of questioning, they have in common the fact that the participant's responses are open-ended and not restricted to choices provided by the interviewer. A fourth type of interview, the closed, fixed-response interview, falls in the realm of quantitative interviewing. In quantitative or structured interviews, the respondent is asked to choose from a predetermined set of response categories. Each type of qualitative interview has advantages and disadvantages. Once your evaluation team has decided to include qualitative interviews as part of your evaluation plan, it is still necessary to consider the strengths and weaknesses of each type in relation to your needs and the resources available for your evaluation:
1) Informal Conversational Interview: This type of interview may occur spontaneously in the course of field work, and the respondent may not know that an "interview" is taking place. Questions emerge from the immediate context, so the wording of questions and even the topics are not predetermined. The major advantage is that the interview is highly individualized and relevant to the individual. Thus, it is likely to produce information or insights that the interviewer could not have anticipated. This type of interview requires an interviewer who is very knowledgeable and experienced in the content area and strong in interpersonal skills, since he or she will have considerable discretion in directing the interview. However, since different information is collected from different people, this kind of interview is not systematic or comprehensive, and it can be very difficult and time-consuming to analyze the data.
2) Interview Guide Approach: This may be the most widely used format for qualitative interviewing. In this approach, the interviewer has an outline of topics or issues to be covered, but is free to vary the wording and order of the questions to some extent. The major advantage is that the data are somewhat more systematic and comprehensive than in the informal conversational interview, while the tone of the interview still remains fairly conversational and informal. Like the conversational interview, this type of interview also requires an interviewer who is relatively skilled and experienced, since he or she will need to know when to probe for more in-depth responses or guide the conversation to make sure that all topics on the outline are covered. A possible drawback is that sticking to the outlined topics will prevent other important topics from being raised by the respondent. Also, while this format is more systematic than the conversational interview, it is still difficult to compare or analyze data because different respondents are responding to somewhat different questions.
3) Standardized Open-Ended Interview: In this format, the interviewers adhere to a strict script, and there is no flexibility in the wording or order of questions. It is still considered a qualitative interview rather than a quantitative interview, because the responses are open-ended. This is the most structured and efficient of the qualitative interviewing techniques and is useful for reducing bias when several interviewers are involved, when interviewers are less experienced or knowledgeable, or when it is important to be able to compare the responses of different respondents. This may be the best choice for an evaluation if you must rely on volunteer or inexperienced interviewers or if you have limited time and money available for analyzing the data. The major drawback is that the interviewer has little flexibility to respond to the particular concerns of the individual, and there is no guarantee that the questions asked tap into the issues that are most relevant to this particular respondent.
Design and Development of Interview Studies for Evaluation
While much of the value of qualitative interviewing lies in its flexibility and openness, it remains extremely important for the evaluation planning team to think through the process and provide the basic structure and framework which will make the study useful and worthwhile. Kvale (1996) describes in detail seven stages in designing and implementing an interview study, which are summarized below. Readers are encouraged to consult the texts by Kvale and others listed below (see Annotated Bibliography).
1) Thematizing - Before even thinking about particular methods or interview formats, the evaluation team needs to be clear on the purpose of the study and the topic to be investigated. The questions of "why" and "what" need to be answered before the question of "how" can be answered. This is as important in a qualitative evaluation study as in a quantitative one.
2) Designing - The overall design for the study, including the later stages of analyzing and reporting, should be planned before the interviewing begins. For example, if there are no funds for transcribing or analyzing interviews, it may be wise to use a more structured format that will be easier to code later.
3) Interviewing - To an extent that is not true in many other methods, the interviewer is the instrument in this type of evaluation (Guba & Lincoln, 1981, as cited in Patton, 1987). The "instrument" can be affected by factors like fatigue, personality, and knowledge, as well as levels of skill, training, and experience. Patton (1987) points out that any face-to-face interview is also an observation. The skilled interviewer is sensitive to nonverbal messages, effects of the setting on the interview, and nuances of the relationship. While these subjective factors are sometimes considered threats to validity, they can also be strengths because the skilled interviewer can use flexibility and insight to ensure an in-depth, detailed understanding of the participant's experience.
4) Transcribing - This important step prepares the material from the interview for analysis. Both Kvale (1996) and Patton (1990) provide detailed practical suggestions for this process, ranging from ensuring that your tape recorder has good batteries to developing a sensitivity to the linguistic differences between oral speech and written text.
5) Analyzing - Data analysis is an issue that should be considered very early in the process of designing a study. Qualitative interviews and their transcripts produce a large volume of material which must be condensed, categorized or otherwise interpreted and made meaningful, and this may turn out to be one of the most costly and time-consuming aspects of the evaluation. If time and resources are limited, you may wish to use more standardized interview formats which are easier to code and interpret.
Methods for analyzing and interpreting qualitative interviews vary widely. Kvale (1996) describes five analysis methods that include 1) meaning condensation, 2) meaning categorization, 3) narrative structuring, 4) meaning interpretation, and 5) generating meaning through ad hoc methods. Patton (1987, 1990) also addresses a number of techniques for quantifiying and analyzing qualitative interview data.
The most appropriate method of analysis for any given study will depend on the purpose of your evaluation and the nature of the material, as well as the time and resources available for this part of the process. Some methods attempt to be more objective, while others depend more heavily on subjective judgements and insights of the researcher. Computer software programs are available that can assist in categorizing interview statements or counting key words, which may allow some forms of quantitative analysis.
6) Verifying - In traditional research terms, this means determining reliability (how consistent the findings are), validity (whether the study really investigates what you intended to investigate), and generalizability (whether the findings apply to anyone outside of this particular program). In qualititative studies, one important way of verifying findings or establishing validity is to actually take transcripts or analyzed results back to some of the interview participants, and ask them if this is really what they meant. Guba and Lincoln (1989) discuss the concepts of confirmability, dependability, credibility and transferability as alternative ways of ensuring quality of data in qualitative evaluations. For more in depth discussions of these important issues in qualitative research, readers are strongly encouraged to consult relevant chapters in Kvale (1996), Patton (1990), or Guba and Lincoln (1989).
7) Reporting - If the evaluation report is to effectively communicate findings, it must a) be in a form that meets some accepted scientific criteria, b) meet ethical standards such as confidentiality and respect, and c) be readable and usable for its intended audiences. In some cases, different reports may be needed for different audiences. An appropriate balance needs to be found between including endless quotations that will bore the reader and just quoting a few entertaining stories that happened to appeal to the researcher.
Ethical Issues in Qualitative Interviewing
The very personal, conversational nature of interview situations highlight many of the basic ethical issues of any research or evaluation method (Patton, 1990). Among these issues are:
1) Confidentiality - Because respondents may be sharing very personal information, it is important to honestly assess how much confidentiality you can promise. Some kinds of disclosures (such as child abuse or threats to the safety of self or others) must be reported, and respondents need to know this from the start. Also consider how the confidentiality of individuals will be preserved when the data are analyzed and reported. Related issues include who has access to the data and who "owns" it.
2) Informed consent - Most studies, including program evaluations, are covered by some kind of human subjects review process. This will usually require that respondents sign a permission form agreeing to participate, after being informed of potential risks and benefits. If children are involved, a parent or legal guardian must provide this permission.
3) Risk assessment - It is important to consider all potential risks and include them in the informed consent process. Even though "just talking" may seem inherently harmless, people who participate in open-ended interviews may experience psychological stress, legal or political repercussions, or ostracism by peers or staff who believe that the participant has said unflattering things about them to the interviewer.
4) Promises and reciprocity - The issue here is what interview participants get in return for sharing their time and insights with you. Will they or their communities benefit in some way from the results of the study? If promises are made (such as copies of reports or monetary payments), those promises should always be kept.
5) Interviewer mental health - Interviewing experiences can be intense interpersonal experiences. Just as participants may experience psychological stress from disclosing more than intended or being reminded of painful experiences, interviewers may be overwhelmed by the sensitive nature of what is seen or heard, especially in home- or field-based interviews. Some form of debriefing after the interview may be necessary. Interviewers should always know who to go to if they need advice or consultation on handling practical or emotional issues that arise from an interview.
Special Issues in Interviewing Children
The skills and logistics involved in interviewing and testing children are not necessarily the same as those required for adults. The performance of young children in testing situations is known to be quite sensitive to factors like the familiarity of the setting and the examiner (Hauser-Cram & Shonkoff, 1988). For ethnic minority children, language and cultural expectations about testing situations may be particularly relevant. Those who are conducting qualitative interviews with children should be trained in child development and familiar with the ways that children at different ages understand and use language. Young children in particular can be very literal and may misunderstand adult language or metaphors. They are also usually unable to respond to "Why?" questions, which require them to analyze or interpret their behavior or thoughts.
There are also special ethical issues involved in interviewing children. Informed consent can be a particularly complicated matter when children are involved. In virtually all cases, it is necessary to have the consent of a parent or legal guardian before interviewing a child for a research or evaluation study. While some schools or agencies ask parents to sign some form of blanket permission for their child to participate in assessments or studies, it should never be taken for granted that this is the case, or that it will cover your particular study. Depending on the age and developmental level of the child, ensuring that the child understands his or her right to refuse participation is also a concern.
ANNOTATED BIBLIOGRAPHY
Callor, S., Betts, S. C., Carter, R., & Marczak, M. (1997). State Strengthening Evaluation Guide. Tucson, AZ: USDA/CSREES & University of Arizona.
Program development and evaluation manual designed by the Evaluation Collaboration for use in State Strengthening Projects. Based on the five-tiered model of evaluation of Jacobs (1988). The manual may be viewed or downloaded from the World Wide Web at: http://ag.arizona.edu/fcr/fs/cyfar/evalgde.htm
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
Includes a good section on reliability and validity issues in qualitative evaluations. While traditional techniques for ensuring reliability, validity and generalizability may not apply to qualitative studies, Guba and Lincoln describe other quality standards that may serve similar purposes. These alternative standards include the concepts of confirmability, dependability, credibility, and transferability.
Guba, E. G. , & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches.. San Francisco: Jossey-Bass.
Guba and Lincoln's book describes a qualitative approach called "fourth generation evaluation". Although specific aspects of their method are somewhat controversial among program evaluators, they make a strong case for personal, individualized approaches to interviewing. They also emphasize the importance of verifying findings with respondents after data is collected, to ensure that the results match what they intended to say.
Hauser-Cram, P., & Shonkoff, J. P. (1988). Rethinking the assessment of child-focused outcomes. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs (73-94). New York: Aldine de Gruyter.
This chapter provides an overview of issues to be considered in evaluating community-based programs for children. Discussion of assessment of children includes general methodology issues and reviews some relevant measures.
Kvale, S. (1996). Inter Views: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.
Although the focus of this book is on the use of qualitative interviewing in research rather than on its use in program evaluation, it provides an in-depth treatment of many important theoretical and practical issues. A unique feature of the book is that it brings theory and techniques from the fields of linguistics and textual criticism to the planning, interpretation, and reporting of interview studies. Qualitative interview studies are placed in the philosophical contexts of hermeneutics, phenomenology, ethics, feminism, and post-modernism. A chapter on reporting of interview studies provides very practical advice on maximizing the readability and usefulness of final reports.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
Patton's book provides a concise practical manual for using qualitative methods in evaluation, although only one chapter deals specifically with interviewing techniques. Could be a useful complement to the more theoretically-oriented treatment of the issues in Kvale's book (see above).
Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.
Meg Sewell
WHAT IS QUALITATIVE INTERVIEWING?
At the most basic level, interviews are conversations (Kvale, 1996). Kvale defines qualitative research interviews as "attempts to understand the world from the subjects' point of view, to unfold the meaning of peoples' experiences, to uncover their lived world prior to scientific explanations." Interviews for research or evaluation purposes differ in some important ways from other familiar kinds of interviews or conversations. Unlike conversations in daily life, which are usually reciprocal exchanges, professional interviews involve an interviewer who is in charge of structuring and directing the questioning. In some professional interview situations, such as job interviews or legal interrogations, the power of the questioner is much greater than the power of the one being questioned. Therapeutic or clinical interviews are another special kind of professional interview, in which the purpose is to increase understanding and produce change in the person being interviewed. While interviews for research or evaluation purposes may also promote understanding and change, the emphasis is on intellectual understanding rather than on producing personal change (Kvale, 1996).
In qualitative program evaluation, open-ended responses to questions provide the evaluator with quotations, which are the main source of raw data. Patton (1987) notes that quotations "reveal the respondents' levels of emotion, the way in which they have organized the world, their thoughts about what is happening, their experiences, and their basic perceptions. The task for the qualitative evaluator is to provide a framework within which people can respond in a way that represents accurately and thoroughly their point of view about the program."
In practice, open-ended, qualitative interview questions are often combined with more closed-ended, structured interview formats. Qualitative interviews may be used as an exploratory step before designing more quantitative, structured questionnaires to help determine the appropriate questions and categories. Conversely, interviews may be used after results of more standardized measures are analyzed to gain insight into interesting or unexpected findings. While quantitative results are sometimes dismissed on political or methodological grounds by those who disagree with the findings, it can be harder to dismiss the actual words of participants which convey their powerful emotions. Patton (1990) gives an example of a school board which dismissed a survey showing teacher dissatisfaction as just the complaints of lazy teachers who did not want to be held accountable in their work. However, when confronted with actual quotations from teachers, reflecting both commitment to their jobs and deep concerns about problems in this particular system, the Board was more willing to hear and respond to their concerns.
QUALITATIVE INTERVIEWING IS MOST USEFUL FOR:
* Evaluating programs that are aimed at individualized outcomes
* Capturing and describing program processes
* Exploring individual differences between participants' experiences and outcomes
* Evaluating programs that are seen as dynamic or evolving
* Understanding the meaning of a program to its participants
* Documenting variations in program implementation at different sites
QUALITATIVE INTERVIEWING IS NOT AS USEFUL FOR:
* Evaluating programs that emphasize common outcomes for all participants
* Measuring specific, predetermined effects of a program on participants
* In impact evaluations, deciding whether your intervention caused changes or effects in participants (since determining causality requires more controlled conditions)
USING QUALITATIVE INTERVIEWING WITH THE STATE STRENGTHENING EVALUATION GUIDE
If you are using the Five-Tiered Approach to Program Evaluation outlined in the State Strengthening Evaluation Guide (Callor, Betts, Carter & Marczak, 1997), qualitative interviewing can be used at several levels:
TIER 1 - Program Definition
A key task in Tier 1 of the planning and evaluation process is to assess and document the need for a particular program in the community. At this stage, qualitative interviews with a few key stakeholders can help document the need. These key people might include potential program participants, parents, and community leaders such as the principal of the neighborhood school. The open-ended and personal nature of qualititative interviews can help ensure that you understand how (or even whether) community members perceive a problem and what they would see as key elements in a program designed to resolve it. Interviews might also alert you to potential barriers to participation. For example, parents may be unable to attend parent meetings unless childcare is provided.
TIER 2 - Accountability
Although qualitative interviews are less likely to play a major role in Tier 2 activities, they may still have a place in assessing implementation of the program at this stage. A few informal, conversational interviews with participants might quickly verify that the program is reaching its intended audience or that key services are being provided in a timely way.
TIER 3 - Understanding and Refining
Interviewing may be especially useful for the Tier 3 activities of understanding and refining the program. Qualitative interviews with staff and participants lend themselves well to the tasks of gathering program satisfaction data, process data, and identifying "lessons learned" that can be used to refine and modify the program.
TIER 4 - Progress Toward Outcomes
In Tier 4, qualitative interviews may be most useful in combination with more objective or quantitative measures and indicators. While the quantitative measures indicate in a literal sense whether outcome objectives are being met, qualitative interviews speak more to how the participants feel about what is happening, and its meaning in their lives. An after-school child care program might be quite successful in its stated objective of enrolling a specified number of children who would otherwise be unsupervised at home in the afternoon. However, it might still be important to know that some parents had concerns because staff seemed to tolerate excessive levels of teasing or bullying behavior by some children, so that their own children did not like to attend.
TIER 5 - Program Impact
Selective use of quotations from participants, staff, and community members in reports, news releases, and publications can be very effective in "getting the word out." When a program, especially a new or innovative program model, has been successful, positive endorsements from those close to the program are good for public relations. Even when feedback was not all positive or significant problems arose along the way, the insights recorded from interviews can be very helpful to those who might wish to replicate the program or model elsewhere. Qualitative interviews might also be useful in sorting out why the program is having greater impact on some participants than others. For example, interviews might reveal that youths who participate voluntarily benefit most from your program, while those who are forced to participate resist making changes and show less impact.
ADVANTAGES OF USING QUALITATIVE INTERVIEWING
* Allows the participant to describe what is meaningful or important to him or her using his or her own words rather than being restricted to predetermined categories; thus participants may feel more relaxed and candid
* Provides high credibility and face validity; results "ring true" to participants and make intuitive sense to lay audiences
* Allows evaluator to probe for more details and ensure that participants are interpreting questions the way they were intended
* Interviewers have the flexibility to use their knowledge, expertise, and interpersonal skills to explore interesting or unexpected ideas or themes raised by participants
* Sometimes no existing standardized questionnaires or outcome measures are available that are appropriate for what your program is trying to accomplish
DISADVANTAGES OF USING QUALITATIVE INTERVIEWING
* May be experienced as more intrusive than quantitative approaches; participants may say more than they intended to say, and later regret having done so
* May be more reactive to personalities, moods, and interpersonal dynamics between the interviewer and the interviewee than methods such as surveys
* Training interviewers and conducting interviews can be expensive and time-consuming, because qualititative interviewing requires considerable skill and experience
* Analyzing and interpreting qualitative interviews is much more time-consuming than analyzing and interpreting quantitative interviews
* More subjective than quantitative interviews because the evaluator/researcher decides which quotes or specific examples to report
HOW TO USE QUALITATIVE INTERVIEWING
Types of Qualitative Interviews
Patton (1990) identifies three basic types of qualitative interviewing for research or evaluation: the informal conversational interview, the interview guide approach, and the standardized open-ended interview. Although these types vary in the format and structure of questioning, they have in common the fact that the participant's responses are open-ended and not restricted to choices provided by the interviewer. A fourth type of interview, the closed, fixed-response interview, falls in the realm of quantitative interviewing. In quantitative or structured interviews, the respondent is asked to choose from a predetermined set of response categories. Each type of qualitative interview has advantages and disadvantages. Once your evaluation team has decided to include qualitative interviews as part of your evaluation plan, it is still necessary to consider the strengths and weaknesses of each type in relation to your needs and the resources available for your evaluation:
1) Informal Conversational Interview: This type of interview may occur spontaneously in the course of field work, and the respondent may not know that an "interview" is taking place. Questions emerge from the immediate context, so the wording of questions and even the topics are not predetermined. The major advantage is that the interview is highly individualized and relevant to the individual. Thus, it is likely to produce information or insights that the interviewer could not have anticipated. This type of interview requires an interviewer who is very knowledgeable and experienced in the content area and strong in interpersonal skills, since he or she will have considerable discretion in directing the interview. However, since different information is collected from different people, this kind of interview is not systematic or comprehensive, and it can be very difficult and time-consuming to analyze the data.
2) Interview Guide Approach: This may be the most widely used format for qualitative interviewing. In this approach, the interviewer has an outline of topics or issues to be covered, but is free to vary the wording and order of the questions to some extent. The major advantage is that the data are somewhat more systematic and comprehensive than in the informal conversational interview, while the tone of the interview still remains fairly conversational and informal. Like the conversational interview, this type of interview also requires an interviewer who is relatively skilled and experienced, since he or she will need to know when to probe for more in-depth responses or guide the conversation to make sure that all topics on the outline are covered. A possible drawback is that sticking to the outlined topics will prevent other important topics from being raised by the respondent. Also, while this format is more systematic than the conversational interview, it is still difficult to compare or analyze data because different respondents are responding to somewhat different questions.
3) Standardized Open-Ended Interview: In this format, the interviewers adhere to a strict script, and there is no flexibility in the wording or order of questions. It is still considered a qualitative interview rather than a quantitative interview, because the responses are open-ended. This is the most structured and efficient of the qualitative interviewing techniques and is useful for reducing bias when several interviewers are involved, when interviewers are less experienced or knowledgeable, or when it is important to be able to compare the responses of different respondents. This may be the best choice for an evaluation if you must rely on volunteer or inexperienced interviewers or if you have limited time and money available for analyzing the data. The major drawback is that the interviewer has little flexibility to respond to the particular concerns of the individual, and there is no guarantee that the questions asked tap into the issues that are most relevant to this particular respondent.
Design and Development of Interview Studies for Evaluation
While much of the value of qualitative interviewing lies in its flexibility and openness, it remains extremely important for the evaluation planning team to think through the process and provide the basic structure and framework which will make the study useful and worthwhile. Kvale (1996) describes in detail seven stages in designing and implementing an interview study, which are summarized below. Readers are encouraged to consult the texts by Kvale and others listed below (see Annotated Bibliography).
1) Thematizing - Before even thinking about particular methods or interview formats, the evaluation team needs to be clear on the purpose of the study and the topic to be investigated. The questions of "why" and "what" need to be answered before the question of "how" can be answered. This is as important in a qualitative evaluation study as in a quantitative one.
2) Designing - The overall design for the study, including the later stages of analyzing and reporting, should be planned before the interviewing begins. For example, if there are no funds for transcribing or analyzing interviews, it may be wise to use a more structured format that will be easier to code later.
3) Interviewing - To an extent that is not true in many other methods, the interviewer is the instrument in this type of evaluation (Guba & Lincoln, 1981, as cited in Patton, 1987). The "instrument" can be affected by factors like fatigue, personality, and knowledge, as well as levels of skill, training, and experience. Patton (1987) points out that any face-to-face interview is also an observation. The skilled interviewer is sensitive to nonverbal messages, effects of the setting on the interview, and nuances of the relationship. While these subjective factors are sometimes considered threats to validity, they can also be strengths because the skilled interviewer can use flexibility and insight to ensure an in-depth, detailed understanding of the participant's experience.
4) Transcribing - This important step prepares the material from the interview for analysis. Both Kvale (1996) and Patton (1990) provide detailed practical suggestions for this process, ranging from ensuring that your tape recorder has good batteries to developing a sensitivity to the linguistic differences between oral speech and written text.
5) Analyzing - Data analysis is an issue that should be considered very early in the process of designing a study. Qualitative interviews and their transcripts produce a large volume of material which must be condensed, categorized or otherwise interpreted and made meaningful, and this may turn out to be one of the most costly and time-consuming aspects of the evaluation. If time and resources are limited, you may wish to use more standardized interview formats which are easier to code and interpret.
Methods for analyzing and interpreting qualitative interviews vary widely. Kvale (1996) describes five analysis methods that include 1) meaning condensation, 2) meaning categorization, 3) narrative structuring, 4) meaning interpretation, and 5) generating meaning through ad hoc methods. Patton (1987, 1990) also addresses a number of techniques for quantifiying and analyzing qualitative interview data.
The most appropriate method of analysis for any given study will depend on the purpose of your evaluation and the nature of the material, as well as the time and resources available for this part of the process. Some methods attempt to be more objective, while others depend more heavily on subjective judgements and insights of the researcher. Computer software programs are available that can assist in categorizing interview statements or counting key words, which may allow some forms of quantitative analysis.
6) Verifying - In traditional research terms, this means determining reliability (how consistent the findings are), validity (whether the study really investigates what you intended to investigate), and generalizability (whether the findings apply to anyone outside of this particular program). In qualititative studies, one important way of verifying findings or establishing validity is to actually take transcripts or analyzed results back to some of the interview participants, and ask them if this is really what they meant. Guba and Lincoln (1989) discuss the concepts of confirmability, dependability, credibility and transferability as alternative ways of ensuring quality of data in qualitative evaluations. For more in depth discussions of these important issues in qualitative research, readers are strongly encouraged to consult relevant chapters in Kvale (1996), Patton (1990), or Guba and Lincoln (1989).
7) Reporting - If the evaluation report is to effectively communicate findings, it must a) be in a form that meets some accepted scientific criteria, b) meet ethical standards such as confidentiality and respect, and c) be readable and usable for its intended audiences. In some cases, different reports may be needed for different audiences. An appropriate balance needs to be found between including endless quotations that will bore the reader and just quoting a few entertaining stories that happened to appeal to the researcher.
Ethical Issues in Qualitative Interviewing
The very personal, conversational nature of interview situations highlight many of the basic ethical issues of any research or evaluation method (Patton, 1990). Among these issues are:
1) Confidentiality - Because respondents may be sharing very personal information, it is important to honestly assess how much confidentiality you can promise. Some kinds of disclosures (such as child abuse or threats to the safety of self or others) must be reported, and respondents need to know this from the start. Also consider how the confidentiality of individuals will be preserved when the data are analyzed and reported. Related issues include who has access to the data and who "owns" it.
2) Informed consent - Most studies, including program evaluations, are covered by some kind of human subjects review process. This will usually require that respondents sign a permission form agreeing to participate, after being informed of potential risks and benefits. If children are involved, a parent or legal guardian must provide this permission.
3) Risk assessment - It is important to consider all potential risks and include them in the informed consent process. Even though "just talking" may seem inherently harmless, people who participate in open-ended interviews may experience psychological stress, legal or political repercussions, or ostracism by peers or staff who believe that the participant has said unflattering things about them to the interviewer.
4) Promises and reciprocity - The issue here is what interview participants get in return for sharing their time and insights with you. Will they or their communities benefit in some way from the results of the study? If promises are made (such as copies of reports or monetary payments), those promises should always be kept.
5) Interviewer mental health - Interviewing experiences can be intense interpersonal experiences. Just as participants may experience psychological stress from disclosing more than intended or being reminded of painful experiences, interviewers may be overwhelmed by the sensitive nature of what is seen or heard, especially in home- or field-based interviews. Some form of debriefing after the interview may be necessary. Interviewers should always know who to go to if they need advice or consultation on handling practical or emotional issues that arise from an interview.
Special Issues in Interviewing Children
The skills and logistics involved in interviewing and testing children are not necessarily the same as those required for adults. The performance of young children in testing situations is known to be quite sensitive to factors like the familiarity of the setting and the examiner (Hauser-Cram & Shonkoff, 1988). For ethnic minority children, language and cultural expectations about testing situations may be particularly relevant. Those who are conducting qualitative interviews with children should be trained in child development and familiar with the ways that children at different ages understand and use language. Young children in particular can be very literal and may misunderstand adult language or metaphors. They are also usually unable to respond to "Why?" questions, which require them to analyze or interpret their behavior or thoughts.
There are also special ethical issues involved in interviewing children. Informed consent can be a particularly complicated matter when children are involved. In virtually all cases, it is necessary to have the consent of a parent or legal guardian before interviewing a child for a research or evaluation study. While some schools or agencies ask parents to sign some form of blanket permission for their child to participate in assessments or studies, it should never be taken for granted that this is the case, or that it will cover your particular study. Depending on the age and developmental level of the child, ensuring that the child understands his or her right to refuse participation is also a concern.
ANNOTATED BIBLIOGRAPHY
Callor, S., Betts, S. C., Carter, R., & Marczak, M. (1997). State Strengthening Evaluation Guide. Tucson, AZ: USDA/CSREES & University of Arizona.
Program development and evaluation manual designed by the Evaluation Collaboration for use in State Strengthening Projects. Based on the five-tiered model of evaluation of Jacobs (1988). The manual may be viewed or downloaded from the World Wide Web at: http://ag.arizona.edu/fcr/fs/cyfar/evalgde.htm
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
Includes a good section on reliability and validity issues in qualitative evaluations. While traditional techniques for ensuring reliability, validity and generalizability may not apply to qualitative studies, Guba and Lincoln describe other quality standards that may serve similar purposes. These alternative standards include the concepts of confirmability, dependability, credibility, and transferability.
Guba, E. G. , & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches.. San Francisco: Jossey-Bass.
Guba and Lincoln's book describes a qualitative approach called "fourth generation evaluation". Although specific aspects of their method are somewhat controversial among program evaluators, they make a strong case for personal, individualized approaches to interviewing. They also emphasize the importance of verifying findings with respondents after data is collected, to ensure that the results match what they intended to say.
Hauser-Cram, P., & Shonkoff, J. P. (1988). Rethinking the assessment of child-focused outcomes. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs (73-94). New York: Aldine de Gruyter.
This chapter provides an overview of issues to be considered in evaluating community-based programs for children. Discussion of assessment of children includes general methodology issues and reviews some relevant measures.
Kvale, S. (1996). Inter Views: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.
Although the focus of this book is on the use of qualitative interviewing in research rather than on its use in program evaluation, it provides an in-depth treatment of many important theoretical and practical issues. A unique feature of the book is that it brings theory and techniques from the fields of linguistics and textual criticism to the planning, interpretation, and reporting of interview studies. Qualitative interview studies are placed in the philosophical contexts of hermeneutics, phenomenology, ethics, feminism, and post-modernism. A chapter on reporting of interview studies provides very practical advice on maximizing the readability and usefulness of final reports.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park, CA: Sage.
Patton's book provides a concise practical manual for using qualitative methods in evaluation, although only one chapter deals specifically with interviewing techniques. Could be a useful complement to the more theoretically-oriented treatment of the issues in Kvale's book (see above).
Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.
1 comment:
This blog gives very important info about SAS Thanks for sharing.Very well written information. Many thanks.
SAS training
SAS course
SAS Certification
SAS Certification course
Online SAS Training
Online Pharmacovigilance course
class room pharmacovilance training
Post a Comment