Which of the following is not a step in the data analysis process for mixed methods studies

Ann Fam Med. 2004 Jan; 2(1): 7–12.

Abstract

BACKGROUND Mixed methods or multimethod research holds potential for rigorous, methodologically sound investigations in primary care. The objective of this study was to use criteria from the literature to evaluate 5 mixed methods studies in primary care and to advance 3 models useful for designing such investigations.

METHODS We first identified criteria from the social and behavioral sciences to analyze mixed methods studies in primary care research. We then used the criteria to evaluate 5 mixed methods investigations published in primary care research journals.

RESULTS Of the 5 studies analyzed, 3 included a rationale for mixing based on the need to develop a quantitative instrument from qualitative data or to converge information to best understand the research topic. Quantitative data collection involved structured interviews, observational checklists, and chart audits that were analyzed using descriptive and inferential statistical procedures. Qualitative data consisted of semistructured interviews and field observations that were analyzed using coding to develop themes and categories. The studies showed diverse forms of priority: equal priority, qualitative priority, and quantitative priority. Data collection involved quantitative and qualitative data gathered both concurrently and sequentially. The integration of the quantitative and qualitative data in these studies occurred between data analysis from one phase and data collection from a subsequent phase, while analyzing the data, and when reporting the results.

DISCUSSION We recommend instrument-building, triangulation, and data transformation models for mixed methods designs as useful frameworks to add rigor to investigations in primary care. We also discuss the limitations of our study and the need for future research.

Keywords: Research design/methods, data collection, methods, investigative techniques, social sciences, qualitative research

INTRODUCTION

With an expanded use of qualitative research in health services investigations, mixed methods or multimethod research holds potential for rigorous, methodologically sound studies in primary care. Mixed methods investigations involve integrating quantitative and qualitative data collection and analysis in a single study or a program of inquiry.1 This form of research is more than simply collecting both quantitative and qualitative data; it indicates that data will be integrated, related, or mixed at some stage of the research process. The underlying logic of mixing is that neither quantitative nor qualitative methods are sufficient in themselves to capture the trends and details of the situation. When used in combination, both quantitative and qualitative data yield a more complete analysis, and they complement each other.

The concept of mixed methods research is not new to primary care. Almost 15 years ago, authors introduced the importance of integrating quantitative and qualitative research in a single study or program of inquiry.2, 3 More recently, investigators emphasized that quantitative and qualitative methods can be mixed, such as in collecting qualitative data before quantitative data where variables are unknown, or using qualitative methods to expand quantitative results to advance study aims.4, 5 This integration calls for collecting quantitative and qualitative data concurrently or in parallel or gathering information sequentially.6

Mixed methods inquiry as a new research paradigm has found a breeding ground in the North American Primary Care Research Group.7 Additionally, in 1999 a National Institutes of Health (NIH) task force in the Office of Behavioral and Social Sciences Research issued guidelines for conducting rigorous qualitative and multimethod investigations.8 In a brief section at the end of the NIH document, the task force noted the “broad appeal” of combining qualitative and quantitative methods in public health research, and it recommended that investigators be specific about how their methods will be combined and how the findings will be integrated.

Although the NIH document suggested several models for combining quantitative and qualitative data, it did not describe specific criteria that primary care researchers might use to design mixed methods investigations. Furthermore, it did not refer to published studies that illustrate mixed methods research or address specific models of mixed methods investigations. Thus the NIH document overlooked discussions about criteria, examples, and models widely accessible in recent social and behavioral science literature.9– 11 The purpose of the present study was to use criteria available in the literature to analyze 5 published mixed methods studies in primary care. Based on our analysis, we recommend 3 models as organizing frameworks for primary care investigators.

METHODS

We began by identifying criteria for designing a mixed methods study reported in the social and behavioral science literature. In the last decade several authors have determined rigorous, interrelated design criteria, such as identifying the reasons for mixing quantitative and qualitative data, the types of data collected and analyzed, the priority given to quantitative or qualitative research in a given study, the implementation sequence (concurrent or sequential), and the phase of research in which the integration or relationship between quantitative and qualitative data collection and analysis occurred.1, 10, 11 These 5 criteria became a coding template for analyzing select mixed methods in primary care studies.

To locate these studies, we conducted a MEDLINE search of the years 1990 to 2001 using the key word descriptors of “multimethod and primary care,” “qualitative and quantitative methods and family medicine/primary care,” and such MeSH terms as “research design” and “research methodology.” From among the 15 studies found, we selected 5 that met the criteria of our coding template.12– 16 They (1) used quantitative and qualitative data collection and analysis; (2) integrated, combined, or related both quantitative and qualitative data at some stage in the research process; (3) collected the data in a primary care setting; and (4) integrated or combined the data within a single study. We excluded for future research sustained programs of inquiry that consisted of multiple quantitative and qualitative investigations, such as the Family Practice Smoking Cessation Project17 or the Direct Observation of Primary Care Study.18

RESULTS

The 5 studies that met our inclusion criteria came from journals reporting primary care research. The content areas ranged from prevention programs and guidelines to communication practices and counseling skills. The analysis showed variation on the 5 criteria. Table 1 ▶ displays a summary of results showing the ways the 5 target articles addressed each of the selection criteria.

Table 1.

Published Mixed Methods Investigations in Primary Care

Rationale

We first examined the studies to locate a stated rationale or reason for mixing advanced by the investigators. More specifically, the authors provided statements indicating why it was important to collect and analyze both quantitative and qualitative data.8 A study became more rigorous when a rationale was present, because data could be seen as included, not because they were available, but because both types of data were important to the study aim. Indeed, in the NIH guidelines, such reasons for integration need to be clearly articulated.9 The rationale for mixing the 2 approaches is usually provided in the introduction to a study, in the study aim, or in an overview of the methods.

Only 3 of the 5 studies clearly articulated a rationale.12– 14 The study by Kutner et al mentioned that qualitative data were collected to identify questions relevant for developing and testing an instrument.13 The 2 other studies by Baskerville et al and McVea et al stated that gathering both forms of data contributed to a comprehensive and complete understanding of the results.12, 14 One study mentioned this rationale in the introduction,12 one in the methods,13 and one in the discussion section at the end of the article.14

Other rationales mentioned in mixed methods studies might state that qualitative data help explore statistical results from quantitative data, or that quantitative outlier or extreme results can be better understood through qualitative data collection. Quantitative results might also help researchers select qualitative cases so they can examine the results in greater depth.1

Data Collection and Analysis

The forms of data collection and analysis in the 5 studies showed data was collected by structured instruments and more open-ended field observations and interviews. Quantitative data collection relied on structured instruments administered by telephone or face-to-face interviews and structured checklists completed by observers. Chart audits also provided numeric data for these studies. The statistical treatment of the data reflected descriptive analysis and some limited inferential analyses (eg, correlation, regression). Qualitative data were collected through open-ended or semistructured interviews and field observations of practices. The qualitative data were coded using predetermined templates and grouped into codes, themes, and categories. These results suggest typical procedures for both quantitative and qualitative data collection and analysis.19, 20

Other types of qualitative data collection procedures, such as videotapes, photographs, or e-mails, did not surface in these projects.20 Aside from case studies,14 the application of other distinct traditions of qualitative inquiry, such as ethnographies, grounded theory studies, or biographies, also were not present.19 Advanced statistical procedures were largely absent.

Priority

Priority is determined by the researchers, who place an emphasis on quantitative data, qualitative data, or an equal priority shared between the 2 forms of data.1, 10 Such priority is detected at the beginning of the study by noting the relative emphasis given to framing the research problem (eg, intent to test a theory, study variables, or explore constructs) or the subservient use of 1 form of data to the other (eg, qualitative data helps to build an instrument). Alternatively, in some studies, investigators might give equal emphasis and status by providing both detailed quantitative and qualitative data collection and data analysis.

Turning to the 5 studies, Kutner et al emphasized equally the quantitative and qualitative components in the study.13 McVea et al emphasized qualitative data collection and analysis,14 whereas the other 3 emphasized quantitative data collection and analysis.12, 15, 16 For example, the study by Nutting et al16 illustrates priority given to quantitative research. In this 2-phase study of barriers encountered by physicians and nurses to implementing guidelines for depression, the investigators focused on developing a structured checklist of 45 barriers to treatment. Their results highlight this checklist and the weighting of the factors; only a small initial qualitative interview phase to identify the barriers cast this study as a mixed methods investigation. For all 5 of the studies, a quantitative rather than a qualitative orientation was given priority.

Implementation

Implementation refers to whether the quantitative and qualitative data are collected in sequential phases or gathered concurrently at roughly the same time during the study.10 In a sequential approach, quantitative or qualitative data collection serves as a basis for the next data collection and analysis stage. This approach is ideal when one phase can contribute to the next phase and enhance the entire study. In the concurrent approach, quantitative and qualitative data are collected at the same time and are brought together in the results or interpretation of the results. Given time limitations for primary care research, concurrent approaches that enable the collection of multiple forms of data at one time might be more practical than a sequential approach. Other important considerations include the types of research questions, sampling goals, and the unit of analysis.

A mixed picture of implementation emerged from the 5 studies analyzed. Two of the studies reported concurrent approaches12, 14 and 3 reported sequential approaches.13, 15, 16 McVea et al used a concurrent approach in which they emphasized the qualitative data collection and analysis, but they collected quantitative encounter data and chart audits at the same time they were in the field gathering qualitative field observations and office interviews.14 Alternatively, Kutner et al conducted a 2-phase sequential investigation in which the initial qualitative phase of data collection and analysis was followed by a second quantitative phase.13 Specifically, they began the study by collecting qualitative interview data (in the first phase) to explore patients information needs, then analyzed the interview data and used the information to build a semistructured instrument that was administered to a sample (in the second phase).

Integration

Priority and implementation decisions in design lead to the logical place in research methods (data collection, analysis, interpretation, or discussion19) where the investigators bring together the quantitative and qualitative data. Integration refers to the point in the process of research procedures at which the investigator mixes or integrates the quantitative and qualitative data collection and analysis.11 Based on the 5 studies, we identified 2 stages in the process where integration is possible: when data analysis leads to further data collection decisions,13, 15, 16 and when results are reported.12, 14

For example, in the study by Nutting et al, integration occurred during data analysis, when the qualitative interviews were used to develop a quantitative checklist.16 Integration thus fl owed from qualitative data analysis into quantitative data collection. Similarly, McIlvain et al transformed the qualitative data by assigning it quantitative codes, thus illustrating another approach to integration at the data analysis stage of research.15 Alternatively, McVea et al integrated the qualitative and quantitative data in their results to portray 3 different types of case office practices.14 The issue of integration—the stage of the research process at which the data are combined—illustrates the complexity of mixed methods research and the need to be explicit about the model of inquiry being used.

DISCUSSION

Criteria about a rationale, the types of data, and their prioritization, implementation, and integration represent rigorous elements that primary care investigators might use in designing their studies. Building on our analysis of the 5 studies reviewed, we advance 3 models using terminology available in the social and behavioral sciences.1 These models do not exhaust the possibilities, but their specification and labeling (Table 1 ▶) add to the rigor of mixed methods designs in primary care research.

Instrument Design Model

In the instrument design model, priority is given to quantitative data collection and analysis. Implementation is a 2-phase project that begins with qualitative data collection and analysis and moves to quantitative instrument design and testing. Integration occurs at the data analysis stage, when the researchers analyze the qualitative data and use this information to develop an instrument for data collection. The intent of this model is to develop an instrument that is grounded in the views of participants (eg, patients) rather than use an off-the-shelf instrument that might not reflect their views. With a sequential approach, the study is logical and easily conducted, but expertise is needed to code and analyze qualitative data, as well as develop a psychometrically sound instrument. The studies by Kutner et al13 and Nutting et al16 illustrate this model of mixed methods design.

Triangulation Design Model

The triangulation design model is frequently used in primary care research, although it is more difficult to implement than the sequential instrument design model because of the need to reconcile and bring together numeric (quantitative) and text (qualitative) data. The intent of this model is to triangulate or gather both quantitative and qualitative data at the same time, and to integrate the two forms of data to best understand a research problem.11 This model typically gives equal priority to quantitative and qualitative data and analysis (often found in separate sections of the report), involves concurrent or simultaneous collection of data, and integrates both quantitative and qualitative data in the results, interpretation, or conclusion phase.

A typical structure for a triangulation study is to have separate sections on quantitative data collection and qualitative data collection, as well as separate sections on quantitative data analysis and qualitative data analysis. The investigators then provide a results, discussion, or conclusion section in which they discuss the results of both analyses. Typically investigators present the 2 forms of results as supporting or conflicting evidence for results, or they might transform one type of data into another form (quantitatively count the codes from qualitative results) to converge results.1 The studies by Baskerville et al12 and the McVea et al14 illustrate the triangulation model in primary care.

Data Transformation Design Model

The data transformation design model is well suited for the primary care investigator because it encompasses correlational (observational) designs, such as prevalence studies, retrospective studies, or prospective studies.21 This model allows the researcher to gather qualitative data, analyze it for codes and themes according to a predetermined codebook or conceptual framework, and (typically) numerically count the codes and themes. The priority favors quantitative data collection and analysis, the implementation is concurrent, and the integration occurs at the data analysis stage of the research process. The study by McIlvain et al15 illustrates this model.

These 3 models illustrate some of the possibilities for mixed methods models in primary care. Others not illustrated, but discussed in the social and behavioral science literature, include the explanatory model and the nested model. In the explanatory model, an initial quantitative phase is conducted to obtain statistical results.1 In a second phase, the investigator gathers qualitative data (eg, open-ended interviews) to help explain the quantitative results. In the nested model, a smaller qualitative data collection phase is embedded within a larger quantitative intervention trial.11 Although this model often fits the criteria for a concurrent and quantitative-priority design, it represents a variation in which the larger component addresses one question (eg, was the intervention successful?) and the smaller component another question (eg, how did patients react from a cultural perspective during the trial?).

CONCLUSIONS

The use of a mixed methods model, with a design based on the rigorous criteria, assumes that investigators possess the resources and the expertise to conduct this form of inquiry. The expertise of the study team conducting the investigation in primary care is an important factor in determining an appropriate type of design. This point is highlighted in the NIH report.8 Unquestionably, mixed methods research is labor-intensive in that it involves multiple stages of data collection and analysis. The 5 studies we analyzed here had multiple authors, external funding support, and study teams with expertise in quantitative and qualitative approaches as well as knowledge about current mixed methods models being discussed in the social and behavioral sciences.22

We recognize several limitations in our discussion of the 5 studies. Our illustrative studies fit the inclusion criteria for selection but limit the number of studies actually discussed and the generalizability of our findings. Furthermore, although the authors from the 5 target articles clearly intended to gather and analyze both quantitative and qualitative data in each study, they might not have intended (or considered) combining or integrating the data, which is a centerpiece of mixed methods designs in the social sciences.1, 20 Our coding template for analysis, drawn from the social sciences, might need to be modified to better address the nuances of mixed methods investigations in primary care. This template, however, seems to add rigor to design decisions that are needed in primary care and advocated by NIH.

The future analysis of primary care mixed methods investigations might focus on models addressed in the literature but not discussed here. The taxonomy should be examined for appropriateness in other health disciplines, such as nursing, allied health, and critical care. Additional studies could help researchers not only develop a better understanding of mixed methods approaches that are used in sustained programs of inquiry consisting of multiple studies but also refine further the criteria for evaluating the quality of primary care mixed methods studies. The discussion initiated here holds promise for designing rigorous proposals for funding and clarifies the complex designs inherent in this form of inquiry.

Notes

Conflict of interest: none reported

Funding support: Dr. Fetter’s involvement in this work was made possible in part by the generous support of the Robert Wood Johnson Generalist Faculty Scholars Program.

REFERENCES

1. Creswell JW, Plano Clark VL, Guttman M, Hanson W. Advanced mixed methods research designs. In: Tashakkori A, Teddlie C, eds. Handbook on Mixed Methods in the Behavioral and Social Sciences. Thousand Oaks, Calif: Sage Publications; 2003:209–240.

2. Blake RL. Integrating quantitative and qualitative methods in family research. Fam Syst Med. 1989;7:411–427. [Google Scholar]

3. Stange KC, Zyzanski SJ. Integrating qualitative and quantitative research methods. Fam Med. 1989;21:448–451. [PubMed] [Google Scholar]

4. Medalie JH, Zyzanski SJ. Problems and issues in family medicine psychological research. Fam Pract. 1992;9:222–230. [PubMed] [Google Scholar]

5. Goering PN, Streiner DL. Reconcilable differences: the marriage of qualitative and quantitative methods. Can J Psychiatry. 1996;41:491–497. [PubMed] [Google Scholar]

6. Stange KC, Miller WL, Crabtree BF, O’Connor PJ, Zyzanski SJ. Multi-method research: approaches for integrating qualitative and quantitative methods. J Gen Intern Med. 1994;9:278–282. [PubMed] [Google Scholar]

7. Stange KC, Miller WL, McWhinney I. Developing the knowledge base of family practice. Fam Med. 2001;33:286–297. [PubMed] [Google Scholar]

8. National Institute of Health. Office of Behavioral and Social Science Research. Qualitative Methods in Health Research. Opportunities and Considerations in Application and Review. Washington, DC: National Institute of Health; 1999.

9. Creswell J. Mixed-method research: introduction and application. In: Cizek GJ, ed. Handbook of Educational Policy. San Diego, Calif: Academic Press; 1999:455–472.

10. Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res. 1998;8:362–376. [PubMed] [Google Scholar]

11. Tashakkori A, Teddlie C. Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, Calif: Sage Publications; 1998.

12. Baskerville NB, Hogg W, Lemelin J. Process evaluation of a tailored multifaceted approach to changing family physician practice patterns improving preventive care. Fam Pract. 2001;50:W242–W249. [PubMed] [Google Scholar]

13. Kutner JS, Steiner JF, Corbett KK, Jahnigen DW, Barton PL. Information needs in terminal illness. Soc Sci Med. 1999;48:1341–1352. [PubMed] [Google Scholar]

14. McVea K, Crabtree BF, Medder JD, et al. An ounce of prevention? Evaluation of ‘Put Prevention into Practice’ program. J Fam Pract. 1996;43:361–369. [PubMed] [Google Scholar]

15. McIlvain HE, Backer EL, Crabtree BF, Lacy N. Physician attitudes and the use of office-based activities for tobacco control. Fam Med. 2002;34:114–119. [PubMed] [Google Scholar]

16. Nutting PA, Rost K, Dickinson M, et al. Barriers to initiating depression treatment in primary care practice. J Gen Intern Med. 2002;17:103–111. [PMC free article] [PubMed] [Google Scholar]

17. Willms DG. A new stage, a new life: individual success in quitting smoking. Soc Sci Med. 1991;33:1365–1371. [PubMed] [Google Scholar]

18. Stange KC, Zyzanski SJ, Jaen CR, et al. Illuminating the ‘black box’. A description of 4454 patient visits to 138 family physicians. J Fam Pract. 1998;46:377–389. [PubMed] [Google Scholar]

19. Creswell JW. Research Design: Quantitative and Qualitative Approaches. 2nd ed. Thousand Oaks, Calif: Sage Publications; 2003.

20. Creswell JW. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research. Upper Saddle River, NJ: Merrill-Pearson Education; 2002.

21. Marvel MK, Staehling S, Hendricks B. A taxonomy of clinical research methods: comparison of family practice and general medical journals. Fam Med. 1991;23:202–207. [PubMed] [Google Scholar]

22. Tashakkori A, Teddlie C, eds. Handbook on Mixed Methods in the Behavioral and Social Sciences. Thousand Oaks, Calif: Sage Publications; 2003.

Articles from Annals of Family Medicine are provided here courtesy of Annals of Family Medicine, Inc.