Policy actors’ perceptions of qualitative research in policymaking: the case of higher education rulemaking in the United States

View author details View Less
  • 1 Hofstra University, , USA
Full Access
Get eTOC alerts
Rights and permissions Cite this article

Background:

As calls for evidence-based policymaking become increasingly common, qualitative research has much to offer the policy community. However, policymakers frequently evidence a preference for quantitative research. By discounting the importance of qualitative research in the policymaking process, resulting policies and their target populations miss out on the benefits that qualitative research uniquely offers.

Aims and objectives:

The purpose of this study was to examine how qualitative research has been perceived and used in the US government’s rulemaking process for creating higher education regulations.

Methods:

This qualitative case study included data from semi-structured interviews with 34 policy actors involved in higher education rulemaking, rulemaking documents, and research reports cited in several key higher education regulations.

Findings:

Many policy actors viewed qualitative research favourably, but qualitative studies have seldom been cited in higher education rulemaking. Several respondents discussed validity concerns and some policymakers’ misunderstandings regarding qualitative methods. Moreover, storytelling can influence policy actors’ perspectives about the content of policies, and qualitative research was viewed as effective at identifying compelling stories. Thus, narratives derived from qualitative research may provide an opportunity for qualitative researchers to have their work considered in policymaking processes.

Discussion and conclusion:

Qualitative research faces challenges with gaining visibility and influence in the development of regulatory policy. However, this study has shown that qualitative research has the potential to be both useful and persuasive to policymakers. Studies that discuss relevant stories may be particularly compelling.

Abstract

Background:

As calls for evidence-based policymaking become increasingly common, qualitative research has much to offer the policy community. However, policymakers frequently evidence a preference for quantitative research. By discounting the importance of qualitative research in the policymaking process, resulting policies and their target populations miss out on the benefits that qualitative research uniquely offers.

Aims and objectives:

The purpose of this study was to examine how qualitative research has been perceived and used in the US government’s rulemaking process for creating higher education regulations.

Methods:

This qualitative case study included data from semi-structured interviews with 34 policy actors involved in higher education rulemaking, rulemaking documents, and research reports cited in several key higher education regulations.

Findings:

Many policy actors viewed qualitative research favourably, but qualitative studies have seldom been cited in higher education rulemaking. Several respondents discussed validity concerns and some policymakers’ misunderstandings regarding qualitative methods. Moreover, storytelling can influence policy actors’ perspectives about the content of policies, and qualitative research was viewed as effective at identifying compelling stories. Thus, narratives derived from qualitative research may provide an opportunity for qualitative researchers to have their work considered in policymaking processes.

Discussion and conclusion:

Qualitative research faces challenges with gaining visibility and influence in the development of regulatory policy. However, this study has shown that qualitative research has the potential to be both useful and persuasive to policymakers. Studies that discuss relevant stories may be particularly compelling.

As calls for evidence-based policymaking become increasingly common (Bowers and Testa, 2019), qualitative research has much to offer policymakers. Qualitative studies make visible the lived experiences of policy implementers, members of underrepresented groups, and people who benefit from policy interventions (Altheide and Johnson, 2011; Gibton, 2016; Clemens and Tierney, 2017; Locock and Boaz, 2019; McAleese and Kilty, 2019). Although quantitative studies can provide ‘what works’ analyses of the effectiveness of interventions (US Department of Education, 2002; Torrance, 2008; Denzin, 2009; Donmoyer, 2012a; Clemens and Tierney, 2017; Saiani, 2018), qualitative research supplies information about the processes through which those practices are developed, the reasons behind their effectiveness, and how such practices have been implemented (Herriott and Firestone, 1983; Altheide and Johnson, 2011; Werner, 2004; Donmoyer, 2012a; Bowers and Testa, 2019; Fielding, 2020).

Despite the valuable contributions that qualitative research provides, policymakers have frequently identified strong preferences for quantitative methods (Kerrigan and Johnson, 2019). Indeed, this preference for quantitative research – particularly randomised controlled trials (RCTs) – has been expressed by governments in the United States, United Kingdom, Australia, and other countries (Torrance, 2008; Saiani, 2018). This methodological preference transcends disciplines and substantive policy areas as well, with policymakers favouring quantitative research in such fields as education (Sallee and Flood, 2012; Saiani, 2018), public health (Nutley et al, 2007; Saiani, 2018), and labour economics (Kugler, 2014). When qualitative methods are sidelined in the policymaking process, resulting policies fail to reap the benefits that qualitative research uniquely provides (Purcell-Gates, 2000; Denzin et al, 2006).

Understanding how policy actors view qualitative research is vital for ensuring that ‘research-based’ policies have an evidentiary foundation that is comprehensive and complete. While previous studies have examined the limited role qualitative research has played in policymaking (for example, Donmoyer, 2012a; 2012b; Sallee and Flood, 2012; McAleese and Kilty, 2019), the study presented here took a deeper look into the issue with an analysis of qualitative research use in the development of regulatory policy for higher education in the US. The process of creating regulations, known as the rulemaking process (Kerwin and Furlong, 2011), governs matters such as accreditation of colleges and universities, institutional reporting requirements, and debt relief for student borrowers (Natow, 2017). This process provides a unique opportunity for research to be considered by policymakers because of the direct avenues for public involvement in the process (Natow, 2017), and the fact that policymakers are required to conduct assessments, which may involve research, during rulemaking (Belfield et al, 2018; Natow, 2020). This study extends upon prior literature by examining qualitative research use in this process. Understanding the perceptions that members of the policy community hold about qualitative research can inform strategies for enhancing the visibility and usability of policy-relevant qualitative research in policymaking for education as well as other fields.

Qualitative research and policymaking

Numerous studies of research utilisation have revealed how research has (and has not) been used in policymaking (for example, Weiss, 1979; Weiss et al, 2005; Nutley et al, 2007; National Research Council, 2012; Oliver et al, 2014). Carol Weiss and others observed that research use tends to cluster into certain categories, including instrumental use (in which research is identified in direct response to a policy problem) and political use (in which research is cited to advocate for a policy the user previously supported) (for example, Weiss, 1979; Weiss et al, 2005; Nutley et al, 2007). Some studies have identified hindrances to research use in policymaking (for example, Natow, 2020; Oliver et al, 2014), and others have identified ways that policymakers can build capacity to enhance the utilisation of research (for example, Haynes et al, 2018). Additionally, some studies have shown the relatively infrequent use of research in educational policymaking (for example, Asen et al, 2011), including in higher education rulemaking (Natow, 2020).

A smaller body of previous literature has examined the limited extent to which qualitative research has been used in policymaking. Observers have long acknowledged the apparent mismatch between qualitative research and policymakers’ preferences. Qualitative research is iterative in that data collection and analysis inform each other and often involve multiple cycles, which can be time-consuming (Srivastava and Hopwood, 2009; Sallee and Flood, 2012; Maxwell, 2013). Findings from qualitative studies often reveal the ‘complexity’ (Donmoyer, 2012a: 663; Donmoyer, 2012b: 799) and ‘multi-layered’ contexts of the social world (Gibton, 2016: 7). However, policymaking tends to move quickly, and policymakers often seek prompt, simplified answers to questions about the impacts of programmes and practices (Nutley et al, 2007; Sallee and Flood, 2012; Donmoyer, 2012b). When policymakers are unfamiliar with qualitative methods, they may hold flawed assumptions about the quality of qualitative research. For example, qualitative research has been discounted as simply ‘anecdotal’ evidence (McAleese and Kilty, 2019: 822–823), and some have questioned the rigour of qualitative studies (Saiani, 2018). Misconceptions about qualitative methods may lead policy actors not to trust the research findings.

Quantitative research has fared better with policymakers (Lather, 2004; Torrance, 2008; Sallee and Flood, 2012; Donmoyer, 2012a; Clemens and Tierney, 2017). The policy community tends to operate according to the positivist viewpoint, shared by many quantitative methodologists, that some objective reality exists and can be known, even if only approximately, through the use of appropriate research methods (Denzin et al, 2006; Torrance, 2008; Denzin, 2009; Gibton, 2016; McAleese and Kilty, 2019; Fielding, 2020). Such methods include large-scale studies that allow for ‘repeated observations of a relationship between two variables or events’ (Maxwell, 2004: 5). Policymakers’ preferences for positivist research have been codified in recent decades alongside increased managerialism and neoliberal practices in government and education (Lather, 2004; 2006). As Lather (2004: 20) notes, positivism supports ‘neoliberal ideologies of the neutrality via proceduralism of’ positivist evaluation techniques. Qualitative studies, meanwhile, are often informed by constructivist, critical, or emancipatory frameworks that conceive of reality as socially constructed and challenge positivist notions of there being one objective and knowable reality (Denzin et al, 2006; Denzin, 2009; Lune and Berg, 2017).

Additionally, the typically larger sample sizes of quantitative studies can lead to a broader generalisation of findings, which appeals to policymakers (Sallee and Flood, 2012; Gibton, 2016; Clemens and Tierney, 2017). Unlike qualitative research, quantitative studies may be conducted relatively quickly, which is more in keeping with the fast-moving pace of the policy world than the longer timeframes needed for qualitative data to be collected and analysed (Sallee and Flood, 2012). The RCT – a form of quantitative research ‘that randomly assign[s] individuals to an intervention group or to a control group, in order to measure the effects of the intervention’ (US Department of Education, 2003: 1) – has been famously characterised as the ‘gold standard’ of evaluation techniques (US Department of Education, 2003: 1, 4; Asen et al, 2011: 202; Clemens and Tierney, 2017, 31; Saiani, 2018: 87). Case studies, focus groups, and similar qualitative methods tend to fall lower on hierarchies of evidence (Evans, 2003; Maxwell, 2004; Asen et al, 2011).

Despite policymakers’ preferences for quantitative studies, qualitative research has the ability to inform policy decisions in valuable ways. Qualitative methods are useful for discovering the perceptions and experiences of individuals who are the targets of policy interventions, and such data can help to inform policy construction and evaluate policy impacts (Locock and Boaz, 2019). The inductive and exploratory nature of qualitative research may ‘reveal unknown or unexpected things about the social world’ (Fielding, 2020: 2), and serve to ‘unearth the unexpected’ (Gibton, 2016: 38). Also, qualitative research is more sensitive to context than is quantitative research (Herriott and Firestone, 1983; Maxwell, 2004; Altheide and Johnson, 2011; Sallee and Flood, 2012; Donmoyer, 2012b; Clemens and Tierney, 2017). Sensitivity to context allows qualitative research to provide information about how social interventions can be tailored to individual situations (Sallee and Flood, 2012; Locock and Boaz, 2019). Similarly, qualitative action research, which involves stakeholders in the research process, can result in findings and recommendations for policy and practice reforms that have been shaped by real-life contexts (Kaplan et al, 2019; Locock and Boaz, 2019).

The flexibility inherent in qualitative research designs (Maxwell, 2013) is also important for studies of policy-relevant issues. As policy contexts evolve while researchers are in the field, a flexible study design allows for an investigation of how phenomena respond to changing contexts (Herriott and Firestone, 1983; Altheide and Johnson, 2011; Sallee and Flood, 2012). Complex political and social contexts are well-served by a research approach that helps to identify and describe their complexity (Donmoyer, 2012a; 2012b; Gibton, 2016; McAleese and Kilty, 2019). Qualitative methods are also useful in the development of theory (Corbin and Strauss, 2008; Creswell and Poth, 2018), and well-developed theories about matters of policy can be useful to policymakers (Bowers and Testa, 2019).

Qualitative studies may be particularly influential when they produce detailed narratives or stories that catch policymakers’ attention and evoke empathy (Sallee and Flood, 2012; Donmoyer, 2012b; Posti-Ahokas, 2013; McAleese and Kilty, 2019; Fielding, 2020). Qualitative methods specifically designed to identify narratives used in policymaking – such as qualitative versions of the narrative policy framework analysis – are useful for identifying stories that have been the subject of policy debates (Gray and Jones, 2016). Moreover, Donmoyer (2012b) has written about how telling stories to policymakers can make a difference in policy content. As a novice researcher, Donmoyer found a letter to a lawmaker among documents he was analysing as part of a research project about an education policy on secondary school exit exams. Donmoyer (2012b: 804–05) wrote that the ‘letter from a parent… told the story of the parent’s special-needs son’, and that the policymaker acknowledged ‘that this story was the only thing he needed to modify his legislation to allow school districts to exempt special-needs students from the’ legislation’s requirements. Qualitative research that makes use of stories to illustrate points may also resonate because policymakers are familiar with the practice of telling stories to argue one’s point (McAleese and Kilty, 2019). The study presented here extends upon this literature by providing empirical evidence of the perceptions about qualitative research held by participants in an educational policymaking process based on policy actors’ own statements, including how persuasive they have found qualitative findings and some misunderstandings about qualitative research. This study also examined the types of qualitative research that have been used in this process for the purpose of better understanding what aspects of qualitative research policymakers find useful.

Opportunities for research use in higher education rulemaking

The higher education rulemaking process allows a wide range of policy actors – government officials as well as other stakeholders – to participate in the policymaking process, often in direct ways. The process includes a series of separate stages. First, the US Department of Education decides to create a rule pursuant to the Higher Education Act. When the rulemaking will affect federal student financial aid programming, the department is required to hold meetings at locations across the US to gain insight from stakeholders about the topics to be regulated. Also for rulemaking involving student financial aid, the department convenes negotiated rulemaking panels, in which a range of higher education stakeholders are selected to meet with Department of Education personnel to develop the wording of a proposed rule (Pelesh, 1994; Natow, 2017). Once a proposed rule is issued, the department receives comments from stakeholders about the prospective rule. The department then decides whether it will make any changes to regulatory language in light of those comments and ultimately publishes the final regulation in the Federal Register (Kerwin and Furlong, 2011; Office of the Federal Register, 2011; Natow, 2017). For regulations that are ‘significant’ in that they are expected to influence the economy by a large amount or are otherwise making major changes to regulatory policy, the US Office of Management and Budget (OMB) reviews the proposed and final rules; during this process, stakeholders may give relevant information to the OMB (Office of the Federal Register, 2011; Sunstein, 2013; Natow, 2017).

There are several opportunities for research to be used in higher education rulemaking. US government officials must conduct regulatory impact assessments of proposed and final rules that are expected to have large economic and other impacts (Executive Order 12866, 1993). These assessments, which can be informed by research, include analyses of projected costs and benefits as well as other possible policy effects (Belfield et al, 2018). Also, because the rulemaking process itself includes several stages that involve the interaction of government policymakers and nongovernmental stakeholders (namely regional meetings, negotiated rulemaking, OMB meetings, and the public comment phase) (Natow, 2017), there are multiple opportunities for policy actors both inside and outside the US government to cite and rely upon research in higher education rulemaking. Indeed, it behooves regulatory policymakers to rely on research to the extent they can when creating rules, as reliance on research or other forms of evidence may strengthen the argument that the rule has a ‘reasoned basis,’ which is required by law (Association of Private Sector Colleges and Universities v. Duncan, 2012: 1, 30; see also Truong, 2000). Thus, policy actors have opportunities and incentives to use research in the higher education rulemaking process. However, normative perspectives about the validity of research methods can influence what types of research policymakers will use (Lee et al, 1997). The extent to which policy actors perceive qualitative research as valid and useful will help to determine whether such research gets used in the rulemaking process.

Methods

This analysis, which was part of a larger case study of how research is used in the US higher education rulemaking process, sought to understand the perceptions that participants in higher education rulemaking had regarding qualitative research, including whether they view qualitative research as valid and the extent to which qualitative studies have been useful to policymakers in the rulemaking process. Specifically, this study examined the following research questions: How, if at all, has qualitative research been used in the higher education rulemaking process? How do participants in this process perceive qualitative research, including its strengths and weaknesses? Because participants in higher education rulemaking include a wide array of policy actors who are often active in other aspects of the policy community as well (Natow, 2017), understanding how rulemaking participants view qualitative research can shed light on how such research is viewed by the education policy community more broadly.

Case-study rules were purposely selected (Creswell and Poth, 2018; Maxwell, 2013), using a sampling approach that identified several key rulemakings: the 2014 Gainful Employment rule that imposed disclosure and performance requirements for career-focused higher education programmes (Program Integrity, 2014); the 2019 rule that rescinded that Gainful Employment rule (Program Integrity, 2019); the 2016 Borrower Defense rule that provided avenues for loan discharge and consumer protections for certain student loan borrowers (Student Assistance General Provisions, 2016); a rule that imposed disclosure and performance requirements for teacher preparation programmes (Teacher Preparation Issues, 2016); and a regulation of an income-based debt repayment programme (Student Assistance General Provisions, 2015). These five case-study rules were selected for two key reasons: (1) they all underwent the typical negotiated rulemaking and public notice process that the US Department of Education uses to create regulations governing federal student aid programmes; and (2) the topics and prominence of these rules varied, with some being more controversial than others, to provide a broad mix of content and contexts for the case study (see also Natow, 2020).

Data collection involved gathering interviews and rulemaking documents. These data sources, commonly used in qualitative case studies, allow for comprehensive understanding of a case and respondents’ perceptions about and experiences with it (Merriam and Tisdell, 2016; Lune and Berg, 2017). In-depth interviews were conducted with 34 higher education policy actors who have had experience with one or more of the case-study rules, either by participating directly in at least one phase of the rulemaking process, or by assisting participants in rulemaking, or by monitoring rulemaking activity as part of their professional work (see also Natow, 2020). Interview participants were purposely sampled to obtain a variety of different roles and backgrounds. Following this strategy, 60 potential interviewees who represented a variety of positions in the higher education policy community both within and outside the US government, based on their current or previous positions, were emailed invitations to participate in this study. Thirty-four individuals provided interviews.1 Table 1 indicates the number of interviewees by role group. Each category of actors represented in the sample played important roles in the rulemaking process: Department of Education and other US government officials developed regulatory policy, and representatives of stakeholder groups (including different types of higher education institutions) participated in negotiated rulemaking, held meetings with the OMB, and/or regularly monitored rulemaking activity. Including participants from a variety of different backgrounds enabled the identification of ‘variable features of a phenomenon as experienced by diverse stakeholders among varied contexts’ (Suri, 2011: 67). The semi-structured interview protocol included questions about what types of research have been used and what the interviewees perceived about the value and proper role of qualitative research in the higher education rulemaking process. Several interviewees were involved with and able to discuss how research was used in the creation of other higher education rules as well, such as the 2019 rule that substantially modified the 2016 Borrower Defense rule (Student Assistance General Provisions, 2019) and a regulation of federal parent ‘PLUS’ loans (William D. Ford Federal Direct Loan Program, 2014). A strength of qualitative research is that the flexible study design allows for the analysis of unanticipated but relevant data such as statements about these additional rulemakings (Sinkovics and Alfoldi, 2012). Statements regarding the additional rules were infrequent but relevant and therefore included in the analysis.

Table 1:

Interviewees’ roles in higher education rulemaking

Interviewee role groupNumber of respondents*
US Department of Education7
Other US government staff5
Higher education institutions/associations
Public four-years3
Public two-years3
Private non-profits6
Private for-profits2
Financial aid administrators: institution/association5
Student lending industry representatives3
Accrediting agency or accreditor association2
Student, youth, veteran, legal aid/consumer advocates6
Think tank4
State government employee/association2

* The total number of interviewees was 34. The table reflects roles held by the respondents whether at the time of their interview or previously, as long as the position had relevance to the higher education rulemaking process.

Documents studied in this analysis included records that reflected matters that were discussed and considered in the development of the case-study rules. Documentary data were used to identify potential interviewees and to address the first research question about how qualitative research has been used in the process. Documentary data included all final case-study rules and research reports cited in the rules for a total of five rules and nearly 200 reports.

Data analysis involved multiple stages of reading, rereading, and coding interview and documentary data (Saldaña, 2016). The first stage was to read through all transcripts for accuracy and then review them again in order to code the data (Savin-Baden and Major, 2013). All interview transcripts were coded using qualitative data analysis software with a coding structure that was created specifically for this study, based on a combination of concepts that appeared in prior literature as well as ‘emergent’ patterns and themes that were identified in this study’s data (Ritchie and Spencer, 1994: 178, 180). Documentary data were also reviewed and coded. Available copies of research reports referenced in final regulations were analysed to identify their sources and methodologies. The second stage of data analysis involved exporting excerpts of coded data into an analytic matrix for a closer analysis of similarly coded data (Miles et al, 2019). This stage of analysis resulted in the identification of patterns and general themes regarding policy actors’ perceptions of the use and value of qualitative research in higher education rulemaking (Gibbs, 2018).

Findings

Qualitative research is seldom used in higher education rulemaking

Qualitative studies have been cited only infrequently in higher education regulations. Final rules that were the focus of this study often contained numerous references to quantitative economic research. More than half the references to academic journal articles cited in the Teacher Preparation rule were published in economics journals. Additionally, four working papers from the National Bureau of Economic Research were cited in that regulation. The rule also referenced RCT studies and mixed-methods research. However, the final Teacher Preparation rule contained no citations to purely qualitative empirical research (Teacher Preparation Issues, 2016). Similarly, the 2016 Borrower Defense rule cited some legal and textual analyses, but most of the research cited involved econometric, statistical, and survey research. The 2014 Gainful Employment rule cited several quantitative studies, a literature review that made use of qualitative data analysis software (Gross et al, 2009), and mixed-methods studies such as one published by the Government Accountability Office that included interviews, document analysis, and a quantitative analysis of licensure exam pass rates (Scott, 2011). But that final rule did not cite any purely qualitative empirical studies (Program Integrity, 2014). This pattern was consistent across the five regulations that were the focus of this case study, with economic and other quantitative research dominating, mixed methods studies sometimes cited, and few references made to purely qualitative research.

On the rare occasions that qualitative research, apart from a mixed methods study, was cited in final higher education rules, the studies typically involved textual analysis of laws and other policy documents (for example, Carter, 2009). One study cited in the Gainful Employment rescission involved focus group research (Bozeman et al, 2017). Statements from some interviewees indicate that these types of qualitative data may be particularly relevant to topics covered by higher education regulations. Analysing policy-relevant documents can help rulemaking participants understand the strengths and weaknesses of existing policies and practices. For example, one participant in the Borrower Defense negotiated rulemaking said:

We had done research on what we knew from publicly available documents about how the current letters of credit were just insufficient to cover any sort of losses for borrowers if the school closed.

Although cited less frequently, focus groups can also provide information useful to rulemaking participants by presenting first-hand accounts from a group of affected stakeholders. For example, a negotiated rulemaking participant identified focus groups as a form of research that would enable policymakers to consider ‘some profiles of individual borrowers and what they go through over a period of many years’.

Yet the consultation of qualitative research was more an exception than the norm: respondents confirmed that pure qualitative research is seldom discussed in the process of developing higher education regulations. A reason for the relative scarcity of qualitative research in higher education rulemaking may be due to the nature and culture of higher education policymaking in the US. Some respondents pointed out that the subject matters of regulations – particularly large-scale student financial aid programmes and student outcomes accountability for institutions – lend themselves to econometric and other quantitative analyses. In the words of one nongovernmental negotiated rulemaking participant:

There’s certainly people at the department that love to just throw a bunch of numbers. They’re there because everything is so numbers-driven. And everything has to, especially in negotiated rulemaking… whatever you decide has to equate to their score, a budget number. So everything is so numbers-driven.

Similarly, a respondent who had worked for the US government reflected that there was ‘definitely a heavy emphasis on the economics discipline’ in higher education rulemaking. Additionally, three respondents – all of whom had worked for the US government either at the time of the interview or previously – echoed the phrase that RCTs are the ‘gold standard’ of educational research methods, a phrase that has been used commonly in the US education policy arena (for example, US Department of Education, 2003; Asen et al, 2011). For example, one US government official said, ‘I think everybody I work with knows that the RCT is going to be the gold standard’. These comments from individuals who have worked in the federal higher education policy space indicate that the culture of this arena and the topics of its regulatory policymaking lean decidedly in the direction of a preference for econometric and quantitative methods.

Qualitative research and the power of stories

Although qualitative studies were rarely cited in final higher education regulations, policy actors’ perceptions of the value and utility of qualitative research were often positive, with fourteen respondents speaking favourably about qualitative research and its use in the process. Respondents perceived qualitative studies as particularly impactful when they produced narratives about the experiences of students and families with college-going, incurring education-related debt, and other matters relevant to the regulations being created. In the words of one respondent who had worked for the federal government, ‘As human beings, we think in stories, and so I think having qualitative research can… help people think about whether there is a need to act, a need for particular regulation, how high-priority [it] should be’. Although respondents did not always make a distinction between a systematic qualitative study design and the recounting of one or more stories, more than a third of the respondents discussed storytelling in the higher education rulemaking process, including its persuasiveness and its fairly common use. A consumer advocate said:

If you can even show profiles of individual people, put a human face on this issue and make the data not seem so impersonal, I think that approach and that application of data could be data-informed without being so data-driven in your language of messaging. I think that’s the way that you start to move people who were kind of in the middle.

Some respondents said that stories presented to policymakers were persuasive. A former US official described a report that:

really laid out in some great detail the stories of some students who had been misled by an institution and wound up in debt and encountered a host of financial problems. I think there were stories like that, that were pretty powerful, that helped to motivate a lot of the regulatory action.

Likewise, a nongovernment respondent who served on a different negotiated rulemaking committee said:

The stories, the compelling stories, are the ones that can sway people. Because numbers are hard and cold and they say one thing, but when you have a strong message from particularly students saying, ‘This is how it impacted me’, that really does speak to the issue for a lot of folks, and you have to balance that somehow with the numbers.

Although some rulemaking participants conflated qualitative research with stories, others recognised the distinction between a true qualitative study and simply sharing anecdotal narratives. One nonfederal negotiated rulemaking participant reflected:

It would be, for me, super compelling if someone from an outside position had done some qualitative evaluation of these issues, and sort of identified the themes, but also tell the stories. The stories are important and compelling, but when you are only being confronted with stories, I think it’s very hard to not become a skeptic, right? To not say to yourself, well, what are the other stories?… I’m a total fan of qualitative research, and I think it could add a much more credible position.

Another nonfederal negotiated rulemaking participant relayed a similar perspective, stating that the negotiators heard ‘assertions based on an anecdote or common sense’ but not ‘grounded in research’. However, this negotiator went on to say:

If we could have sat as a group… to sit down a group of people that graduated from college five years ago and have a focus group about, how do you actually view the wage outcomes and other kinds of outcomes from your college experience? I think that would have been – I certainly would have taken it seriously.

These statements indicate that stories resonate with policymakers, but research that identifies stories may be particularly compelling due to the respect some policymakers have for research.

Concerns about validity

Concerns relating to the validity of qualitative research were also identified by more than a quarter of this study’s respondents as a reason that qualitative research is not used more frequently in higher education rulemaking. Validity refers to ‘the correctness or credibility of a’ study’s findings (Maxwell, 2013: 122), including issues relating to generalisability (the extent to which findings apply to other situations or contexts) and bias (the extent to which beliefs, positionality, or other subjective perceptions influence research methods or reporting) (Malterud, 2001; Maxwell, 2013; Creswell and Poth, 2018).

Some respondents viewed the limited generalisability of small-scale qualitative studies as problematic, indicating that rulemaking participants have been sceptical about studies or other evidence that presented findings based on a small sample. A former government official who had worked on education regulations said, ‘The challenge of qualitative research is how representative it is’. One respondent who had worked for the Department of Education said that even well-designed studies have been treated sceptically when they involved a small sample size. This respondent said, ‘A lot of times the more rigorous studies were less convincing to some people because they were very small numbers of people’.

Rulemaking participants’ perceptions of bias in qualitative research may also be a reason why it is not used more frequently in the rulemaking process. Specifically, respondents expressed that quantitative research may be granted more credibility in rulemaking because policy actors view statistics as inherently more objective than qualitative data. For example, a nongovernmental negotiator stated that when it comes to statistical data, policymakers might:

quibble with the data and what it shows and what the trend lines might say. But when it comes to the qualitative stuff, I think it would just sort of devolve into an argument over the reliability of the source.

Similarly, a different nongovernmental negotiator said that ‘there is a place for’ qualitative research in rulemaking; however:

how seriously it would be taken depends on who brought the data, where the data come from, is it a trusted source by all, and if you can find that balance. I think that’s why people go back to numbers sometimes, because if it’s numbers coming directly from the department in response to a direct question, it’s just data.

These quotations reflect a perception that quantitative data, being ‘numbers’, are ‘just data’, whereas participants may have concerns about bias in qualitative data, depending on its source.

Misunderstandings about qualitative research

About one-fifth of this study’s respondents made statements evidencing some fundamental misunderstandings about what qualitative research is and how it is conducted. When asked about whether and what kinds of qualitative research were presented or discussed in the higher education rulemaking process, some respondents identified personal testimony from stakeholders or the sharing of anecdotal evidence as examples of qualitative research. One negotiator from a public university said:

We do have a testimony from students or even from student advocacy groups or… we had some testimony on one, maybe two of the days, from a prior negotiator even…. So that’s one way in which qualitative evidence is projected.

Also, while some respondents distinguished between qualitative research and mere stories (as described above), others conflated storytelling or gathering stories with conducting qualitative research, even if there was no evidence of systematic data collection or analysis, the use of formal narrative inquiry, or other qualitative research methods. For example, one student advocate said:

At the Borrower Defense negotiated rulemaking, there were many negotiators who had compiled reams of stories. So these are stories from people who have been defrauded by their schools. The legal aid community also came with real stories from borrowers. There were stories shared at the public comment period at the end of each day, from actual borrowers. I would call that qualitative research.

Another respondent indicated that one of the reasons qualitative research was not used more frequently in the rulemaking process may be because participants in the process are not very knowledgeable about qualitative methods. This respondent – who had worked for the US government – said that qualitative research was not used ‘as formally as quantitative research’ perhaps because of ‘a lack of expertise in performing… strong, rigorous qualitative research’. This respondent went on to say, ‘Policymakers are… by and large, people who come from the field, rather than people who come from research backgrounds…. But we didn’t really have a rigorous qualitative research programme’.

Discussion and conclusion

More than a dozen of the 34 participants in this study spoke favourably about qualitative research and its utility in the higher education rulemaking process, particularly because of the power of stories. Storytelling influenced policy actors’ perspectives about the content of regulations, and qualitative research was viewed as effective at producing stories of individuals who have been affected by issues that were relevant to the rulemaking. However, qualitative research has seldom been cited in final higher education rules, and several respondents voiced concerns about the validity of qualitative research with regard to bias and a lack of generalisability. Some policy actors evidenced fundamental misunderstandings about qualitative research methods.

These findings demonstrate that qualitative research faces challenges with gaining influence in the development of regulatory policy affecting higher education. However, findings also indicate that qualitative research has the potential to be both useful and persuasive in higher education rulemaking. Qualitative researchers sometimes identify stories within their data and discuss those stories as illustrative examples of their findings. Such studies that identify relevant stories may be particularly compelling to policymakers. Importantly, some policy actors recognised that qualitative research is research and that mere anecdotes are not. This distinction is crucial: as many of this study’s respondents observed, storytelling can have power in policymaking processes. But qualitative research findings – if produced through systematic inquiries with sound study designs – can be even more convincing to policymakers who respect and are persuadable by research. Indeed, the mere identification of something as research or evidence often confers upon it respect that non-evidentiary products are less likely to enjoy (Denzin et al, 2006; Nutley et al, 2007; Denzin, 2009; Henig, 2012).

The production of stories can also help qualitative researchers to break through the neoliberal, positivist tendencies of policymaking that may serve as a barrier to the consideration of qualitative studies. This research found that some participants in the rulemaking process view numbers as unbiased, a perspective shared by neoliberal views of social research. Because these same participants are implementers of rulemaking, this perspective of numbers-based research contributes to the positivist tendencies of the process. But this study also found that policy actors are receptive to storytelling in policymaking, including in the form of qualitative research that makes use of relevant stories about real people to substantiate findings. As Delgado (1989) noted, storytelling can help to illuminate the socially constructed nature of reality and can empower marginalised individuals and communities. Thus, narratives derived from qualitative research may provide an opportunity for constructivist, critical, and emancipatory studies to be considered during policymaking.

This study contributes to the literature on evidence and policymaking in several respects. First, this study confirmed what others have found about the dominant position of quantitative research in policymaking, a cultural identification of RCTs as the ‘gold standard’ of evaluation approaches, a preference for generalisable research, and the persuasiveness of stories in policymaking (see also Asen et al, 2011; Sallee and Flood, 2012; Donmoyer, 2012b; Gibton, 2016; Clemens and Tierney, 2017; Saiani, 2018; McAleese and Kilty, 2019; Fielding, 2020). This study has also highlighted the prominent role that econometric research plays in the development of higher education regulations. This focus on economics is unsurprising given governments’ increased adoption of market-based, neoliberal policies for higher education in recent decades (for example, Dougherty and Natow, 2020).

Moreover, by identifying policy actors’ perceptions of qualitative research via their own firsthand statements, this study determined that a number of policy actors view qualitative research favourably and believe it can be valuable in policymaking. Document analysis, legal analysis, and focus groups may be particularly useful. However, this study also demonstrates that some policymakers misunderstand qualitative research. For example, some respondents appeared to conflate qualitative research with anecdotal evidence, and others expressed perceptions that qualitative research is more biased than quantitative research. But quantitative research is not inherently objective: numbers, scales, and even RCTs have all been found to present bias in some ways (for example, Torgerson and Torgerson, 2003; Rivera and Tilcsik, 2019). Findings from this study indicate that actors may be open to using qualitative research in policymaking and would benefit from learning more about the utility, strengths, and weaknesses of both qualitative and quantitative methodologies. This study has also shown that widely held norms and expectations in the policy arena – such as a common perspective that RCTs represent a research ‘gold standard’ – may serve as a barrier to more reliance on qualitative research in policymaking.

This study has implications for those who would like to see qualitative research become more influential in policymaking. In light of the power of storytelling in policy development, qualitative researchers who seek to influence policy should identify relevant and compelling stories in their data and take steps to communicate those stories in a format accessible to policymakers. For example, researchers who conduct case studies that make use of stories to illustrate key findings can write policy briefs that highlight those stories and share them with the policy community. Institutions that educate qualitative researchers can provide instruction on policymaking processes so that their graduates will be better equipped to identify points in the policymaking process – such as the points for public engagement in the rulemaking process discussed above – where research findings can be introduced (National Research Council, 2012). Qualitative researchers may also participate in conferences attended by policymakers. For example, the Postsecondary National Policy Institute (PNPI) in Washington, DC hosts ‘professional development for current and prospective policymakers who work on federal higher education issues’, including seminars that researchers also attend (Postsecondary National Policy Institute, n.d.). Researchers should attempt to educate policy communities about what qualitative research is, how it is conducted, and the ways it can be useful in policymaking. This may be done by providing workshops, webinars, or other training for policymakers. Moreover, some respondents identified concerns with validity as a reason why qualitative research is not used more frequently in rulemaking. To address these concerns, qualitative researchers should be as transparent as possible about their methodologies, and should fully report the details of their sampling, data collection, and analysis procedures (Ospina et al, 2018).

Finally, qualitative researchers should promote their work to policy audiences. This study has shown that many policy actors are likely to be receptive to qualitative research. As Sallee and Flood (2012: 142) wrote: ‘Qualitative researchers, in particular, need to become vocal proponents of the utility of their research’. Enhancing the role of qualitative research in policymaking begins with qualitative researchers themselves, by promoting and advocating for their own work.

Note

1

Because this study targeted policymakers and other ‘elite’ respondents, this response rate and total number of participants were respectable figures (Lovell et al, 2018: 230).

Funding

This research received financial support from the Hofstra College of Liberal Arts and Sciences Faculty Research and Development Grant.

Acknowledgements

The author is grateful to Dr Hana Lahr for providing valuable feedback on a prior version of this manuscript.

Research ethics statement

This study received approval from the Hofstra University Institutional Review Board on February 12, 2018.

Contributor statement

RSN was the sole author and researcher for this manuscript.

Conflict of interest

The author declares that there is no conflict of interest.

References

  • Altheide, D.L. and Johnson, J.M. (2011) Reflections on interpretive adequacy in qualitative research, SAGE Handbook of Qualitative Research, vol 4, Los Angeles, CA: SAGE, pp 58194.

    • Search Google Scholar
    • Export Citation
  • Asen, R., Gurke, D., Solomon, R., Conners, P. and Gumm, E. (2011) ‘The research says’: definitions and uses of a key policy term in federal law and local school board deliberations, Argumentation and Advocacy, 47(4): 195213.

    • Search Google Scholar
    • Export Citation
  • Association of Private Sector Colleges and Universities v. Duncan (2012) Memorandum Opinion, Civil Action 11–1314 (RC), D.D.C.

  • Belfield, C.R., Bowden, A.B. and Rodriguez, V. (2018) Evaluating regulatory impact assessments in education policy, American Journal of Evaluation, 40(3): 33553.

    • Search Google Scholar
    • Export Citation
  • Bowers, J. and Testa, P.F. (2019) Better government, better science: the promise of and challenges facing the evidence-informed policy movement, Annual Review of Political Science, 22(1): 52142.

    • Search Google Scholar
    • Export Citation
  • Bozeman, H., Mingo, M. and Hershey-Arista, M. (2017) Summary report for the 2017 gainful employment focus groups, report for the US department of education, Rockville, MD: Westat, https://www2.ed.gov/about/offices/list/ope/summaryrpt2017gefocus317.pdf.

    • Search Google Scholar
    • Export Citation
  • Carter, C.L. (2009) Consumer Protection in the States: A 50-state Report On Unfair and Deceptive Acts and Practices Statutes, Report, Boston, MA: National Consumer Law Center, https://www.nclc.org/images/pdf/udap/report_50_states.pdf.

    • Search Google Scholar
    • Export Citation
  • Clemens, R.F. and Tierney, W.G. (2017) The utility of qualitative research to inform public policy, in P.A. Pasque and V.M. Lechuga (eds) Qualitative Inquiry in Higher Education Organization and Policy Research, New York: Routledge, pp 2747.

    • Search Google Scholar
    • Export Citation
  • Corbin, J. and Strauss, A. (2008) Basics of Qualitative Research, 3rd edn, Los Angeles, CA: SAGE.

  • Creswell, J.W. and Poth, C.N. (2018), Qualitative Inquiry and Research Design: Choosing Among Five Approaches, 4th edn, Thousand Oaks, CA: SAGE.

    • Search Google Scholar
    • Export Citation
  • Delgado, R. (1989) Storytelling for oppositionists and others: a plea for narrative, Michigan Law Review, 87(8): 241141.

  • Denzin, N.K. (2009) The elephant in the living room: or extending the conversation about the politics of evidence, Qualitative Research, 9(2): 13960.

    • Search Google Scholar
    • Export Citation
  • Denzin, N.K., Lincoln, Y.S. and Giardina, M.D. (2006) Disciplining qualitative research, International Journal of Qualitative Studies in Education, 19(6): 76982.

    • Search Google Scholar
    • Export Citation
  • Donmoyer, R. (2012a) Can qualitative researchers answer policymakers’ What-Works Question?, Qualitative Inquiry, 18(8): 66273.

  • Donmoyer, R. (2012b) Two (very) different worlds: the cultures of policymaking and qualitative research, Qualitative Inquiry, 18(9): 798807.

    • Search Google Scholar
    • Export Citation
  • Dougherty, K.J. and Natow, R.S. (2020) Performance-based funding for higher education: how well does neoliberal theory capture neoliberal practice?, Higher Education, 80(3): 45778.

    • Search Google Scholar
    • Export Citation
  • Evans, D. (2003) Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions, Journal of Clinical Nursing, 12(1): 7784.

    • Search Google Scholar
    • Export Citation
  • Executive Order No. 12866, 1993 comp., Title 3 C.F.R. 638.

  • Fielding, N.G. (2020) Critical qualitative research and impact in the public sphere, Qualitative Inquiry, 26(2): 14252.

  • Gibbs, G.R. (2018) Analyzing Qualitative Data, 2nd edn, Thousand Oaks, CA: SAGE.

  • Gibton, D. (2016) Researching Education Policy, Public Policy, and Policymakers: Qualitative Methods and Ethical Issues, New York: Routledge.

    • Search Google Scholar
    • Export Citation
  • Gray, G. and Jones, M.D. (2016) A qualitative narrative policy framework? Examining the policy narratives of US campaign finance regulatory reform, Public Policy Administration, 31(3): 193220.

    • Search Google Scholar
    • Export Citation
  • Gross, J.P., Cekic, O., Hossler, D. and Hillman, N. (2009) What matters in student loan default: a review of the research literature, Journal of Student Financial Aid, 39(1): 1929.

    • Search Google Scholar
    • Export Citation
  • Haynes, A., Rowbotham, S.J., Redman, S., Brennan, S., Williamson, A. and Moore, G. (2018) What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review, Health Research Policy & Systems, 16(1): 31.

    • Search Google Scholar
    • Export Citation
  • Henig, J. (2012) The politics of data use, Teachers College Record, 114(11): 132.

  • Herriott, R.E. and Firestone, W.A. (1983) Multisite qualitative policy research: optimizing description and generalizability, Educational Researcher, 12(2): 1419.

    • Search Google Scholar
    • Export Citation
  • Kaplan, R.G., Riedy, R., Van Horne, K. and Penuel, W. (2019) Going on a statewide listening tour: involving education leaders in the process of research to enhance the practical value of qualitative research, Evidence & Policy, 15(2): 17996.

    • Search Google Scholar
    • Export Citation
  • Kerrigan, M.R. and Johnson, A.T. (2019) Qualitative approaches to policy research in education: contesting the evidence-based, neoliberal regime, American Behavioral Scientist, 63(3): 28795.

    • Search Google Scholar
    • Export Citation
  • Kerwin, C.M. and Furlong, S.R. (2011) Rulemaking: How Government Agencies Write Law and Make Policy, 4th edn, Washington, DC: CQ Press.

  • Kugler, A. (2014) Labor market analysis and labor policymaking in the nation’s capital, ILR Review, 67(Suppl 3): 594607.

  • Lather, P. (2004) This is your father’s paradigm: government intrusion and the case of qualitative research in education, Qualitative Inquiry, 10(1): 1534.

    • Search Google Scholar
    • Export Citation
  • Lather, P. (2006) Foucauldian scientificity: rethinking the nexus of qualitative research and educational policy analysis, International Journal of Qualitative Studies in Education, 19(6): 78391.

    • Search Google Scholar
    • Export Citation
  • Lee, H., Lindquist, J.D., and Acito, F. (1997) Managers’ evaluation of research design and its impact on the use of research: An experimental approach, Journal of Business Research, 39(3): 23140.

    • Search Google Scholar
    • Export Citation
  • Locock, L. and Boaz, A. (2019) Drawing straight lines along blurred boundaries: qualitative research, patient and public involvement in medical research, co-production and co-design, Evidence & Policy, 15(3): 40921.

    • Search Google Scholar
    • Export Citation
  • Lovell, M., Guthrie, J., Simpson, P. and Butler, T. (2018) Navigating the political landscape of Australian criminal justice reform: senior policy-makers on alternatives to incarceration, Current Issues in Criminal Justice, 29(3): 22741.

    • Search Google Scholar
    • Export Citation
  • Lune, H. and Berg, B.L. (2017) Qualitative Research Methods for the Social Sciences, 9th edn, New York: Pearson.

  • Malterud, K. (2001) Qualitative research: standards, challenges, and guidelines, The Lancet, 358(9280): 48388.

  • Maxwell, J.A. (2004) Causal explanation, qualitative research, and scientific inquiry in education, Educational Researcher, 33(2): 311.

    • Search Google Scholar
    • Export Citation
  • Maxwell, J.A. (2013) Qualitative Research Design: An Interactive Approach, 3rd edn, Thousand Oaks, CA: SAGE.

  • McAleese, S. and Kilty, J.M. (2019) Stories matter: reaffirming the value of qualitative research, The Qualitative Report, 24(4): 82245.

    • Search Google Scholar
    • Export Citation
  • Merriam, S.B. and Tisdell, E.J. (2016) Qualitative Research: A Guide to Design and Implementation, San Francisco, CA: Jossey-Bass.

  • Miles, M.B., Huberman, A.M. and Saldaña, J. (2019) Qualitative Data Analysis: A Methods Sourcebook, 4th edn, Thousand Oaks, CA: SAGE.

  • National Research Council (2012) Using Science as Evidence in Public Policy, Washington, DC: National Academies Press.

  • Natow, R.S. (2017) Higher Education Rulemaking: The Politics of Creating Regulatory Policy, Baltimore, MD: Johns Hopkins University Press.

    • Search Google Scholar
    • Export Citation
  • Natow, R.S. (2020) Research utilization in higher education rulemaking: a multi-case study of research prevalence, sources, and barriers, Education Policy Analysis Archives, 28(5): 132.

    • Search Google Scholar
    • Export Citation
  • Nutley, S.M., Walter, I. and Davies, H.T.O. (2007) Using Evidence: How Research Can Inform Public Services, Bristol: Policy Press.

  • Office of the Federal Register (2011) A Guide to the Rulemaking Process, www.federalregister.gov/uploads/2011/01/the_rulemaking_process.pdf.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 2.

    • Search Google Scholar
    • Export Citation
  • Ospina, S.M., Esteve, M. and Lee, S. (2018) Assessing qualitative studies in public administration research, Public Administration Review, 78(4): 593605.

    • Search Google Scholar
    • Export Citation
  • Pelesh, M.L. (1994) Regulations under the higher education amendments of 1992: a case study in negotiated rulemaking, Law and Contemporary Problems, 57(4): 15170.

    • Search Google Scholar
    • Export Citation
  • Posti-Ahokas, H. (2013) Empathy-based stories capturing the voice of female secondary school students in Tanzania, International Journal of Qualitative Studies in Education, 26(10): 127792.

    • Search Google Scholar
    • Export Citation
  • Postsecondary National Policy Institute (n.d.) Overview, https://pnpi.org/wp-content/uploads/2019/08/UpdatedPNPI_OnePager_Doc.pdf.

  • Program Integrity: Gainful Employment, Final Regulations (2014) 79 Fed. Reg. 64890.

  • Program Integrity: Gainful Employment, Final Regulations (2019) 84 Fed. Reg. 31392.

  • Purcell-Gates, V. (2000) The Role of Qualitative and Ethnographic Research in Educational Policy, Opinion paper, Newark, NJ: International Reading Association.

    • Search Google Scholar
    • Export Citation
  • Ritchie, J. and Spencer, L. (1994) Qualitative data analysis for applied policy research, in A. Bryman and R.G. Burgess (eds) Analyzing Qualitative Data, New York: Routledge, pp 17394.

    • Search Google Scholar
    • Export Citation
  • Rivera, L.A. and Tilcsik, A. (2019) Scaling down inequality: rating scales, gender bias, and the architecture of evaluation, American Sociological Review, 84(2): 24874.

    • Search Google Scholar
    • Export Citation
  • Saiani, P.P. (2018) Doing sociology in the age of ‘evidence-based research’: scientific epistemology versus political dominance, The American Sociologist, 49(1): 8097.

    • Search Google Scholar
    • Export Citation
  • Saldaña, J. (2016) The Coding Manual for Qualitative Researchers, 3rd edn, Los Angeles, CA: SAGE.

  • Sallee, M.W. and Flood, J.T. (2012) Using qualitative research to bridge research, policy, and practice, Theory Into Practice, 51(2): 13744.

    • Search Google Scholar
    • Export Citation
  • Savin-Baden, M. and Major, C.H. (2013) Qualitative Research: The Essential Guide to Theory and Practice, New York: Routledge.

  • Scott, G.A. (2011) Postsecondary Education: Student Outcomes Vary at For-Profit, Nonprofit, and Public Schools. Report to Congressional Requesters, Report no. GAO-12–143, Washington, DC: US Government Accountability Office.

    • Search Google Scholar
    • Export Citation
  • Sinkovics, R.R. and Alfoldi, E.A. (2012) Progressive focusing and trustworthiness in qualitative research, Management International Review, 52(6): 81745.

    • Search Google Scholar
    • Export Citation
  • Srivastava, P. and Hopwood, N. (2009) A practical iterative framework for qualitative data analysis, International Journal of Qualitative Research Methods, 8(1): 7684.

    • Search Google Scholar
    • Export Citation
  • Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Final Regulations (2015) 80 Fed. Reg. 67204.

    • Search Google Scholar
    • Export Citation
  • Student Assistance General Provisions, Federal Perkins Loan Program, Federal Family Education Loan Program, William D. Ford Federal Direct Loan Program, and Teacher Education Assistance for College and Higher Education Grant Program: Final Regulations (2016) 81 Fed. Reg. 75926.

  • Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Final Regulations (2019) 84 Fed. Reg. 49788.

  • Sunstein, C.R. (2013) The office of information and regulatory affairs: myths and realities, Harvard Law Review, 126(7): 183878.

  • Suri, H. (2011) Purposeful sampling in qualitative research synthesis, Qualitative Research Journal, 11(2): 6375.

  • Teacher Preparation Issues: Final Regulations (2016) 81 Fed. Reg. 75494.

  • Torgerson, D.J. and Torgerson, C.J. (2003) Avoiding bias in randomised controlled trials in educational research, British Journal of Educational Studies, 51(1): 3645.

    • Search Google Scholar
    • Export Citation
  • Torrance, H. (2008) Building confidence in qualitative research: engaging the demands of policy, Qualitative Inquiry, 14(4) 50727.

  • Truong, D.H. (2000) Daubert and judicial review: how does an administrative agency distinguish valid science from junk science, Akron Law Review, 33(3): 36590.

    • Search Google Scholar
    • Export Citation
  • US Department of Education (2002) No Child Left Behind: A Desktop Reference, Washington, DC: US Department of Education.

  • US Department of Education (2003) Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide, Washington, DC: US Department of Education.

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 42631.

  • Weiss, C.H., Murphy-Graham, E. and Birkeland, S. (2005) An alternate route to policy influence: How evaluations affect D.A.R.E., American Journal of Evaluation, 26(1): 1230.

    • Search Google Scholar
    • Export Citation
  • Werner, A. (2004) A Guide to Implementation Research, Washington, DC: Urban Institute Press.

  • William D. Ford Federal Direct Loan Program: Final Regulations (2014) 79 Fed. Reg. 63317.

  • Altheide, D.L. and Johnson, J.M. (2011) Reflections on interpretive adequacy in qualitative research, SAGE Handbook of Qualitative Research, vol 4, Los Angeles, CA: SAGE, pp 58194.

    • Search Google Scholar
    • Export Citation
  • Asen, R., Gurke, D., Solomon, R., Conners, P. and Gumm, E. (2011) ‘The research says’: definitions and uses of a key policy term in federal law and local school board deliberations, Argumentation and Advocacy, 47(4): 195213.

    • Search Google Scholar
    • Export Citation
  • Association of Private Sector Colleges and Universities v. Duncan (2012) Memorandum Opinion, Civil Action 11–1314 (RC), D.D.C.

  • Belfield, C.R., Bowden, A.B. and Rodriguez, V. (2018) Evaluating regulatory impact assessments in education policy, American Journal of Evaluation, 40(3): 33553.

    • Search Google Scholar
    • Export Citation
  • Bowers, J. and Testa, P.F. (2019) Better government, better science: the promise of and challenges facing the evidence-informed policy movement, Annual Review of Political Science, 22(1): 52142.

    • Search Google Scholar
    • Export Citation
  • Bozeman, H., Mingo, M. and Hershey-Arista, M. (2017) Summary report for the 2017 gainful employment focus groups, report for the US department of education, Rockville, MD: Westat, https://www2.ed.gov/about/offices/list/ope/summaryrpt2017gefocus317.pdf.

    • Search Google Scholar
    • Export Citation
  • Carter, C.L. (2009) Consumer Protection in the States: A 50-state Report On Unfair and Deceptive Acts and Practices Statutes, Report, Boston, MA: National Consumer Law Center, https://www.nclc.org/images/pdf/udap/report_50_states.pdf.

    • Search Google Scholar
    • Export Citation
  • Clemens, R.F. and Tierney, W.G. (2017) The utility of qualitative research to inform public policy, in P.A. Pasque and V.M. Lechuga (eds) Qualitative Inquiry in Higher Education Organization and Policy Research, New York: Routledge, pp 2747.

    • Search Google Scholar
    • Export Citation
  • Corbin, J. and Strauss, A. (2008) Basics of Qualitative Research, 3rd edn, Los Angeles, CA: SAGE.

  • Creswell, J.W. and Poth, C.N. (2018), Qualitative Inquiry and Research Design: Choosing Among Five Approaches, 4th edn, Thousand Oaks, CA: SAGE.

    • Search Google Scholar
    • Export Citation
  • Delgado, R. (1989) Storytelling for oppositionists and others: a plea for narrative, Michigan Law Review, 87(8): 241141.

  • Denzin, N.K. (2009) The elephant in the living room: or extending the conversation about the politics of evidence, Qualitative Research, 9(2): 13960.

    • Search Google Scholar
    • Export Citation
  • Denzin, N.K., Lincoln, Y.S. and Giardina, M.D. (2006) Disciplining qualitative research, International Journal of Qualitative Studies in Education, 19(6): 76982.

    • Search Google Scholar
    • Export Citation
  • Donmoyer, R. (2012a) Can qualitative researchers answer policymakers’ What-Works Question?, Qualitative Inquiry, 18(8): 66273.

  • Donmoyer, R. (2012b) Two (very) different worlds: the cultures of policymaking and qualitative research, Qualitative Inquiry, 18(9): 798807.

    • Search Google Scholar
    • Export Citation
  • Dougherty, K.J. and Natow, R.S. (2020) Performance-based funding for higher education: how well does neoliberal theory capture neoliberal practice?, Higher Education, 80(3): 45778.

    • Search Google Scholar
    • Export Citation
  • Evans, D. (2003) Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions, Journal of Clinical Nursing, 12(1): 7784.

    • Search Google Scholar
    • Export Citation
  • Executive Order No. 12866, 1993 comp., Title 3 C.F.R. 638.

  • Fielding, N.G. (2020) Critical qualitative research and impact in the public sphere, Qualitative Inquiry, 26(2): 14252.

  • Gibbs, G.R. (2018) Analyzing Qualitative Data, 2nd edn, Thousand Oaks, CA: SAGE.

  • Gibton, D. (2016) Researching Education Policy, Public Policy, and Policymakers: Qualitative Methods and Ethical Issues, New York: Routledge.

    • Search Google Scholar
    • Export Citation
  • Gray, G. and Jones, M.D. (2016) A qualitative narrative policy framework? Examining the policy narratives of US campaign finance regulatory reform, Public Policy Administration, 31(3): 193220.

    • Search Google Scholar
    • Export Citation
  • Gross, J.P., Cekic, O., Hossler, D. and Hillman, N. (2009) What matters in student loan default: a review of the research literature, Journal of Student Financial Aid, 39(1): 1929.

    • Search Google Scholar
    • Export Citation
  • Haynes, A., Rowbotham, S.J., Redman, S., Brennan, S., Williamson, A. and Moore, G. (2018) What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review, Health Research Policy & Systems, 16(1): 31.

    • Search Google Scholar
    • Export Citation
  • Henig, J. (2012) The politics of data use, Teachers College Record, 114(11): 132.

  • Herriott, R.E. and Firestone, W.A. (1983) Multisite qualitative policy research: optimizing description and generalizability, Educational Researcher, 12(2): 1419.

    • Search Google Scholar
    • Export Citation
  • Kaplan, R.G., Riedy, R., Van Horne, K. and Penuel, W. (2019) Going on a statewide listening tour: involving education leaders in the process of research to enhance the practical value of qualitative research, Evidence & Policy, 15(2): 17996.

    • Search Google Scholar
    • Export Citation
  • Kerrigan, M.R. and Johnson, A.T. (2019) Qualitative approaches to policy research in education: contesting the evidence-based, neoliberal regime, American Behavioral Scientist, 63(3): 28795.

    • Search Google Scholar
    • Export Citation
  • Kerwin, C.M. and Furlong, S.R. (2011) Rulemaking: How Government Agencies Write Law and Make Policy, 4th edn, Washington, DC: CQ Press.

  • Kugler, A. (2014) Labor market analysis and labor policymaking in the nation’s capital, ILR Review, 67(Suppl 3): 594607.

  • Lather, P. (2004) This is your father’s paradigm: government intrusion and the case of qualitative research in education, Qualitative Inquiry, 10(1): 1534.

    • Search Google Scholar
    • Export Citation
  • Lather, P. (2006) Foucauldian scientificity: rethinking the nexus of qualitative research and educational policy analysis, International Journal of Qualitative Studies in Education, 19(6): 78391.

    • Search Google Scholar
    • Export Citation
  • Lee, H., Lindquist, J.D., and Acito, F. (1997) Managers’ evaluation of research design and its impact on the use of research: An experimental approach, Journal of Business Research, 39(3): 23140.

    • Search Google Scholar
    • Export Citation
  • Locock, L. and Boaz, A. (2019) Drawing straight lines along blurred boundaries: qualitative research, patient and public involvement in medical research, co-production and co-design, Evidence & Policy, 15(3): 40921.

    • Search Google Scholar
    • Export Citation
  • Lovell, M., Guthrie, J., Simpson, P. and Butler, T. (2018) Navigating the political landscape of Australian criminal justice reform: senior policy-makers on alternatives to incarceration, Current Issues in Criminal Justice, 29(3): 22741.

    • Search Google Scholar
    • Export Citation
  • Lune, H. and Berg, B.L. (2017) Qualitative Research Methods for the Social Sciences, 9th edn, New York: Pearson.

  • Malterud, K. (2001) Qualitative research: standards, challenges, and guidelines, The Lancet, 358(9280): 48388.

  • Maxwell, J.A. (2004) Causal explanation, qualitative research, and scientific inquiry in education, Educational Researcher, 33(2): 311.

    • Search Google Scholar
    • Export Citation
  • Maxwell, J.A. (2013) Qualitative Research Design: An Interactive Approach, 3rd edn, Thousand Oaks, CA: SAGE.

  • McAleese, S. and Kilty, J.M. (2019) Stories matter: reaffirming the value of qualitative research, The Qualitative Report, 24(4): 82245.

    • Search Google Scholar
    • Export Citation
  • Merriam, S.B. and Tisdell, E.J. (2016) Qualitative Research: A Guide to Design and Implementation, San Francisco, CA: Jossey-Bass.

  • Miles, M.B., Huberman, A.M. and Saldaña, J. (2019) Qualitative Data Analysis: A Methods Sourcebook, 4th edn, Thousand Oaks, CA: SAGE.

  • National Research Council (2012) Using Science as Evidence in Public Policy, Washington, DC: National Academies Press.

  • Natow, R.S. (2017) Higher Education Rulemaking: The Politics of Creating Regulatory Policy, Baltimore, MD: Johns Hopkins University Press.

    • Search Google Scholar
    • Export Citation
  • Natow, R.S. (2020) Research utilization in higher education rulemaking: a multi-case study of research prevalence, sources, and barriers, Education Policy Analysis Archives, 28(5): 132.

    • Search Google Scholar
    • Export Citation
  • Nutley, S.M., Walter, I. and Davies, H.T.O. (2007) Using Evidence: How Research Can Inform Public Services, Bristol: Policy Press.

  • Office of the Federal Register (2011) A Guide to the Rulemaking Process, www.federalregister.gov/uploads/2011/01/the_rulemaking_process.pdf.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 2.

    • Search Google Scholar
    • Export Citation
  • Ospina, S.M., Esteve, M. and Lee, S. (2018) Assessing qualitative studies in public administration research, Public Administration Review, 78(4): 593605.

    • Search Google Scholar
    • Export Citation
  • Pelesh, M.L. (1994) Regulations under the higher education amendments of 1992: a case study in negotiated rulemaking, Law and Contemporary Problems, 57(4): 15170.

    • Search Google Scholar
    • Export Citation
  • Posti-Ahokas, H. (2013) Empathy-based stories capturing the voice of female secondary school students in Tanzania, International Journal of Qualitative Studies in Education, 26(10): 127792.

    • Search Google Scholar
    • Export Citation
  • Postsecondary National Policy Institute (n.d.) Overview, https://pnpi.org/wp-content/uploads/2019/08/UpdatedPNPI_OnePager_Doc.pdf.

  • Program Integrity: Gainful Employment, Final Regulations (2014) 79 Fed. Reg. 64890.

  • Program Integrity: Gainful Employment, Final Regulations (2019) 84 Fed. Reg. 31392.

  • Purcell-Gates, V. (2000) The Role of Qualitative and Ethnographic Research in Educational Policy, Opinion paper, Newark, NJ: International Reading Association.

    • Search Google Scholar
    • Export Citation
  • Ritchie, J. and Spencer, L. (1994) Qualitative data analysis for applied policy research, in A. Bryman and R.G. Burgess (eds) Analyzing Qualitative Data, New York: Routledge, pp 17394.

    • Search Google Scholar
    • Export Citation
  • Rivera, L.A. and Tilcsik, A. (2019) Scaling down inequality: rating scales, gender bias, and the architecture of evaluation, American Sociological Review, 84(2): 24874.

    • Search Google Scholar
    • Export Citation
  • Saiani, P.P. (2018) Doing sociology in the age of ‘evidence-based research’: scientific epistemology versus political dominance, The American Sociologist, 49(1): 8097.

    • Search Google Scholar
    • Export Citation
  • Saldaña, J. (2016) The Coding Manual for Qualitative Researchers, 3rd edn, Los Angeles, CA: SAGE.

  • Sallee, M.W. and Flood, J.T. (2012) Using qualitative research to bridge research, policy, and practice, Theory Into Practice, 51(2): 13744.

    • Search Google Scholar
    • Export Citation
  • Savin-Baden, M. and Major, C.H. (2013) Qualitative Research: The Essential Guide to Theory and Practice, New York: Routledge.

  • Scott, G.A. (2011) Postsecondary Education: Student Outcomes Vary at For-Profit, Nonprofit, and Public Schools. Report to Congressional Requesters, Report no. GAO-12–143, Washington, DC: US Government Accountability Office.

    • Search Google Scholar
    • Export Citation
  • Sinkovics, R.R. and Alfoldi, E.A. (2012) Progressive focusing and trustworthiness in qualitative research, Management International Review, 52(6): 81745.

    • Search Google Scholar
    • Export Citation
  • Srivastava, P. and Hopwood, N. (2009) A practical iterative framework for qualitative data analysis, International Journal of Qualitative Research Methods, 8(1): 7684.

    • Search Google Scholar
    • Export Citation
  • Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Final Regulations (2015) 80 Fed. Reg. 67204.

    • Search Google Scholar
    • Export Citation
  • Student Assistance General Provisions, Federal Perkins Loan Program, Federal Family Education Loan Program, William D. Ford Federal Direct Loan Program, and Teacher Education Assistance for College and Higher Education Grant Program: Final Regulations (2016) 81 Fed. Reg. 75926.

  • Student Assistance General Provisions, Federal Family Education Loan Program, and William D. Ford Federal Direct Loan Program: Final Regulations (2019) 84 Fed. Reg. 49788.

  • Sunstein, C.R. (2013) The office of information and regulatory affairs: myths and realities, Harvard Law Review, 126(7): 183878.

  • Suri, H. (2011) Purposeful sampling in qualitative research synthesis, Qualitative Research Journal, 11(2): 6375.

  • Teacher Preparation Issues: Final Regulations (2016) 81 Fed. Reg. 75494.

  • Torgerson, D.J. and Torgerson, C.J. (2003) Avoiding bias in randomised controlled trials in educational research, British Journal of Educational Studies, 51(1): 3645.

    • Search Google Scholar
    • Export Citation
  • Torrance, H. (2008) Building confidence in qualitative research: engaging the demands of policy, Qualitative Inquiry, 14(4) 50727.

  • Truong, D.H. (2000) Daubert and judicial review: how does an administrative agency distinguish valid science from junk science, Akron Law Review, 33(3): 36590.

    • Search Google Scholar
    • Export Citation
  • US Department of Education (2002) No Child Left Behind: A Desktop Reference, Washington, DC: US Department of Education.

  • US Department of Education (2003) Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide, Washington, DC: US Department of Education.

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 42631.

  • Weiss, C.H., Murphy-Graham, E. and Birkeland, S. (2005) An alternate route to policy influence: How evaluations affect D.A.R.E., American Journal of Evaluation, 26(1): 1230.

    • Search Google Scholar
    • Export Citation
  • Werner, A. (2004) A Guide to Implementation Research, Washington, DC: Urban Institute Press.

  • William D. Ford Federal Direct Loan Program: Final Regulations (2014) 79 Fed. Reg. 63317.

  • 1 Hofstra University, , USA

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 46 46 46
PDF Downloads 24 24 24

Altmetrics

Dimensions