Abstract
Background:
Although lawmakers play an essential role in policymaking, there is no systematic review on the use of research evidence in legislatures.
Aims and objectives:
To examine types of research use and factors facilitating and hindering use in legislatures.
Methods:
We conducted a systematic review of studies in legislatures, regardless of geographical region or year of publication. We included empirical studies irrespective of the methodology employed. Thematic synthesis was used to synthesise the type of use and the facilitating and hindering factors to using research evidence in parliaments. We included 21 studies.
Findings:
The most frequently observed type of utilisation was the use for symbolic or tactical purposes. Forms of use specific to legislatures were also identified, such as to prepare questions and debates and to help build consensus. Four categories of factors seen as facilitators or barriers were found: institution and organisation, research characteristics, policy and political context, and individual characteristics. Some factors had already been identified in previous reviews, while others seem to apply exclusively to legislatures.
Discussion and conclusions:
The review identified types of use of research evidence observed in legislatures and developed a new categorisation of factors that may promote or hinder evidence use in this institutional setting. It highlighted a need for more research beyond the US, in unicameral legislatures and in countries with a parliamentary form of government. Content analysis of parliamentary debates in legislative assembly or committee to examine the use of research evidence seems to be underused.
Background
The effort to inject research evidence into policymaking is a challenge for the executive branch of government and its public service (Landry et al, 2003), but also for the legislative branch (Rose et al, 2020). Like governments, legislatures are involved in public policymaking, whereas they differ from governments in their functions and activities. Although governments initiate several bills, the role of the legislature lies primarily with the drafting, adoption and amendment of laws that provide general frameworks for the government’s development of public policies. The function of oversight of the government through votes of confidence and no confidence, question-and-answer sessions, legislative committees and the work of legislators from the opposition parties are also what makes legislatures different from governments. Furthermore, unlike governments, legislatures (especially national ones) draft, adopt and revise the constitution. But perhaps more importantly, legislatures carry the representative function of linking citizens to the political system (on the core functions of legislatures and governments, see Balzacq et al, 2014: 269–305). The critical role of the legislative branch in public policymaking raises questions about the use of research evidence within this specific setting. In particular, for what purposes is research evidence used in legislatures, and what are the facilitating and hindering factors? These are the main questions that this article seeks to answer.
An update of a systematic review of the empirical literature on the factors hindering and facilitating the use of research evidence in policymaking was conducted eight years ago (Oliver et al, 2014). The approach of studying facilitating or hindering factors for informing the process of developing solutions to reduce the evidence-policy divide has been criticised for not being sufficiently grounded in theoretical and empirical work in policy studies, leading to a misunderstanding of the requirements and constraints of the work of policymakers (Cairney, 2016). However, identifying and categorising the factors associated with the use of research evidence in policymaking provides a comparative basis for researchers about to research this topic, bearing in mind that these factors may be highly dependent on the local context and are therefore not necessarily transposable to all settings.
One of the main strengths of the systematic review produced by Oliver et al (2014) is that it identifies empirical studies conducted in various policymaking environments (for example, ministries, agencies, parliaments, and so on). On the other hand, this strength can also be a weakness, as the nature and constraints of policy work that may influence the use of research evidence could differ between the executive and legislative branches. Covering 145 studies, Oliver et al’s (2014)’s systematic review makes few references to the legislative branch.
The systematic review, whose results are presented in this article, is the first phase of a larger project on the use of research evidence that will be examined in over 20 member legislatures of the Assemblée Parlementaire de la Francophonie. Although the focus of the study in these legislatures will be primarily on institutional and organisational factors influencing the use of research evidence, our review focuses on any category of factors that may affect the use of research evidence in any type of legislature. More specifically, the objectives of the review were to (1) examine the type of evidence use in the legislative branch that seems to be most frequently observed in empirical studies; and (2) produce a thematic synthesis of factors facilitating or hindering the use of research evidence in legislatures.
The article is structured as follows. The following section presents the methods used to achieve the research objectives. The presentation of the results follows the methods section. In this section, we first report on the literature search results and the characteristics of the included studies. Next, we present results on the types of evidence use observed in the studies (first objective), after which we report results on the factors that facilitate or hinder use (second objective). Finally, we discuss the findings, present the limitations of the review and propose some avenues for future research.
Methods
To be included, studies had to be:
On utilisation. Therefore, the focus was on the demand side rather than the supply side. Studies focusing solely on the dissemination or brokering practices of research evidence producers or intermediaries have been excluded.
On utilisation of research evidence, defined globally as research results, regardless of the discipline, research design (observational or experimental) or analysis method (quantitative, qualitative or mixed). Studies for which it was unclear whether research evidence was included in the evidence or information whose use was under examination were excluded from the review (for example, use of government data, internal policy analysis, programme evaluation, benchmarking studies, performance indicators, and so on).
Primary research (that is, systematic reviews were used to identify primary studies) that describes types of utilisation of research evidence or facilitating or hindering factors. Studies describing sources of information used without describing the kind of use or facilitating or hindering factors were excluded from the review. The same applies to studies focusing on perceived solutions to promote the use of research evidence.
Empirical, regardless of the research design. However, studies presenting little or no information on the observation or experimentation methodology were excluded. Studies that simply report on unstructured discussions at meetings or workshops (except for focus groups and interviews) were excluded. Essays or reflective texts of a conceptual, theoretical or debate nature were also excluded.
Conducted in any type of national or sub-national legislatures and countries. Studies not covering the legislative branch or focusing solely on government ministries or agencies were excluded. To be included, studies had to report results that could be directly linked to the legislative branch. For example, in a study covering policymaking within both the executive and legislative branches, subgroup analyses for the legislative branch would have to be presented for the study to be included.
Like Oliver et al (2014), we did not exclude studies based on study design or policy sectors. However, our review differs from Oliver et al (2014) in several respects. Our review focuses on one type of policymaking environment, namely legislatures. In contrast, Oliver et al’s (2014) review includes studies focusing on various settings (for example, hospitals, prisons, government agencies, legislatures, international organisations, and so on). Furthermore, our review focuses on the type of use and factors, whereas Oliver et al (2014) cover several other aspects, such as ideas for solutions to promote use.
The following English electronic databases were searched on 9 May 2021: Web of Science (SSCI), International Bibliography of the Social Sciences (IBSS), PAIS Index, Sociological Abstracts, Worldwide Political Science Abstracts (WPSA), ABI/Inform Collection, Canadian Business and Current Affairs Database (CBCA), Business Source Premier, Public Administration Abstracts, Social Science Abstracts, and International Political Science Abstracts. All OVID Medline, which includes PubMed, was searched on 11 May 2022 during the review process of the manuscript. Furthermore, as phase 2 of the larger project plans to collect data in French-speaking legislatures, we also searched the following French-speaking databases on 22 May 2021: CAIRN and Erudit (including Persée). Conducted by a database search specialist with many years of experience in systematic review projects, the queries were organised by combining two concepts: (1) evidence use, and (2) parliament (or legislature). The search strings for the evidence use concept were designed to be broad to minimise the losses that could be caused by using a smaller number of search terms. The same logic was applied to the search strings for the concept of parliament or the legislative branch. In some databases, we limited the searches to peer-reviewed publications to reduce the number of results. We did not use limitations by year of publication. The complete list of Boolean programming queries is available in supplementary data online (https://www.openicpsr.org/openicpsr/project/180061). We also scrutinised the 145 publications included in the systematic review by Oliver et al (2014) and the publications cited in the studies included in the review. Finally, we also searched Google to find studies published as reports.
All studies were first double-screened on title and abstract. Relevant studies were retrieved and screened on full text by two reviewers. Studies were screened and extracted using the web-based Covidence software. Two reviewers extracted all studies. A third reviewer double-checked the extraction for all studies. Data extraction fields included characteristics of studies, methods and findings related to the type of use and facilitating and hindering factors. During the extraction phase, publications reporting the results from the same data were merged into a single study. The information extracted for the type of use and factors comes exclusively from the results section of the publications reviewed.
We used the thematic synthesis approach to synthesise studies’ findings, which involves grouping the results into categories (themes) (Gough et al, 2012). For the type of use, we conduct the thematic analysis deductively using the instrumental, conceptual and symbolic (tactical or political) use categorisation (Amara et al, 2004). When a type of use could not be placed under one of these three categories, it was allocated to the ‘other’ category. Concerning facilitating or hindering factors, we conducted thematic syntheses inductively by extracting the statements describing the findings on a spreadsheet. The findings statements were grouped into an analytical category. The analysis of these statements led us to create four analytical categories, namely institution and organisation, research characteristics, policy and political context, and individual characteristics. We then created analytical subcategories called analytical themes, under which studies were classified into sub-themes describing the findings.
Finally, to examine whether the use of research evidence differs between the executive and legislative branches, we summarised the results on the type of use and factors from the subset of studies that compared these aspects between the two branches.
Limitations
As shown in the flow diagram discussed in the results section, several empirical studies were discarded as they did not appear to involve the use of research evidence. This choice does not mean that research evidence is the only or best source of evidence that legislators should use in their work. Indeed, it is well established that legislators use a range of information that goes well beyond research evidence. However, focusing specifically on this type of evidence allows for more accurate results in the kind of use and facilitating and hindering factors, as these aspects may vary according to the type of evidence.
Theoretically, there is no limit to the effort reviewers can put into finding publications and studies in a systematic review project. The literature search strategy mainly focused on bibliographic databases and corollary peer-reviewed studies. Searches on Google (not Google Scholar) were more limited in their coverage. It is, therefore, possible that eligible studies published as reports available on the Internet were not identified. Also, we did not survey the authors who published on the subject to identify publications that might have been missed. However, as mentioned, the entire screening process was done by double-coding and the final inclusion decisions were all validated by a third coder.
Findings
The search strategy led to the identification of 3804 unique records, of which 255 were screened on full text. Twenty-one studies published between 1979 and September 2021 were included in the review. The flow chart describing the disposition of documents is presented in Figure 1.
Flow chart describing the disposition of documents
Citation: Evidence & Policy 20, 2; 10.1332/174426421X16656568731041
Characteristics of included studies
For a complete description of the included studies, see Supplementary data (https://www.openicpsr.org/openicpsr/project/180061). A summary of the characteristics of these studies is presented in Figure 2. Before 2010, only three empirical studies on the type of use or factors influencing the use of research evidence showing specific results for the legislative branch had been published. The number of such studies exploded in the 2010s, and this upward trend looks set to continue in the current decade. Most studies were conducted in the US (17 of 21 studies), a presidential political system. The other three countries covered by the review (the UK, Canada and Netherlands) operate under a parliamentary political system. Ten studies examined the use of research evidence at the national level, while 11 studies have done so at a sub-national level.
Characteristics of the 21 studies included in the review
Citation: Evidence & Policy 20, 2; 10.1332/174426421X16656568731041
All 21 studies examined bicameral legislatures (two chambers including the Senate or its equivalent). Most studies (13 of 21) collected data on both chambers, while seven studies did not specify the chamber(s) studied. One study focused exclusively on the lower chamber, while no study focused exclusively and precisely on the upper chamber.
Only four of the 21 studies examined the use of research evidence in general without focusing on one or more policy areas. More than half of the studies included in the review examined the health sector. It is the policy area that has been studied the most. This result is similar to that of the systematic review by Oliver et al (2014), which examined the use of research evidence in a wide variety of settings beyond legislatures.
Qualitative methodologies mainly based on semi-structured interviews of elected officials or parliamentary staff were used in most studies (13 out of 21). The review includes five cross-sectional studies, two randomised controlled trials, and one mixed-methods study. The 21 studies included in the review report results on factors influencing the use of research evidence. Twelve of these studies also reported results on the type of use. Finally, five of the 21 studies reported results comparing the use or its determinants between the executive and legislative branches.
Type of use
Of the 21 studies included in the review, 12 report results on the type of use of research evidence in legislatures. In the literature on the use of research evidence, the study of the kind of use is often guided by the use of typologies. Since Weiss (1979)’s seminal work on the subject, several typologies of use have been developed, each with its variants. For this review, segments relating to how research evidence is used have been classified into three types of use frequently used in the literature, namely, instrumental use to facilitate decision making; conceptual use to facilitate understanding or reflection on problems; issues or solutions and, finally, symbolic or tactical use which consists of using research evidence to achieve strategic or political objectives. An ‘other type’ category has been added to the classification grid to avoid forcing segments into any of the three categories and possibly discovering new purposes for use that would apply to the legislative branch.
Figure 3 shows the distribution of the number of studies with at least one outcome segment in each of the use categories. The detailed list of segments for each of the 12 studies is available in the supplementary data online (https://www.openicpsr.org/openicpsr/project/180061). Symbolic or tactical use was observed in all but one of the studies that examined the type of use (11 of 12 studies), while conceptual and instrumental use were observed in five and four studies, respectively.
Type of use of research evidence
Citation: Evidence & Policy 20, 2; 10.1332/174426421X16656568731041
Four of the 12 studies reported results on other types of research evidence use. Interestingly, these purposes of use apply specifically to the legislative environment. Table 1 shows the segments relating to these other purposes for using research evidence.
Other types of evidence use that apply specifically to the legislative branch
| Study | Other type of evidence use |
|---|---|
| Bogenschneider et al (2019b: 783) | ‘Improving the decision-making process by enhancing debate, dialogue, collaboration, and compromise.’
‘Earning the trust of colleagues as a knowledgeable and credible information source.’ ‘Asking important questions for policy or political purposes.’ ‘Assessing the political and economic feasibility of policy decisions.’ ‘Explaining a vote on an issue.’ |
| Geddes et al (2017: 268)
Geddes (2020: 50) |
‘These quotes reveal that academic research is used, alongside other sources, to set out the parameters of contemporary debate.’
‘POST (Parliamentary Office of Science & Technology) is more likely to evaluate scientific evidence before it to provide authoritative “scientific” consensus.’ ‘… evidence plays an important role in driving committee work forward and links closely to consensus. It is used to ensure that party politics is at the margin of committee scrutiny (and could potentially be threatened through greater party polarization).’ |
| Girling and Gibbs (2019: 26) | ‘…several MPs discussed the importance of research in helping to verify or fact-check information.’ |
| Rose et al (2020: 625) | ‘Overall, the dominant theme, put forward by around a third of interviewees, was the use of research to directly inform their immediate representative, legislative or scrutiny work, such as questioning of select committee witnesses, tabling an amendment to a Bill, or dealing with a constituency issue.’ |
Bogenschneider et al (2019b: 783) report that research evidence is used for ‘enhancing debate, dialogue, collaboration, and compromise’, ‘earning the trust of colleagues as a knowledgeable and credible information source’, ‘asking important questions for policy or political purposes’, ‘assessing the political and economic feasibility of policy decisions’, and ‘explaining a vote on an issue’. The data collected by Geddes and colleagues, that was the subject of two publications, show that research evidence was used to ‘provide authoritative scientific consensus’ (Geddes et al, 2017: 268), in ‘driving committee work forward and links closely to consensus’ (Geddes, 2020: 50) and ‘ensure that party politics is at the margin of committee scrutiny’ (Geddes, 2020: 50). A third study found that members of legislature use research evidence to fact-check information (Girling and Gibbs, 2019: 26). Finally, Rose et al (2020: 625) found that a third of their interviewees use research evidence ‘to directly inform their immediate representative, legislative or scrutiny work, such as the questioning of select committee witnesses, tabling an amendment to a Bill, or dealing with a constituency issue’.
Factors affecting the use of research evidence in legislatures
The 21 studies in the review report results on facilitating factors, hindering factors or both. The inductive thematic synthesis led to the identification of categories for which there are analytical themes and sub-themes called descriptive themes. The number of studies that reported categories (themes) and associated analytical themes for facilitators and hindrances is presented in Figure 4. The descriptive themes related to the analytical themes are not shown in this figure so as not to overload it. The descriptive themes are presented in the supplementary data (https://www.openicpsr.org/openicpsr/project/180061) and summarised below.
Categories and analytical themes for facilitators of and hindrances to the use of research evidence in legislatures
Citation: Evidence & Policy 20, 2; 10.1332/174426421X16656568731041
The findings on factors affecting the use of research evidence have been grouped into the following four broad categories: (1) institution and organisation (legislature characteristics, social relations and trust, organisational support, and availability and access); (2) research characteristics (form and content); (3) policy and political context (issue characteristics, interest groups and work-related context); and (4) individual characteristics (experience, knowledge and skills, and attitude or beliefs towards research evidence). It should be noted that no categories for researcher or intermediary characteristics were created, as the review only focused on the demand side (see selection criteria in the methods section). The figure shows a clustered bar chart displaying the number of studies that report results for each analytical theme, whether they are facilitators or hindrances. The influence or importance of a factor is not a function of the number of studies reporting it. Indeed, identifying factors may be influenced by the aims or methods used in the studies (Oliver et al, 2014). A factor identified in a single study could therefore be significant. The number of studies associated with each factor is thus presented as an indication and should be interpreted with caution.
Institution and organisation
Legislature characteristics, social relations and trust, organisational support, and availability and access are the four analytical themes identified under the institution and organisation category. Let us now examine the descriptive themes that fall under each of these four analytical themes, beginning with the characteristics of the legislature. Some characteristics of legislatures are seen as facilitators, others as barriers. Much legislative work is done in committees, which is seen as a facilitator to the use of research evidence in legislatures (Jewell and Bero, 2008; Girling and Gibbs, 2019; Bogenschneider et al, 2021; Rose et al, 2020). Committees with a research focus (Jewell and Bero, 2008) and those that receive and deal with research evidence at the beginning of their work (Rose et al, 2020) facilitate evidence use. The library is also seen as a facilitator as this service seems to be positively viewed by legislators (Feller et al, 1979; Girling and Gibbs, 2019). On the other hand, the lack of homogeneity in legislature and the absence of an obligation to use research evidence were seen as an obstacle (Geddes et al, 2017). The tendency to concentrate power on senior members who are not always familiar with evidence-based policy was also seen as a hindrance (Jewell and Bero, 2008), as is the frequent replacement of legislators, which hinders the development and sustainability of expertise within legislatures (Jewell and Bero, 2008). Finally, the consensus culture restricting the propensity of committee members to challenge information is also seen as a barrier to the use of research evidence (Kenny et al, 2017).
Relationships and trust can be seen as critical social institutions forming working communities. This analytical theme is not new and is widely discussed in the field. One of the two randomised controlled trials in the review tested the effect of a multifaceted intervention, one component of which was to match researchers with legislators to feed them with research evidence (Crowley et al, 2021). This study found a positive and statistically significant effect on conceptual research use (that is, no effect on instrumental and symbolic use). Several other observational studies show that evidence use is facilitated when the legislator knows and trusts the information provider (Guston et al, 2005; Dodson et al, 2015; Brownson et al, 2016; Purtle et al, 2016; Gollust et al, 2017; Bogenschneider et al, 2019a; Bogenschneider et al, 2019b; Woodruff and Roberts, 2019; Bogenschneider and Bogenschneider, 2020; Rose et al, 2020; Woodruff and Roberts, 2020; Bogenschneider et al, 2021; Crowley et al, 2021). On the other hand, studies show that social relations and trust can also be a barrier to the use of research evidence for the following reasons: legislators’ tendency to trust people they know more than research evidence (Woodruff and Roberts, 2019; Woodruff and Roberts, 2020); reduced critical thinking as a result of social relations with their peers (Jewell and Bero, 2008; Jones and Louis, 2018); and researchers not communicating in a way that is appropriate to the policymaking environment (Gollust et al, 2017). Lack of mutual trust between researchers and legislators (Gollust et al, 2017), and lack of trust in the credibility and neutrality of researchers, were also seen as barriers to use (Feller et al, 1979; Jewell and Bero, 2008; Bogenschneider et al, 2019a; Bogenschneider et al, 2019b; Bogenschneider and Bogenschneider, 2020; Bogenschneider et al, 2021).
Organisational support is also an analytical theme that contains facilitating and hindering factors. On the one hand, the presence of staff who source research evidence on committees (Kenny et al, 2017; Rose et al, 2020), and an intervention combining needs assessment and linking offices of the legislature with researchers previously trained in evidence-based policymaking (Crowley et al, 2021), were found to be facilitating factors. On the other hand, the challenge for staff to select and translate research evidence (Guston et al, 2005), the lack of staff (Jewell and Bero, 2008; Kenny et al, 2017; Girling and Gibbs, 2019; Rose et al, 2020), and the lack of funding for the library or staff (Guston et al, 2005; Jewell and Bero, 2008; Kenny et al, 2017; Girling and Gibbs, 2019; Rose et al, 2020), were seen as hindrances to the use of research evidence.
Availability and accessibility are the last analytical theme identified under the category of institution and organisation. The accessibility of research evidence (Kenny et al, 2017; Geddes et al, 2017; Bogenschneider et al, 2019a; Bogenschneider et al, 2019b; Rose et al, 2020; Geddes, 2020), research evidence that is available and accessible when a window of opportunity opens (Purtle et al, 2016; Kenny et al, 2017; Mosley and Gibson, 2017; Geddes et al, 2017; Rose et al, 2020; Geddes, 2020), and capacity of information provider to rapidly respond as issues arose (Jewell and Bero, 2008), were seen as facilitators of use. Lack of timeliness and rarely receiving unsolicited research evidence (Kenny et al, 2017; Rose et al, 2020), as well as lack of access or subscription to databases (Guston et al, 2005; Dodson et al, 2015; Dodson et al, 2015; Bogenschneider et al, 2019a; Bogenschneider and Bogenschneider, 2020), were seen as hindrances to the use of research evidence.
Research characteristics
Research characteristics are a well-known category of factors in the field of study. We have identified two analytical themes under this category, namely form and content. Let us look at the descriptive themes under these two analytical themes. Firstly, in terms of form, included studies only report findings on facilitating factors (see Figure 4). More precisely, research evidence that is communicated in a concise, clear and understandable way for non-specialists (Guston et al, 2005; Brownson et al, 2016; Purtle et al, 2016; Kenny et al, 2017; Geddes et al, 2017; Purtle et al, 2018; Rose et al, 2020), and packaged to highlight how it affects specific groups or individuals (Jewell and Bero, 2008), were seen as facilitators of evidence use in legislators.
On the content side, some of the included studies found facilitating factors and/or barriers. The inclusion of recommendations for action (Brownson et al, 2016; Purtle et al, 2016), statistics (Brownson et al, 2011; Kenny et al, 2017; Geddes et al, 2017; Rose et al, 2020), including those collected in a routine and standardised manner (Gerrits et al, 2019), cost-benefit analysis (Jewell and Bero, 2008; Brownson et al, 2016; Purtle et al, 2016; Purtle et al, 2018), and budgetary impact analysis (Purtle et al, 2018), as well as case studies and international comparative analysis (Kenny et al, 2017; Rose et al, 2020), were seen as facilitating the use of research evidence in legislatures. Research evidence linked with stories or anecdotes was also seen as a facilitator (Mosley and Gibson, 2017; Geddes et al, 2017; Woodruff and Roberts, 2019; Woodruff and Roberts, 2020; Geddes, 2020).
Concerning hindrances related to research content, research that is not up to date (Dodson et al, 2015), or that includes qualitative verbose (Kenny et al, 2017), was seen as a hindrance to use. Evidence use was also found to be hindered by findings that do not align with conclusions of other sources (Kenny et al, 2017; Girling and Gibbs, 2019; Bogenschneider and Bogenschneider, 2020; Rose et al, 2020), and with the constituents’ needs or feelings (Girling and Gibbs, 2019), are not accepted by a lobby trusted by the legislator (Guston et al, 2005) or that proves legislators wrong (Kenny et al, 2017; Rose et al, 2020).
Policy and political context
The context of policymaking and the political dimension of legislative work also affect the use of research evidence. Three analytical themes fall under this category: issue characteristics, interest groups, and the constraints of the legislator’s work. Issue characteristics concern facilitators and barriers, whereas the other two themes were only seen as hindrances (see Figure 4). Let us now examine the descriptive themes that fall under each of these three analytical themes, beginning with issue characteristics.
The unfamiliarity of the legislator with the issue or the fact that they do not yet have a clear position makes the use of research evidence more likely (Asen and Gent, 2019; Bogenschneider and Bogenschneider, 2020; Rose et al, 2020). The same is true when the issue is important to legislators or their constituents (Brownson et al, 2016; Purtle et al, 2016; Rose et al, 2020). The fact that an issue resonates with existing evidence can also facilitate the use of research evidence (Jewell and Bero, 2008). On the other hand, characteristics of the issues seem to hinder the use of research evidence in legislatures. This is the case for issues that are visible and high-profile so that they capture constituent, media or lobbying attention (Asen and Gent, 2019), polarising and controversial issues that solicit passions (Jewell and Bero, 2008; Kenny et al, 2017; Asen and Gent, 2019; Bogenschneider et al, 2019a), or well-known issues that provide legislators with default position (Asen and Gent, 2019). The use of research evidence was also found less likely when issues are defined by narratives that are not linked to research evidence but capture the attention of policymakers (Mosley and Gibson, 2017).
With regard to the analytical theme of interest groups, some studies found that the actions of interest groups are seen as undermining the use of research evidence by legislators. Specifically, legislators reported that interest groups might disseminate poor-quality evidence, promote poorly-designed studies, or attack research evidence (Jewell and Bero, 2008). Consultants and interest groups also excel at tackling targeted questions on a truncated timeline in a way that meets the needs of legislators (Jones and Louis, 2018; Rose et al, 2020).
Finally, hindrances concerning the constraints inherent in the work of legislators (third analytical theme) have been observed in several studies. These are workload, lack of time and pressure to make quick decisions (Guston et al, 2005; Jewell and Bero, 2008; Dodson et al, 2015; Gollust et al, 2017; Jou et al, 2018; Girling and Gibbs, 2019; Bogenschneider and Bogenschneider, 2020; Rose et al, 2020), information overload (Jewell and Bero, 2008; Dodson et al, 2015; Gollust et al, 2017; Girling and Gibbs, 2019; Rose et al, 2020), and the fact that policymaking is an activity that involves other elements than evidence and especially competing goals (Guston et al, 2005; Jewell and Bero, 2008; Jones and Louis, 2018; Girling and Gibbs, 2019).
Individual characteristics
The last category of factors from the thematic synthesis is the individual characteristics of legislators. The results of this category have been grouped under two analytical themes, namely experience, knowledge and skills and attitude or beliefs towards research evidence. Some individual characteristics were seen as facilitators, others as barriers. As we have done with the previous three categories of factors, we will now examine the descriptive themes that fall under these two analytical themes starting with experience, knowledge and skills.
Under the analytical theme of experience, knowledge and skills, facilitating factors include the level of experience of members of legislature in knowing how research can be used during oral sessions (Kenny et al, 2017), the expertise of legislators and their staff (Kenny et al, 2017), and knowing how to investigate basic research quality issues (Jewell and Bero, 2008). On the side of the barriers, there is a poor understanding of research, including the difficulty of distinguishing research evidence from other types of information (Feller et al, 1979; Jewell and Bero, 2008; Geddes et al, 2017; Girling and Gibbs, 2019; Rose et al, 2020), the difficulty of locating sources and searching for research evidence (Dodson et al, 2015; Jou et al, 2018; Girling and Gibbs, 2019; Rose et al, 2020), and difficulty of discerning what is valid (Guston et al, 2005; Jewell and Bero, 2008; Girling and Gibbs, 2019).
The attitudes or beliefs of legislators toward research evidence have been a recurring theme in the literature for many years. This analytical theme has both enablers and hindrances. Positive attitudes and beliefs towards research evidence predispose legislators to use research evidence. These favourable predispositions are the perception that research evidence meets the concerns, needs or priorities of the constituents (Brownson et al, 2016; Purtle et al, 2016) and the legislators or their staff (Purtle et al, 2016; Kenny et al, 2017; Mosley and Gibson, 2017), is credible (Rose et al, 2020), allows new ways of thinking (Bogenschneider et al, 2013), and can lead to practical applications (Geddes et al, 2017). About attitudinal predispositions that hinder the use of research evidence, studies found a lack of interest in research evidence (Guston et al, 2005; Jewell and Bero, 2008), preference for anecdotes over research evidence (Jewell and Bero, 2008; Woodruff and Roberts, 2019; Woodruff and Roberts, 2020), a perception that research is biased, lacks credibility and neutrality (Feller et al, 1979; Dodson et al, 2015; Gollust et al, 2017; Geddes et al, 2017; Girling and Gibbs, 2019; Bogenschneider and Bogenschneider, 2020; Woodruff and Roberts, 2020; Bogenschneider et al, 2021), has limited practical implications or answers the wrong questions (Feller et al, 1979; Guston et al, 2005; Dodson et al, 2015; Kenny et al, 2017; Geddes et al, 2017), and that evidence is too uncertain (Guston et al, 2005; Geddes et al, 2017).
How does this differ from the executive branch?
A few studies have compared evidence use between the legislative and executive branches. A cross-sectional study conducted in the US shows significantly more heavy users of evidence among public servants than among legislators or their staff. It concludes that civil servants have to make more efforts to find research evidence (Bogenschneider et al, 2013). As for the type of use, Jones and Louis (2018: 386) found that ‘politicians were more often described as seeking evidence to support the conclusions they had already come to, whereas bureaucrats were described as seeking evidence to inform what decision to make’.
One of the two experimental studies included in the review assessed the effect of the type of data contained in a policy brief on the use of the brief in three distinct populations, namely legislators, legislative staff and executives in the executive branch (Brownson et al, 2011). This study uses breast cancer prevention (mammography) as a use case. The experimental manipulation consisted of testing different brief versions, including data vs stories and local vs state information. The results show higher use of the brief among legislators than among legislative staff and executives. Furthermore, legislators preferred briefs with statistics to those with stories. This finding could be explained by the fact that one of the facilitators of use is research evidence including statistics (Kenny et al, 2017; Geddes et al, 2017; Rose et al, 2020), while qualitative verbose was seen as an obstacle (Kenny et al, 2017). Another interesting finding was that the more educated legislative staff used the brief less frequently than those without a graduate degree. The authors suggest that this could be explained by the greater familiarity of the more educated staff with the issue. This interpretation converges with the findings from our thematic synthesis, showing that familiarity with the issue was seen as a hindrance to evidence use (Asen and Gent, 2019).
Finally, a qualitative study of legislators, executive officials and actors with advocacy roles in the state of Minnesota found that while legislators appear to be more concerned about lack of time and resources, the public servants interviewed saw communication and networking problems as more critical challenges (Jou et al, 2018).
Discussion and conclusions
This systematic review aimed to identify and describe research about the type of use and the barriers and facilitators of using research evidence in the legislative branch. This policymaking environment is by nature more politicised than the public service serving the executive branch of the government. It found that of the three types of use well documented in the literature, symbolic or tactical use is the one that has been observed in most studies, followed by conceptual and instrumental use. The preponderance of symbolic or tactical use could be explained by the very nature of legislative work, where adversarial debate plays an important role. The review captured other types of use exclusively applicable to legislatures, notably to prepare for questions and debates in the House and committees and to help build consensus.
The review also identified several categories of factors grouped into analytical and descriptive themes. Although the results of our review are not directly comparable with those of previous ones (for example, Oliver et al, 2014), one similarity is the importance of factors such as availability and access, organisational support, research characteristics (form and content), social relationships and trust, the attitude of policymakers towards research evidence, and knowledge and skills. These factors were found to play a role in explaining research use in legislatures. On the other hand, our review found factors that do not seem to emerge as clearly from non-legislature-specific reviews. This is the case for factors related to the characteristics of legislatures (that is, organisation of committee work, role of the library) and for factors related to the policy and political context, particularly issue characteristics and interest groups. Concerning interest groups, studies captured in our review show that their actions can undermine the use of research evidence, notably through their ability to gain the trust of legislators and provide biased or incomplete information in a timely manner.
Most studies were based on interviews or surveys of legislators or their staff. Qualitative interviews were the dominant method of data collection. It is understandable and possibly a good thing, as interviews allow factors to emerge naturally from legislators’ mouths. In contrast, in surveys with closed lists of factors, the researchers somehow dictate the factors listed and thus the results. Few studies have used content analysis of legislative proceedings in chambers or committees, which allows for an objective examination of how legislators mobilise research evidence publicly in debates. Moreover, identifying two randomised controlled trials proves that it is possible to run experiments in legislatures.
The precision of the selection criteria combined with a professional database search strategy means that the review can be considered reasonably comprehensive. Although the review did not follow a pre-published protocol, a protocol was used to guide the review process. Furthermore, all review process steps were carried out with double or triple coders. The main limitations of the review are the less systematic and structured use of databases to identify grey literature (for example, Google), and the fact that we did not survey authors in the field to identify references. Another limitation is that we did not analyse the risk of bias in the included studies. Like Oliver et al (2014), we aimed to identify and categorise factors rather than synthesise measurements of causal relationships between variables.
We would have wished to examine the presence of patterns in the results by type of legislature. It was impossible to do so, as all the included studies were conducted in bicameral legislatures. It would be interesting for future research to look at differences in usage patterns and factors between the lower and upper chambers (House of Representatives and Senate). Furthermore, the imbalance in the number of studies carried out under presidential vs parliamentary systems is such that it was impossible to examine differences in the type of use and factors between the two forms of government. This research niche is still predominantly American. We did not find any studies in French-speaking legislatures that met the selection criteria, although we searched French-language databases.
Identifying and synthesising ideas for optimising the use of research evidence in legislatures was not within the scope of the review. Nevertheless, the review findings can provide some avenues to explore. Legislators are very busy, short of time, and have to inform themselves and make decisions on short notice according to the day’s issues. They tend to trust and rely on intra-legislative non-partisan support units such as the library and committee staff. As one of the common obstacles is the perception that research evidence is biased, uncertain or lacking in neutrality, strengthening the organisational capacity of legislatures to collect, evaluate and synthesise research evidence in a non-partisan way is likely to be a promising way forward. Equipping legislatures with a neutral strike force for rapid assessment and dissemination of research evidence would give legislators a means of validating the quality of some of the information they are bombarded with, notably by interest groups and through the mass media.
Research ethics statement
The authors of this paper have declared that research ethics approval was not required since the paper does not present or draw directly on data/findings from empirical research.
Funding
This work was supported by the Fonds de recherche du Québec (3 sectors) under Grant No. GQ130925.
Contributor statement
MO, EM, SJ, FG and MB conceptualised and designed the study protocol. SR developed and implemented the literature search. MB coordinated the selection and extraction processes. MB, AC and AC screened the papers on title and abstract and on full text, performed the branching, and data extraction. MO double-checked selection and extraction. MO and MB produced the thematic synthesis which was then reviewed and approved by EM, SJ, FG, AC and AC. MO wrote the manuscript. MB, EM, SJ, FG, AC, AC and SR reviewed and approved the manuscript.
Acknowledgements
We thank the anonymous reviewers for the quality and professionalism of their work. Their comments and suggestions have helped to improve the manuscript.
Supplemental data
Conflict of interest
The authors declare that there is no conflict of interest.
References
Amara, N., Ouimet, M. and Landry, R. (2004) New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies, Science Communication, 26(1): 75–106. doi: 10.1177/1075547004267491
Asen, R. and Gent, W. (2019) Reconsidering symbolic use: a situational model of the use of research evidence in polarised legislative hearings, Evidence & Policy, 15(4): 525–41. doi: 10.1332/174426418X15378681033440
Balzacq, T., Baudewyns, P., Jamin, J., Legrand, V., Paye, O. and Schiffino, N. (2014) Fondements de Science Politique, Louvain-la-Neuve: De Boeck.
Bogenschneider, K. and Bogenschneider, B.N. (2020) Empirical evidence from state legislators: how, when, and who uses research, Psychology, Public Policy, and Law, 26(4): 413–24. doi: 10.1037/law0000232
Bogenschneider, K., Little, O.M. and Johnson, K. (2013) Policymakers’ use of social science research: looking within and across policy actors, Journal of Marriage and Family, 75(2): 263–75. doi: 10.1111/jomf.12009
Bogenschneider, K., Corbett, T.J. and Parrott, E. (2019a) Realizing the promise of research in policymaking: theoretical guidance grounded in policymaker perspectives, Journal of Family Theory & Review, 11(1): 127–47. doi: 10.1111/jftr.12310
Bogenschneider, K., Day, E. and Parrott, E. (2019b) Revisiting theory on research use: turning to policymakers for fresh insights, American Psychologist, 74(7). doi: 10.1037/amp0000460
Bogenschneider, K., Day, E. and Bogenschneider, B.N. (2021) A window into youth and family policy: state policymaker views on polarization and research utilization, American Psychologist, 76(7): 1143–58. doi: 10.1037/amp0000681
Brownson, R.C., Dodson, E.A., Stamatakis, K.A., Casey, C.M., Elliott, M.B., Luke, D.A., Wintrode, C.G. and Kreuter, M.W. (2011) Communicating evidence-based information on cancer prevention to state-level policy makers, Journal of the National Cancer Institute, 103(4): 306–16. doi: 10.1093/jnci/djq529
Brownson, R.C., Dodson, E.A., Kerner, J.F. and Moreland-Russell, S. (2016) Framing research for state policymakers who place a priority on cancer, Cancer Causes & Control, 27(8): 1035–41. doi: 10.1007/s10552-016-0771-0
Cairney, P. (2016) The Politics of Evidence-Based Policy Making, Basingstoke: Palgrave MacMillan.
Crowley, D.M., Taylor Scott, J., Long, E.C. and Giray, C. (2021) Lawmakers’ use of scientific evidence can be improved, Proceedings of the National Academy of Sciences, 118(9): e2012955118. doi: 10.1073/pnas.2012955118
Dodson, E.A., Geary, N.A. and Brownson, R.C. (2015) State legislators’ sources and use of information: bridging the gap between research and policy, Health Education Research, 30(6): 840–48. doi: 10.1093/her/cyv044
Feller, I., King, M.R., Menzel, D.C., O’Connor, R.E., Wissel, P.A. and Ingersoll, T. (1979) Scientific and technological information in state legislatures, American Behavioral Scientist, 22(3): 417–36. doi: 10.1177/00027642790220
Geddes, M. (2020) The webs of belief around ‘evidence’ in legislatures: the case of select committees in the UK House of Commons, Public Administration, 99(1): 40–54. doi: 10.1111/padm.12687
Geddes, M., Dommett, K. and Prosser, B. (2017) A recipe for impact? Exploring knowledge requirements in the UK Parliament and beyond, Evidence & Policy, 14(2): 259–76. doi: 10.1332/174426417X14945838375115
Gerrits, R.G., van den Berg, M.J., Klazinga, N.S. and Kringos, D.S. (2019) Statistics in Dutch policy debates on health and healthcare, Health Research Policy and Systems, 17: 55. doi: 10.1186/s12961-019-0461-y
Girling, K. and Gibbs, K. (2019) An Analysis of Information Gathering and Use By Canadian Parliamentarians, Ottawa: Evidence for Democracy.
Gollust, S.E., Seymour, J.W., Pany, M.J., Goss, A., Meisel, Z.F. and Grande, D. (2017) Mutual distrust: perspectives from researchers and policy makers on the research to policy gap in 2013 and recommendations for the future, INQUIRY:The Journal of Health Care Organization Provision and Financing, 54. doi: 10.1177/0046958017705465
Gough, D., Oliver, S. and Thomas, J. (2012) An Introduction to Systematic Reviews, Singapore, Washington, DC, New York, NY, Thousand Oaks, CA, New Delhi: Sage Publications.
Guston, D.H., Jones, M. and Branscomb, L.M. (2005) The demand for and supply of technical information and analysis in state legislatures, Policy Studies Journal, 25(3): 451–69. doi: 10.1111/j.1541-0072.1997.tb00034.x
Jewell, C.J. and Bero, L.A. (2008) ‘Developing good taste in evidence’: facilitators of and hindrances to Evidence-informed health policymaking in state government, Milbank Quarterly, 86(2): 177–208. doi: 10.1111/j.1468-0009.2008.00519.x.
Jones, D.K. and Louis, C.J. (2018) Using evidence to inform state health policy making: lessons from four states comparing Obamacare and infant mortality, Journal of Health Politics Policy and Law, 43(3): 377–99. doi: 10.1215/03616878-4366148
Jou, J., Nanney, M.S., Walker, E., Callanan, R., Weisman, S. and Gollust, S. (2018) Using obesity research to shape obesity policy in minnesota: stakeholder insights and feasibility of recommendations, Journal of Public Health Management and Practice, 24(3): 195–203. doi: 10.1097/PHH.0000000000000637
Kenny, C., Rose, D.C., Hobbs, A., Tyler, C. and Blackstock, J. (2017) The Role of Research in the UK Parliament, London: Houses of Parliament.
Landry, R., Lamari, M. and Amara, N. (2003) The extent and determinants of the utilization of university research in government agencies, Public Administration Review, 63(2): 192–205. doi: 10.1111/1540-6210.00279
Mosley, J.E. and Gibson, K. (2017) Strategic use of evidence in State-level policymaking: matching evidence type to legislative stage, Policy Sciences, 50(4): 697–719. doi: 10.1007/s11077-017-9289-x
Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 1–26. doi: 10.1186/1472-6963-14-2
Purtle, J., Dodson, E.A. and Brownson, R.C. (2016) Uses of research evidence by state legislators who prioritize behavioral health issues, Psychiatric Services, 67(12): 1355–61. doi: 10.1176/appi.ps.201500443
Purtle, J., Lê-Scherban, F., Xi Wang, Shattuck, P.T., Proctor, E.K. and Brownson, R.C. (2018) Audience segmentation to disseminate behavioral health evidence to legislators: an empirical clustering analysis, Implementation Science, 13: 121. doi: 10.1186/s13012-018-0816-8
Rose, D.C., Kenny, C., Hobbs, A. and Tyler, C. (2020) Improving the use of evidence in legislatures: the case of the UK parliament, Evidence & Policy, 16(4): 619–38. doi: 10.1332/174426420X15828100394351
Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 426–31. doi: 10.2307/3109916
Woodruff, K. and Roberts, S.C.M. (2019) ‘Alcohol during pregnancy? Nobody does that anymore’: state legislator’s use of evidence in making policy on alcohol use in pregnancy, Journal of Studies on Alcohol and Drugs, 80(3): 380–88. doi: 10.15288/jsad.2019.80.380
Woodruff, K. and Roberts, S.C.M. (2020) ‘My good friends on the other side of the aisle aren’t bothered by those facts’: US State legislators’ use of evidence in making policy on abortion, Contraception, 101(4): 249–55. doi: 10.1016/j.contraception.2019.11.009
