How are evidence and policy conceptualised, and how do they connect? A qualitative systematic review of public policy literature

View author details View Less
  • 1 University of Hagen, , Germany
  • | 2 Leiden University, , Netherlands
Full Access
Get eTOC alerts
Rights and permissions Cite this article

Background:

While current public policy scholarship can take advantage of a decades-long accumulated knowledge base on the relationship between evidence and policy, it is hard to keep the overview across different literatures. Over time, the ever more differentiated branches of public policy research have developed their own perspectives, languages, and conceptualisations of ‘evidence’ and ‘policy’, as well as their connections.

Aims and objectives:

Existing reviews have stressed that studies often do not provide clear definitions of ‘policy’ or ‘evidence’, and have outlined the importance of investigating underlying conceptualisations in the literature. Against this backdrop, this article investigates how present-day public policy scholarship approaches the concepts of ‘evidence’, ‘policy’, and their connections.

Methods:

We conducted a qualitative systematic review following the PRISMA method. Using a keyword search, we identified relevant articles (n=85) in eleven Q1 and Q2 policy journals included in Web of Science in the period 2015 to 2019.

Findings:

The synthesis confirms that ‘evidence’ and ‘policy’ are often not clearly defined, yet different trends regarding understandings can be identified. There are two approaches taken on the evidence and policy connection: a ‘use of evidence’ or a ‘use for policy’ perspective.

Discussion and conclusions:

Research on evidence and policy would benefit from more explicit conceptual discussions. This review may provide a heuristic for explicating conceptual choices when working with the notions of ‘evidence’, ‘policy’, and their connections. It also suggests several avenues that are worth exploring in future research.

Abstract

Background:

While current public policy scholarship can take advantage of a decades-long accumulated knowledge base on the relationship between evidence and policy, it is hard to keep the overview across different literatures. Over time, the ever more differentiated branches of public policy research have developed their own perspectives, languages, and conceptualisations of ‘evidence’ and ‘policy’, as well as their connections.

Aims and objectives:

Existing reviews have stressed that studies often do not provide clear definitions of ‘policy’ or ‘evidence’, and have outlined the importance of investigating underlying conceptualisations in the literature. Against this backdrop, this article investigates how present-day public policy scholarship approaches the concepts of ‘evidence’, ‘policy’, and their connections.

Methods:

We conducted a qualitative systematic review following the PRISMA method. Using a keyword search, we identified relevant articles (n=85) in eleven Q1 and Q2 policy journals included in Web of Science in the period 2015 to 2019.

Findings:

The synthesis confirms that ‘evidence’ and ‘policy’ are often not clearly defined, yet different trends regarding understandings can be identified. There are two approaches taken on the evidence and policy connection: a ‘use of evidence’ or a ‘use for policy’ perspective.

Discussion and conclusions:

Research on evidence and policy would benefit from more explicit conceptual discussions. This review may provide a heuristic for explicating conceptual choices when working with the notions of ‘evidence’, ‘policy’, and their connections. It also suggests several avenues that are worth exploring in future research.

Key messages

  • Review studies of evidence and policy research have stressed the need of investigating underlying conceptualisations.

  • This article presents the results of a qualitative systematic PRISMA review.

  • The synthesis reveals significant differences in the conceptualisations of evidence and policy.

  • Their connections are approached either from a ‘use of evidence’ or a ‘use for policy’ perspective.

Introduction: the conceptual challenges in the evidence and policy literature

Public policy scholarship has puzzled over the evidence and policy relationship for decades (see, for example, Caplan, 1979; Weiss, 1979; Gieryn, 1983; Lindquist, 1990; Landry et al, 2001). While it can thus take advantage of a long-accumulated knowledge base, it is hard to keep the overview across different literatures dealing with this relationship, such as the evidence-based policymaking (EBPM) literature (for example, Nutley et al, 2009), the literature on policy-advisory systems (for example, Halligan, 1995), or the more constructivist and critical literature on evidence and policy (for example, Freeman, 2010). Over time, ever more differentiated branches of public policy research have developed, which complement and enrich each other. However, they have also developed their own perspectives, languages, and understandings of ‘evidence’, ‘policy’, and their connections. Attentiveness to careful conceptualisations is warranted, as they are important vehicles for advances in theory development and empirical research. They help in setting clear boundaries around what is and is not being studied, help to avoid misunderstandings, and can be useful guides for robust operationalisations (PytlikZillig and Kimbrough, 2016). Also practitioners (for example, policymakers or knowledge brokers) interested in the evidence and policy relationship may benefit from clear conceptualisations, which can help them to keep a clearer overview of existing literature. This article traces different conceptualisations of ‘evidence’, ‘policy’, and their connections through a qualitative systematic review.

The literature on evidence use in policy already includes various systematic meta-reviews which aim to identify more general lessons across contexts. Such systematic reviews on evidence use have been particularly widespread in the field of public health (Innvaer et al, 2002; Liverani et al, 2013; Oliver et al, 2014), mostly using the PRISMA methodology (Preferred Reporting Items for Systematic Reviews and Meta-Analysis). While the main focus of these studies (for example, Innvaer et al, 2002; updated by Oliver et al, 2014) has been on identifying determinants for effective evidence utilisation, Oliver et al (2014: 1) also highlight that only ‘few studies provide clear definitions of policy, evidence or policymaker’. Oliver’s summary stresses that the differentiation and heterogeneity, demonstrated for the ‘evidence’ end of the evidence and policy relationship, equally applies to its ‘policy’ end. Also in other policy fields, literature reviews on evidence-based policy can be identified (for example, Nutley et al, 2009; Strydom et al, 2010), albeit not following the systematic approach of the PRISMA method. Other reviews have taken a more normative stance. Based on a rigorous review of 86 academic and ‘grey’ publications, Oliver and Cairney (2019) and Cairney and Oliver (2020), for instance, studied ‘how to influence policy’. Yet another type of review study questioned the policy potential of evidence-based policy. Through an extensive analysis of nearly 400 publications, French (2019) identified major schools of thought which differ in their perspective on evidence-based policy.

While existing review studies have thus continuously advanced the field, they have rarely followed an explicit conceptual interest, for example, mapping and asking what is understood as ‘evidence’ or ‘policy’ in the first place. Including different conceptual interpretations of those notions in a literature review is not problematic per se, but one cannot rule out the possibility that reviews would reach other conclusions if they would exclusively consider a particular type of evidence, or policy, for instance. Most recently, conceptual – not always systematic – review studies have been published about particular notions or actors carrying evidence, such as experts and expert knowledge (Christensen, 2021; Neal et al, 2021). These studies indicate the relevance of providing a systematic overview of how different conceptual manifestations of ‘evidence’ and ‘policy’ are tied together. Also Cairney (2016) highlighted the importance of clarity about what evidence and policy exactly mean. To put it in his words (2016: 2): ‘the term “evidence-based policymaking” or EBPM has a “bewitching” effect and seems like a valence issue: who could not want it? Yet, we cannot want something if we do not really know what it means, or what each word means’. His concise summary with examples of possible meanings of each of these words (Cairney, 2016: 2–4) indicates a diversity of conceptualisations circulating, but a systematic overview is (to the best of our knowledge) not yet available.

It is precisely here that our qualitative systematic review sets in: its main aim is to investigate how present-day public policy scholarship approaches the concepts of ‘evidence’, ‘policy’, and their connections. Gerring’s (1999: 357–8) work tells us that conceptualisation could include different aspects of a concept (that is, the phenomenon to be defined, its properties or attitudes, as well as a respective term) – but also that we may not necessarily expect engagement with all those aspects. Nevertheless, even if unwittingly and implicitly, all ‘authors make lexical and semantic choices as they write and thus participate… in an ongoing interpretive battle’ (Gerring, 1999). The review can thus contribute to a critical reflection of terminology that is at times taken for granted (for example, evidence, policy). Shedding light on different conceptualisations is not just a semantic discussion. Instead, having a clear understanding of concepts used can help bridging across fragmented literatures, and further advance cumulative theory development and empirical research on the use of evidence in public policy. Precise conceptualisations also foster valid measurement (Eagly and Chaiken, 2007: 583).

To gain a better understanding of the meanings of evidence, policy, and their connections, we focus on recent public policy literature on evidence use. With the study of policy in all its dimensions and various manifestations being the key focus of public policy scholars, we expect most varied and differentiated uses of those terms here, which can then contribute to other fields of the interdisciplinary evidence and policy scholarship. As such, we respond to calls to better capture lessons being learned within and across different evidence and policy-related disciplines (Oliver and Boaz, 2019). While our review confirmed the existence of a plethora of approaches to evidence and policy without even a commonly shared ‘essence’ across the article corpus, scholars may draw inspiration from the overview of conceptualisations we identified. As we will show, for instance, studies focusing on ‘use of evidence’ can benefit from more precise conceptualisations of policy from ‘use for policy’ studies, and vice versa. The list of conceptualisations resulting from our review (see Table 2) can as such also guide scholars in explicating conceptual choices.

By using the PRISMA methodology, we synthesise the 2015–2019 public policy literature on evidence use (n=85) published in eleven Q1 and Q2 policy journals indexed in Web of Science. Although we do not capture longitudinal trends, we expect this recent literature to represent the ‘sediments’ accumulated by the decade-long theoretical exchanges on evidence and policy. As mentioned, Oliver et al’s (2014) review also highlighted the importance of conceptual questions for the field, but called for reviewing those more systematically. Our review thus follows this call and extends to the more recent time period; with its focus on public policy journals following Cairney’s (2016) call for acknowledging policy theories when dealing with evidence and policy issues.

Our analysis proceeds in three steps. First, we identify the main attributes of the public policy community currently active in this field, in terms of geographical scope, and substantive policy areas. This helps to better understand the background in which different conceptual approaches are taken. Second, we explore the conceptual puzzle in depth, and investigate different understandings of ‘evidence’, and ‘policy’. Third, we present what the synthesis revealed about the connections between the two, and how these connections are conceptualised and theoretically approached.

The next section details the methodology, the article corpus, and the analytical categories of this qualitative systematic review. Following this we characterise the article corpus and identify trends. We then present the results of our analysis, before finally drawing some general conclusions and several specific directions for further research.

Methodology

There are many different forms of literature reviews, from which the method of a qualitative systematic review seems most fitting for our purposes. It allows for seeking themes ‘that lie in or across individual qualitative studies’ (Grant and Booth, 2009: 94), comparing and integrating those studies’ findings. Through their systematic character, they have the advantage to give a transparent account for inclusion or exclusion of certain parts of the literature. Concretely, we use the PRISMA methodology (Moher et al, 2009). This choice is further justified through the observation that systematic (PRISMA) reviews have increasingly exceeded the area of public health research, and been successfully applied in public policy scholarship (see, for example, van der Heijden and Kuhlmann, 2017; Moyson et al, 2017; Verweij and Trell, 2019; MacKillop et al, 2020; Acciai and Capano, 2021; Riche et al, 2021).

In qualitative systematic reviews, the evidence synthesis is presented in a qualitative, narrative style. This offers greater depth of analysis, but also goes along with a smaller number of possibly-included publications (also acknowledging research-pragmatic considerations of resource limitations). A central question was thus how to meaningfully select the article corpus, given the broad nature of the evidence and policy literature. Considering that our main purpose is to identify conceptual understandings, and the highly differentiated nature of the policy literature and its journals, we decided to delimit the corpus in time rather than in scope. Against this backdrop, articles published over five recent years (2015 to 2019) were included. While admittedly the review could thus miss key articles of present-day evidence and policy literature, we believe that an investigation over five years still allows us to detect the presence of distinct conceptual understandings and patterns (and is in that sense representative also for a longer time period).

To arrive at the sample of included studies, qualitative systematic reviews may involve selective or purposive sampling. For our systematic search of peer-reviewed publications, we decided for a term search in the Web of Science Database; no additional records were identified through other sources (see Figure 1). There is no policy category in Web of Science (nor in Scopus), therefore the included journals build on the ‘public administration’ category. As we are interested in the core public policy literature on evidence use, all Q1 and Q2 were chosen which have ‘policy’ explicitly mentioned in their main title.1 This led to 11 policy journals included in Web of Science (20182), namely:

  • Climate Policy

  • Policy Sciences

  • Policy Studies Journal (PSJ)

  • Journal of Policy Analysis and Management

  • Journal of European Public Policy (JEPP)

  • Journal of Accounting and Public Policy

  • Journal of Comparative Policy Analysis (JCPA)

  • Journal of European Social Policy (JESP)

  • Journal of Social Policy

  • Policy & Politics, as well as

  • Policy and Society.

The PRISMA flow chart which portrays the steps through which we arrived at the studies included in the qualitative synthesis.
Figure 1:

PRISMA flow chart

Citation: Evidence & Policy 2022; 10.1332/174426421X16397411532296

This list allows coverage of both more general policy journals as well as two journals exclusively dealing with distinct policy sectors (climate/environmental policy, as well as social policy). To be sure, these were the only journals with such substantive focus that the Public Administration category (Q1 and Q2) included. While sharing the interest in policy, these journals also cover different branches of public policy research. We decided to include only English-language publications. Our analysis builds on previous systematic literature review studies, but does not include them in the article corpus, as they do not always allow retrieval of the original conceptualisations used in the underlying studies. Nor did our review include generic introductions of special issues. Conceptual articles instead were explicitly kept in the analysis and coded in the same way as empirical contributions.3 While journals in other Web of Science categories may also occasionally publish policy research, we believe that this approach allows us to have a reliable picture of different conceptualisations characterising present-day public policy scholarship.

In the next step, the keywords for Web of Science’s Advanced Search function were determined. As the common denominator of the articles at interest for this qualitative systematic review is an interest in ‘evidence’ (and related concepts), this is how the keyword search was approached. Deductively following notions of evidence and ‘expert knowledge’ that have been identified for different literature strands (see, for example, Christensen, 2021, Table 1), the search terms covered ‘evidence’, ‘expert/expertise’, ‘research’, ‘knowledge’, ‘advice/advisory’, and/or ‘epistemic/epistemology’. Those terms were scanned via topic search (TS) in Web of Science, meaning that they needed to show in title, abstract, or keywords of the article.4

In the first round of identification (Figure 1), the Web of Science search yielded 1,082 records.5 The topics (title, abstracts, keywords) of those records were saved and proceeded into the next round of screening. In this round, records were screened for matching our inclusion criteria. We decided to include a record if it met the basic definition of addressing some sort of ‘evidence’ for example, knowledge, scientific advice and so on), some sort of ‘policy’ (for example, legal regulation, policy idea and so on), and some sort of connection between the two (for example, use, uptake). This meant that, for instance, those records were excluded which addressed certain theories or theoretical frameworks whose elements are prone to investigating evidence use (for example, policy transfer, policy learning), where the three elements named above were not explicitly addressed. Yet the criteria were applied generously at the screening stage, and when in doubt the record was moved to the next stage. To ensure inter-coder reliability, the screening round proceeded in several steps. First, Authors 1 and 2 scanned the same 30 records separately (the first six published in each of the five years); and then discussed the choice. This step also helped fine-tuning the inclusion/exclusion criteria. Second, the rest of the records were divided equally, scanned separately, again followed by discussing the choice (especially doubtful cases) and inclusion/exclusion criteria.

Overall, 130 records proceeded to the next stage of checking eligibility of the full text articles. To begin with, a first version of a coding scheme was developed collaboratively using Microsoft Excel. To refine this coding scheme and to again take measures to ensure inter-coder reliability, Authors 1 and 2 coded the first ten full-text records, then exchanged on this and revised the coding scheme into its final version, which contained the categories summarised in Table 1. Afterwards, the remaining full-text records were again equally divided between Authors 1 and 2, fully read and coded separately, and results as well as critical cases were discussed. With this, we arrived at a number of 85 records included in the synthesis. This decision on the final article corpus was taken on 11 May 2021.

Table 1:

Summary of coding scheme

ContentCategories
General information
  • Article ID

  • Author(s)

  • Title

  • Journal

  • Year of publication

  • Country and affiliation of first author

  • Keywords

Research design
  • Type: empirical; conceptual

  • Substantive policy field8 (based on categories from Comparative Agendas Project)

  • Geographical scope: country/countries; supra-/international level

  • Methodology: qualitative; quantitative; mixed methods

  • Data collection and data analysis: interviews; document analysis; observation; survey; statistical data analysis; other (specific) method

  • Research question

  • ‘Evidence’ as independent, dependent variable, or both

Concepts, analysis, findings (see Supplemental data provided at: https://doi.org/10.6084/m9.figshare.17294207)
  • Understanding of ‘evidence’ + conceptualisation

  • Understanding of ‘policy’ + conceptualisation

  • Understanding of ‘connection’ + conceptualisation

  • Main theoretical framework / literature used

For classifying the conceptualisations we followed an inductive approach, corresponding to our ‘open’ question of which conceptualisations emerge from the public policy literature on evidence and policy. This is with the exception of the ‘policy’ classification, where we also rely on existing literature to distinguish between policy as substance and policy as discourse (cf, for example, Bacchi, 2000; Knill and Tosun, 2020) (see below). The lines of distinction between different types of conceptualisation are in the first place to be understood as a continuum with fuzzy borders rather than clear-cut categories.6 We now turn to describe the results of this synthesis.

Sample characteristics

Overall, the number of records meeting our – somewhat narrower – inclusion criteria, and their spread over years and journals, shows an established interest of the public policy literature in the evidence and policy relationship. Yet a closer look also reveals clear characteristics and trends of this literature. Of the 11 journals screened, most articles were published in Policy Sciences (18) and the Journal of Comparative Policy Analysis (16) (see Table 3, Appendix), followed by Climate Policy (12) and Policy and Society (11). The substantial number of relevant articles identified in Climate Policy confirms the salience of evidence and policy research in this field. In the Journal of Social Policy, however, another field-specific journal of our sample, we did not find many articles (2).

Table 2:

Categories and foci identified in the articles

Lines of distinctionTypical terms and conceptsExample research questions
Understandings of evidence
Broad understandingknowledge, information, expertise‘Seeking a more sophisticated understanding of the know-how that animates engagement practice’ (*Escobar, 2015)
Narrow understandingscientific evidence, research, poverty indicators‘How is PISA and other evidence used in highly ideological policy debates?’ (*Schlaufer, 2016)
Supply-side orientationadvice, knowledge claims‘Explore how an epistemic community of agricultural scientists advising the European Commission reacted to the politicization of learning surrounding the use of hormones in meat production’ (*Dunlop, 2017b)
Demand-side orientation(certified) expertise‘What kind of work is policy-making, and what kinds of knowledge do public administrators draw on in practice?’ (*Maybin, 2015)
Understandings of policy
Broad focus on policyPolicy, policy change‘This article discusses the role of think tanks in the production of ideas guiding recent change in Brazil’s economic policy’ (*Ladi et al, 2019)
Specific focus on policypolicy innovations, wicked policies, policy experimentation‘This article investigates how framing processes lead to polarization in the public debate on a large infrastructure project’ (*Wolf and Van Dooren, 2017)
Policy as substancelegislative change, policy proposals, policy direction‘How are cross-national ideas and comparative analyses used by policymakers and other key stakeholders in the Turkish health reform of 2003?’ (*Agartan, 2015)
Policy as discoursepolicy discourse, framing, ideas‘The article focuses on the role of economic experts and their policy advice in how the hegemony of ‘austerian thinking’ has been achieved through the lens of think tank networks’ (*Plehwe et al, 2018)
Policy governance and institutional configurationsgovernance, policy-advisory system‘What do experiments do for governance?’ (*Voß and Simmons, 2018)
Understandings of the connection
Use-of-evidence orientationevidence production, provision of evidence, translation, uptake, science-policy interaction‘Our article aims at making a contribution to better understanding in what ways influence is exercised by expert advisory bodies and what factors shape their influence on morality policies’ (*Schiffino and Krieger, 2019)
Use-for-policy orientationuptake, influence policy, instrumental use, conceptual use, political/symbolic use, non-use‘How is expertise mobilized in the continuing development of drug policy at the national level?’ (*Nekola and Moravek, 2015)

This being said, studying the use of evidence strongly cuts across different branches of public policy research and types of problems. This is particularly apparent when analysing the substantive policy fields addressed by the authors of empirical papers (Figure 2). The field of environment (27%) clearly stands out. Also health is a field which is traditionally marked by strong interest in evidence and policy (and evidence-based policymaking in particular), which is also confirmed in our review (with 19% of the articles).

A pie chart that shows the substantive fields addressed in the empirical articles (in per cent).
Figure 2:

Substantive policy fields; source: PRISMA review (n=85)

Citation: Evidence & Policy 2022; 10.1332/174426421X16397411532296

Despite the English-exclusive focus of our search, the study of the evidence and policy link is relatively globally spread (Table 4, Appendix). Yet there is a predominance of Anglo-Saxon scholars in our sample, with 60% of the articles being published by first authors based in the UK, USA, or Australia. Another 29% is drafted by European scholars (UK excluded). Other countries are much less represented. While it would exceed the scope of the article to investigate whether authors’ conceptualisations of evidence, policy and their linkage is influenced by the setting in which they operate, this bias is striking.

Conceptualisations of evidence and policy

Even from the starting point of a fairly brief retrospect of public policy scholarship on evidence and policy, we can observe that scholars increasingly realised the difficulties of ‘speaking truth to power’ (Wildavsky, 1979) and of translating scientific evidence into practice in a direct, instrumental way (Caplan, 1979; Weiss, 1979). While moving away from a linear model of ‘knowledge transfer’ towards more interactive models, research has investigated forms of advice (Gieryn, 1983; Lindquist, 1990), advisory systems in different countries (Craft and Howlett, 2013; Pattyn et al, 2019), as well as the extent and ways in which policymakers use evidence (see, for example, Landry et al, 2001; Newman et al, 2015). Based on studies of evidence use in policy work and policy practices (Freeman, 2010), other types of evidence use have been identified in policy processes, including symbolic and political-legitimising use (Boswell, 2009). More constructivist and critical approaches towards the role of knowledge for policy also called attention to political influences on the production of evidence itself, in the sense of policy-based evidence making (Strassheim and Kettunen, 2014).

Turning to present-day public policy scholarships, this section presents the results of the qualitative systematic review on how ‘evidence’ and ‘policy’ are conceptualised therein. In the following, an asterisk * marks articles included in this review, a full list of which is provided online as supplementary data (available at: https://doi.org/10.6084/m9.figshare.17294207). The following sections are titled by ‘understandings’ rather then ‘conceptualisations’, as – confirming the findings of previous review studies (see for example, Oliver et al, 2014; Cairney, 2016) – ‘evidence’ and ‘policy’ are in many cases not clearly defined in the literature. Nevertheless, trends could be identified.

Understandings of ‘evidence’

While, in many of the articles, there are several terms mentioned (for example, ‘research’, and ‘evidence’), there is often one term that stands out (for example, being named in the title, or research question), and whose understanding is further specified by the author(s). Along which lines can these different understandings and conceptualisations be distinguished?

A first line of distinction is that the understandings range from broad to narrow approaches towards evidence. This concerns a definitory aspect as well as an empirical aspect. On the conceptual level, ‘evidence’ is defined more or less widely. On the empirical level, the perspective is set broader or more focused to include more/less different sorts of ‘evidence’.

Conceptually, articles either take a broad approach towards ‘knowledge’ or ‘information’ (among the scientific ones). For instance, Fleming and Rhodes (*2018: 4) describe how they use the police as a ‘case study to highlight the way in which experience and inherited knowledges are shared and assimilated in an organisation’. Other articles’ focus on ‘scientific evidence’ or ‘research’ is more strictly defined. Among these is Schlaufer (*2016: 91), who even examines how ‘the same scientific study – namely the Program for International Student Assessment (PISA)’ has been used narratively by opposing coalitions. In some articles, different sorts of ‘evidence’ are explicitly distinguished. Surprisingly rarely, such distinction is conceptualised based on the literature – when it is, the different types of knowledge base (Flyvberg, 2001; Tenbensel, 2008) are a prominent classification (for example, *Maybin, 2015). More often, the understanding is specified by distinguishing ‘evidence’ from ‘non-evidence’, for example, through considering scientific knowledge alongside administrative knowledge or practical knowledge (for example, *Cloutier et al, 2015).

Empirically, the broadest approach takes an encompassing interest in various kinds and types of ‘evidence’ (albeit with varying terminology) that are detectable – often without specifying or limiting those further. For instance, Nekola and Moravek (*2015: 2–3) ask broadly how expertise by ‘experts of various sorts… is mobilized in the continuing development of drug policy’. In the focused empirical approach, the interest is in one particular kind of ‘evidence’, specific, for example, through its disciplinary background, or methodology. Notably, articles on economic expertise (for example, *Flickenschild and Afonso, 2019), behavioural research (for example, *Gopalan & Pirog, 2017), and the role of experiments (for example,*Lee & Ma, 2019), are particularly prominent in our sample.

A second line of distinction ranges from more supply-side to more demand-side oriented approaches, which corresponds to a well-known distinction in the evidence and policy literature (see, for example, Landry et al, 2001). Articles which take a supply-side perspective understand ‘evidence’ as being brought into policymaking, for example, through different actors of policy-advisory systems (see *Craft & Wilder, 2017). These articles often use the concept of ‘(policy) advice’, which already includes a notion of knowledge-for-policy (efforts). Li (*2019), for instance, focuses on non-governmental expert advisors, specifically their role perceptions, and how they ‘manage the tension between scientific knowledge versus value/interests’ when providing advice. Another group of articles tends towards a demand-side perspective, characterised by understanding ‘evidence’ as something that is drawn on by the policymaking side, and integrated (or not) into political decision making. A typically used concept here is that of ‘expertise’, with the notion of ‘expert status’ being granted or drawn from. Respective terms in this category include ‘expert input’ or the concept of ‘certified expertise’, which as Baekkeskov and Öberg (*2017: 4) highlight is appointed by the polity as ‘particularly qualified to guide policy’. There is a third perspective discernible, which understands ‘evidence’ neither as something that is clearly supplied for policymaking, nor drawn on by policymaking, but that is deliberated and thereby ‘co-created’. An article that can be assigned to this group is that of Richards (*2019), whose aim is ‘to map the influences of science-policy relationships on climate policy decisions’.

Understandings of ‘policy’

The variation in understandings does extend also, if not particularly, to the ‘policy’ end of the evidence and policy relationship. The majority of articles in our corpus take policy as the dependent variable of this relationship, but what exactly is understood to be explained, influenced, affected, and so on, by (some sort of) ‘evidence’ differs hugely. To throw some light into those different understandings, again we tried to identify lines of distinction.

We are – as Dunlop and Radaelli (2020) recently put it for the policy-learning literature – thrown back onto Thomas Dye’s (1972) foundational question of ‘what policy is’. As a word of caution, it must be highlighted that policy scholars have often questioned the possibility of defining policy (cf Colebatch, 1998; Cairney, 2020: 17), given the complexity of the involved phenomena and perspectives. Apart from a more-or-less general consensus on policy as ‘what policy actors do’, the nature of policy is conceived very variably (Knill and Tosun, 2020: 4). Nevertheless, attempts to define and conceptualise policy abound in the literature (cf Cairney, 2020: 17–36). On this basis, we can distinguish three different perspectives, which are also reflected in our article corpus (see supplementary data).

A first understanding is that of policy as substance (cf, for example, Tosun and Workman, 2017), related interest being in policy characteristics and issues of policy design. A large proportion of our included articles adopts such understanding, which is mirrored in the above-mentioned broad notions (‘policy options’, ‘policy alternatives’, and so on), but also more specific understandings. For instance, Cairney and Yamazaki (*2018: 254) provide an ‘in-depth analysis of the mechanisms of key changes in specific policy instruments’ (of smoking bans as regulative legislation in particular). Fisher et al.’s (*2016) interest is in specific ‘policy goals’.

A second understanding is that of policy as discourse (see, for example, Bacchi, 2000), related interest being in policy ideas and policy narratives (Cairney, 2020: 22–24). In that direction, Flickenschild and Afonso (*2019: 1293) ‘focus on… the social organisation of economic expertise in the form of social networks between economists’, and on how policy ideas are created and diffused. Plehwe et al (*2018: 189) ask for the role of economic experts and their policy advice in how the hegemony of ‘austerian thinking’ has been achieved, and other articles, too, show an underlying conceptualisation of policy as discourse (see, for example, *Salas and Porras, 2018; *Schlaufer, 2016).

While some articles show pure forms of these two understandings, others overlap, and are, for example, interested in both substantive policy goals and underlying policy paradigms (see, for example, *Entzinger and Scholten, 2015). Moreover, there are a number of articles which, potentially form a third category (cf also Tosun and Workman, 2017), and are mainly interested in policy-related aspects of institutional configurations and governance, such as ‘organisational capacity’ (*Dunlop, 2017a), or ‘models of governance’ (*Voß and Simmons, 2018).

As a second line of distinction, one can identify articles ranging from broad to specific approaches towards policy. We see a broad focus, for example, in articles that wish to ‘examine scientific approaches to flood forecasting… as input to flood policy’ (*Wasson, 2015) generically; analyse ‘Brazil’s economic policy’ (*Ladi et al, 2018), or the role of expert input for ‘policy alternatives’ (*Baekkeskov and Öberg, 2017). Another group of articles use specific concepts that have been developed in the public policy literature, and examine, for instance, ‘policy innovations’ (*Rocle and Salles, 2018; *Brown and Cohen, 2019), ‘wicked policies’ (*Parkhurst, 2016), or ‘policy failure/success’ (*Bovens and t’Hart, 2016).7 Specific approaches can also serve to investigate policy as substance, policy as discourse, or both in a procedural perspective – such as with the classic questions of ‘legislative change’ (*Hunter and Boswell, 2015), or ‘policy (dis)continuity’ (*Skogstad, 2017)

Evidence and policy: conceptualisations and theoretical approaches to their connection

What characterises the evidence and policy literature – by our definition and operationalisation for screening the articles’ eligibility – is that there is some sort of ‘connection’ between (some sort of) evidence on the one hand, and policy on the other. Understandings of this ‘connection’ are therefore crucial. The terms used to describe this connection are numerous – some articles follow more abstract and conceptual terms (‘knowledge uses’, in particular), whereas many terms are more practical or colloquial. We also noticed that when evidence-based policy is the dependent variable, it was by trend more difficult to identify the type of connection with ‘policy’.

How evidence and policy are tied together of course resonates in the theoretical frameworks and literature strands which authors resort to. Somewhat surprisingly, in only a limited number of articles, a clear theoretical framework was used that guided the whole analysis. More often, authors referred to a selection of literatures – often the literature on knowledge production and utilisation more broadly – without a clear common denominator that could be discerned. Overall, a significant proportion of the articles did not incorporate core evidence and policy literature (that is, knowledge utilisation, STS [Science and Technology Studies], evidence-based policy, or epistemic communities) in their conceptual-theoretical foundation. This again reflects the multiple theoretical angles available to evidence and policy.

Overall, there seem to be two broader orientations on the evidence and policy connection: a ‘use of evidence’ orientation as well as a ‘use for policy’ orientation.

First, what unites the use of evidence oriented articles is that their point of departure seems to be an interest in ‘evidence’, and how it is either provided or taken up for policymaking. That being said, different foci are found under this umbrella. Some articles focus on evidence development, and understand the connection as ‘knowledge accumulation’ or ‘diffusion’ (see, for example, *Gonnet, 2019). Others approach the provision of evidence from a supply-side perspective, showing in terms such as ‘dissemination’ (see, for example, *Ayling and Cunningham, 2017). Its counterpart is the uptake of evidence as connection, showing in concrete terms such as ‘mobilisation’, or ‘perception as credible’ (see, for example, *Doberstein, 2017). Yet another approach follows the interactive understanding of the evidence and policy relationship, and terms the linkage as ‘participation’, or ‘blurring of boundaries between science and policy’ (see, for example, Brown and Cohen, 2019).

When it comes to theoretical frameworks, these use of evidence oriented articles are typically embedded in the literatures on knowledge utilisation, knowledge production, and sometimes also more specific strands such as policy work, or policy-advisory systems (PAS). It is here that we find also articles which are interested in evidence as the dependent variable, for example, asking how politicisation of a policy issue affects boundary work and knowledge production (*Entzinger and Scholten, 2015).

Second, the use for policy oriented articles’ point of departure is rather the effect of evidence use or its functions for policy. Again, this includes different orientations, such as problem- and policy-oriented uses, like ‘framing’, or ‘develop regulations’ (see, for example, *Béland et al, 2018). Other articles also cover political uses, for example, ‘legitimising’ policy (see, for example, *Agartan, 2015). Connections here can also be understood as uses for policymaking more generally, mirrored in terms such as ‘reducing uncertainty’ (see, for example, Wasson, 2016), or the concept of ‘narrative uses’ (*Schlaufer, 2016). Connections here are conceptualised, for instance, by drawing on the classical distinction of instrumental, conceptual and symbolic/political use, but also, for example, different types of policy learning. Notably, an additional understanding of the connection is that of non-use, which typically appears alongside other uses, and which is reflected in more specific terms such as ‘failure of epistemic learning’ (*Dunlop, 2017a).

When it comes to theoretical approaches, the use for policy oriented articles are more firmly embedded in the policy (process) literature. One set of articles employs different theories of the policy process to study the evidence use, here in particular the Multiple-streams Framework (*Cairney, 2018), the Advocacy Coalition Framework (*Béland et al, 2018), or the Narrative Policy Framework (*Schlaufer, 2016). Another set of the policy literature on evidence use applies theories of policy feedback and policy learning (*Dunlop, 2017b). Authors also rely on specific theoretical foci, when zooming into particular carriers of evidence; in particular this was visible for epistemic communities (*Mavrot and Sager, 2018), think tanks (*Plehwe et al, 2018), and the respective, rather established literatures. Additionally, a quite separate set of articles is more embedded in the literature and frameworks of governance (*Elken, 2017), (urban) planning, and participation, including that of scientific actors.

Conclusions

The quest to understand the evidence and policy relationship has intrigued policy researchers for decades, and has received reinvigorated attention in recent years, triggered by discussions on post-truth and post-factual policymaking. Scholars, however, highlighted the siloed nature of the field (Oliver and Boaz, 2019), with evidence being scattered across different disciplines, and have drawn attention to the fact that such notions as evidence and policy are often not clearly conceptualised or defined in literature (Oliver et al 2014: 1). This makes it challenging to come to systematic conclusions across individual studies about, for instance, the factors explaining different extents and types of evidence use.

In this article, we presented the results of a qualitative systematic review of recent scholarship on the evidence and policy relationship in the broad field of public policy, particularly with the ambition of taking stock of existing conceptualisations currently circulating. Table 2 provides a summative overview of the main conceptual approaches to evidence, policy, and their connections derived from the literature. The overview confirms that notions of evidence, policy and their connections come in many flavours and guises, resulting also in a myriad of possible configurations. Within each of these lines of inquiry, we also identified instances of variability, adding further to the conceptual complexity. It is telling as well that we could not retrieve any commonly shared ‘essence’ that returns across the entire article corpus.

While reaching consensus about conceptualisations on evidence, policy, and their connections is not what we call for per se, and scholars may have good reasons to apply a specific conceptual focus, research on evidence and policy would benefit from more explicit conceptual discussions. As a minimum, scholars should make their conceptual stance explicit about evidence, policy, and the precise connection. Table 2 provides scholars with a heuristic to explicate those choices when working with these notions. Interestingly, when it comes to conceptualisations, notions of ‘evidence’ are usually more defined in a specific way in ‘use of evidence’ studies. By contrast, ‘policy’ tends to be more specifically conceptualised in ‘use for policy’ research. While not necessarily surprising, this finding may advise scholars to draw inspiration from the respective approaches to fine-tune conceptualisations.

Based on the review, we can suggest several main avenues for further research that are worth exploring.

First, we identified different conceptualisations of evidence, policy and their connections. For a better understanding of conceptual trends, it would be interesting to see whether specific conceptualisations prevail among authors from certain geographical areas, or among scholars focusing on a specific policy field. With knowledge regimes and policy styles being shown to vary across countries and policy areas (Strassheim and Kettunen, 2014; Howlett and Tosun, 2018; Pattyn et al, 2019), this may affect how evidence and policy, and their connection, are conceived. Follow-up literature reviews may also benefit from including other scholarship in the sample (for example, not covered in Web of Science; or not written in English) to allow for a broader perspective on evidence, policy and their connections. We do not know, for instance, whether our findings are influenced by the predominance of Anglo-Saxon scholars in our sample, who may well be in frequent dialogue with each other, but not with other language communities which often have their own venues for exchanging concepts and ideas. Also, in this contribution we focused on present-day public policy scholarship, which can be expected to build on earlier generations of researchers investigating the evidence and policy relationship. Follow-up studies may longitudinally track whether and how conceptualisations have shifted over time within the different branches of public policy research, or within particular policy fields. Such study could also scrutinise potential transfer of concepts across different fields and public policy branches. Worth considering as well is a study of conceptualisations circulating among practitioners, comparing these with those applied by researchers. Not least, it may be useful to investigate whether different conceptualisations also relate to what scholars themselves conceive as ‘good evidence’ for policymaking (for example, highlighting experiments, or rather the value of stakeholder input).

Second, scholars need to theorise and study the relationships between specific approaches to policy, to evidence, and the connection between them. As we have shown, in only a limited number of articles, a clear theoretical framework was used that guided the overall analysis. In many instances, authors only referred to a selection of literatures, without a clear denominator. A large proportion of the articles did not have a precisely delineated theoretical foundation. Where specific frameworks were indeed mentioned, these could mainly be found in articles adopting a ‘use for policy’ lens. Generally speaking, frameworks for the ‘use of evidence’ lens are often of a relatively generic nature (for example, policy advisory systems). In comparative terms, frameworks at the ‘use for policy’ side lend themselves more towards rigorous testing (for example, Advocacy Coalition Framework; Narrative Policy Framework) which is beneficial for detecting the causal mechanisms linking evidence with policy.

Third, follow-up research can engage in a meta-analysis of available knowledge on the influence of some categories of evidence (for example, advice; scientific knowledge) on particular policy categories (for example, policy design). By the same token, and complementary to such review studies, research could engage in systematic analyses of the relative importance of particular types of evidence on specific policy categories in particular policy contexts, and vice versa. Such studies can constitute the basis for developing middle range theories (Merton, 1967) on how particular types of evidence ‘work’ in various contexts.

Fourth, it may be wise to study the influence of the framing of specific evidence notions on public policy categories (that is, a study of the role of conceptualisations themselves). Conceptualisations remain constructs after all, which may come with certain interpretations, also among practitioners. It may well be the case, for instance, that framing evidence as ‘research’ may be differently received by the ‘demand side’ compared to framing it as ‘advice’, and which conceptualisations are prioritised may also depend on a particular policy setting. Applying and triangulating different qualitative and quantitative methods could help us to shed more light on this. Besides the potential to contribute to theory development, such explicit conceptual studies may also foster the use of evidence in practice.

Finally, it is clear that even within the public policy field, the evidence and policy relationship is treated in very different literatures that have often remained disconnected, and it is also not always addressed at the evidence and policy literature and debate. A citation analysis could be a next logical step to probe deeper into our initial observations; and therewith to engage with Oliver and Boaz’s (2019: 1) argument that ‘practical and empirical lessons are not shared across disciplinary boundaries and theoretical and conceptual leaps remain contained’.

Notes

1

Again, other journals of the public administration category should also be expected to occasionally publish articles on evidence and policy matters, yet as we are interested in trends rather than giving an exhaustive overview, we believe that the explicit ‘policy journals’ offer an overview of core public policy research.

2

This was the most recent edition at the time of the research.

3

If categories were not relevant, they were left empty (for example, methodology, geographical scope).

4

The exact search string used via advanced search, and including the ISSN numbers of the named journals was: ‘TS=(evidence OR expert* OR research OR knowledge OR advi* OR epistem*) AND IS= (14693062 OR 02768739 OR 13501763 OR 02784254 OR 13876988 OR 09589287 OR 0190292X OR 00472794 OR 03055736 OR 14494035)’.

5

The search was conducted on 21 January 2020; the articles from Policy Sciences were added through an additional search conducted on 20 April 2021.

6

For instance, when it comes to approaches to ‘policy’, some articles combine an interest in policy discourse (for example, guiding paradigms) with an interest in policy substance (for example, policy proposals).

7

This does not, however, necessarily mean that these notions are also operationalised in very specific ways.

8

Those contain: macroeconomics/monetary policy/industry policy; civil rights/political rights/gender issues; health; agriculture/food policy; labour; education; environment; energy; immigration/integration/ethnic minorities; transportation; crime/law and order; social welfare; housing; domestic trade/commerce/financial sector; defence; technology (including telecommunications); foreign trade; international affairs/development aid/EU; government and public administration organisation/electoral reforms; public works/urban planning; culture.

Research ethics statement

The authors of this paper have declared that research ethics approval was not required since the paper does not present or draw directly on data/findings from empirical research.

Contributor statement

SB and VP jointly developed the idea for this article and contributed in equal parts to its conception, data collection, analysis, and writing.

Acknowledgements

We are grateful to Paul Cairney and the two anonymous reviewers for their valuable feedback on previous versions of this article. For research assistance, we would like to thank Julia Özdemir, Alena Oleinik, Robin-Michel van Parijs, and Paul-Philipp Schnase.

Conflict of interest

The authors declare that there is no conflict of interest.

References

  • * References for the articles included in the review are provided as supplementary data, which is available online at: https://doi.org/10.6084/m9.figshare.17294207

  • Acciai, C. and Capano, G. (2021) Policy instruments at work: a meta‐analysis of their applications, Public Administration, 99(1): 11836. doi: 10.1111/padm.12673

    • Search Google Scholar
    • Export Citation
  • Bacchi, C. (2000) Policy as discourse: what does it mean? Where does it get us? Discourse: Studies in the Cultural Politics of Education, 21(1): 4557. doi: 10.1080/01596300050005493

    • Search Google Scholar
    • Export Citation
  • Boswell, C. (2009) The Political Uses of Expert Knowledge. Immigration Policy and Social Research, Cambridge: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Cairney, P. (2016) The Politics of Evidence-based Policy Making, London: Palgrave Macmillan.

  • Cairney, P. (2020) Understanding Public Policy. Theories and Issues, 2nd edn, New York: Red Globe Press.

  • Cairney, P. and Oliver, K. (2020) How should academics engage in policymaking to achieve impact? Political Studies Review, 18(2): 22844. doi: 10.1177/1478929918807714

    • Search Google Scholar
    • Export Citation
  • Caplan, N. (1979) The Two-communities theory and knowledge utilization, American Behavioral Scientist, 22(3): 45970. doi: 10.1177/000276427902200308

    • Search Google Scholar
    • Export Citation
  • Christensen, J. (2021) Expert knowledge and policymaking: a multi-disciplinary research agenda, Policy & Politics, 49(3): 45571, doi: 10.1332/030557320X15898190680037.

    • Search Google Scholar
    • Export Citation
  • Colebatch, H. (1998) Policy, Buckingham: Open University Press.

  • Craft, J. and Howlett, M. (2013) The dual dynamics of policy advisory systems: The impact of externalization and politicization on policy advice. Policy and Society, 32(3): 18797.

    • Search Google Scholar
    • Export Citation
  • Dunlop, C.A. and Radaelli, C.M. (2020) Policy learning in comparative policy analysis, Journal of Comparative Policy Analysis: Research and Practice, doi: 10.1080/13876988.2020.1762077.

    • Search Google Scholar
    • Export Citation
  • Dye, T. (1972) Understanding Public Policy, Englewood Cliffs, NJ: Prentice Hall.

  • Eagly, A.H. and Chaiken, S. (2007) The advantages of an inclusive definition of attitude, Social Cognition, 25(5): 582602. doi: 10.1521/soco.2007.25.5.582

    • Search Google Scholar
    • Export Citation
  • Flyvbjerg, B. (2001) Making Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again, Cambridge: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Freeman, R. (2010) Epistemological bricolage: how practitioners make sense of learning, Administration and Society, 39(4): 47696. doi: 10.1177/0095399707301857

    • Search Google Scholar
    • Export Citation
  • French, R.D. (2019) Is it time to give up on Evidence-based policy? Four answers, Policy & Politics, 47(1): 15168.

  • Gerring, J. (1999) What makes a concept good? A criterial framework for understanding concept formation in the social sciences, Polity, 31(3): 35793. doi: 10.2307/3235246

    • Search Google Scholar
    • Export Citation
  • Gieryn, T.F. (1983) Boundary-work and the demarcation of science from non-science: strains and interests in professional ideologies of scientists, American Sociological Review, 48(6): 78195. doi: 10.2307/2095325

    • Search Google Scholar
    • Export Citation
  • Halligan, J. (1995) Policy advice and the public service, in B.G. Peters and D.J. Savoie (eds) Governance in a Changing Environment, Québec: Canadian Centre for Management Development, pp 13872.

    • Search Google Scholar
    • Export Citation
  • Howlett, M. and Tosun, J. (2018) Policy styles: a new approach, in M. Howlett and J. Tosun (eds) Policy Styles and Policy-Making. Exploring the Linkages, Abingdon: Routledge.

    • Search Google Scholar
    • Export Citation
  • Innvaer, S., Vist, G., Trommald, M. and Oxman, A. (2002) Health policy-makers’ perceptions of their use of evidence: a systematic review, Journal of Health Services Research and Policy, 7(4): 23944. doi: 10.1258/135581902320432778

    • Search Google Scholar
    • Export Citation
  • Knill, C. and Tosun, J. (2020) Public Policy: A New Introduction, 2nd edn, New York: Red Globe Press.

  • Landry, R., Amara, N. and Lamari, M. (2001) Utilization of social science research knowledge in Canada, Research Policy, 30(2): 33349. doi: 10.1016/S0048-7333(00)00081-0

    • Search Google Scholar
    • Export Citation
  • Lindquist, E.A. (1990) The third community, policy inquiry, and social scientists, in S. Brooks and A. Gagnon (eds) Social Scientists, Policy, and the State, New York: Praeger, pp 2151.

    • Search Google Scholar
    • Export Citation
  • Liverani, M., Hawkins, B. and Parkhurst, J.O. (2013) Political and institutional influences on the use of evidence in public health policy: a systematic review, PLoS ONE, 8(10): e77404. doi: 10.1371/journal.pone.0077404

    • Search Google Scholar
    • Export Citation
  • MacKillop, E., Quarmby, S. and Downe, J. (2020) Does knowledge brokering facilitate evidence-based policy? A review of existing knowledge and an agenda for future research, Policy & Politics, 48(2): 33553.

    • Search Google Scholar
    • Export Citation
  • Merton, R.K. (1967) On sociological theories of the middle range, in R.K. Merton, On Theoretical Sociology, New York: Free Press, pp 3972.

    • Search Google Scholar
    • Export Citation
  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G. and Prisma Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, PLoS Medicine, 6(7): e1000097. doi: 10.1371/journal.pmed.1000097

    • Search Google Scholar
    • Export Citation
  • Moyson, S., Raaphorst, N., Groeneveld, S. and Van de Walle, S. (2017) Organizational socialization in public administration research: a systematic review and directions for future research, American Review of Public Administration, 48(6), doi: 10.1177/0275074017696160.

    • Search Google Scholar
    • Export Citation
  • Neal, J.W., Neal, Z. and Brutzman, B. (2021) Defining brokers, intermediaries, and boundary spanners: a systematic review, Evidence & Policy, doi: 10.1332/174426420X16083745764324.

    • Search Google Scholar
    • Export Citation
  • Newman, L., Rowley, J., Vander Hoorn, S., Wijesooriya, N.S., Unemo, M. and Low, N. (2015) Global estimates of the prevalence and incidence of four curable sexually transmitted infections in 2012 based on systematic review and global reporting, PLos One, 10(12): e0143304.

    • Search Google Scholar
    • Export Citation
  • Nutley, S., Walter, I. and Davies, H.T. (2009) Promoting evidence-based practice: models and mechanisms from cross-sector review, Research on Social Work Practice, 19(5): 55259. doi: 10.1177/1049731509335496

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Boaz, A. (2019) Transforming evidence for policy and practice: creating space for new conversations, Palgrave Communications, 5(60): 110. doi: 10.1057/s41599-018-0199-0

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Cairney, P. (2019) The dos and don’ts of influencing policy: a systematic review of advice to academics, Palgrave Communications, 5(21): 111.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvaer, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(2): 112. doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation
  • Pattyn, V., Blum, S., Fobé, E., Pekar-Milicevic, M. and Brans, M. (2019) Academic policy advice in Consensus-seeking countries: the cases of Belgium and Germany, International Review of Administrative Science, online first, doi: 10.1177/0020852319878780.

    • Search Google Scholar
    • Export Citation
  • PytlikZillig, L.M. and Kimbrough, C. (2016) Consensus on conceptualizations and definitions of trust: are we there yet? in E. Shockley, T.M. Neal, L.M. PytlikZillig and B.H. Bornstein (eds) Interdisciplinary Perspectives on Trust, Berlin: Springer, pp 1747.

    • Search Google Scholar
    • Export Citation
  • Riche, C., Aubin, D. and Moyson, S. (2021) Too much of a good thing? A systematic review about the conditions of learning in governance networks, European Policy Analysis, 7(1): 14764. doi: 10.1002/epa2.1080

    • Search Google Scholar
    • Export Citation
  • Strassheim, H. and Kettunen, P. (2014) When does evidence-based policy turn into policy-based evidence? Configurations, contexts and mechanisms, Evidence & Policy, 10(2): 25977.

    • Search Google Scholar
    • Export Citation
  • Strydom, W.F., Funke, N., Nienaber, S., Nortje, K. and Steyn, M. (2010) Evidence-based policymaking: a review, South African Journal of Science, 106(5–6): 1724. doi: 10.4102/sajs.v106i5/6.249

    • Search Google Scholar
    • Export Citation
  • Tenbensel, T. (2008) The role of evidence in policy: how the mix matters, Paper presented at the Twelfth Annual Conference of the Research Society for Public Management, Brisbane.

    • Search Google Scholar
    • Export Citation
  • Tosun, J. and Workman, S. (2017) Struggle and triumph in fusing policy process and comparative research, in C.M. Weible and P.A. Sabatier (eds) Theories of the Policy Process, 4th edn, Boulder, CO: Westview Press, pp 32962.

    • Search Google Scholar
    • Export Citation
  • van der Heijden, J. and Kuhlmann, J. (2017) Studying incremental institutional change: a systematic and critical meta-review of the literature from 2005 to 2015, Policy Studies Journal, 45(3): 53554. doi: 10.1111/psj.12191

    • Search Google Scholar
    • Export Citation
  • Verweij, S. and Trell, E.M. (2019) Qualitative Comparative Analysis (QCA) in spatial planning research and related disciplines: a systematic literature review of applications, Journal of Planning Literature, 34(3): 300317. doi: 10.1177/0885412219841490

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 42631. doi: 10.2307/3109916

  • Wildavsky, A.B. (1979) Speaking Truth to Power the Art and Craft of Policy Analysis, 2nd edn, Piscataway, NJ: Transaction Publishers.

Appendix

Table 3:

Distribution of articles across journals

JournalsNumber
Policy Sciences18
Journal of Comparative Policy Analysis16
Climate Policy12
Policy and Society11
Policy & Politics8
Journal of European Public Policy7
Policy Studies Journal7
Journal of Policy Analysis and Management2
Journal of Social Policy2
Journal of Accounting and Public Policy1
Public Policy1
Table 4:

Country affiliation of first authors

Affiliation of the first authorCountPercentage
African countries11%
Anglo-Saxon countries5160%
Asian countries33%
European countries2631%
Latin American countries and Mexico45%

*Note: Turkey is included in European countries.

  • * References for the articles included in the review are provided as supplementary data, which is available online at: https://doi.org/10.6084/m9.figshare.17294207

  • Acciai, C. and Capano, G. (2021) Policy instruments at work: a meta‐analysis of their applications, Public Administration, 99(1): 11836. doi: 10.1111/padm.12673

    • Search Google Scholar
    • Export Citation
  • Bacchi, C. (2000) Policy as discourse: what does it mean? Where does it get us? Discourse: Studies in the Cultural Politics of Education, 21(1): 4557. doi: 10.1080/01596300050005493

    • Search Google Scholar
    • Export Citation
  • Boswell, C. (2009) The Political Uses of Expert Knowledge. Immigration Policy and Social Research, Cambridge: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Cairney, P. (2016) The Politics of Evidence-based Policy Making, London: Palgrave Macmillan.

  • Cairney, P. (2020) Understanding Public Policy. Theories and Issues, 2nd edn, New York: Red Globe Press.

  • Cairney, P. and Oliver, K. (2020) How should academics engage in policymaking to achieve impact? Political Studies Review, 18(2): 22844. doi: 10.1177/1478929918807714

    • Search Google Scholar
    • Export Citation
  • Caplan, N. (1979) The Two-communities theory and knowledge utilization, American Behavioral Scientist, 22(3): 45970. doi: 10.1177/000276427902200308

    • Search Google Scholar
    • Export Citation
  • Christensen, J. (2021) Expert knowledge and policymaking: a multi-disciplinary research agenda, Policy & Politics, 49(3): 45571, doi: 10.1332/030557320X15898190680037.

    • Search Google Scholar
    • Export Citation
  • Colebatch, H. (1998) Policy, Buckingham: Open University Press.

  • Craft, J. and Howlett, M. (2013) The dual dynamics of policy advisory systems: The impact of externalization and politicization on policy advice. Policy and Society, 32(3): 18797.

    • Search Google Scholar
    • Export Citation
  • Dunlop, C.A. and Radaelli, C.M. (2020) Policy learning in comparative policy analysis, Journal of Comparative Policy Analysis: Research and Practice, doi: 10.1080/13876988.2020.1762077.

    • Search Google Scholar
    • Export Citation
  • Dye, T. (1972) Understanding Public Policy, Englewood Cliffs, NJ: Prentice Hall.

  • Eagly, A.H. and Chaiken, S. (2007) The advantages of an inclusive definition of attitude, Social Cognition, 25(5): 582602. doi: 10.1521/soco.2007.25.5.582

    • Search Google Scholar
    • Export Citation
  • Flyvbjerg, B. (2001) Making Social Science Matter: Why Social Inquiry Fails and How it Can Succeed Again, Cambridge: Cambridge University Press.

    • Search Google Scholar
    • Export Citation
  • Freeman, R. (2010) Epistemological bricolage: how practitioners make sense of learning, Administration and Society, 39(4): 47696. doi: 10.1177/0095399707301857

    • Search Google Scholar
    • Export Citation
  • French, R.D. (2019) Is it time to give up on Evidence-based policy? Four answers, Policy & Politics, 47(1): 15168.

  • Gerring, J. (1999) What makes a concept good? A criterial framework for understanding concept formation in the social sciences, Polity, 31(3): 35793. doi: 10.2307/3235246

    • Search Google Scholar
    • Export Citation
  • Gieryn, T.F. (1983) Boundary-work and the demarcation of science from non-science: strains and interests in professional ideologies of scientists, American Sociological Review, 48(6): 78195. doi: 10.2307/2095325

    • Search Google Scholar
    • Export Citation
  • Halligan, J. (1995) Policy advice and the public service, in B.G. Peters and D.J. Savoie (eds) Governance in a Changing Environment, Québec: Canadian Centre for Management Development, pp 13872.

    • Search Google Scholar
    • Export Citation
  • Howlett, M. and Tosun, J. (2018) Policy styles: a new approach, in M. Howlett and J. Tosun (eds) Policy Styles and Policy-Making. Exploring the Linkages, Abingdon: Routledge.

    • Search Google Scholar
    • Export Citation
  • Innvaer, S., Vist, G., Trommald, M. and Oxman, A. (2002) Health policy-makers’ perceptions of their use of evidence: a systematic review, Journal of Health Services Research and Policy, 7(4): 23944. doi: 10.1258/135581902320432778

    • Search Google Scholar
    • Export Citation
  • Knill, C. and Tosun, J. (2020) Public Policy: A New Introduction, 2nd edn, New York: Red Globe Press.

  • Landry, R., Amara, N. and Lamari, M. (2001) Utilization of social science research knowledge in Canada, Research Policy, 30(2): 33349. doi: 10.1016/S0048-7333(00)00081-0

    • Search Google Scholar
    • Export Citation
  • Lindquist, E.A. (1990) The third community, policy inquiry, and social scientists, in S. Brooks and A. Gagnon (eds) Social Scientists, Policy, and the State, New York: Praeger, pp 2151.

    • Search Google Scholar
    • Export Citation
  • Liverani, M., Hawkins, B. and Parkhurst, J.O. (2013) Political and institutional influences on the use of evidence in public health policy: a systematic review, PLoS ONE, 8(10): e77404. doi: 10.1371/journal.pone.0077404

    • Search Google Scholar
    • Export Citation
  • MacKillop, E., Quarmby, S. and Downe, J. (2020) Does knowledge brokering facilitate evidence-based policy? A review of existing knowledge and an agenda for future research, Policy & Politics, 48(2): 33553.

    • Search Google Scholar
    • Export Citation
  • Merton, R.K. (1967) On sociological theories of the middle range, in R.K. Merton, On Theoretical Sociology, New York: Free Press, pp 3972.

    • Search Google Scholar
    • Export Citation
  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G. and Prisma Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement, PLoS Medicine, 6(7): e1000097. doi: 10.1371/journal.pmed.1000097

    • Search Google Scholar
    • Export Citation
  • Moyson, S., Raaphorst, N., Groeneveld, S. and Van de Walle, S. (2017) Organizational socialization in public administration research: a systematic review and directions for future research, American Review of Public Administration, 48(6), doi: 10.1177/0275074017696160.

    • Search Google Scholar
    • Export Citation
  • Neal, J.W., Neal, Z. and Brutzman, B. (2021) Defining brokers, intermediaries, and boundary spanners: a systematic review, Evidence & Policy, doi: 10.1332/174426420X16083745764324.

    • Search Google Scholar
    • Export Citation
  • Newman, L., Rowley, J., Vander Hoorn, S., Wijesooriya, N.S., Unemo, M. and Low, N. (2015) Global estimates of the prevalence and incidence of four curable sexually transmitted infections in 2012 based on systematic review and global reporting, PLos One, 10(12): e0143304.

    • Search Google Scholar
    • Export Citation
  • Nutley, S., Walter, I. and Davies, H.T. (2009) Promoting evidence-based practice: models and mechanisms from cross-sector review, Research on Social Work Practice, 19(5): 55259. doi: 10.1177/1049731509335496

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Boaz, A. (2019) Transforming evidence for policy and practice: creating space for new conversations, Palgrave Communications, 5(60): 110. doi: 10.1057/s41599-018-0199-0

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Cairney, P. (2019) The dos and don’ts of influencing policy: a systematic review of advice to academics, Palgrave Communications, 5(21): 111.

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvaer, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(2): 112. doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation
  • Pattyn, V., Blum, S., Fobé, E., Pekar-Milicevic, M. and Brans, M. (2019) Academic policy advice in Consensus-seeking countries: the cases of Belgium and Germany, International Review of Administrative Science, online first, doi: 10.1177/0020852319878780.

    • Search Google Scholar
    • Export Citation
  • PytlikZillig, L.M. and Kimbrough, C. (2016) Consensus on conceptualizations and definitions of trust: are we there yet? in E. Shockley, T.M. Neal, L.M. PytlikZillig and B.H. Bornstein (eds) Interdisciplinary Perspectives on Trust, Berlin: Springer, pp 1747.

    • Search Google Scholar
    • Export Citation
  • Riche, C., Aubin, D. and Moyson, S. (2021) Too much of a good thing? A systematic review about the conditions of learning in governance networks, European Policy Analysis, 7(1): 14764. doi: 10.1002/epa2.1080

    • Search Google Scholar
    • Export Citation
  • Strassheim, H. and Kettunen, P. (2014) When does evidence-based policy turn into policy-based evidence? Configurations, contexts and mechanisms, Evidence & Policy, 10(2): 25977.

    • Search Google Scholar
    • Export Citation
  • Strydom, W.F., Funke, N., Nienaber, S., Nortje, K. and Steyn, M. (2010) Evidence-based policymaking: a review, South African Journal of Science, 106(5–6): 1724. doi: 10.4102/sajs.v106i5/6.249

    • Search Google Scholar
    • Export Citation
  • Tenbensel, T. (2008) The role of evidence in policy: how the mix matters, Paper presented at the Twelfth Annual Conference of the Research Society for Public Management, Brisbane.

    • Search Google Scholar
    • Export Citation
  • Tosun, J. and Workman, S. (2017) Struggle and triumph in fusing policy process and comparative research, in C.M. Weible and P.A. Sabatier (eds) Theories of the Policy Process, 4th edn, Boulder, CO: Westview Press, pp 32962.

    • Search Google Scholar
    • Export Citation
  • van der Heijden, J. and Kuhlmann, J. (2017) Studying incremental institutional change: a systematic and critical meta-review of the literature from 2005 to 2015, Policy Studies Journal, 45(3): 53554. doi: 10.1111/psj.12191

    • Search Google Scholar
    • Export Citation
  • Verweij, S. and Trell, E.M. (2019) Qualitative Comparative Analysis (QCA) in spatial planning research and related disciplines: a systematic literature review of applications, Journal of Planning Literature, 34(3): 300317. doi: 10.1177/0885412219841490

    • Search Google Scholar
    • Export Citation
  • Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 42631. doi: 10.2307/3109916

  • Wildavsky, A.B. (1979) Speaking Truth to Power the Art and Craft of Policy Analysis, 2nd edn, Piscataway, NJ: Transaction Publishers.

  • 1 University of Hagen, , Germany
  • | 2 Leiden University, , Netherlands

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 27 27 27
PDF Downloads 15 15 15

Altmetrics

Dimensions