Analysing use of evidence in public policymaking processes: a theory-grounded content analysis methodology

Authors:
Itzhak Yanovitzky Rutgers University School of Communication and Information, USA

Search for other papers by Itzhak Yanovitzky in
Current site
Google Scholar Close
and
Matthew Weber Rutgers University School of Communication and Information, USA

Search for other papers by Matthew Weber in
Current site
Google Scholar Close
Article Category:
Research Article
Copyright:
© Policy Press 2020
Online Publication Date:
01 Nov 2018
Pages:
65–82
Publisher:
Policy Press
Volume/Issue:
Volume 16: Issue 1
DOI:
https://doi.org/10.1332/174426418X15378680726175
Restricted access
Get eTOC alerts
Rights and permissions Cite this article
Buy article/chapter (
GBP  £25.00
)

A broad range of methodologies is routinely employed to track and analyse use of research evidence in public policymaking. However, available methods and tools are mostly equipped to track ‘evidence’ but are less equipped to capture and represent ‘use’ of evidence. In particular, existing methods overlook policymakers’ frequent use of evidence for political persuasion and bargaining. Drawing on established theories and research tools from the field of persuasive communication, we developed a content analysis instrument for tracking and analysing policymakers’ use of evidence based on the information contained in policy documents. We provide a specific example of applying this tool and demonstrate how it may be used to generate rich and nuanced insights regarding the scope, nature, and timing of policymakers’ evidence use at different levels and phases of the public policymaking process. We conclude with a discussion of potential strengths and limitations and offer recommendations regarding the optimal application of this methodology.

  • Altheide, DL, 2000, Tracking discourse and qualitative document analysis, Poetics, 27, 287–99 doi: 10.1016/S0304-422X(00)00005-X

  • Baumgartner, FR, 2013, Ideas and policy change, Governance, 26, 239–258. doi: 10.1111/gove.12007

  • Benoit, K, Herzog, A, 2017, Text analysis: estimating policy preferences from written and spoken words, in Bachner, J, Hill, KW, Ginsberg, B (eds), Analytics, policy and governance, New Haven, CT: Yale University Press

    • Search Google Scholar
    • Export Citation

  • Boaz, A, 2013, The role of stakeholders in promoting the use of evidence, Evidence & Policy 9, 455–6

  • Bogenschneider, K, Corbett, T, 2010, Evidence-based policymaking: insights from policy-minded researchers and research-minded policymakers, New York: Routledge

    • Search Google Scholar
    • Export Citation

  • Boswell, C. 2008. The political functions of expert knowledge: knowledge and legitimation in European Union immigration policy, European Public Policy 15, 471–88 doi: 10.1080/13501760801996634

    • Search Google Scholar
    • Export Citation

  • Boswell, C, Smith, K, 2017, Rethinking policy ‘impact’: four models of research-policy relations, Palgrave Communications 3, 1–10 doi: 10.1057/s41599-017-0001-8

    • Search Google Scholar
    • Export Citation

  • Bowen, GA, 2009, Document analysis as a qualitative research method, Qualitative Research 9, 27–40 doi: 10.3316/QRJ0902027

  • Cacciatore, MA, Scheufele, DA, Iyengar, S, 2016, The end of framing as we know it… and the future of media effects, Mass Communication and Society 19, 7–23 doi: 10.1080/15205436.2015.1068811

    • Search Google Scholar
    • Export Citation

  • Castellani, T, Valente, A, Cori, L, Bianchi, F, 2016, Detecting the use of evidence in a meta-policy, Evidence & Policy 12, 91–107

  • Caulley, DN, 1983, Document analysis in program evaluation, Evaluation and Program Planning 6, 19–29 doi: 10.1016/0149-7189(83)90041-1

    • Search Google Scholar
    • Export Citation

  • Davies, H, Nutley, S, Walter, I, 2008, Why ‘knowledge transfer’ is misconceived for applied social research, Health Services Research & Policy 13, 188–90

    • Search Google Scholar
    • Export Citation

  • Dedoose, 2017, Web application for managing, analyzing, and presenting qualitative and mixed method research data (Version 7.0.23), Los Angeles, CA: SocioCultural Research Consultants, LLC, www.dedoose.com

    • Search Google Scholar
    • Export Citation

  • Epstein, D, Farina, C, Heidt, J, 2014, The value of words: narrative as evidence in policymaking, Evidence & Policy 10, 243–58

  • Feldman, R, 2013, Techniques and applications for sentiment analysis, Communications of the ACM 56, 82–9 doi: 10.1145/2436256.2436274

  • Finnigan, KS, Daly, AJ, Che, J, 2013, Systemwide reform in districts under pressure: the role of social networks in defining, acquiring, using, and diffusing research evidence, Educational Administration 51, 476–97 doi: 10.1108/09578231311325668

    • Search Google Scholar
    • Export Citation

  • Freeman, R, Griggs, S, Boaz, A, 2011, The practice of policymaking, Evidence & Policy 7, 127–36

  • Freeman, R, Maybin, J, 2011, Documents, practices and policy, Evidence & Policy 7, 155–70

  • Goffman, E, 1974, Frame analysis: an essay on the organization of experience, New York: Harper Colophon

  • Goldie, D, Linick, M, Jabbar, H, Lubienski, C, 2014, Using bibliometric and social media analyses to explore the ‘echo chamber’ hypothesis, Educational Policy 28, 281–305 doi: 10.1177/0895904813515330

    • Search Google Scholar
    • Export Citation

  • Gormley, WT, 2011, From science to policy in early childhood education, Science 333, 978–81 doi: 10.1126/science.1206150

  • Haunschild, R, Bornmann, L, 2017, How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data, Scientometrics 110, 1209–16 doi: 10.1007/s11192-016-2237-2

    • Search Google Scholar
    • Export Citation

  • Hayes, AF, Krippendorff, K, 2007, Answering the call for a standard reliability measure for coding data, Communication Methods and Measures 1, 77–89 doi: 10.1080/19312450709336664

    • Search Google Scholar
    • Export Citation

  • Jackson, S, Jacobs, S, 1980, Structure of conversational argument: pragmatic bases for the enthymeme, Quarterly Journal of Speech 66, 251–65 doi: 10.1080/00335638009383524

    • Search Google Scholar
    • Export Citation

  • Kingdon, JW, 2011, Agendas, alternatives, and public policies, Boston: Longman

  • Krippendorff, K, 2013, Content analysis: an introduction to its methodology, Los Angeles and London: Sage

  • Majone, G, 1989, Evidence, argument, and persuasion in the policy process, New Haven, CT: Yale University Press

  • Makkar, SR, Brennan, S, Turner, T, Williamson, A, Redman, S, Green, S, 2016, The development of SAGE: a tool to evaluate how policymakers engage with and use research in health policymaking, Research Evaluation 25, 315–28 doi: 10.1093/reseval/rvv044

    • Search Google Scholar
    • Export Citation

  • Meagher, L, Lyall, C, Nutley, S, 2008. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research, Research Evaluation 17, 163–73 doi: 10.3152/095820208X331720

    • Search Google Scholar
    • Export Citation

  • Moat, KA, Lavis, JN, Abelson, J, 2013, How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis, Milbank Quarterly 91, 604–48 doi: 10.1111/1468-0009.12026

    • Search Google Scholar
    • Export Citation

  • National Research Council, 2012, Using science as evidence in public policy, Washington, DC: National Academies Press

  • Neal, JW, Neal, ZP, Kornbluh, M, Mills, KJ, Lawlor, JA, 2015, Brokering the research–practice gap: a typology, American Journal of Community Psychology 56, 422–35 doi: 10.1007/s10464-015-9745-8

    • Search Google Scholar
    • Export Citation

  • Nutley, SM, Walter, I, Davies, HTO, 2007, Using evidence: how research can inform public services, Bristol: Policy Press

  • O’Keefe, DJ, 2002, Persuasion: theory and research, Thousand Oaks, CA: Sage

  • Obama, M, 2012, Let’s move! Raising a healthier generation of kids, Childhood Obesity 8, 1

  • Oliver, K, Innvar, S, Lorenc, T, Woodman, J, Thomas, J, 2014, A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research 14, 1–12 doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation

  • Oliver, KA, de Vocht, F, 2017, Defining ‘evidence’ in public health: a survey of policymakers’ uses and preferences, European Journal of Public Health 27, 112–17

    • Search Google Scholar
    • Export Citation

  • Oxman, AD, Lavis, JN, Lewin, S, Fretheim, A, 2009, SUPPORT Tools for evidence-informed health policymaking (STP) 1: what is evidence-informed policymaking? Health Research Policy and Systems, 7, S1

    • Search Google Scholar
    • Export Citation

  • Palinkas, LA, Garcia, AR, Aarons, GA, Finno-Velasquez, M, Holloway, IW, Mackie, TI, Leslie, LK, Chamberlain, P, 2016, Measuring use of research evidence: the structured interview for evidence use, Research on Social Work Practice 26, 550–64 doi: 10.1177/1049731514560413

    • Search Google Scholar
    • Export Citation

  • Pearce, W, Wesselink, A, Colebatch, H, 2014, Evidence and meaning in policymaking, Evidence & Policy 10, 161–5

  • Petty, RE, Cacioppo, JT, 1981, Attitudes and persuasion – classic and contemporary approaches, Dubuque, IA: WC Brown Co

  • Prior, L, 2003, Using documents in social research, Thousand Oaks, CA: Sage

  • Ritter, A, Lancaster, K, 2013, Measuring research influence on drug policy: a case example of two epidemiological monitoring systems, International Journal of Drug Policy 24, 30–37 doi: 10.1016/j.drugpo.2012.02.005

    • Search Google Scholar
    • Export Citation

  • Scheufele, DA, Tewksbury, D, 2007, Framing, agenda setting, and priming: the evolution of three media effects models, Communication 57, 9–20

    • Search Google Scholar
    • Export Citation

  • Schlesinger, M, Lau, RR, 2000, The meaning and measure of policy metaphors, American Political Science Review 94, 611–26 doi: 10.2307/2585834

    • Search Google Scholar
    • Export Citation

  • Sproles, C, 2011, Federal Digital System (Fdsys), Government Information Quarterly 28, 129 doi: 10.1016/j.giq.2010.07.004

  • Stiff, JB, Mongeau, PA, 2003, Persuasive communication, New York: Guilford Press

  • Stryker, JE, Wray, RJ, Hornik, RC, Yanovitzky, I, 2006, Validation of database search terms for content analysis: the case of cancer news coverage, Journalism & Mass Communication Quarterly 83, 413–30 doi: 10.1177/107769900608300212

    • Search Google Scholar
    • Export Citation

  • Toulmin, S, 2003, The uses of argument, Cambridge: Cambridge University Press

  • Tseng, V, 2012, The uses of research in policy and practice, Social Policy Report 26, 1–16 doi: 10.1002/j.2379-3988.2012.tb00071.x

  • Tversky, A, Kahneman, D, 1981, The framing of decisions and the psychology of choice, Science 211, 453–8 doi: 10.1126/science.7455683

  • Van De Goor, I, Hamalainen, RM, Syed, A, Lau, CJ, Sandu, P, Spitters, H, Karlsson, LE, Dulf, D, Valente, A, Castellani, T, Aro, AR, Consortium, R, 2017, Determinants of evidence use in public health policymaking: results from a study across six EU countries, Health Policy 121, 273–81 doi: 10.1016/j.healthpol.2017.01.003

    • Search Google Scholar
    • Export Citation

  • Weiss, CH, 1979, The many meanings of research utilization, Public Administration Review 39, 426–31 doi: 10.2307/3109916

  • Zardo, P, Collie, A, 2014, Measuring use of research evidence in public health policy: a policy content analysis, BMC Public Health 14, 1–10 doi: 10.1186/1471-2458-14-1

    • Search Google Scholar
    • Export Citation
Itzhak Yanovitzky Rutgers University School of Communication and Information, USA

Search for other papers by Itzhak Yanovitzky in
Current site
Google Scholar Close
and
Matthew Weber Rutgers University School of Communication and Information, USA

Search for other papers by Matthew Weber in
Current site
Google Scholar Close

itzhak@rutgers.edu
matthew.weber@rutgers.edu

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 3145 1419 53
Full Text Views 186 51 0
PDF Downloads 231 81 0

Altmetrics

Dimensions

Cover Evidence & Policy
Evidence & Policy
A journal of research, debate and practice