Analysing use of evidence in public policymaking processes: a theory-grounded content analysis methodology

Authors:
Itzhak Yanovitzky Rutgers University School of Communication and Information, USA

Search for other papers by Itzhak Yanovitzky in
Current site
Google Scholar
Close
and
Matthew Weber Rutgers University School of Communication and Information, USA

Search for other papers by Matthew Weber in
Current site
Google Scholar
Close
Restricted access
Get eTOC alerts
Rights and permissions Cite this article

A broad range of methodologies is routinely employed to track and analyse use of research evidence in public policymaking. However, available methods and tools are mostly equipped to track ‘evidence’ but are less equipped to capture and represent ‘use’ of evidence. In particular, existing methods overlook policymakers’ frequent use of evidence for political persuasion and bargaining. Drawing on established theories and research tools from the field of persuasive communication, we developed a content analysis instrument for tracking and analysing policymakers’ use of evidence based on the information contained in policy documents. We provide a specific example of applying this tool and demonstrate how it may be used to generate rich and nuanced insights regarding the scope, nature, and timing of policymakers’ evidence use at different levels and phases of the public policymaking process. We conclude with a discussion of potential strengths and limitations and offer recommendations regarding the optimal application of this methodology.

  • Altheide, DL, 2000, Tracking discourse and qualitative document analysis, Poetics, 27, 28799 doi: 10.1016/S0304-422X(00)00005-X

  • Baumgartner, FR, 2013, Ideas and policy change, Governance, 26, 239258. doi: 10.1111/gove.12007

  • Benoit, K, Herzog, A, 2017, Text analysis: estimating policy preferences from written and spoken words, in Bachner, J, Hill, KW, Ginsberg, B (eds), Analytics, policy and governance, New Haven, CT: Yale University Press

    • Search Google Scholar
    • Export Citation
  • Boaz, A, 2013, The role of stakeholders in promoting the use of evidence, Evidence & Policy 9, 4556

  • Bogenschneider, K, Corbett, T, 2010, Evidence-based policymaking: insights from policy-minded researchers and research-minded policymakers, New York: Routledge

    • Search Google Scholar
    • Export Citation
  • Boswell, C. 2008. The political functions of expert knowledge: knowledge and legitimation in European Union immigration policy, European Public Policy 15, 47188 doi: 10.1080/13501760801996634

    • Search Google Scholar
    • Export Citation
  • Boswell, C, Smith, K, 2017, Rethinking policy ‘impact’: four models of research-policy relations, Palgrave Communications 3, 110 doi: 10.1057/s41599-017-0001-8

    • Search Google Scholar
    • Export Citation
  • Bowen, GA, 2009, Document analysis as a qualitative research method, Qualitative Research 9, 2740 doi: 10.3316/QRJ0902027

  • Cacciatore, MA, Scheufele, DA, Iyengar, S, 2016, The end of framing as we know it… and the future of media effects, Mass Communication and Society 19, 723 doi: 10.1080/15205436.2015.1068811

    • Search Google Scholar
    • Export Citation
  • Castellani, T, Valente, A, Cori, L, Bianchi, F, 2016, Detecting the use of evidence in a meta-policy, Evidence & Policy 12, 91107

  • Caulley, DN, 1983, Document analysis in program evaluation, Evaluation and Program Planning 6, 1929 doi: 10.1016/0149-7189(83)90041-1

    • Search Google Scholar
    • Export Citation
  • Davies, H, Nutley, S, Walter, I, 2008, Why ‘knowledge transfer’ is misconceived for applied social research, Health Services Research & Policy 13, 18890

    • Search Google Scholar
    • Export Citation
  • Dedoose, 2017, Web application for managing, analyzing, and presenting qualitative and mixed method research data (Version 7.0.23), Los Angeles, CA: SocioCultural Research Consultants, LLC, www.dedoose.com

    • Search Google Scholar
    • Export Citation
  • Epstein, D, Farina, C, Heidt, J, 2014, The value of words: narrative as evidence in policymaking, Evidence & Policy 10, 24358

  • Feldman, R, 2013, Techniques and applications for sentiment analysis, Communications of the ACM 56, 829 doi: 10.1145/2436256.2436274

  • Finnigan, KS, Daly, AJ, Che, J, 2013, Systemwide reform in districts under pressure: the role of social networks in defining, acquiring, using, and diffusing research evidence, Educational Administration 51, 47697 doi: 10.1108/09578231311325668

    • Search Google Scholar
    • Export Citation
  • Freeman, R, Griggs, S, Boaz, A, 2011, The practice of policymaking, Evidence & Policy 7, 12736

  • Freeman, R, Maybin, J, 2011, Documents, practices and policy, Evidence & Policy 7, 15570

  • Goffman, E, 1974, Frame analysis: an essay on the organization of experience, New York: Harper Colophon

  • Goldie, D, Linick, M, Jabbar, H, Lubienski, C, 2014, Using bibliometric and social media analyses to explore the ‘echo chamber’ hypothesis, Educational Policy 28, 281305 doi: 10.1177/0895904813515330

    • Search Google Scholar
    • Export Citation
  • Gormley, WT, 2011, From science to policy in early childhood education, Science 333, 97881 doi: 10.1126/science.1206150

  • Haunschild, R, Bornmann, L, 2017, How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data, Scientometrics 110, 120916 doi: 10.1007/s11192-016-2237-2

    • Search Google Scholar
    • Export Citation
  • Hayes, AF, Krippendorff, K, 2007, Answering the call for a standard reliability measure for coding data, Communication Methods and Measures 1, 7789 doi: 10.1080/19312450709336664

    • Search Google Scholar
    • Export Citation
  • Jackson, S, Jacobs, S, 1980, Structure of conversational argument: pragmatic bases for the enthymeme, Quarterly Journal of Speech 66, 25165 doi: 10.1080/00335638009383524

    • Search Google Scholar
    • Export Citation
  • Kingdon, JW, 2011, Agendas, alternatives, and public policies, Boston: Longman

  • Krippendorff, K, 2013, Content analysis: an introduction to its methodology, Los Angeles and London: Sage

  • Majone, G, 1989, Evidence, argument, and persuasion in the policy process, New Haven, CT: Yale University Press

  • Makkar, SR, Brennan, S, Turner, T, Williamson, A, Redman, S, Green, S, 2016, The development of SAGE: a tool to evaluate how policymakers engage with and use research in health policymaking, Research Evaluation 25, 31528 doi: 10.1093/reseval/rvv044

    • Search Google Scholar
    • Export Citation
  • Meagher, L, Lyall, C, Nutley, S, 2008. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research, Research Evaluation 17, 16373 doi: 10.3152/095820208X331720

    • Search Google Scholar
    • Export Citation
  • Moat, KA, Lavis, JN, Abelson, J, 2013, How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis, Milbank Quarterly 91, 60448 doi: 10.1111/1468-0009.12026

    • Search Google Scholar
    • Export Citation
  • National Research Council, 2012, Using science as evidence in public policy, Washington, DC: National Academies Press

  • Neal, JW, Neal, ZP, Kornbluh, M, Mills, KJ, Lawlor, JA, 2015, Brokering the research–practice gap: a typology, American Journal of Community Psychology 56, 42235 doi: 10.1007/s10464-015-9745-8

    • Search Google Scholar
    • Export Citation
  • Nutley, SM, Walter, I, Davies, HTO, 2007, Using evidence: how research can inform public services, Bristol: Policy Press

  • O’Keefe, DJ, 2002, Persuasion: theory and research, Thousand Oaks, CA: Sage

  • Obama, M, 2012, Let’s move! Raising a healthier generation of kids, Childhood Obesity 8, 1

  • Oliver, K, Innvar, S, Lorenc, T, Woodman, J, Thomas, J, 2014, A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research 14, 112 doi: 10.1186/1472-6963-14-1

    • Search Google Scholar
    • Export Citation
  • Oliver, KA, de Vocht, F, 2017, Defining ‘evidence’ in public health: a survey of policymakers’ uses and preferences, European Journal of Public Health 27, 11217

    • Search Google Scholar
    • Export Citation
  • Oxman, AD, Lavis, JN, Lewin, S, Fretheim, A, 2009, SUPPORT Tools for evidence-informed health policymaking (STP) 1: what is evidence-informed policymaking? Health Research Policy and Systems, 7, S1

    • Search Google Scholar
    • Export Citation
  • Palinkas, LA, Garcia, AR, Aarons, GA, Finno-Velasquez, M, Holloway, IW, Mackie, TI, Leslie, LK, Chamberlain, P, 2016, Measuring use of research evidence: the structured interview for evidence use, Research on Social Work Practice 26, 55064 doi: 10.1177/1049731514560413

    • Search Google Scholar
    • Export Citation
  • Pearce, W, Wesselink, A, Colebatch, H, 2014, Evidence and meaning in policymaking, Evidence & Policy 10, 1615

  • Petty, RE, Cacioppo, JT, 1981, Attitudes and persuasion – classic and contemporary approaches, Dubuque, IA: WC Brown Co

  • Prior, L, 2003, Using documents in social research, Thousand Oaks, CA: Sage

  • Ritter, A, Lancaster, K, 2013, Measuring research influence on drug policy: a case example of two epidemiological monitoring systems, International Journal of Drug Policy 24, 3037 doi: 10.1016/j.drugpo.2012.02.005

    • Search Google Scholar
    • Export Citation
  • Scheufele, DA, Tewksbury, D, 2007, Framing, agenda setting, and priming: the evolution of three media effects models, Communication 57, 920

    • Search Google Scholar
    • Export Citation
  • Schlesinger, M, Lau, RR, 2000, The meaning and measure of policy metaphors, American Political Science Review 94, 61126 doi: 10.2307/2585834

    • Search Google Scholar
    • Export Citation
  • Sproles, C, 2011, Federal Digital System (Fdsys), Government Information Quarterly 28, 129 doi: 10.1016/j.giq.2010.07.004

  • Stiff, JB, Mongeau, PA, 2003, Persuasive communication, New York: Guilford Press

  • Stryker, JE, Wray, RJ, Hornik, RC, Yanovitzky, I, 2006, Validation of database search terms for content analysis: the case of cancer news coverage, Journalism & Mass Communication Quarterly 83, 41330 doi: 10.1177/107769900608300212

    • Search Google Scholar
    • Export Citation
  • Toulmin, S, 2003, The uses of argument, Cambridge: Cambridge University Press

  • Tseng, V, 2012, The uses of research in policy and practice, Social Policy Report 26, 116 doi: 10.1002/j.2379-3988.2012.tb00071.x

  • Tversky, A, Kahneman, D, 1981, The framing of decisions and the psychology of choice, Science 211, 4538 doi: 10.1126/science.7455683

  • Van De Goor, I, Hamalainen, RM, Syed, A, Lau, CJ, Sandu, P, Spitters, H, Karlsson, LE, Dulf, D, Valente, A, Castellani, T, Aro, AR, Consortium, R, 2017, Determinants of evidence use in public health policymaking: results from a study across six EU countries, Health Policy 121, 27381 doi: 10.1016/j.healthpol.2017.01.003

    • Search Google Scholar
    • Export Citation
  • Weiss, CH, 1979, The many meanings of research utilization, Public Administration Review 39, 42631 doi: 10.2307/3109916

  • Zardo, P, Collie, A, 2014, Measuring use of research evidence in public health policy: a policy content analysis, BMC Public Health 14, 110 doi: 10.1186/1471-2458-14-1

    • Search Google Scholar
    • Export Citation
Itzhak Yanovitzky Rutgers University School of Communication and Information, USA

Search for other papers by Itzhak Yanovitzky in
Current site
Google Scholar
Close
and
Matthew Weber Rutgers University School of Communication and Information, USA

Search for other papers by Matthew Weber in
Current site
Google Scholar
Close

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 1247 734 57
Full Text Views 111 56 3
PDF Downloads 130 70 5

Altmetrics

Dimensions

Evidence & Policy
A journal of research, debate and practice