Does evaluation quality enhance evaluation use?
Background:
Evaluations are a useful tool to learn more about the effectiveness of public measures. In the era of evidence-based policymaking, recent research suggests that quality is an important determinant of the utilisation of evaluations. Despite this claim, hardly any empirical study has investigated whether the quality of an evaluation – measured by a meta-evaluation – influences its perceived utilisation by decision makers.
Aims and objectives:
This article asks how the quality of an evaluation study is related to its perceived use, and investigates the relationship between the quality of an evaluation, assessed through a meta-evaluation, and how the evaluation is perceived and accepted by the parties concerned.
Methods:
The basis for the empirical analyses were 34 external evaluations, conducted from 2006 to 2014, of upper secondary schools in the canton of Zurich, as well as a standardised survey conducted among 307 representatives of these schools (teachers, administrators, members of quality development teams, and the heads of school oversight commissions).
Findings:
We conclude that the quality of the evaluation, as assessed in a meta-evaluation, is not particularly associated with the perception of evaluation quality and the perceived use of the evaluation. The perceived quality, however, is related to the perceived impact of an evaluation.
Discussion and conclusion:
These findings are relevant for evaluation research and practice, since they show that the quality of an evaluation and evaluation use do not necessarily go hand in hand.
-
Alkin, M.C. and King, J.A. (2016) The historical development of evaluation use, American Journal of Evaluation, 37(4): 568–79. doi: 10.1177/1098214016665164
-
Alkin, M.C. and Taut, S.M. (2003) Unbundling evaluation use, Studies in Educational Evaluation, 29(1): 1–12. doi: 10.1016/S0191-491X(03)90001-0
-
Ayers, T.D. (1987) Stakeholders as partners in evaluation: a stakeholder-collaborative approach, Evaluation and Program Planning, 10(3): 263–71. doi: 10.1016/0149-7189(87)90038-3
-
Böhm-Kasper, O., Selders, M. and Lambrecht, M. (2016) Schulinspektion und Schulentwicklung: Ergebnisse der quantitativen Schulleitungsbefragung, in Arbeitsgruppe Schulinspektion (eds) Schulinspektion als Steuerungsimpuls?, Wiesbaden: Springer, pp 1–50. doi: 10.1007/978-3-658-10872-4_1
-
Böttcher, W. and Keune, M. (2010) Funktionen und Effekte der Schulinspektion, in W. Böttcher, J.N. Dicke and N. Hogrebe (eds) Evaluation, Bildung und Gesellschaft, Münster: Waxmann, pp 151–64.
-
Bundi, P. (2016) What do we know about the demand for evaluation? Insights from the parliamentary arena, American Journal of Evaluation, 37(4): 522–41. doi: 10.1177/1098214015621788
-
Bundi, P., Varone, F., Gava, R. and Widmer, T. (2018) Self-selection and misreporting in legislative surveys, Political Science Research and Methods, 6(4): 771–89. doi: 10.1017/psrm.2016.35
-
Bustelo, M. (2002) Metaevaluation as a Tool for the Improvement and Development of the Evaluation Function in Public Administrations, Paper presented at the 5th European Evaluation Society Biennial Conference, Seville, Spain.
-
Cook, T.D. and Gruder, C.L. (1978) Metaevaluation research, Evaluation Quarterly, 2(1): 5–51. doi: 10.1177/0193841X7800200101
-
Cook, T.D. and Shadish, W.R. (1982) Metaevaluation: an assessment of the congressionally mandated evaluation system for community mental health centers, in G.J. Stahler and W.R. Tash (eds) Innovative Approaches to Mental Health Evaluation, New York: Academic Press, pp 221–53.
-
Cooksy, L.J. and Caracelli, V.J. (2009) Metaevaluation in practice: selection and application of criteria, Journal of MultiDisciplinary Evaluation, 6(11): 1–15.
-
Cousins, J.B. (1995) Assessing program needs using participatory evaluation: a comparison of high and marginal success cases, in J.B. Cousins and L.M. Earl (eds) Participatory Evaluation in Education: Studies in Evaluation Use and Organizational Learning, London: Routledge, pp 55–71.
-
Cousins, J.B. and Leithwood, K.A. (1986) Current empirical research on evaluation utilization, Review of Educational Research, 56: 331–65. doi: 10.3102/00346543056003331
-
Cousins, J.B. and Leithwood, K.A. (1992) Enhancing knowledge utilization as a strategy for school improvement, Knowledge: Creation, Diffusion, Utilization, 14(3): 305–33.
-
Cronbach, L.J. (2000) Course improvement through evaluation, in D.L. Stufflebeam, G.F. Madaus and T. Kellaghan (eds) Evaluation Models: Evaluation in Education and Human Services, Dordrecht: Springer.
-
Daigneault, P.M. (2014) Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: a systematic map, Evaluation and Program Planning, 45(2): 171–81. doi: 10.1016/j.evalprogplan.2014.04.003
-
Davies, H.T.O., Nutley, S.M. and Smith, P.C. (2000) What Works? Evidence-based Policy and Practice in Public Services, Bristol: Policy Press.
-
Dedering, K. and Mueller, S. (2011) School improvement through inspections? First empirical insights from Germany, Journal of Educational Change, 12(3): 301–22. doi: 10.1007/s10833-010-9151-9
-
Donaldson, M.L., Woulfin, S., LeChasseur, K. and Cobb, C.D. (2016) The structure and substance of teachers’ opportunities to learn about teacher evaluation reform: promise or pitfall for equity?, Equity and Excellence in Education, 49(2): 183–201. doi: 10.1080/10665684.2016.1144831
-
Earl, L.M. (1995) District-wide evaluation of school improvement: a system partners approach, in J.B. Cousins and L.M. Earl (eds) Participatory Evaluation in Education: Studies in Evaluation Use and Organizational Learning, London: Routledge, pp 21–32.
-
Ehren, M.C.M. and Visscher, A.J. (2006) Towards a theory on the impact of school inspections, British Journal of Educational Studies, 54(1): 51–72. doi: 10.1111/j.1467-8527.2006.00333.x
-
Ehren, M.C.M., Gustafsson, J.E., Altrichter, H., Skedsmo, G., Kemethofer, D. and Huber, S.G. (2015) Comparing effects and side effects of different school inspection systems across Europe, Comparative Education, 51(3): 375–400. doi: 10.1080/03050068.2015.1045769
-
Fleischer, D.N. and Christie, C.A. (2009) Evaluation use: results from a survey of U.S. American evaluation association members, American Journal of Evaluation, 30(2): 158–75. doi: 10.1177/1098214008331009
-
Frey, K. (2010) Revising road safety policy: the role of systematic evidence in Switzerland, Governance, 23(4): 667–90. doi: 10.1111/j.1468-0491.2010.01503.x
-
Gaertner, H., Wurster, S. and Pant, H.A. (2014) The effect of school inspections on school improvement, School Effectiveness and School Improvement: An International Journal of Research, Policy and Practice, 25(4): 489–508. doi: 10.1080/09243453.2013.811089
-
Gustafsson, J.E., Ehren, M.C.M., Conyngham, G., McNamara, G., Altrichter, H. and O’Hara, J. (2015) From inspection to quality: ways in which school inspection influences change in schools, Studies in Educational Evaluation, 47(1): 47–57. doi: 10.1016/j.stueduc.2015.07.002
-
Hatry, H.P. (1980) Pitfalls of evaluation, in G. Majone and E.S. Quade (eds) Pitfalls of Analysis, Chichester: John Wiley, pp 159–78.
-
Head, B.W. (2010) Reconsidering evidence-based policy: key issues and challenges, Policy and Society, 29(2): 77–94. doi: 10.1016/j.polsoc.2010.03.001
-
Henry, G.T. and Mark, M.M. (2003) Beyond use: understanding evaluation’s influence on attitudes and actions, American Journal of Evaluation, 24(3): 293–314.
-
House, E.R. (1980) Evaluating with Validity, Beverly Hills, CA: Sage.
-
House, E.R. (1993) Professional Evaluation: Social Impact and Political Consequences, Beverly Hills, CA: Sage.
-
Husfeldt, V. (2011) Wirkungen und Wirksamkeit der externen Schulevaluation. Überblick zum Stand der Forschung, Zeitschrift für Erziehungswissenschaft, 14(2): 259–82.
-
Jacobson, M.R. and Azzam, T. (2016) Methodological credibility: an empirical investigation of the public’s perception of evaluation findings and methods, Evaluation Review, 40(1): 29–60. doi: 10.1177/0193841X16657728
-
Johnson, K., Greenseid, L.O., Toal, S.A., King, J.A., Lawrenz, F. and Volkov, B. (2009) Research on evaluation use: a review of the empirical literature from 1986 to 2005, American Journal of Evaluation, 30(3): 377–410. doi: 10.1177/1098214009341660
-
Joint Committee on Standards for Educational Evaluation (1981) Standards for Evaluations of Educational Programs, Projects, and Materials, New York: McGraw-Hill.
-
Joint Committee on Standards for Educational Evaluation (1994) The Program Evaluation Standards: How to Assess Evaluations of Educational Programs, 2nd edn, Thousand Oaks, CA: Sage.
-
Joint Committee on Standards for Educational Evaluation (2011) The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd edn, Los Angeles, CA: Sage.
-
King, J. A., and Alkin, M. C. (2019) The centrality of use: Theories of evaluation use and influence and thoughts on the first 50 years of use research, American Journal of Evaluation, 40(3): 431–58.
-
Kirkhart, K.E. (2004) Reconceptualizing evaluation use: an integrated theory of influence, New Directions for Evaluation, 2000(88): 5–23. doi: 10.1002/ev.1188
-
Lafleur, C. (1995) A participatory approach to district-level program evaluation: the dynamics of internal evaluation, in J.B. Cousins and L.M. Earl (eds) Participatory Evaluation in Education: Studies in Evaluation Use and Organizational Learning, London: Routledge, pp 33–54.
-
Ledermann, S. (2012) Exploring the necessary conditions for evaluation use in program change, American Journal of Evaluation, 33(2): 159–78. doi: 10.1177/1098214011411573
-
Lee, L.E. and Cousins, J.B. (1995) Participation in evaluation of funded school improvement: effects and supporting conditions, in J.B. Cousins and L.M. Earl (eds) Participatory Evaluation in Education: Studies in Evaluation use and Organizational Learning, London: Routledge, pp 72–85.
-
Leithwood, K.A. (1986) The Role of the Secondary School Principal in Policy Implementation and School Improvement. Publication Sales, The Ontario Institute for Studies in Education, 252 Bloor Street West, Toronto, Ontario M5S 1V6 Canada.
-
Mark, M.M. and Henry, G.T. (2004) The mechanisms and outcomes of evaluation influence, Evaluation, 10(1): 35–57. doi: 10.1177/1356389004042326
-
Marsh, D.D. and Glassick, J.M. (1988) Knowledge utilization in evaluation efforts: the role of recommendations, Knowledge, 9(3): 323–41. doi: 10.1177/107554708800900301
-
Mayring, P. (2015) Qualitative Inhaltsanalyse: Grundlagen und Techniken, 12. Auflage, Weinheim: Beltz.
-
Miller, R.L. (2015) How people judge the credibility of information: lessons for cognitive and information sciences, in S.I. Donaldson, C.A. Christie and M.M. Mark (eds) Credible and Actionable Evidence: The Foundation for Rigorous and Influential Evaluations, Thousand Oaks, CA: Sage, pp 39–62.
-
Newman, D.L., Brown, R.D. and Rivers, L. (1987) Factors influencing the decision-making process: an examination of the effect of contextual variables, Studies in Educational Evaluation, 13(2): 199–209. doi: 10.1016/S0191-491X(87)80034-2
-
Nutley, S., Walter, I. and Davies, H.T.O. (2003) From knowing to doing: a framework for understanding the evidence-into-practice agenda, Evaluation, 9(2): 125–48. doi: 10.1177/1356389003009002002
-
Odom, S.L., Brantlinger, E., Gersten, R., Horner, R.H., Thompson, B. and Harris, K.R. (2005) Research in special education: scientific methods and evidence-based practices, Exceptional Children, 71(2): 137–48. doi: 10.1177/001440290507100201
-
Oliver, K., Lorenc, T. and Tinkler, J. (2020) Evaluating unintended consequences: New insights into solving practical, ethical and political challenges of evaluation, Evaluation, 26(1): 61–75.
-
Olsen, R.B., Orr, L.L., Bell, S.H. and Stuart, E.A. (2013) External validity in policy evaluations that choose sites purposively, Journal of Policy Analysis and Management, 32(1): 107–21. doi: 10.1002/pam.21660
-
Ouimet, M., Landry, R., Ziam, S. and Bédard, P. O. (2009) The absorption of research knowledge by public civil servants, Evidence & Policy: A Journal of Research, Debate and Practice, 5(4): 331–50.
-
Patton, M.Q. (1997) Utilization-focused Evaluation: The New Century Text, 3rd edn, Thousand Oaks, CA: Sage.
-
Pawson, R. (2006) Evidence-based Policy: A Realist Perspective, London: Sage.
-
Penninckx, M., Vanhoof, J., De Maeyer, S. and Van Petegem, P. (2016) Explaining effects and side effects of school inspections: a path analysis, School Effectiveness and School Improvement, 27(3): 333–47. doi: 10.1080/09243453.2015.1085421
-
Petrosino, A., Petrosino, C.T. and Buehler, J. (2005) ‘Scared Straight’ and other juvenile awareness programs for preventing juvenile delinquency, Campbell Systematic Reviews, 1(1): 1–62.
-
Potts, S.A.K. (1998) Impact of Mixed Method Designs on Knowledge Gain, Credibility, and Utility of Program Evaluation Findings, Phoenix: Arizona State University.
-
Quesel, C., Husfeldt, V. and Bauer, F.D. (2011a) Externe Schulevaluation aus der Sicht von Lehrpersonen, Schulleitungen und lokalen Schulbehörde: Eine explorative Untersuchung zur Education Governance im Kanton Aargau, Aarau: Fachhochschule Nordwest Schweiz.
-
Quesel, C., Husfeldt, V., Landwehr, N. and Steiner, P. (2011b) Wirkungen und Wirksamkeit der externen Schulevaluation, Bern: h.e.p.-Verlag.
-
Rickinson, M., Sharples, J. and Lovell, O. (2020) Towards a better understanding of quality of evidence use, in S. Gorard (ed) Getting Evidence into Education, Abingdon: Routledge, pp 219–33.
-
Rickinson, M., Walsh, L., De Bruin, K. and Hall, M. (2019) Understanding evidence use within education policy: a policy narrative perspective, Evidence & Policy, 15(2): 235–52.
-
Rogers, W. (1993) Quantile regression standard errors, Stata Technical Bulletin, 2(9): 1–28.
-
Rossi, P.H., Lipsey, M.W. and Henry, G.T. (2018) Evaluation: A Systematic Approach, Thousand Oaks, CA: Sage.
-
Russon, C. and Russon, G. (eds) (2004) International perspectives on evaluation standards, New Directions for Evaluation, San Francisco: Jossey-Bass, 104.
-
Sanders, W.L. and Horn, S.P. (1998) Research findings from the Tennessee Value-Added Assessment System (TVAAS) database: implications for educational evaluation and research, Journal of Personnel Evaluation in Education, 12(3): 247–56. doi: 10.1023/A:1008067210518
-
Schwartz, R. and Mayne, J. (2005) Assuring the quality of evaluative information: theory and practice, Evaluation and Program Planning, 28(1): 1–14. doi: 10.1016/j.evalprogplan.2004.10.001
-
Scriven, M. (1969) An introduction to meta-evaluation, Educational Product Report, 2(5): 36–8.
-
Seidel, T., Mok, S.Y., Hetmanek, A. and Knogler, M. (2017) Meta-Analysen zur Unterrichtsforschung und ihr Beitrag für die Realisierung eines Clearing House Unterricht für die Lehrerbildung, Zeitschrift für Bildungsforschung, 7(3): 311–25. doi: 10.1007/s35834-017-0191-6
-
Slavin, R.E. (2008) Perspectives on evidence-based research in education: what works? Issues in synthesizing educational program evaluations, Educational Researcher, 37(1): 5–14. doi: 10.3102/0013189X08314117
-
Steenbergen, M.R. and Jones, B.S. (2002) Modeling multilevel data structures, American Journal of Political Science, 46(1):218–37. doi: 10.2307/3088424
-
Stufflebeam, D.L. (1974) Meta-evaluation, Evaluation Center Occasional Paper Series #3, Kalamazoo, MI: Western Michigan University.
-
Stufflebeam, D.L. (2001) The metaevaluation imperative, American Journal of Evaluation, 22(2): 183–209. doi: 10.1177/109821400102200204
-
Turnbull, B. (1999) The mediating effect of participation efficacy on evaluation use, Evaluation and program Planning, 22(2): 131–40.
-
Verhoest, K., Verschuere, B. and Bouckaert, G. (2007) Pressure, legitimacy, and innovative behavior by public organizations, Governance, 20(3): 469–97. doi: 10.1111/j.1468-0491.2007.00367.x
-
Weiss, C.H. (1977) Research for policy’s sake: the enlightenment function of social research, Policy Analysis, 3(4): 531–45.
-
Weiss, C.H. (1998) Have we learned anything new about the use of evaluation?, American Journal of Evaluation, 19(1): 21–33. doi: 10.1177/109821409801900103
-
Weiss, C.H. and Bucuvalas, M.J. (1980) Truth tests and utility tests: decision-makers’ frames of reference for social science research, American Sociological Review, 45(2): 302–13. doi: 10.2307/2095127
-
Weiss, C.H., Murphy-Graham, E. and Birkeland, S. (2005) An alternate route to policy influence: how evaluations affect DARE, American Journal of Evaluation, 26(1): 12–30. doi: 10.1177/1098214004273337
-
Weiss, C.H., Murphy-Graham, E., Petrosino, A. and Gandhi, A.G. (2008) The fairy godmother – and her warts: making the dream of evidence-based policy come true, American Journal of Evaluation, 29(1): 29–47. doi: 10.1177/1098214007313742
-
Widmer, T. (1996) Meta-Evaluation: Kriterien zur Bewertung von Evaluationen, Bern: Haupt.
-
Widmer, T. (2004) The development and status of evaluation standards in Western Europe, New Directions for Evaluation, 2004(104): 31–42. doi: 10.1002/ev.134
-
Widmer, T. (2005) Instruments and procedures for assuring evaluation quality: a Swiss perspective, in R. Schwartz and J. Mayne (eds) Quality Matters. Seeking Confidence in Evaluating, Auditing and Performance Reporting, New Brunswick: Transaction Publishers, pp 41–68.
-
Widmer, T., Landert, C. and Bachmann, N. (2000) Evaluations-Standards der Schweizerischen Evaluationsgesellschaft (SEVAL-Standards), Bern: SEVAL.
-
Widmer, T., Frey, K., Rüegg, N., Stadter, C., Bundi, P. and Stein, J. (2015) Qualität der IFES-Schulevaluationen und deren Nutzung im Kanton Zürich, Zürcher Politik- & Evaluationsstudien Nr. 13, Zürich: Institut für Politikwissenschaft.
| May 2022 onwards | Past Year | Past 30 Days | |
|---|---|---|---|
| Abstract Views | 625 | 624 | 60 |
| Full Text Views | 76 | 76 | 1 |
| PDF Downloads | 32 | 32 | 2 |
We are a leading social science publisher
committed to making a difference.
Bristol University Press and
Policy Press
bristoluniversitypress.co.uk