This chapter explores what data, evidence and feedback practices are helpful or unhelpful for knowing whether initiatives are having their intended impact. It considers some of the cultures and practices that can get in the way of having the required feedback for understanding change.
It sets out the kinds of data and evidence that are best suited to understanding complex change. Drawing on experience of working in different settings, this chapter explores the concept of ‘data cultures’ and how these data cultures can help or hinder initiatives in getting the data they need. Examples highlight ways of applying this thinking to any specific work, especially making the case for using mixed data that include qualitative data. The concept of a data improvement journey is introduced, along with practical ways of thinking about the data and evidence needs in any specific setting.
Ahmed, I.K. and Dantata, B.S. (2016) ‘Problems and challenges of policy implementation for national development’, Research on Humanity and Social Sciences, 6(15): 60–65.
Archibald, T., Sharrock, G., Buckley, J. and Cook, N. (2016) ‘Assumptions, conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development’, Evaluation and Program Planning, 59: 119–127.
Australian Government (2019) ‘Delivering for Australians. A world-class Australian public service: The government’s APS reform agenda’, Department of the Prime Minister and Cabinet [online] 13 December. Available from: https://www.pmc.gov.au/resource-centre/government/delivering-for-australians [Accessed 4 October 2021].
Bicket, M., Christie, I., Gilbert, N., Hills, D., Penn, A. and Wilkinson, H. (2020) Magenta Book 2020: Supplementary Guide: Handling Complexity in Policy Evaluation, London: HM Treasury.
Black, L. (2021) ‘Funding Utopia’ [online]. Available from: https://leah-black.medium.com/funding-utopia-2-ten-fifteen-years-unrestricted-outcomes-community-wealth-and-more-b32760b32dd3 [Accessed 15 March 2022].
Blamey, A. and Mackenzie, M. (2007) ‘Theories of change and realistic evaluation: Peas in a pod or apples and oranges?’, Evaluation, 13(4): 439–455.
Bozarth, J.D. (2013) ‘Unconditional positive regard’, in M. Cooper, M. O’Hara, P.F. Schmid and A.C. Bohart (eds) The Handbook of Person-Centred Psychotherapy and Counselling, New York: Palgrave Macmillan/Springer Nature, pp 180–192.
Cameron, W.B. (1969) Informal Sociology: A casual introduction to sociological thinking, New York: Random House.
CECAN – Centre for Evaluating Complexity Across the Nexus (2020) ‘Supplementary Guide: Handling complexity in policy evaluation’, HM Treasury [online]. Available from: https://www.cecan.ac.uk/news/handling-complexity-in-policy-evaluation-magenta-book-2020-supplementary-guide/ [Accessed 4 October 2021].
Christie, C. (2011) ‘Christie Commission on the future delivery of public services’, Scottish Government [online] 29 June. Available from: https://www.gov.scot/publications/commission-future-delivery-public-services/ [Accessed 4 October 2021].
Cook, A. (2015) ‘Partnership working across UK public services: Evidence review’, Edinburgh: What Works Scotland [online]. Available from: http://whatworksscotland.ac.uk/wp-content/uploads/2015/12/WWS-Evidence-Review-Partnership-03-Dec-2015-.pdf [Accessed 7 October 2021].
Cook, A. (2017) ‘Outcome based approaches in public service reform’, Edinburgh: What Works Scotland [online]. Available from: http://whatworksscotland.ac.uk/wp-content/uploads/2017/04/OutcomeBasedApproachesinPublicServiceReform.pdf [Accessed 25 February 2022].
Cook, A. and Miller, E. (2012) Talking Points Personal Outcomes Approach: A practical guide for organisations, Edinburgh: Joint Improvement Team, Scottish Government.
Crawford, C., Edwards, A., Farquharson, C., Greaves, E., Trevelyan, G., Wallace, E. and White, C. (2016) Magic Breakfast: Evaluation report and executive summary, London: The Education Endowment Foundation.
Crewe, E. and Young, J. (2002) Bridging Research and Policy: Context, evidence and links, London: Overseas Development Institute.
Darnton, A. and Horne, J. (2013) ‘Influencing behaviours – moving beyond the individual: ISM user guide’, Scottish Government [online]. Available from: https://www.gov.scot/publications/influencing-behaviours-moving-beyond-individual-user-guide-ism-tool/ [Accessed 1 March 2022].
Davies, H.T. and Nutley, S.M. (2000) ‘Healthcare: Evidence to the fore’, in H.T. Davies, S.M. Nutley and P.C. Smith (eds) What Works: Evidence-based policy and practice in public services, Bristol: Policy Press, pp 43–67.
Gabbay, J. and le May, A. (2004) ‘Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care’, BMJ, 329: 1013 https://doi.org/10.1136/bmj.329.7473.1013
Garvin, D.A. (1993) ‘Building a learning organization’, Harvard Business Review, 71(4): 78–91 [online]. Available from: https://hbr.org/1993/07/building-a-learning-organization [Accessed 5 October 2021].
Glendinning, C., Clarke, S., Hare, P., Kitchetkova, I., Maddison, J. and Newbronner, L. (2006) Outcomes-focused Services for Older People, Bristol: Policy Press.
Government of Canada. ‘Theory-based approaches to evaluation: Concept and practices’ [online]. Available from: https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/theory-based-approaches-evaluation-concepts-practices.html [Accessed 5 October 2021].
Huff, D. and Geis, I. (1954) How to Lie with Statistics, New York: Norton.
Matter of Focus ‘Understand the unique context of your work’ [online]. Available from: https://www.matter-of-focus.com/understand-the-unique-context-of-your-work/ [Accessed 5 October 2021].
Matter of Focus ‘3 feedback tools to help you track your outcomes and impact’ [online]. Available from: https://www.matter-of-focus.com/3-feedback-tools/ [Accessed 7 October 2021].
Mayne, J. (2008) ‘Contribution analysis: An approach to exploring cause and effect’, ILAC Brief 16, pp 1–4. Available from: https://hdl.handle.net/10568/70124 [Accessed 7 October 2021].
Miller, E. and Barrie, K. (2016) ‘Learning from the Meaningful and Measurable Project: Strengthening links between identity, action and decision-making’, Healthcare Improvement Scotland [online]. Available from: https://ihub.scot/media/2126/po-learning.pdf [Accessed 25 February 2022].
Montague, S. (2012) ‘Theory-based approaches for practical evaluation’ [online]. Available from: https://www.pmn.net/wp-content/uploads/Theory-based-Approaches-for-Practical-Evaluation.pdf [Accessed 5 October 2021].
Morton, S. (2012) Exploring and Assessing Social Research Impact: A case study of a research partnership’s impacts on policy and practice, Edinburgh: University of Edinburgh.
Morton, S. (2015) ‘Creating research impact: The roles of research users in interactive research mobilisation’, Evidence & Policy: A Journal of Research, Debate and Practice, 11(1): 35–55.
Morton, S. and Casey, T. (2017) Changing National Policy on Violence Affecting Children: An impact assessment of UNICEF and partners’ multi-country study on the drivers of violence affecting children in Peru, Edinburgh: University of Edinburgh.
Morton, S., Grant, A., Cook, A., Berry, H., McMellon, C., Robbin, M. and Ipince, A. (2019) Children’s Experiences Online: Building global understanding and action, Innocenti [online]. Available from: https://www.unicef-irc.org/publications/1065-childrens-experiences-online-building-global-understanding-and-action.html [Accessed 7 October 2021].
National Performance Framework, Scottish Government website [online]. Available from: https://nationalperformance.gov.scot/ [Accessed 5 October 2021].
Needham, C. and Mangan, C. (2016) ‘The 21st century public servant: Working at three boundaries of public and private’, Public Money and Management, 36(4): 265–272.
Osborne, S. and Strokosch, K. (2013) ‘It takes two to tango? Understanding the co-production of public services by integrating the services management and public administration perspectives’, British Journal of Management, 24: 1–29.
Patton, M.Q. (2011) Developmental Evaluation: Applying complexity concepts to enhance innovation and use, New York: Guilford Press.
Public Health England (2019) ‘Health matters: Whole systems approach to obesity’, UK Government Publication [online] 25 July. Available from: https://www.gov.uk/government/publications/health-matters-whole-systems-approach-to-obesity/health-matters-whole-systems-approach-to-obesity [Accessed 4 October 2021].
Rogers, P.J. (2008) ‘Using programme theory to evaluate complicated and complex aspects of interventions’, Evaluation, 14(1): 29–48.
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H. and Jinks, C. (2018) ‘Saturation in qualitative research: Exploring its conceptualization and operationalization’, Quality & Quantity, 52(4): 1893–1907.
Schein, E.H. (1985) ‘Defining organizational culture’, Classics of Organization Theory, 3(1): 490–502.
Senge, P.M. (1990) The Fifth Discipline, New York: Random House Business.
Shove, E., Pantzar, M. and Watson, M. (2012) The Dynamics of Social Practice: Everyday life and how it changes, London: SAGE Publications Ltd.
Social Care Wales (2022) ‘Friend not foe: Supporting meaningful outcomes focused recording in social care in Wales’, Welsh Government [online]. Available from: https://socialcare.wales/cms_assets/file-uploads/Friend-not-Foe-English.pdf [Accessed 25 February 2022].
The Manchester Partnership (2021) ‘Public Service Reform’ [online]. Available from: https://www.manchester.gov.uk/manchesterpartnership/info/5/public_service_reform [Accessed 4 October 2021].
United Nations Sustainable Development Goals [online]. Available from: https://sdgs.un.org/ [Accessed 6 October 2021].
Weiss, C.H. (1998) Evaluation: Methods for studying programs and policies, Upper Saddle River, NJ: Prentice Hall.
Whitebloom, S. (2020) ‘“Care” can and should work but it is a postcode lottery for young care leavers’, Oxford News Blog, Oxford: University of Oxford website [online]. Available from: https://www.ox.ac.uk/news/arts-blog/care-can-and-should-work-it-postcode-lottery-young-care-leavers [Accessed 5 October 2021].
World Health Organization (2016) ‘INSPIRE: 7 strategies to end violence against children’ [online]. Available from: https://www.who.int/publications/i/item/inspire-seven-strategies-for-ending-violence-against-children [Accessed 7 October 2021].
|May 2022 onwards||Past Year||Past 30 Days|
|Full Text Views||0||0||0|