Abstract
Evidence-based policy has at its foundation a set of ideas about what makes evidence valid so that it can be trusted in the creation of policy. This validity is frequently conceptualised in terms of rigour deriving from scientific studies which adhere to highly structured processes around data collection, analysis and inscription. In comparison, the knowledge gained from lived experience, while viewed as important for ensuring that policy meets the needs of the people it is trying to serve, is characterised by its tacit nature, unstructure and difficulty in transferring from one actor to another. Validity of experiential knowledge in policy arises from the connection of policy knowledge to the lived experience of individuals. This paper considers validity in this context through exploring four modes in which experiential knowledge is currently utilised within policy. The tensions surrounding validity in the policy context find resolution through the development of a situated notion of validity decoupled from structural rigour and recoupled to context.
Key messages
Evidence-based policy relies heavily on a validity drawn from very highly structured processes which assure rigour.
Experiential knowledge is validated through its connection to the lived experience of individuals and does not easily fit into the structured processes of validity ascribed for evidence-based policy.
In order for the inclusion of experiential knowledge in policy to be useful it must be not only a reflection of lived experience but transferred into policy in a way that is interpretable by policy actors.
Processes that do this are valid because they meet the knowledge needs of both contexts.
Introduction
Knowledge from expertise by experience is increasingly utilised in the discourse and actions of policy and practice documentation and consultation processes (for example, Australian Government, 2016; NSW Government, 2016). Through my own work at the intersection between policy, practice and academia, and as an academic who regularly collaborates with ‘consumers’1, I have become interested in the ways in which such knowledge becomes validated in a context which otherwise expresses validation of very different, highly-structured forms of knowledge under the banner of evidence-based policy. Marston and Watts (in Lancaster et al, 2017) write of evidence-based policy as ‘a powerful metaphor in shaping what forms of knowledge are considered closest to the “truth” in decision-making processes’. My concern in this paper is to discuss the validity and acceptability of experiential knowledge in relation to the structured notions of rigour and validity evident in standard definitions of evidence-based policy. I do this by discussing four practices by which experiential knowledge is currently enacted in policy. I focus my discussion on the fields of mental health and disability because these are fields where consumer knowledge has been readily accepted as a form of evidence over the last decade. The approach I take to explore experiential knowledge in this paper broadly sits under the field of interpretive policy analysis. An interpretive approach to policy focuses on the creation and enactment of knowledge for policy as contextually situated, with work in the field generally focusing on interpretation of meaning, process and outcomes of policy by different actors, which can lead to different policy outcomes.
Background
Evidence-based policy
Critical policy researchers have shown over a considerable period that instrumental accounts of policy, such as that described in Laswell’s staged account of the policy cycle, do not reflect the reality of practice (for example, Colebatch 2005; 2017). Alternative accounts show policy as comprising of multiple sets of rationalities seemingly irrational in comparison to each other but with logics individual to those involved. Policy is an ‘arena’ where ‘interactive, discursive, negotiation’ around these rationalities takes place (Williams, 2010: 197). Despite such counter discourses those people working to create policy do often strive for an objective, instrumental process, as reflected in guidelines for the creation of evidence-based policy (Colebatch, 2017).
The effects of a policy landscape dominated by evidence-based policy has been one of the key points of debate amongst critical policy analysts over the past 15 years. The very first edition of Evidence & Policy included an article by Hammersley (2005) which reflected on the growing hegemony of evidence-based policy making and calls to change research practice in response. He critiqued Iain Chalmers, a founder of the Cochrane Collaboration, for his role in articulating that the priority of research should be in creating ‘rigorous, transparent, up-to-date evaluations’ to inform practice and policy and consequently avoid harm (Chalmers in Hammersley, 2005: 86). In response Hammersley (2005) argues that harm avoidance is not just constituted in the way evidence is gathered, but in the way it is used by humans going about their work. Evidence, he and other critical policy researchers argue, must be linked to the knowledge of experience of the individuals who do the work of policy and practice (for example, Smith, 2013).
Mainstream accounts of evidence-based policy do not universally ignore human experience over more structured evidence. The origins of evidence-based policy lie in a 1992 article by Guyatt and colleagues outlining the new paradigm of medical education termed ‘evidence-based medicine’. This new way of working ‘de-emphasise[d] intuition [and] unsystematic clinical experience’ but did not actually do away with it altogether (Guyatt et al, 1992: 2420). Subsequent manifestations of evidence-based ‘work’ (that is, medicine, practice, policy) have resulted in varying definitions of the validity of expertise. NICE Guidelines in the UK and those developed by WHO, for example, all have a focus on the involvement of expert stakeholders in the process. However, expertise is often solely conceptualised in terms of the experience of expert clinicians (for example, Richter-Sundberg et al, 2017). This expertise is valued and valid because of the expert training that sits behind this knowledge and allows them to sort and classify knowledge as valid or otherwise.
Various models have been proposed which create hierarchies of evidence for policy, including The National Health and Medical Research Council (NHMRC)’s Hierarchy of Evidence for use in Australia (Becker et al, 2010; Cairney, 2017; for example, NHMRC, 2009). This is presented in Figure 1 below. Neither this hierarchy nor the guidance surrounding it describe inclusion of any evidence outside of peer-reviewed journal articles (NHMRC, 2009: 14).
National Health and Medical Research Council Hierarchies of Evidence (NHMRC, 2009:15).
Citation: Evidence & Policy 16, 2; 10.1332/174426419X15700265131524
National Health and Medical Research Council Hierarchies of Evidence (NHMRC, 2009:15).
Citation: Evidence & Policy 16, 2; 10.1332/174426419X15700265131524
National Health and Medical Research Council Hierarchies of Evidence (NHMRC, 2009:15).
Citation: Evidence & Policy 16, 2; 10.1332/174426419X15700265131524
While recent research has pointed to the illusory nature of such highly-structured definitions of evidence-based policy when ‘behind the scenes’ practice settings are studied, the discourses surrounding it remain very powerful (Colebatch, 2017; Cairney, 2017). These official discourses serve to deprioritise and separate valid (scientific, rigorous) and non-valid (experiential, non-rigorous) evidence (Bacchi, 2009; Lancaster et al, 2017). Yet within many policy contexts, including disability and mental health, there has been a revalidation of experiential knowledge. With this has come questions about how the validity of these forms of knowledge can be understood in relation to evidence-based policy (Beresford, 2007; Restall et al, 2011).
Experiential knowledge
Discourse around consumer knowledge in policy and practice has developed significantly over the past 50 years. In Talcott Parsons’ (1951) articulation of the sick role, people interacting with the health system were viewed as sick bodies empty of knowledge about their own illness or experiences of ‘treatment’. These empty, sick bodies could not provide knowledge that was of utility for policy. The combined work of mad-pride, ‘patient’ and disability rights movements over the past 40 years, along with changing practice from within professions, has served to elevate the consumer voice within health policy discourse. It is now seen as an ‘intrinsic good’ which enables a policy to reflect the needs of those who are subject to it (Stewart, 2016).
This potential utility is tempered by the peculiarities of experiential knowledge which make it difficult to grasp and utilise in policy making. Experiential knowledge is held by each person and developed over years as we take on each new experience and integrate it with the knowledge we already have about the world and those around us (Blume, 2017). It is ‘embodied’ in that it is largely held within the body and the mind and sometimes only known through doing (Boardman, 2014; Freeman and Sturdy, 2014; Smith-Merry, 2014). Experiential knowledge is thus difficult to truly be grasped by others, particularly when the receiver doesn’t have a shared set of experiences or repertoire of actions upon which to build understanding (Boardman, 2014). It is also difficult to inscribe into documents (Freeman and Sturdy, 2014). These difficulties with understanding, accounting for and transmitting experiential knowledge in a structured fashion means that it is frequently ‘delegitimised’ or ‘marginalised’ as lacking rigour in the face of evidence-based practice which is easier to access and compare (Lancaster et al, 2017).
Rigour and validity
It might seem antithetical to examine rigour and validity given that experiential knowledge is characterised by unstructure. The focus of the discussion here is on the uneasy assertion of experiential knowledge within the evidence-based policy context dominated by scientific knowledge where validity arises from the rigour of structured knowledge practices.
Rigour is concerned with processes of data collection, inscription and interpretation and is used as a de facto measure of truth or validity in academic data collection and policy. Rigour is a key object in the development of evidence-based policy as it is used to determine the validity of knowledge as a basis for policy. The NHMRC Hierarchy of Evidence, for example, lists standard measures of rigour used for testing studies, including types of study designs that when followed faithfully are viewed as rigorous. Key to rigour in these designs is a strict internal validity represented by the ability researchers have to control data subjects and context. Internal validity is less straightforward to achieve in other methodologies (Sandelowski, 1986), such as interview or ethnographic studies where the ‘messiness’ of people’s lives and interactions is harder to control, and thus rigour is perceived to be reduced.
In standard methodological definitions validity is proven through tools such as reproducibility, reflexivity and internal consistency. However, more critical approaches to methodology question this and link validity to context rather than a set of structured processes (Lub, 2015; for example, Sandelowski, 1993; Rolfe, 2006). Writing about validity as a research method, Sandelowski (1993: 2) comments that if we take a contextual approach and ‘…validity is viewed as a culturally and historically situated process, both experimentalist and interpretivist can be recognised as relying on contextually grounded linguistic and interpretive practices rather than on rules assumed to be sufficiently abstract and universal for every project’. Other authors have built on this and provided a ‘restructuring’ of context by proposing methodological frameworks for internal validity where validity relies on the ability to generalise within a context, rather than beyond (Lub, 2015; for example, Maxwell and Chmiel, 2013). My argument, expressed through the discussion below, is that validity should be similarly contextualised, with validity coming not from the rules of policy but from the context in which knowledge is enacted.
Discussion
This discussion maps out several current strategies for asserting the consumer voice within policy and practice. These are both ‘invited’ and ‘uninvited’ voices into the policy process (Stewart, 2016). I have deliberately chosen not to focus on external policy activism but instead focus on attempts to make consumer voices part of the existing mechanisms of evidence-based policy. These examples are by no means the entirety of current strategies, but they allow current practice to be mapped sufficiently to develop the discussion around validity. In doing this I draw heavily on my own projects and my observations as an insider at the consumer-policy nexus. Examples one and two draw from a series of projects focusing on the creation and implementation of mental health policy in Scotland and Australia (for example, Smith-Merry et al, 2011; Smith-Merry, 2014; Smith-Merry and Gillespie, 2016). Example three is drawn from a review of ‘inclusive’ disability research (Smith-Merry, 2017; Smith-Merry, 2018), and example four from a co-designed project collecting personal narratives from people with mental ill-health (Wayland et al, 2017; Hancock et al, 2018).
Employment of experts by experience in policy
Consumers are frequently employed in paid or voluntary roles as experts by experience within policy organisations. They are usually included within dedicated roles or as members of steering or advisory committees devising and reviewing policy directions, and draw on their own lived experience to ensure that consumer interests are included in policy. These experts are present to represent the consumer voice in proceedings which have, in the past, only included consumer needs via the experience of practitioners. Their inclusion works both to help policy practitioners come to understand the experience of those subject to their policies and to validate policy processes by demonstrating that they are striving towards creating policy appropriate to the context in which it will be implemented (Meriluoto, 2018).
Borkman (1979), who coined the term ‘experiential knowledge’, likened the validity of experiential knowledge to ‘truth’ for an individual in that it is knowledge of an individual’s true experiences (Blume, 2017). The validity of this truth only comes into question when it moves from personal to shared knowledge, through enactment or inscription where truth may be lost in translation (Restall et al, 2011; Smith-Merry, 2014; Freeman and Sturdy, 2014; Blume, 2017). Experiential knowledge must be enacted in a policy-making context very different from that in which it was experienced, and there is no guarantee that just because a person with lived experience is present they will be heard or be able to speak in a way in which their experiential knowledge is effectively received (Blume, 2017; Lancaster et al 2017; Meriluoto, 2018).
The specificity of experiential knowledge makes it hard for one individual consumer to represent the multiple and divergent realities of people with disability or mental illness (Blume, 2017). Some people with valid expertise are also routinely left out of these types of positions, including people who are non-verbal, have high care needs, cognitive processing disorders or severe anxiety. Some experts are not (yet) equipped to understand the context of policy or practice and this can lead people to feel unable to take part in discussions (Stewart, 2016). Consumer representatives can also be subject to stigma if introduced into an organisation that associates consumers with negative attributes, which is common in mental health (Browne and Hemsley, 2008). Stigma distorts the reception of experiential knowledge. Conversely the problem of the ‘professional consumer’ is also spoken about as a problem for the validity of these roles (Butler and Greenhalgh, 2010; Carey, 2011). The professional consumer is accused of becoming co-opted into the policy environment and losing their connection with the ‘authentic’ knowledge of experience. As these examples show, validity of the experiential knowledge of the consumer advocate is open to questions from both a policy and consumer perspective.
Advocacy organisations in policy
Advocacy groups, which may or may not be led by people with a lived experience, often stand in for the consumer voice in policy processes as representative of their clientele or membership. Their knowledge is valid for policy because the members of the organisation are able to communicate with each other and develop ‘empathic’ experience to build up an organisational repertoire of experiential knowledge (Borkman in Boardman, 2014; Smith-Merry, 2014). Rabeharisoa et al (2014) write that consumer ‘organisations which engage in evidence-based activism collect experiences and build experiential knowledge’ which is then actively translated into other settings. Knowledge is translated from those with experience, through the bodies of those in policy or practice engagement roles who have built up the knowledge to be able to speak in policy settings (Borkman in Smith-Merry, 2014; Blume, 2017). In one example from my research, a consumer advocacy organisation developed a complex system of local-level knowledge- gathering processes where one key person with both lived experience and policy knowledge gathered the experience of organisation members and enacted it within policy settings (Smith-Merry, 2014). People in such roles have a window to both worlds, understanding both the context of those with lived experience and the peculiarities of the policy-making world. They are key to the whole endeavour, looking Janus-like into both worlds and transitioning knowledge through their own bodies.
Harrison and Mott (1998, in Stewart, 2016) refer to the ‘unstable legitimacy of user groups’ which is directly linked to their ability to be valid in two contexts. Because advocacy organisations live in two worlds they walk a fine line between both and are at risk of quickly losing validity if they fail to present the consumer voice in a way that has validity to people with disability. They are also at risk of legitimacy in a policy setting and possibly excluded if they make claims which are viewed as unrealistic or unworkable.
Taking up tools to create scientific evidence for policy
Consumers are increasingly involved in policy development by leading or becoming partners in the development of ‘scientific’ evidence as researchers developing policy-oriented research (Callard et al, 2012). Co-created research has been presented as a public good and as such a worthy goal that consultation with consumers as part of the research process has been mandated in many grant-funding processes over recent years (Pollard and Evans, 2013). Research gains validity from the inclusion of consumers within, for example, the research design process, data collection and analysis, and in presentation of research findings (Beresford, 2007; for example, Tuffrey-Wijne and Butler, 2010). Another contrasting and perhaps paradoxical form of validity comes in the validation of the consumer voice by the ‘rigour’ of the research process. If consumers create or co-create rigorous research it can become part of the canon of evidence supporting policy, thereby bringing together the validity that comes from both the consumer voice and the scientific evidence valued in standard definitions of evidence-based practice.
Evidence reviews are used by government and non-government organisations to develop policy or practice and focus on peer-reviewed papers, rapidly assessed for validity by examining content, context and quality (Tricco et al, 2015). Funders generally ask for an assessment of research quality against a standard scale to determine the strength of evidence, and only include ‘strong’ evidence as the basis of policy. In a recent review I created on co-produced research I was required to provide an assessment of the quality of the research against the NHMRC Hierarchies of Evidence guidelines. Against these guidelines the vast majority of co-produced research did not meet any level on the NHMRC matrix. The problematic association of validity with co-produced research has also been raised by others (Becker et al, 2010; Flinders et al, 2016). Consumers may have little interest in validity or rigour from a traditional academic perspective and take a standpoint that deliberately rejects these academic rules of evidence, which have been previously used to actively exclude or marginalise them (Beresford, 2007). My own co-creation review had a consumer advisory group, and through them I met with a group of Australian researchers with primary intellectual disability. Their frame for understanding validity focused on the extent to which the research targeted an issue of concern to them and that they were heard, including through presenting the results of their research themselves. They wanted to influence policy but believed that the voices of lived experience were enough to do this without other structured criteria being necessary. Research may thus be considered invalid as evidence from the perspective of scientific rigour but be valid from the perspective of consumer researchers (Smith-Merry, 2017).
Narrative approaches to influencing policy
While less prominent outside of the context of mental health, narratives of individual experiences have been used to incorporate the experience of consumers into policy (Smith-Merry et al, 2011; for example, Lapsley et al, 2002; Brown and Kandirikirira, 2007). Narratives have been used to develop the conceptual basis of policy in mental health and to develop frameworks of practice underpinned by local understandings of mental health (Smith-Merry et al, 2011). Narratives aim to present the unadulterated consumer voice to let the experience of the consumer stand as a ‘true’ telling of experience. Examples of this are the development of ‘Kia Mauri Tau!’ Narratives of Recovery from Disabling Mental Health Problems in New Zealand, and by the Scottish Recovery Network in Recovering Mental Health in Scotland (Lapsley et al, 2002; Brown and Kandirikirira, 2007). These were used to develop national policies and practices around mental health including, in Scotland, a tool used to measure service performance as part of the national policy framework.
The co-produced narrative project I co-led was funded by a local health authority and the aim was to use this knowledge to create local policy and practices to better meet the needs of clients enrolled in a major mental health policy being implemented nationwide at that time (Hancock et al, 2018). Following a narrative research methodology we conducted interviews and then compiled them into narratives acceptable to the interview respondents. However, the narratives spoke about recovery being aided by negative coping tools, such as drug use, and others were difficult to interpret for meaning. The narratives also revealed considerable amounts of self-stigma and stigma against people with mental illness more generally. This made them unsuitable for their intended use. In our interactions with others we generally represent our knowledge in a way which has been refined for the context in which it will be received (Cook and Wagenaar, 2012). These narratives did not do that and were not able to be used in policy. The respondents felt that the narratives they produced were valid representations of their experience, but from a policy and practice perspective they were not valid when transplanted into those contexts.
Conclusions
Earlier in the paper I commented on the contradiction between a public striving for objective instrumental policy processes, as evidenced in the structured notions of validity in evidence-based policy which exclude non-scientific forms of knowledge, and the changing imperative towards inclusion of the voices of lived experience which are viewed as a valid and important part of ensuring that policy is relevant to those who are subject to it. In accounting for this contradiction, we can conceptualise multiple types of validity as existing side by side even within policy deliberations. This is because validity is contextual and dependent on the context in which evidence is created, shared and used (Andrews, 2007). As Fischer (2007) comments, ‘Validation is an interpretive mode of reasoning that takes place within the frameworks of the normative belief systems brought to bear on the problem situation’. Sets of evidence sorted and refined according to hierarchies are valid in the normative belief system of evidence-based policy they were created for, which seeks highly structured, comparable and generalisable knowledge for the creation of guidelines for practice. Similarly, experiential evidence is valid in a policy environment which seeks to understand the lived experience of consumers in order to develop more appropriate policy.
Examining the examples of practice above shows that knowledge valid when gathered, enacted and inscribed in one context, may not be valid in another and that this may limit the utility of some modes in which experiential knowledge is currently enacted to influence policy. Co-created knowledge, both through consumer-led research and narrative exercises in policy influence, is valid when gathered, but can have limited validity in the context in which it is inscribed and put to use. There are similar issues of validity in the context of consumers employed to influence policy based on their own experience. Validity was most easily asserted in the context of consumer-led advocacy group involvement in policy. This is because the knowledge practices of advocacy organisations appeared to be set up to both draw on the experiential knowledge of a range of consumers and translate this into the knowledge practices of policy making.
Rigour and validity are notions that need to be decoupled from a strict understanding of methods and recoupled to knowledge and measured according to the knowledge context within which it is both gathered and being applied. Rolfe (2006), writing about rigour in qualitative research, speaks about the disassociation between rigour and broad definitions of methods, such as ‘qualitative’ or ‘quantitative’, but instead associating it with particular paradigms of knowledge. This can be shifted slightly and applied to the context of consumer knowledge in policy. Consumer knowledge fits within a paradigm where experience is central, and this is a very different paradigm to that in which frameworks of evidence hierarchies structure valid knowledge. While both work within the context of ‘evidence-based policy’ they have different purposes within this context. Bochner (2018: 361–362) comments that the ‘… trouble with inherited words like ‘rigor’ is that they impede our use of other words that better express the beliefs, goals and standards or the members of our community’s ways of life”. The overt focus on scientific knowledge within the evidence-based policy paradigm has connected validity to rigour as the antithesis of the unformed, variable nature of experiential knowledge. The discussion here has disconnected validity from rigour and connected it to context as a way of more productively understanding the uses of evidence from experience in policy.
Acknowledgements
Thanks to Dr Tanja Riches and the Inclusive Research Network at the Centre for Disability Studies, who kindly gave their time to discussing the concept of rigour and validity with me. This discussion helped to shape my thinking about co-research.
Note
I acknowledge the contested nature of nomenclature surrounding people with a lived experience of mental ill-health and/or disability. I use the term ‘consumer’ because that it what is generally accepted in Australian discourse where this paper is written. Consumer is the term used by the Australian National Mental Health Consumer and Carer Forum https://nmhccf.org.au/.
Conflict of interest
The author declares that there is no conflict of interest.
References
Andrews, C. (2007) Rationality in public policy decision making, In F. Fischer, G. Miller and M.S. Sidney (eds), Handbook of public policy analysis: Theory, politics, and methods, Boca Raton, FL: Taylor & Francis, pp 161–171.
Australian Government (2016) National disability agreement: A strong commitment to people with disability, www.dss.gov.au/disability-and-carers/programmes-services/government-international/national-disability-agreement.
Bacchi, C. (2009) Analysing policy: What’s the problem represented to be?, Sydney: Pearson Education.
Becker, S., Sempik, J. and Bryman, A. (2010) Advocates, agnostics and adversaries: researchers’ perceptions of service user involvement in social policy research, Social Policy and Society, 9(3): 355–66. doi: 10.1017/S1474746410000072
Beresford, P. (2007) The role of service user research in generating knowledge-based health and social care: from conflict to contribution, Evidence & Policy, 3(3): 329–41.
Blume, S. (2017) In search of experiential knowledge, Innovation: The European Journal of Social Science Research, 30(1): 91–103. doi: 10.1080/13511610.2016.1210505
Boardman, F.K. (2014) Knowledge is power? the role of experiential knowledge in genetically ‘risky’ reproductive decisions, Sociology of Health and Illness, 36(1): 137–50. doi: 10.1111/1467-9566.12048
Bochner, A.P. (2018) Unfurling rigor: on continuity and change in qualitative inquiry, Qualitative Inquiry, 24(6): 359–68. doi: 10.1177/1077800417727766
Borkman, T. (1979) Experiential knowledge: a new concept for the analysis of self-help groups’ Social Service Review, 50(3): 445–456. doi: 10.1086/643401
Brown, W. and Kandirikirira, N. (2007) Recovering mental health in Scotland. Report on narrative investigation of mental health recovery, Glasgow: Scottish Recovery Network.
Browne, G. and Hemsley, M. (2008) Consumer participation in mental health in Australia: what progress is being made?, Australasian Psychiatry, 16(6): 446–9. doi: 10.1080/10398560802357063
Butler, C. and Greenhalgh, T. (2010) What is already known about involving users in service transformation?, In F. Greenhalgh, C. Humphrey and F. Woodard (eds), User involvement in health care, London: Wiley Blackwell, pp. 10–27.
Cairney, P. (2017) Evidence-based best practice is more political than it looks: a case study of the “Scottish approach”, Evidence & Policy, 13(3): 499–515.
Callard, F., Rose, D. and Wykes, T. (2012) Close to the bench as well as at the bedside: involving service users in all phases of translational research, Health Expectations, 15: 389–400. doi: 10.1111/j.1369-7625.2011.00681.x
Carey, M. (2011) Should I stay or should I go? Practical, ethical and political challenges to “Service user” participation within social work research, Qualitative Social Work, 10(2): 224–43. doi: 10.1177/1473325010362000
Colebatch, H. (2005) Policy analysis, policy practice and political science, Australian Journal of Public Administration, 64(3): 14–23. doi: 10.1111/j.1467-8500.2005.00448.x
Colebatch, H. (2017) The idea of policy design: intention, process, outcome, meaning and validity, Public Policy and Administration, 33(4): 365–83. doi: 10.1177/0952076717709525
Cook, S.D.N. and Wagenaar, H. (2012) Navigating the eternally unfolding present: toward an epistemology of practice, The American Review of Public Administration, 42(1): 3–38. doi: 10.1177/0275074011407404
Fischer, F. (2007) Deliberative policy analysis as practical reason: integrating empirical and normative arguments, In F. Fischer, G. Miller, and M.S. Sidney (eds), Handbook of public policy analysis: Theory, politics, and methods, Boca Raton, FL: Taylor and Francis, pp. 223–36.
Flinders, M., Wood, M. and Cunningham, M. (2016) The politics of co-production: risks, limits and pollution, Evidence & Policy, 12(2): 261–279.
Freeman, R. and Sturdy, S. (2014) Introduction, In R. Freeman and S. Sturdy (eds), Knowledge in policy: embodied, inscribed, enacted, Bristol: Policy Press, pp. 1–14.
Guyatt, G., Cairns, J., Churchill, D., Cook, D., Haynes, B., Hirsh, J. and Evidence-Based Medicine Working Group (1992) Evidence-based medicine: a new approach to teaching the practice of medicine, JAMA, 268(17): 2420–5. doi: 10.1001/jama.1992.03490170092032
Hammersley, M. (2005) Is the evidence-based practice movement doing more good than harm? reflections on Iain Chalmers’ case for research-based policy making and practice, Evidence & Policy, 1(1): 85–100.
Hancock, N., Smith-Merry, J., Jessop, G., Wayland, S. and Kokany, A. (2018) Understanding the ups and downs of living well: the voices of people experiencing early mental health recovery, BMC Psychiatry, 18(121). doi: 10.1186/s12888-018-1703-1
Lancaster, K., Seear, K., Treloar, C. and Ritter, A. (2017) The productive techniques and constitutive effects of “evidence-based policy” and “consumer participation” discourses in health policy processes, Social Science and Medicine, 176: 60–68. doi: 10.1016/j.socscimed.2017.01.031
Lapsley, H., Nikora, L. and Black, R. (2002) Kia Mauri Tau! Narratives of recovery from disabling mental health problems, Wellington: New Zealand Mental Health Commission.
Lub, V. (2015) Validity in qualitative evaluation: linking purposes, paradigms, and perspectives, International Journal of Qualitative Methods, 14(5): 1–8. doi: 10.1177/1609406915621406
Maxwell, J. and Chmiel, M. (2013) Generalization in and from qualitative analysis, In U. Flick (ed), The Sage handbook of qualitative data analysis, London: Sage.
Meriluoto, T. (2018) Turning experience into expertise: technologies of the self in Finnish participatory social policy, Critical Policy Studies, 12(3): 294–313. doi: 10.1080/19460171.2017.1310051
NHMRC (National Health and Medical Research Council) (2009) NHMRC additional levels of evidence and grades for recommendations for developers of guidelines, Canberra: Commonwealth of Australia.
NSW Government (2016) Mental health consumer advocacy and participation, www.health.nsw.gov.au/mentalhealth/Pages/consumer-mh-advocacy.aspx
Parsons, T. (1951) The social system, New York: The Free Press.
Pollard, K.C. and Evans, D. (2013) Theorising service user involvement from a researcher perspective, In P. Staddon (ed), Mental health service users in research: Critical sociological perspectives, Chicago, IL: University of Chicago Press, pp. 39–51.
Rabeharisoa, V., Moreira, T. and Akrich, M. (2014) Evidence-based activism: patients’, users’ and activists’ groups in knowledge society, Biosocieties, 9(2): 111–28. doi: 10.1057/biosoc.2014.2
Restall, G., Cooper, J.E. and Kaufert, J.M. (2011) Pathways to translating experiential knowledge into mental health policy, Psychiatric Rehabilitation Journal, 35(1): 29–36. doi: 10.2975/35.1.2011.29.36
Richter-Sundberg, L., Garvare, R. and Nystrom, M. (2017) Reaching beyond the review of research evidence: a qualitative study of decision making during the development of clinical practice guidelines for disease prevention in healthcare, BMC Health Services Research, 17(1): 344. doi: 10.1186/s12913-017-2277-1
Rolfe, G. (2006) Validity, trustworthiness and rigour: quality and the idea of qualitative research, Journal of Advanced Nursing, 53(3): 304–10. doi: 10.1111/j.1365-2648.2006.03727.x
Sandelowski, M. (1986) The problem of rigor in qualitative research, ANS. Advances in Nursing Science, 8(3): 27–37. doi: 10.1097/00012272-198604000-00005
Sandelowski, M. (1993) Rigor or rigor mortis: the problem of rigor in qualitative research revisited, ANS. Advances in Nursing Science, 16(2): 1–8. doi: 10.1097/00012272-199312000-00002
Smith, K. (2013) Beyond evidence-based policy in public health: The interplay of ideas, Basingstoke: Palgrave Macmillan.
Smith-Merry, J. (2014) Seeing knowledge in mental health in Scotland, In R. Freeman, S. Sturdy (eds), Knowledge in policy: Embodied, inscribed, enacted, Bristol: Policy Press, pp. 21–41.
Smith-Merry, J. (2017) Inclusive research: research to action guide, rapid review, Sydney: Centre for Applied Disability Research.
Smith-Merry, J. (2018) Inclusive research: how to do it, who does it and why, In P. Liamputtong (ed), Research methods in health social sciences (Vol 2), Berlin: Springer.
Smith-Merry, J. and Gillespie, J. (2016) Embodying policy making in mental health: the implementation of partners in recovery, Health Sociology Review, 25(2): 187–201. doi: 10.1080/14461242.2016.1171120
Smith-Merry, J., Freeman, R. and Sturdy, S. (2011) Implementing recovery: an analysis of the key technologies in Scotland, International Journal of Mental Health Systems, 5(1). doi: 10.1186/1752-4458-5-11
Stewart, E. (2016) Publics and their health systems: Rethinking participation, London: Palgrave Macmillan.
Tricco, A.C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J. and Straus, S.E. (2015) A scoping review of rapid review methods, BMC Medicine, 13(1): 224. doi: 10.1186/s12916-015-0465-6
Tuffrey‐Wijne, I. and Butler, G., (2010) Co‐researching with people with learning disabilities: an experience of involvement in qualitative data analysis’, Health Expectations, 13(2): 174–84.
Wayland, S., Smith-Merry, J., Kolany, A. and Hancock, N. (2017) What recovery means to us: Understanding real-life recovery, Blacktown: Western Sydney Partners in Recovery.
Williams, A. (2010) Is evidence-based policy making really possible? Reflections for policymakers and academics on making use of research in the work of policy, In H.K. Colebatch, R. Hoppe and M. Noordegraaf (eds), Working for policy, Amsterdam: Amsterdam University Press, pp. 195–210.
