A research protocol for studying participatory processes in the use of evidence in child welfare systems

View author details View Less
  • 1 University of North Carolina at Chapel Hill, , USA
  • | 2 Kingston University London and St. George’s University of London, , UK
  • | 3 University of North Carolina at Chapel Hill, , USA
Open access
Get eTOC alerts
Rights and permissions Cite this article

This paper presents a protocol for a funded study of technical assistance strategies used to support the use of evidence, and, in particular, how participatory processes contribute to the use of evidence to improve outcomes for populations. Findings from the study will increase understanding of the relationships between technical assistance, stakeholder participation and evidence use in child and family services. The authors argue that publishing such a protocol can increase transparency between researchers and practitioners and raise awareness of the need for research on how stakeholder participation can strengthen evidence use in child welfare service settings. The authors also reflect on the potential value and limitations of published protocols. This study will systematically gather input from stakeholders with expertise in technical assistance to develop a compilation of strategies that can be used to support the use of evidence. The study will identify strategies that include stakeholder involvement and assess which strategies under what conditions facilitated the use of research evidence. The study will address four research questions: What technical assistance strategies are used to support the use of research evidence? What are the consensus-driven terms and definitions of identified strategies? To what extent do technical assistance strategies involve stakeholders and for what purpose?

Abstract

This paper presents a protocol for a funded study of technical assistance strategies used to support the use of evidence, and, in particular, how participatory processes contribute to the use of evidence to improve outcomes for populations. Findings from the study will increase understanding of the relationships between technical assistance, stakeholder participation and evidence use in child and family services. The authors argue that publishing such a protocol can increase transparency between researchers and practitioners and raise awareness of the need for research on how stakeholder participation can strengthen evidence use in child welfare service settings. The authors also reflect on the potential value and limitations of published protocols. This study will systematically gather input from stakeholders with expertise in technical assistance to develop a compilation of strategies that can be used to support the use of evidence. The study will identify strategies that include stakeholder involvement and assess which strategies under what conditions facilitated the use of research evidence. The study will address four research questions: What technical assistance strategies are used to support the use of research evidence? What are the consensus-driven terms and definitions of identified strategies? To what extent do technical assistance strategies involve stakeholders and for what purpose?

key messages

  • The current paper presents a research protocol for empirically studying under what conditions stakeholder participation supports use of research evidence.

  • The paper provides a rationale for the study of how participatory processes, such as co-creation, contribute to evidence use, a proposed design for the study, and potential contributions and limitations.

Introduction

The study of implementation strategies to integrate evidence into practice has garnered increased attention over the last 20 years (Proctor et al, 2013; Powell et al, 2019). Recently, Leeman and colleagues (2017) have called for a classification system that specifies the implementation actor and the implementation target, in order to build knowledge related to how specific strategies delivered by certain actors can facilitate the use of evidence to improve outcomes. One set of actors that has received minimal attention in the literature is individuals who provide a system of support for implementation, including building delivery systems’ general and intervention specific capacity (Wandersman et al, 2008) to adopt and integrate research evidence into day-to-day practice. These actors reside outside of the delivery system, providing external support as an ‘outside expert’, and may be referred to as a technical assistance providers, implementation specialists, consultants, or intermediaries. There is increasing interest in understanding what these actors do, and in what ways the external support they provide relies on participatory processes to engage key stakeholders in decisions regarding evidence use (Metz et al, 2018).

Given the interest in how this external support is provided, and whether this support results in greater evidence use, this paper presents a research protocol for studying the relationship between external support and evidence use. The research protocol focuses on a specific type of external support referred to as technical assistance, which includes capacity building for service delivery and systems change. The paper first provides a conceptual rationale for studying the role of technical assistance in advancing evidence use and, specifically, under what conditions technical assistance strategies that include stakeholder participation, such as co-creation, may result in greater evidence use. For example, how do technical assistance providers engage stakeholders within the service delivery system, including practitioners, and the people who are intended to benefit from the use of evidence in the service system? Does stakeholder participation lead to greater use of research evidence and improved outcomes?

Following the presentation of a conceptual famework for linking evidence use, technical assistance, and stakeholder participation, we provide an overview of the study including study questions and research methods. In the methods section we define key terms including evidence, use of evidence, technical assistance, and stakeholder participation. A contribution of the study protocol is the operationalisation of commonly used terms that are inconsistently defined. The methods will include a description of how the current study builds upon previous frameworks and research. The discussion will conclude with how the study will contribute to the field, as well as how this particular paper contributes to this special issue on evidence use and co-creation.

Figure 1 depicts the potential relationship between technical assistance, stakeholder participation, and evidence use. Technical assistance providers may use context-based, stakeholder participation strategies to engage communities and people in selecting and adapting research evidence to meet population needs, resulting in sustainable and equitable outcomes for populations. However, further empirical work is needed to explore the potential relationships among technical assistance, participatory processes (for example, co-creation), and evidence use to help us understand if and how specific technical assistance strategies used under certain conditions yield improved outcomes for people and communities.

Figure 1:
Figure 1:

Technical assistance, stakeholder participation, and evidence use

Citation: Evidence & Policy 15, 3; 10.1332/174426419X15579811791990

In sharing the current protocol, we seek to extend the idea of protocol publication, which is a common feature in healthcare research, to other fields (for example, child and family services) and to different types of study designs (for example, exploratory). We have published this protocol at the outset of the study to increase transparency in our methods. We hope that publishing this protocol will encourage discussion, strengthen the quality of reporting, improve transparency of the research, reduce the likelihood of duplications, and mitigate the challenges of study bias where only studies with positive findings are published.

The approach has also been used in the field of evidence use. For example, the protocol for an intervention study designed to build organisational capacity to use research in the Australian health system has been widely cited (The CIPHER Investigators, 2014). The publication of protocols is also becoming more commonplace in implementation science. For example, Hasson and colleagues (2018) have recently published a protocol to study the de-implementation process of low-value practices in the healthcare system.

This paper makes a contribution to the literature alongside the other papers in this special issue. Other papers in this issue provide conceptual, theoretical, and practical considerations for determining the conditions under which co-creation and involvement strategies are useful and effective. The current paper presents a research protocol for empirically studying this very question. Following, we provide a rationale for the study of how participatory processes, such as co-creation, contribute to evidence use, a proposed design for the study, and potential contributions and limitations. The study is funded by the William T. Grant Foundation based in the US.

Evidence use, technical assistance, and stakeholder participation

Despite the growing emphasis on the use of evidence-based practices and programmes to improve outcomes, the mobilisation of research evidence on the frontlines of child welfare has been quite limited, especially in public agencies serving the vast majority of children, youth and families (Aarons and Palinkas, 2007; Durlak and Dupree, 2008; Raghavan et al, 2010; Saldana, et al., 2015). The use of evidence has tended to rely on ‘a unidirectional flow from research to practice’ (Cabassa, 2016) without a clear understanding of how context, community needs, and resources shape the use of research in practice. Even when service systems make the investment in evidence-based programmes, sustaining the potential of these programmes over the long term continues to be a challenge internationally (Chambers et al, 2013). Only 37% of widely disseminated evidence-based models are sustained in the long-term (Ghate, 2016). Indeed, in many jurisdictions, ‘services as usual’ remain untouched by research evidence.

A range of strategies are emerging to support the uptake of research evidence, including knowledge-brokering activities, the establishment of research networks, and technical assistance. We discuss technical assistance as a strategy to assist service systems with using research evidence to improve population outcomes. In the US, federal agencies and private foundations spend many millions of dollars on technical assistance to support the use of evidence in practice. Historically, technical assistance has been defined as the transfer of new knowledge along with new technology to others who do not know about it (Blase, 2009). More recently, the goals of technical assistance have aligned with goals to use evidence-based approaches, and technical assistance has been redefined as building capacity for the service and system changes needed to enable the use of evidence-based approaches. However, even with this revised definition and focus, findings from a research synthesis on technical assistance demonstrated that quality standards for technical assistance are essentially nonexistent and that technical assistance is rarely delivered systematically (Katz and Wandersman, 2016).

There is emerging consensus that research evidence generated outside of a practice context struggles to be successfully pushed into a system to achieve positive outcomes for children and families (Holmes et al, 2016). Positive outcomes are, instead, the result of the interaction between evidence and the context within which the evidence is used (Pawson, 2013). Therefore, it would make sense that technical assistance strategies that embrace context, rather than attempt to ignore or minimise context, would be more successful in supporting the use of evidence. Understanding context requires stakeholder participation. Stakeholder participation concerns the involvement of individuals and organisation with a ‘stake’ in the selection and use of evidence to improve population and community outcomes (Boaz and Metz, in press). Critical stakeholders may include public agencies, funders, families, and experts. Technical assistance strategies grounded in stakeholder participation emphasise the importance of capitalising on the unique knowledge of stakeholders to develop and use the best available contextualised evidence. Co-creation, a type of stakeholder participation, is the active involvement of stakeholders in all stages of the production and implementation process, resulting in service models, approaches, and practices that are contextualised and tailored to settings (Vargo and Lusch, 2004; Metz and Bartley, 2016). The contextualisation of evidence refers to increasing the match between evidence-based practices and the local delivery setting, including those who deliver interventions, systems stakeholders, and children and families expected to benefit from evidence-based practices (Horner et al, 2014; Van Kerkhoff and Lebel, 2015)

Figure 2 describes how technical assistance strategies can promote the use of research evidence through the evolution of exchange models. Push models attempt to move research evidence into communities without a focus on community stakeholders or context. Pull models endeavour to respond to community needs through responsive technical assistance strategies, such as gathering feedback from stakeholders. However, pull models do not promote ongoing exchange between the evidence use and community context. The ‘pull’ model is used to initially determine community needs, but once these needs are determined technical assistance providers select evidence to meet these needs, and there is limited community input following that process to contextualise the evidence.

Figure 2:
Figure 2:

Technical assistance and exchange models

Citation: Evidence & Policy 15, 3; 10.1332/174426419X15579811791990

Based on the work of Lavis (2006)

Exchange models create reciprocal dialogue among stakeholders, researchers, and technical assistance providers to improve the use of contextualised and relevant evidence in practice. The exchange happens during all phases of evidence selection, use, and improvement. For example, Metz and Bartley (2016) found that higher levels of mutual consultation among researchers, practitioners, and policy makers were associated with increased levels of research use. Mutual consultation was described as activities that promote developing a shared understanding of the problem space, taking into account different perspectives through communication, negotiation, and mutual adjustment, and using formal communication mechanisms to promote feedback loops (Edelenbos et al, 2011; Prager and McKee, 2015). Moreover, the systems change required to use research evidence in child welfare requires learning from different stakeholders how elements of the system are interrelated to produce current outcomes (Senge, 2014).

Study description

This study will systematically gather input from stakeholders with expertise in technical assistance to develop a compilation of technical assistance strategy terms and definitions that can be used to support the use of research evidence in child welfare. The study will identify technical assistance strategies that include stakeholder involvement and assess which strategies under what conditions facilitated research use. The study will address four research questions:

  1. What technical assistance strategies are used to support the use of research evidence?

  2. What are the consensus-driven terms and definitions of identified strategies?

  3. To what extent do technical assistance strategies involve stakeholders and for what purpose?

  4. Under what conditions have specific technical assistance strategies, including strategies that foster stakeholder participation, contributed to supporting research evidence use in child welfare?

The study will produce a taxonomy for technical assistance that will set the stage for further research. The taxonomy will categorise technical assistance strategies by the extent to which they are participatory and inclusive (see Table 1) and contribute to the use of evidence-based practices through strategic processes that promote evidence use, continuous quality improvement processes, and the regular use of assessments and feedback to contextualise evidence-based practices to local populations.

Table 1:

Participatory processes

InformConsultInvolveCollaborateCo-create
To provide informationTo obtain feedbackTo work with to develop alternativesTo partner in each aspect of decision makingTo empower to make decisions

Adapted from © International Association for Public Participation

Taxonomies have proven useful in other fields for providing the foundation for further research. For example, Michie and colleagues’ taxonomy of behaviour change techniques (Abraham and Michie, 2008, Michie, et al, 2013) developed a taxonomy that has promoted consistency in international reporting on behaviour change interventions. Powell and colleagues (2012; 2015) have developed a taxonomy for implementation strategies that also promotes consistency in reporting on implementation studies and paves the way for more rigorous research designs that test under what conditions specific implementation strategies are most effective. This study will yield a similar contribution to the field of technical assistance and, specifically, develop a consensus-driven research agenda on studying how, when, and with whom different technical assistance strategies promote the use of research evidence.

The protocol focuses on the use of evidence in child welfare specifically for a few reasons, including the need to narrow the study context, the growing use of evidence-based practices in child welfare settings, and the struggle to sustain the use of evidence long enough to achieve population outcomes (Saldana, et al., 2015). It is anticipated that findings from the study will increase understanding of the relationship between technical assistance, participatory processes, and evidence use in child and family services and, specifically, in public child welfare systems.

The study will allow for the investigation of a range of participatory processes (described below in Table 1) and, in line with current debate, will attempt to draw out what ‘counts’ as meaningful involvement in technical assistance strategies. Other papers in this special issue (see Locock and Boaz) seek to understand the differences among participatory processes and the implications of labelling certain approaches as potentially better than other approaches. Locock and Boaz (this issue) note that all participatory processes can be challenged on the grounds of tokenism. However, as we use case examples to describe participatory processes in technical assistance, we will identify processes that embody co-creation, moving beyond shared decision making and empowering consumer-based decision making.

Building on previous studies in the field

One promising area to consider for strengthening the body of research accessible to technical assistance providers is implementation science. Powell and colleagues (2015) have compiled implementation strategies through the Expert Recommendations for Implementing Change (ERIC) project. The ERIC project identified 73 discrete strategies that can be used in isolation or combination to implement effective practices. Strategies range from facilitation to identifying and preparing champions to modelling and simulating change to promoting network weaving to using data experts. Follow-up studies from ERIC have identified how implementation strategies might be used over time, estimating the dose, temporal order, and inclusion of implementation actors to facilitate change (for example, Bunger et al, 2017; Rogal et al, 2017; Boyd et al, 2018; Perry et al, 2019).

This study will use these strategies as a foundation for defining technical assistance strategies, expanding on this description by more fully operationalising what these strategies look like in practice settings where technical assistance is delivered. For example, Metz and colleagues (2018) have recently defined implementation skills and competencies needed to support the use of research including activities to engage stakeholders, support continuous improvement, and sustain change. As another example, Boaz and colleagues (2016) have described the value of participatory processes in supporting research use, which will be examined in relation to technical assistance strategies.

Operational definitions and criteria

The study will define technical assistance as strategies to build service and system capacity to use research evidence to improve population outcomes. Research use will include the adoption and use of evidence-based and evidence-informed programmes and practices, and the ongoing use of research and evaluation methods, along with collaborative stakeholder involvement, to support necessary and dynamic adaptations to evidence-based and evidence-informed practices and programmes at practice, organisational, systems, and policy levels to achieve desired child and family outcomes (Aarons et al, 2012). The study will focus on the child welfare context. Given the current inconsistencies in how technical assistance is defined or measured, it is expected that strategies will include a wide range of tactics including training, coaching, facilitation, technology and infrastructure development, team development, data analysis, and assessment.

A summary of key study terms includes:

  • Evidence – Evidence-based and evidence-informed programmes and practices

  • Use of evidence – The ongoing use of research and evaluation methods, along with collaborative stakeholder involvement, to support necessary and dynamic adaptations to evidence-based and evidence-informed programmes and practices to inform practice and policy settings

  • Technical assistance – Building capacity for the service and systems changes needed to enable the use of evidence-based approaches. The study will include activities that are self-defined as technical assistance, and it is expected that such activities will be similar to other activities such as consultation and facilitation, but with explicit focus on building capacity to use evidence-based practices.

  • Stakeholder participation – participatory processes ranging from informing, to consulting, to involving, to collaborating, to co-creating (see Table 1 for descriptions)

The study will focus on what Nutley and colleagues (2007) describe as instrumental use of research evidence, studying how technical assistance strategies contribute to the direct application of research evidence to decision making by policy makers and practitioners. The study is exploratory in nature and will consider whether certain technical assistance strategies, under specific conditions, support research use. Criteria will be established during the case study review related to how dynamic, inclusive, and research-driven were adaptations to evidence-based and evidence-informed practices. Descriptions of these constructs include:

  • Dynamic – the regularity of ongoing assessments, feedback, and improvement at practice, organisational and systems level to determine the contextual fit of the evidence-based practices and whether children and families benefit

  • Inclusive – the extent to which key stakeholders, including families, are engaged in the production, adaptation, and ongoing improvement of evidence-based practice

  • Research-driven – the level of rigour of the design and methods used to support adaptations and contextualisation of research evidence

  • Strategic – the extent to which a systematic approach has been developed to support research use

Research methods

The study will utilise a two-step methodology. First, a modified Delphi process (Hasson et al, 2000) with three rounds will be used to gather expert knowledge on technical assistance and address the first two research questions. The basis for the Delphi process will include strategies identified through previous studies that identified and defined implementation strategies in health and behavioural health (Powell et al, 2012; 2015), as well as an effort to identify competencies for implementation technical assistance providers (Metz et al, 2018). The Delphi process will also be used to identify case studies. Second, case studies will be analysed through a performance story methodology (Mayne, 2008) to address the third and fourth research questions.

Table 2 provides more detailed information on methods and data sources that will be used to address each research question.

Table 2:

Research design overview

Research questionMethodData source
What technical assistance strategies are used to support the use of research evidence and how are they defined?• Strategy review• Research literature
• Delphi process• Documents (for example, government reports)
• Expert panel including funders, TA providers, TA recipients, and researchers
To what extent do technical assistance strategies engage stakeholders, and for what purpose?• Strategy review• Research literature
• Delphi process• Documents (for example, government reports)
• Case study reviews including interviews and document review• Expert panel including funders, TA providers, and researchers, TA recipients
• Key stakeholders from case studies including TA providers, TA recipients, researchers
• 3–5 telephone interviews per case study with 5 case studies
Under what conditions have specific types of technical assistance strategies contributed to supporting research evidence use in child welfare?• Case study reviews including interviews and document review• Key stakeholders from case studies including TA providers, TA recipients, researchers
• 3–5 telephone interviews per case study

Delphi process

During the first two rounds of the Delphi process, panelists will be presented with a list of technical assistance strategies via a web-based survey, and they will be given the opportunity to provide edits to names and definitions as well as to suggest additional strategies and definitions. They will also be asked to nominate case studies that demonstrate the use of these strategies to improve research use. After each of the first two rounds, data will be summarised and fed back to participants, with iterative refinements made based upon participant feedback. The third round will involve a live meeting and web-based polling process in which panellists will have the opportunity to discuss earlier rounds of feedback, arrive at consensus for the final compilation of technical assistance strategies and definitions, and choose 5–10 case studies to be included in the second part of the study. These procedures are similar to those employed in the ERIC project, the methods of which are described in detail elsewhere (Waltz et al, 2014; Powell et al, 2015).

Case studies

Selected case studies will be analysed through a contribution analysis methodology referred to as performance stories (Mayne, 2006; Mayne, 2008). Contribution analysis is a type of performance measurement used to explore the contribution an approach makes to observed results. Contribution analysis for this study will include six steps: 1) specify challenges for evidence use in public child welfare systems; 2) develop a theory of change for using co-creative technical assistance strategies to support evidence use and specify risks to the theoretical assumptions; 3) gather information from case examples that supports (or fails to support) the theory of change; 4) assemble findings across case examples and develop a contribution story that expresses whether or not it is reasonable to assume (or not) that co-creative technical assistance strategies, under certain conditions, contribute to evidence use; 5) seek additional evidence as necessary (for example, document reviews, interviews, focus groups); and 6) revise the contribution story as needed.

Contribution analysis is increasingly being used to study evidence use (Morton, 2015), and is particularly relevant when assessing the contribution of complex initiatives in complex settings without a clear counterfactual. Contribution analysis does not set out to provide definitive proof of cause-effect; rather, in this case, it aims to reduce uncertainty about the difference specific technical assistance strategies are making, by mapping the intended causal pathways to increase research evidence use, and by gathering evidence to support and/or refute an agreed theory of change and alternative explanations. Addressing the alternative explanations explicitly is one way of appraising and weighing an argument in favour of different technical assistance strategies’ impact on research use.

Each case study analysis will assess whether outcomes related to use of research evidence were achieved if:

  • There is a plausible theory of change where underlying assumptions for how technical assistance strategies facilitate evidence use are clearly articulated

  • The technical assistance strategies were implemented according to a plan

  • There is evidentiary confirmation that the key elements of the theory of change are upheld based on findings from the case study

  • Other influencing factors on evidence use have been identified and accounted for; and

  • The most relevant alternative explanations have been considered in relation to the contribution of well-described technical assistance strategies to evidence use.

Sampling and recruitment

Expert panellists working in child welfare contexts will include a stratified sample that represents the perspectives of researchers who study technical assistance, intermediaries and consultants who provide technical assistance, and funders who support technical assistance.

For the researcher sample, Google Scholar and web analytics will be used to determine publications and impact. For funders, the percentage of their grants portfolio dedicated to technical assistance will be used to assess experience. Finally, for technical assistance providers, the number and scope of technical assistance projects will be used to assess expertise, along with the range of methods used by technical assistance providers (didactic vs participatory and interactive). While this information is not readily accessible in the public domain, we will conduct outreach with potential participants to gather information needed to select the expert panel.

Expert panellists will be included from the US and UK. It is expected that the majority of the sample will represent the US, but given technical assistance is an emerging concept in other countries, the sample will include a sample of participants from the UK. Each panel will include 10–18 people. Prior to applying our metrics to select our sample, an initial pool of experts will be identified from implementation and technical assistance round tables, forums, networks, conferences and relevant professional bodies (Okali and Pawlowski, 2004).

Discussion

This exploratory research will advance the field by developing a taxonomy for technical assistance strategies that will improve the conceptual clarity, relevance and comprehensiveness of technical assistance strategies that support research use in child welfare settings and systems, and by distinguishing technical assistance strategies that utilise participatory processes. Findings will also describe how technical assistance strategies contributed to evidence use, and the nature of the methods to promote the use of evidence to meet population needs. These findings will set the stage for future studies that can rank each strategy’s importance and feasibility for impacting use of research evidence. As conceptually distinct categories of technical assistance strategies evolve, future research designs can continue to test the effectiveness of strategies in supporting research use and identify how strategies may be combined or packaged given the context, goals, and interventions being implemented.

The publication of this protocol provides an opportunity to increase transparency between researchers studying co-creation methods and practitioners enacting co-creation strategies to support evidence use and positive benefits for communities and people. A principle of co-creation work is transparency, and sharing this protocol ensures that the research on co-creation is aligned with the practice of co-creation. The authors invite engagement from practitioners and researchers and hope that in sharing this protocol they raise awareness for the need for research on technical assistance strategies and on how participatory processes, in particular, can strengthen evidence use in child welfare service settings and social welfare broadly.

This study offers an important opportunity to assess the relationships among technical assistance, co-creation, and evidence use. Inherent in the approach of the study is the observation that context, including the values, resources, and priorities of stakeholders, must be considered when developing and using co-creative technical assistance strategies intended to support sustainable use of evidence in communities. The case examples included in the study protocol allow for researchers to demonstrate the extent to which specific technical assistance strategies engage stakeholders, and under what conditions specific strategies are more or less effective in promoting evidence use.

When there is misalignment between context and evidence, the evidence-based practice will not be implemented or sustained as intended. Understanding the structural, contextual, and human factors that impact evidence use, and can be mitigated by co-creative technical assistance strategies, will provide actionable knowledge for enhancing such strategies and improving population outcomes.

If we want to see the investment in using evidence-based practices pay off, we need to make sure to put in the hard work it takes to implement and scale up evidence-based programmes, including engaging stakeholders to support the ongoing adaptation, improvement and sustainability of these programmes in local contexts. The study protocol seeks to answer questions related to the role of stakeholders in addressing this challenge and, consequently, will have policy implications. Funders and policy makers have an important leadership role in that they have the opportunity and authority to build evidence- and implementation-informed agendas and funding programmes. Policy makers can request accountability from public agencies and community partners to move toward a model of active stakeholder engagement among agencies, families, and other stakeholders. This engagement strategy can form an ‘opportunity structure’ for co-creating the infrastructure needed to improve contextual fit between evidence-based programmes and local communities.

This article serves as an important puzzle piece in the group of articles for this special issue on co-creation. Theoretical articles include conceptual models for evolving the distinction of co-creation from other participation processes. Empirical articles demonstrate how co-creation activities contribute to the implementation of projects and initiatives in real world contexts. Practice articles discuss strategies for promoting co-creation in collaborative and multi-sector service efforts. All articles, though, point to a need for understanding under what conditions specific participation methods, including co-creation, may be most beneficial for improving the use of evidence and innovations to improve the lives of people. This article offers a protocol for studying how, when, and with whom co-creation may have its intended benefit. Implications of this study will include guidance for practitioners, policy makers, and funders who seek to support change in service to better population outcomes.

Acknowledgements

This article was developed with generous support from SESYNC, the National Socio-Environmental Synthesis Center, under the theme Building Resources for Complex, Action-Oriented Team Science and the William T. Grant Foundation under grant number 188777.

The authors are grateful for the support of SESYNC colleagues and the Pursuit to Advance Co-Creative Capacity, and Gabriele Bammer, Australian National University.

Conflict of interest

The authors declare that there is no conflict of interest.

References

  • Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., Silovsky, J. F., Hecht, D. B., Chaffin, M. J. (2012) ‘Dynamic adaptation process to implement an evidence-based child maltreatment intervention’, Implementation Science, 7, 32.

    • Search Google Scholar
    • Export Citation
  • Aarons, G. A., Palinkas, L. A. (2007) ‘Implementation of evidence-based practice in child welfare: service provider perspectives’, Administration and Policy in Mental Health and Mental Health Services, 34, 411419. doi: 10.1007/s10488-007-0121-3

    • Search Google Scholar
    • Export Citation
  • Abraham, C., Michie, S. (2008) ‘A taxonomy of behavior change techniques used in interventions’, Health Psychology, 27(3), 379. doi: 10.1037/0278-6133.27.3.379

    • Search Google Scholar
    • Export Citation
  • Blase, K. (2009) Technical Assistance to Promote Service and System Change: Roadmap to Effective Intervention Practices #4 (Tampa, FL: University of South Florida, Technical Assistance Center on Social Emotional Intervention for Young Children).

    • Search Google Scholar
    • Export Citation
  • Boaz, A., Metz, A. (in press) ‘Engaging Stakeholders in the Implementation of Evidence-Based Programs, in E. Zimmerman (ed), Researching Together: Engaging Patients and Stakeholders in Health Research, from Topic Identification to Policy Change, Sage Press.

  • Boaz, A., Robert, G., Locock, L., Sturmey, G., Gager, M., Vougioukalou, S., Ziebland, S., Fielden, J. (2016) ‘What patients do and their impact on implementation: an ethnographic study of participatory quality improvement projects in English acute hospitals’, Journal of Health Organization and Management, 30(2), 258278. doi: 10.1108/JHOM-02-2015-0027

    • Search Google Scholar
    • Export Citation
  • Boyd, M. R., Powell, B. J., Endicott, D., Lewis, C. C. (2018) ‘A method for tracking implementation strategies: an exemplar implementing measurement-based care in community behavioral health clinics’, Behavior Therapy, 49(4), 525537. doi: 10.1016/j.beth.2017.11.012

    • Search Google Scholar
    • Export Citation
  • Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H., Birken, S. A., Shea, C. (2017) ‘Tracking implementation strategies: a description of a practical approach and early findings’, Health Research Policy and Systems, 15, 15. doi: 10.1186/s12961-017-0175-y

    • Search Google Scholar
    • Export Citation
  • Cabassa, L. J. (2016) ‘Implementation science: why it matters for the future of social work’, Journal of Social Work Education, 52(suppl 1), 3850. doi: 10.1080/10437797.2016.1174648

    • Search Google Scholar
    • Export Citation
  • Chambers, D. A., Glasgow, R. E., Stange, K. C. (2013) ‘The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change’, Implementation Science, 8(1), 117. doi: 10.1186/1748-5908-8-117

    • Search Google Scholar
    • Export Citation
  • Durlak, J. A., Dupree, E. P. (2008) ‘Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation’, American Journal of Community Psychology, 41(3–4), 327350. doi: 10.1007/s10464-008-9165-0

    • Search Google Scholar
    • Export Citation
  • Edelenbos, J., Van Buuren, A., Van Schie, N. (2011) ‘Co-producing knowledge: joint knowledge production between experts, bureaucrats and stakeholders in Dutch water management projects’, Environmental Science & Policy, 14, 675684.

    • Search Google Scholar
    • Export Citation
  • Ghate, D. (2016) ‘From programs to systems: deploying implementation science and practice for sustained real world effectiveness in services for children and families’, Journal of Clinical Child & Adolescent Psychology, 45(6), 812826.

    • Search Google Scholar
    • Export Citation
  • Hasson, F., Keeney, S., McKenna, H. (2000) ‘Research guidelines for the Delphi survey technique’, Journal of Advanced Nursing, 32(4), 10081015.

    • Search Google Scholar
    • Export Citation
  • Hasson, H., Nilsen, P., Augustsson, H., von Thiele Schwarz, U. (2018) ‘Empirical and conceptual investigation of de-implementation of low-value care from professional and health care system perspectives: a study protocol’, Implementation Science, 13(1), 67. doi: 10.1186/s13012-018-0760-7

    • Search Google Scholar
    • Export Citation
  • Holmes, B. J., Best, A., Davies, H., Hunter, D., Kelly, M., Marshall, M., Rycroft-Malone, J. (2016) ‘Mobilising knowledge in complex health systems: a call to action’, Evidence & Policy, 13(3), 539560.

    • Search Google Scholar
    • Export Citation
  • Horner, R., Blitz, C., Ross, S. W. (2014) The Importance of Contextual Fit When Implementing Evidence-Based Interventions, Issue Brief (Washington, DC: Office of the Assistant Secretary for Planning and Evaluation, Office of Human Services Policy, US Department of Health and Human Services).

    • Search Google Scholar
    • Export Citation
  • Katz, J., Wandersman, A. (2016) ‘Technical assistance to enhance prevention capacity: a research synthesis of the evidence base’, Prevention Science, 17, 417428. doi: 10.1007/s11121-016-0636-5

    • Search Google Scholar
    • Export Citation
  • Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., Shea, C. M. (2017) ‘Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice’, Implementation Science, 12, 125. doi: 10.1186/s13012-017-0657-x

    • Search Google Scholar
    • Export Citation
  • Mayne, J. (2008). ‘Contribution analysis: an approach to exploring cause and effect’, Institutional Learning and Change Initiative (ILAC), Brief, 16, 4.

    • Search Google Scholar
    • Export Citation
  • Mayne, J., Rist, R. C. (2006) ‘Studies are not enough: the necessary transformation of evaluation’, The Canadian Journal of Program Evaluation, 21(3), 93120.

    • Search Google Scholar
    • Export Citation
  • Metz, A., Bartley, L. (2016) ‘Co-creating the conditions to sustain the use of research evidence in public child welfare’, Child Welfare, 94(2), 115139.

    • Search Google Scholar
    • Export Citation
  • Metz, A., Boaz, A. (2016) ‘Where are the stakeholders in implementation science? Integration and implementation insights’, https://i2insights.org/2016/09/08/co-creation-and-implementation-science/

    • Search Google Scholar
    • Export Citation
  • Metz, A., Louison, L., Ward, C., Burke, K. (2018) Implementation Specialist Practice Profile: Skills and Competencies for Implementation Practitioners, Working Draft (Chapel Hill, NC: National Implementation Research Network, University of North Carolina at Chapel Hill).

    • Search Google Scholar
    • Export Citation
  • Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., Wood, C. E. (2013) ‘The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions’, Annals of Behavioral Medicine, 46(1), 8195. doi: 10.1007/s12160-013-9486-6

    • Search Google Scholar
    • Export Citation
  • Morton S. (2015) ‘Research impact assessment: a ‘Contributions’ approach’, Research Evaluation, 24(4), 405419. doi: 10.1093/reseval/rvv016

    • Search Google Scholar
    • Export Citation
  • Nutley, S. M., Walter, I., Davies, H. T. (2007) Using Evidence: How Research Can Inform Public Services (Bristol: Policy Press).

  • Okali, C., Pawlowski, S. D. (2004) ‘The Delphi method as a research tool: an example, design considerations and applications’, Information & Management, 42(1), 1529.

    • Search Google Scholar
    • Export Citation
  • Pawson, R. (2013) The Science of Evaluation: A Realist Manifesto. (Los Angeles, CA: Sage).

  • Perry, C. K., Damschroder, L. J., Hemler, J. R., Woodson, T. T., Ono, S. S., Cohen, D. J. (2019) ‘Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory’, Implementation Science, 14, 32. doi: 10.1186/s13012-019-0876-4

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., Weiner, B. J. (2019) ‘Enhancing the impact of implementation strategies in healthcare: a research agenda’, Frontiers in Public Health, 7(3), 19. doi: 10.3389/fpubh.2019.00001

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T, Bunger, A. C., Glass, J. E., York, J. L. (2012) ‘A compilation of strategies for implementing clinical innovations in health and mental health’, Medical Care Research and Review, 69(2), 123157. doi: 10.1177/1077558711430690

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Kirchner, J. E. (2015) ‘A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project’, Implementation Science, 10, 21. doi: 10.1186/s13012-015-0209-1

    • Search Google Scholar
    • Export Citation
  • Prager, K., McKee, A. (2015) ‘Co-production of knowledge in soils governance’, International Journal of Rural Law and Policy, Special edition 1.

    • Search Google Scholar
    • Export Citation
  • Proctor, E. K., Powell, B. J., McMillen, J. C. (2013) ‘Implementation strategies: recommendations for specifying and reporting’, Implementation Science, 8, 139. doi: 10.1186/1748-5908-8-139

    • Search Google Scholar
    • Export Citation
  • Raghavan, R., Inoue, M., Ettner, S. L., Hamilton, B. H. (2010) ‘A preliminary analysis of the receipt of mental health services consistent with national standards among children in the child welfare system’, American Journal of Public Health, 100(4), 742749. doi: 10.2105/AJPH.2008.151472

    • Search Google Scholar
    • Export Citation
  • Rogal, S.S., et al (2017) ‘The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample’, Implementation Science, 12(60), 113. doi: 10.1186/s13012-016-0533-0

    • Search Google Scholar
    • Export Citation
  • Saldana, L., Schaper, H., Campbell, M., Chapman, J. (2015) ‘Standardized measurement of implementation: the universal SIC’, Implementation Science, 10, A73. doi: 10.1186/s13012-015-0262-9

    • Search Google Scholar
    • Export Citation
  • Senge, P. M. (2014) The Fifth Discipline: The Art and Practice of the Learning Organization (New York: Knopf Doubleday Publishing Group).

    • Search Google Scholar
    • Export Citation
  • The CIPHER Investigators (2014) ‘Supporting policy in health with research: an intervention trial (SPIRIT) – protocol for a stepped wedge trial’, BMJ Open, 4(7), e005293. doi: 10.1136/bmjopen-2014-005293

    • Search Google Scholar
    • Export Citation
  • Van Kerkhoff, L., Lebel, L. (2015) ‘Coproductive capacities: rethinking science-governance relations in a diverse world’, Ecology and Society, 20(1), 14. doi: 10.5751/ES-07188-200114

    • Search Google Scholar
    • Export Citation
  • Vargo, S. L., Lusch, R. F. (2004) ‘Evolving to a new dominant logic for marketing’, Journal of Marketing, 68(1), 117. doi: 10.1509/jmkg.68.1.1.24036

    • Search Google Scholar
    • Export Citation
  • Waltz, T. J., Powell, B. J., Chinman, M. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Damschroder, L. J., Kirchner, J. E. (2014) ‘Expert recommendations for implementing change (ERIC): protocol for a mixed methods study’, Implementation Science, 9(39), 112. doi: 10.1186/1748-5908-9-1

    • Search Google Scholar
    • Export Citation
  • Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., Blachman, M., Dunville, R., Saul, J. (2008) ‘Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation’, American Journal of Community Psychology, 41(3–4), 171181. doi: 10.1007/s10464-008-9174-z

    • Search Google Scholar
    • Export Citation
  • Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., Silovsky, J. F., Hecht, D. B., Chaffin, M. J. (2012) ‘Dynamic adaptation process to implement an evidence-based child maltreatment intervention’, Implementation Science, 7, 32.

    • Search Google Scholar
    • Export Citation
  • Aarons, G. A., Palinkas, L. A. (2007) ‘Implementation of evidence-based practice in child welfare: service provider perspectives’, Administration and Policy in Mental Health and Mental Health Services, 34, 411419. doi: 10.1007/s10488-007-0121-3

    • Search Google Scholar
    • Export Citation
  • Abraham, C., Michie, S. (2008) ‘A taxonomy of behavior change techniques used in interventions’, Health Psychology, 27(3), 379. doi: 10.1037/0278-6133.27.3.379

    • Search Google Scholar
    • Export Citation
  • Blase, K. (2009) Technical Assistance to Promote Service and System Change: Roadmap to Effective Intervention Practices #4 (Tampa, FL: University of South Florida, Technical Assistance Center on Social Emotional Intervention for Young Children).

    • Search Google Scholar
    • Export Citation
  • Boaz, A., Metz, A. (in press) ‘Engaging Stakeholders in the Implementation of Evidence-Based Programs, in E. Zimmerman (ed), Researching Together: Engaging Patients and Stakeholders in Health Research, from Topic Identification to Policy Change, Sage Press.

  • Boaz, A., Robert, G., Locock, L., Sturmey, G., Gager, M., Vougioukalou, S., Ziebland, S., Fielden, J. (2016) ‘What patients do and their impact on implementation: an ethnographic study of participatory quality improvement projects in English acute hospitals’, Journal of Health Organization and Management, 30(2), 258278. doi: 10.1108/JHOM-02-2015-0027

    • Search Google Scholar
    • Export Citation
  • Boyd, M. R., Powell, B. J., Endicott, D., Lewis, C. C. (2018) ‘A method for tracking implementation strategies: an exemplar implementing measurement-based care in community behavioral health clinics’, Behavior Therapy, 49(4), 525537. doi: 10.1016/j.beth.2017.11.012

    • Search Google Scholar
    • Export Citation
  • Bunger, A. C., Powell, B. J., Robertson, H. A., MacDowell, H., Birken, S. A., Shea, C. (2017) ‘Tracking implementation strategies: a description of a practical approach and early findings’, Health Research Policy and Systems, 15, 15. doi: 10.1186/s12961-017-0175-y

    • Search Google Scholar
    • Export Citation
  • Cabassa, L. J. (2016) ‘Implementation science: why it matters for the future of social work’, Journal of Social Work Education, 52(suppl 1), 3850. doi: 10.1080/10437797.2016.1174648

    • Search Google Scholar
    • Export Citation
  • Chambers, D. A., Glasgow, R. E., Stange, K. C. (2013) ‘The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change’, Implementation Science, 8(1), 117. doi: 10.1186/1748-5908-8-117

    • Search Google Scholar
    • Export Citation
  • Durlak, J. A., Dupree, E. P. (2008) ‘Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation’, American Journal of Community Psychology, 41(3–4), 327350. doi: 10.1007/s10464-008-9165-0

    • Search Google Scholar
    • Export Citation
  • Edelenbos, J., Van Buuren, A., Van Schie, N. (2011) ‘Co-producing knowledge: joint knowledge production between experts, bureaucrats and stakeholders in Dutch water management projects’, Environmental Science & Policy, 14, 675684.

    • Search Google Scholar
    • Export Citation
  • Ghate, D. (2016) ‘From programs to systems: deploying implementation science and practice for sustained real world effectiveness in services for children and families’, Journal of Clinical Child & Adolescent Psychology, 45(6), 812826.

    • Search Google Scholar
    • Export Citation
  • Hasson, F., Keeney, S., McKenna, H. (2000) ‘Research guidelines for the Delphi survey technique’, Journal of Advanced Nursing, 32(4), 10081015.

    • Search Google Scholar
    • Export Citation
  • Hasson, H., Nilsen, P., Augustsson, H., von Thiele Schwarz, U. (2018) ‘Empirical and conceptual investigation of de-implementation of low-value care from professional and health care system perspectives: a study protocol’, Implementation Science, 13(1), 67. doi: 10.1186/s13012-018-0760-7

    • Search Google Scholar
    • Export Citation
  • Holmes, B. J., Best, A., Davies, H., Hunter, D., Kelly, M., Marshall, M., Rycroft-Malone, J. (2016) ‘Mobilising knowledge in complex health systems: a call to action’, Evidence & Policy, 13(3), 539560.

    • Search Google Scholar
    • Export Citation
  • Horner, R., Blitz, C., Ross, S. W. (2014) The Importance of Contextual Fit When Implementing Evidence-Based Interventions, Issue Brief (Washington, DC: Office of the Assistant Secretary for Planning and Evaluation, Office of Human Services Policy, US Department of Health and Human Services).

    • Search Google Scholar
    • Export Citation
  • Katz, J., Wandersman, A. (2016) ‘Technical assistance to enhance prevention capacity: a research synthesis of the evidence base’, Prevention Science, 17, 417428. doi: 10.1007/s11121-016-0636-5

    • Search Google Scholar
    • Export Citation
  • Leeman, J., Birken, S. A., Powell, B. J., Rohweder, C., Shea, C. M. (2017) ‘Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice’, Implementation Science, 12, 125. doi: 10.1186/s13012-017-0657-x

    • Search Google Scholar
    • Export Citation
  • Mayne, J. (2008). ‘Contribution analysis: an approach to exploring cause and effect’, Institutional Learning and Change Initiative (ILAC), Brief, 16, 4.

    • Search Google Scholar
    • Export Citation
  • Mayne, J., Rist, R. C. (2006) ‘Studies are not enough: the necessary transformation of evaluation’, The Canadian Journal of Program Evaluation, 21(3), 93120.

    • Search Google Scholar
    • Export Citation
  • Metz, A., Bartley, L. (2016) ‘Co-creating the conditions to sustain the use of research evidence in public child welfare’, Child Welfare, 94(2), 115139.

    • Search Google Scholar
    • Export Citation
  • Metz, A., Boaz, A. (2016) ‘Where are the stakeholders in implementation science? Integration and implementation insights’, https://i2insights.org/2016/09/08/co-creation-and-implementation-science/

    • Search Google Scholar
    • Export Citation
  • Metz, A., Louison, L., Ward, C., Burke, K. (2018) Implementation Specialist Practice Profile: Skills and Competencies for Implementation Practitioners, Working Draft (Chapel Hill, NC: National Implementation Research Network, University of North Carolina at Chapel Hill).

    • Search Google Scholar
    • Export Citation
  • Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., Wood, C. E. (2013) ‘The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions’, Annals of Behavioral Medicine, 46(1), 8195. doi: 10.1007/s12160-013-9486-6

    • Search Google Scholar
    • Export Citation
  • Morton S. (2015) ‘Research impact assessment: a ‘Contributions’ approach’, Research Evaluation, 24(4), 405419. doi: 10.1093/reseval/rvv016

    • Search Google Scholar
    • Export Citation
  • Nutley, S. M., Walter, I., Davies, H. T. (2007) Using Evidence: How Research Can Inform Public Services (Bristol: Policy Press).

  • Okali, C., Pawlowski, S. D. (2004) ‘The Delphi method as a research tool: an example, design considerations and applications’, Information & Management, 42(1), 1529.

    • Search Google Scholar
    • Export Citation
  • Pawson, R. (2013) The Science of Evaluation: A Realist Manifesto. (Los Angeles, CA: Sage).

  • Perry, C. K., Damschroder, L. J., Hemler, J. R., Woodson, T. T., Ono, S. S., Cohen, D. J. (2019) ‘Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory’, Implementation Science, 14, 32. doi: 10.1186/s13012-019-0876-4

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., Fernandez, M. E., Williams, N. J., Aarons, G. A., Beidas, R. S., Lewis, C. C., McHugh, S. M., Weiner, B. J. (2019) ‘Enhancing the impact of implementation strategies in healthcare: a research agenda’, Frontiers in Public Health, 7(3), 19. doi: 10.3389/fpubh.2019.00001

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T, Bunger, A. C., Glass, J. E., York, J. L. (2012) ‘A compilation of strategies for implementing clinical innovations in health and mental health’, Medical Care Research and Review, 69(2), 123157. doi: 10.1177/1077558711430690

    • Search Google Scholar
    • Export Citation
  • Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Kirchner, J. E. (2015) ‘A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project’, Implementation Science, 10, 21. doi: 10.1186/s13012-015-0209-1

    • Search Google Scholar
    • Export Citation
  • Prager, K., McKee, A. (2015) ‘Co-production of knowledge in soils governance’, International Journal of Rural Law and Policy, Special edition 1.

    • Search Google Scholar
    • Export Citation
  • Proctor, E. K., Powell, B. J., McMillen, J. C. (2013) ‘Implementation strategies: recommendations for specifying and reporting’, Implementation Science, 8, 139. doi: 10.1186/1748-5908-8-139

    • Search Google Scholar
    • Export Citation
  • Raghavan, R., Inoue, M., Ettner, S. L., Hamilton, B. H. (2010) ‘A preliminary analysis of the receipt of mental health services consistent with national standards among children in the child welfare system’, American Journal of Public Health, 100(4), 742749. doi: 10.2105/AJPH.2008.151472

    • Search Google Scholar
    • Export Citation
  • Rogal, S.S., et al (2017) ‘The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample’, Implementation Science, 12(60), 113. doi: 10.1186/s13012-016-0533-0

    • Search Google Scholar
    • Export Citation
  • Saldana, L., Schaper, H., Campbell, M., Chapman, J. (2015) ‘Standardized measurement of implementation: the universal SIC’, Implementation Science, 10, A73. doi: 10.1186/s13012-015-0262-9

    • Search Google Scholar
    • Export Citation
  • Senge, P. M. (2014) The Fifth Discipline: The Art and Practice of the Learning Organization (New York: Knopf Doubleday Publishing Group).

    • Search Google Scholar
    • Export Citation
  • The CIPHER Investigators (2014) ‘Supporting policy in health with research: an intervention trial (SPIRIT) – protocol for a stepped wedge trial’, BMJ Open, 4(7), e005293. doi: 10.1136/bmjopen-2014-005293

    • Search Google Scholar
    • Export Citation
  • Van Kerkhoff, L., Lebel, L. (2015) ‘Coproductive capacities: rethinking science-governance relations in a diverse world’, Ecology and Society, 20(1), 14. doi: 10.5751/ES-07188-200114

    • Search Google Scholar
    • Export Citation
  • Vargo, S. L., Lusch, R. F. (2004) ‘Evolving to a new dominant logic for marketing’, Journal of Marketing, 68(1), 117. doi: 10.1509/jmkg.68.1.1.24036

    • Search Google Scholar
    • Export Citation
  • Waltz, T. J., Powell, B. J., Chinman, M. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., Damschroder, L. J., Kirchner, J. E. (2014) ‘Expert recommendations for implementing change (ERIC): protocol for a mixed methods study’, Implementation Science, 9(39), 112. doi: 10.1186/1748-5908-9-1

    • Search Google Scholar
    • Export Citation
  • Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., Blachman, M., Dunville, R., Saul, J. (2008) ‘Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation’, American Journal of Community Psychology, 41(3–4), 171181. doi: 10.1007/s10464-008-9174-z

    • Search Google Scholar
    • Export Citation

Supplementary Materials

  • 1 University of North Carolina at Chapel Hill, , USA
  • | 2 Kingston University London and St. George’s University of London, , UK
  • | 3 University of North Carolina at Chapel Hill, , USA

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 82 82 55
PDF Downloads 27 27 21

Altmetrics

Dimensions