Abstract
Background / aims / objectives
‘Embedded’ approaches to knowledge mobilisation are gaining currency, as health and social care services come under increasing pressure to redesign services now rather than wait for research. One such approach is the ‘Researcher-in-Residence’ (RiR) model which seeks to co-produce knowledge for action. The aim of this paper is to extend the evidence base regarding mechanisms of impact.
Methods
A two-year mixed-method case study of the experience and impact of two part-time RiRs, embedded within an Integrated Care Organisation to support the implementation of new models of care. Data included the results of an anonymous impact survey sent to 80 key stakeholders, field notes of meetings (n=112), and observations of naturally occurring events (n=68).
Findings
Impacts were identified in relation to use of co-produced evidence, capacity building, changes in ways of working, and to a lesser degree changes in operations or strategy. Impact involved learning which was mediated by three non-linear, non-predictable, positive and negative feedback cycles (expectations, access, learning/ improvement). A mixture of technical skills, personal attributes and behaviours were identified as key to this mediation.
Discussion/conclusion
The RiR model promises a timely, applied and transferable research model that contributes to the development, evaluation and adaptation of innovations that seek to integrate services where the evidence base is weak and uncertain. However, the model is not without challenges. These could be addressed by flexibility of research design and funding, and adequately supporting and developing key attributes of RiRs.
key messages
RiRs are embedded in care systems to co-produce knowledge for action.
Integrated care case study evidences mechanisms of impact and attributes from a stakeholder perspective.
Impact was mediated through three non-linear feedback cycles: Expectations, Access, and Learning and Improvement.
The RIR model is a complex implementation intervention that merits roll-out combined with further study.
Introduction
The translation of research-based knowledge into practice – knowledge mobilisation - has become the focus of policymakers worldwide, as health and care services struggle to cope with rising demand, greater patient expectations and stagnating budgets. How to bridge the ‘know-do’ gap has led to the emergence of new academic disciplines, such as Implementation Science. Typically, this portrays the implementation gap as a failure of the delivery mode, frontline staff or institutions to use the latest knowledge, a problem that can be fixed by studying the factors that facilitate or hinder models of evidence roll-out (Greenhalgh, 2017). To date and internationally, the impact of efforts to understand and improve such knowledge transfers appears to have fallen short of expectations (Kreindler, 2016; Lockett et al, 2014; Reed et al, 2017).
In integrated health and social care services, the translation problem is exacerbated by the fact that the evidence base to support many proposed changes may be weak methodologically, ambiguous in its findings or absent altogether. A case in point is the evidence base to support the UK policy (National Audit Office, 2017). Many studies lack conceptual clarity, are poorly designed, or operate in very different health service contexts, limiting their generalisability (Damery, 2016).
Such concerns have given rise to a shift in models of knowledge mobilisation from passive, linear and instrumental conceptions that see ‘knowledge’ as a transferable product; through a more relational approach connecting researchers and decision makers; to the idea that knowledge is neither fixed nor privileged (Davies et al, 2015; Ward et al, 2009). Reflecting the growing emphasis on the ‘Zeitgeist’ of co-production (Palmer et al, 2018), embedded researchers such as Researchers-in-Residence (RiRs) have gained traction (Marshall, 2014; Marshall et al, 2016; Marshall et al, 2014), supported by international calls to action (Holmes et al, 2017). Researchers-in-Residence can sit in the ‘middle ground’ (Guthrie et al, 2017) between research and evaluation. In our case, the model combines embedded, operational service evaluation with an action-orientated, participatory approach to research that seeks to impact the development and implementation of service innovations in real time.
The RiR model seeks to co-produce (Clarke et al, 2017) ‘knowledge or theory in practice’ (Argyris and Schön, 1974; Van de Ven, 2007). To achieve this (Vindrola Padros et al, 2017) suggest researchers should: 1) be co-located with or embedded in the services for significant amounts of time and have a stake in the research or improvement initiative they are supporting; and 2) negotiate the meaning and use of research-based knowledge and co-produced knowledge that is sensitive to the local context and responsive to the continuously changing needs of the system.
However, the evidence base to support this model is limited, comprising a few case studies (Eyre et al, 2017; Lalani et al, 2018) and a narrative review (Vindrola Padros et al, 2017). These have sought to clarify the model and identify challenges such as relationship building, flexibility around the scope of the work, and tensions with maintaining professional identity (Vindrola Padros et al, 2017). These have mainly been based on experiences of RiR teams rather than the partners they work with. Furthermore, questions still remain about mechanisms of how the RiR model impacts on stakeholders and systems. This paper reports on the impact of two RiRs working on care model innovations in an integrated care provider organisation, as perceived by stakeholders. It seeks to identify attributes and behaviours of the RiRs and stakeholders that facilitate effective interaction and several key processes important to delivering impact, laying foundations for theory development.
Setting
In Torbay and South Devon, the acute NHS Trust and adult social care and community services merged in October 2015 to create an Integrated Care Organisation (ICO) – among the first in the UK to adopt this new organisational form – implementing a new care model, as part of the integration of services. This vertical integration builds on the history of horizontal (that is, across sectors) integration between organisations working in the Torbay area (Wilding, 2010), following the development of the ‘Mrs Smith model’ and the creation of the Torbay Care Trust in 2005 (Thistlethwaite, 2011). The ICO’s integrated model of care incorporates over 30 service innovations and enabling functions that cover prevention, voluntary sector social prescribing, self-care, proactive transition management and intermediate care.
Against this background, strong emphasis has been placed on the need for research and evaluation to evidence and support desired changes. Torbay Medical Research Fund, the ICO and the The National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South West Peninsula invested in research and evaluation through the RiR model, initially over a three-year period from April 2016 until 2019.
The two RiRs were employed and supervised by the University of Plymouth and based within the Quality Improvement team for reporting, support and development purposes. The two RiRs were provided with honorary contracts which gave them full access to all services and data. The RiRs based themselves in various locations across the ICO footprint to enable them to work alongside teams.
The RiRs were mixed methods specialists and mid-career research fellows, including a public health consultant with analytics skills and a social scientist with ethnographic skills, working collaboratively on all aspects of the projects. They were supported by the QI (Quality Improvement) Lead and held regular meetings off-site with the principal investigators, a health policy researcher and a GP / primary care researcher (all co-authors of this paper). Governance and meaningful co-design was also assured through executive sponsorship by the Deputy Medical Director and regular attendance / reporting to the bi-monthly cross-system Care Model Delivery Group.
The RiRs have been working together with managers and frontline teams to support the evaluation of specific care model service innovations and the development of a metrics dashboard, as well as conducting research into how these and other service innovations integrate to impact on person-centred, coordinate care (PCCC) (Lloyd et al, 2017). The research was a mixed methods study involving a phased, iterative process of collecting and analysing quantitative and qualitative data with the aim of developing mid-range theories (to be reported elsewhere). The core of the RiR activity focused on two care model innovations: 1) the Enhanced Intermediate Care Service (EIC) provided by multidisciplinary teams in the community looking to avoid hospital admission or bridging journeys back to people’s homes; and 2) the co-located holistic link-worker / social prescribing, Wellbeing Coordinators Programme (WBC), ideally preventing people from becoming unwell in the first place. The scope also included a case study of how these contribute to integration in one of the five ICO geographical locality hubs.
Methods
This paper is based on data from a stakeholder online impact survey and reflective field notes collected by the two RiRs during the first two years (2016–2018).
The survey design was based on impact domains identified through publications by the department of Quality Improvement (QI) at University College London (Eyre et al, 2017, Holmes et al, 2017). It was developed by the two RiRs and the QI lead (co-author) with input from Principal Investigators and tested with two further colleagues. The purpose of the survey was twofold. Firstly, to explore the impact of the RiRs and through this to develop a business case for continued investment in researching and evaluating the care model as it develops. Secondly, to more fully understand the mechanisms of impact and attributes required of a successful RiR model. The survey contained nine questions, of which six contained 5-point Likert Scale response options. The six quantitative response options captured: 1) overall impact; 2) new ways of thinking; 3) evidence use; 4) up-skilling staff; (5) impact on operations / planning; and 6) impact on organisational strategy / culture. As well as open comments boxes for each quantitative question, there were also three qualitative questions asking: where the RiR approach worked best, what might be improved, and the required attributes of a RiR. The survey was conducted anonymously, but respondents were invited to identify their job role in the last question (available on request from author).
The online survey was sent to 80 stakeholders who were recorded as active partners through the matched funding capture system and deemed to have engaged meaningfully with the RiRs. The sample comprised a mix of professional backgrounds and roles from commissioning, primary, community and acute services, social care, voluntary services, and patient and public representatives. Noting that some had mixed roles as health professionals and managers, the distribution of main job roles in this sample was: 44 managers, 16 health professionals, 12 voluntary sector representatives, 5 administrative staff and 3 patient and public representatives. The heterogeneity and overlap of stakeholder attribution with various evaluation projects (both service innovations were co-located in community hubs) made it difficult to look for meaningful quantitative response patterns by potential subgroups. The survey ran from April 2018 and was open for three months. Two automated reminders were sent, and non-responders were followed up via email or verbally by the QI Lead.
Reflective field notes were recorded independently by both RiRs. Typically, notes were made following meetings and written up in Word, and stored on a secure, password-protected server using NHS devices and servers. Notes were related to dedicated meetings (n=112), or at naturally occurring events (n=68).
The analytic strategy collated the quantitative results of the survey and interpreted the findings in light of: a) an analysis of the qualitative survey comments (using basic content and thematic analysis in NVIVO and Excel, analysing prevalence of codings for three qualitative questions in particular); b) automated search queries run on the field notes in NVIVO; c) comparing and contrasting the factors that RiRs felt influenced their impact in three selected project areas; and d) triangulation of data sources, iterative sense-checking of categories and themes, discussion and reflection between co-authors.
Individual and organisational consent was obtained, while ethical approval was granted through ‘Proportionate Review’ by the NHS Health Research Authority (Research Ethics Committee reference: 17/LO/1745; Protocol number: PSMD-208147-SA-FG-034; Integrated Research Application System project ID: 208147).
Findings
The following section comprises quantitative findings of the stakeholder online survey, which helped build an understanding of how RiRs brought about change through their activities. The section then goes on to illustrate identified impacts through a range of feedback cycles.
Impact survey
Fifty stakeholders responded to the online survey, a response rate of 62.5% (N=50/80). Qualitative comments were reported from a minimum of 22 respondents to a maximum of 42 respondents for respective open comment boxes and the three qualitative questions. Of the 50 respondents only 35 stakeholders partly disclosed their role, which made it hard to find and analyse response patterns based on possible subgroups.
Figure 1 shows that over half of respondents (53%) reported a great or moderate impact on ‘Improved use of evidence (research findings, guidelines, dashboards, evaluation data and so on)’, which was better than expected. Respondents reported a similar level of impact on ‘Encouraging new ways of thinking / working, for example on their perspectives, work or practice’, as well as on ‘Improved research and evaluation skills (that is, through advice, practice or training)’. Impact perceived as going beyond the RiR’s influence on the individual, such as on ‘Influencing strategic direction or culture (communications, business cases, organisational learning)’ was less at 44% of stakeholders reporting a great or moderate impact, and lesser still when respondents considered the impact on ‘Influencing operations and planning (Multidisciplinary Teams, care delivery and so on)’ (39% great or moderate).
Assessment of RiR impact across specific domains (stakeholder survey)
Citation: Evidence & Policy 15, 2; 10.1332/174426419X15538508969850
Qualitative analysis
Key to facilitating impact were three interacting, positive and negative, feedback and feed-forward cycles: Expectations cycles emerged through managed and negotiated input, and were sensitive to organisational structure and sector; Access cycles were grounded in successful and widening engagement and relationship building; and Learning and Improvement cycles facilitated a process of mutual learning about the RiR approach and integrated care, that assisted partners to co-create impact directly and indirectly at the level of individuals, teams and organisations. This included impacts generated on a continuum of attribution, which were more direct on individuals and teams (in terms of capability and capacity building), and more indirect in relation to operational planning, strategy and organisational culture.
The interaction between cycles must be considered non-linear, and impact was not necessarily dependent on each cycle, for example, learning could happen at any stage of the process or aligning expectations might lead to learning and improvement. The dynamics of each of the cycles and their relationship with learning and improvement, and RIR attributes are discussed below.
Expectations
The RiR contract and research questions, developed to satisfy both the ICO and the research funding body, were broad in spectrum, in part recognising that the care model was new and would develop over time. This provided flexibility to iterate and co-design the evaluation and develop the research questions. However it also made managing expectations challenging. Expectations evolved over time and sometimes differed in relation to the scope and focus of the work, the approach and role of RiRs, clarifying research and evaluation aims, and the required resources and capacity to deliver. Resolving these differences required an iterative process of dialogue, negotiation and compromise. Alignment of expectations facilitated engagement to increase potential for new ways of (joint) working and mutual learning. This management and negotiation worked better through increasing trust between the RiRs and stakeholders, and where expectations were unresolved or conflicting, disengagement could arise. Repair was possible through more engagement in further activities to build trust. It could also result in work opportunities and activity, increased through its intersection with the Access Loop.
Scope and focus
The original research and evaluation protocols identified four areas of evaluation as well as a system-wide piece of research on implementation. It became apparent early on that the focus of the research and evaluation were too broad and ambitious, given the resources, time and capacity available to the RiR and stakeholders, and the emerging, iterative approach the RiRs were taking.
I think the impact issue is not about the RiRs themselves, but about what the strategic management is willing to hear from their reports (good, bad and ugly) and recommendations and what improvements the management team wants to implement.
This ongoing debate became most apparent at the 12-month reporting stage, and led to renegotiating the focus of the research on one geographic locality and reducing the number of evaluations of innovations to focus on the two with most novelty and investment (EIC, WBC).
Approach and role
When the funding was applied for, the aim from the beginning was to try a more novel approach to research, using the RiR model. However, the host organisation and the team appointing the posts had little experience of this role and therefore it took time and learning from all parties to embed the role. This was, perhaps, no surprise given the absence of an accepted blueprint for the model. Nevertheless, these differences had to be negotiated at several stages.
More clarity on role at the beginning would have helped – although it was an iterative process of understanding together so it’s important to have enough time to do this.
Understanding research and evaluation
Long short: difficult but amiable enough meeting as (again) mostly about managing expectations about evaluation and what might be research going forward. It was important for building trusting relationships, which is a key message we emphasised (as well as our role in driving culture change). Didn’t go into detail of findings and not sure if they read it, did not agree concrete action about further dissemination yet. Most interested in whether ‘it’ [the care model] ‘works’ and overall disappointment that our report does not deliver this.
Resource and capacity
Survey respondents also highlighted differences in expectations related to resources and capacity to undertake joint evaluation activities. The participatory approach placed more demands on managers and frontline staff to commit time and resources to evaluation at a time when the organisation was committed to implementing its service innovations ‘at pace and scale’ (Extraordinary Joint Management Meeting, Observation note 005, 16.05.16; All Manager Meeting, Observation Note 019, 25.07.16).
Working quickly (is important) when doing direct work with clinical staff to avoid excess time spent taking teams away from their roles. [There needs to be] realism in terms of staff capacity; a balance between operational pressures and what we would do academically, with what is realistically achievable. (Clinical lead)
Access
Access to and building relationships with multiple stakeholders in the health and care system was crucial to undertaking evaluation and research activities, particularly as the focus of the RiR’s work was on integrated care horizontally and vertically. Spending time engaging with or embedding in different parts of the system formally (that is, through co-location) and informally (that is, supporting related evaluations beyond the host organisation) enabled the RiRs to build networks and develop trusting relationships, principally through generating capacity and sharing expertise. This in turn enabled the RiR to identify championing leaders or staff. In several instances, this circular process may have led to an increase in engagement between the host and the wider community.
A key to access was acquiring honorary contractual status with the ICO. An outwardly visible effect of this was the issuing of NHS email addresses as well as blue NHS lanyards. These were used always while engaging with NHS stakeholders to demonstrate insider status, and more selectively used when engaging with non-NHS stakeholders, where RiRs might use University lanyards instead to express credibility and independence (Observation Note 059, ICO Board Away Day, 13.07.18, RiR: “I wear this badge with pride”).
Time to build relationships and expand networks
A crucial aspect of gaining access was the RiRs’ ability to spend time with people in different parts of the system. Over the first two years, the RiRs engaged with 124 stakeholders (totalling 874 hours of co-production) across the system (as active partners not research subjects). This enabled the RiRs to identify relevant stakeholders, including the public, and to build working relationships and a ‘platform to influence’ (Stakeholder survey: Senior clinical manager). This happened mainly through supporting senior clinical managers on service evaluation or system transformation. Effective working relationships were also strengthened when the RiRs formally co-located with the care model operational transformation team.
Working informally with system stakeholders also served to develop a network of links and relationships that in turn led to increased access, as confidence and trust in the RiRs grew. The impact on access was further increased when the RiRs followed guidance from senior NHS managers to ‘go where the energy is’ (Meeting note 005, 23.02.16) and identify champions and receptive teams or settings in which they could co-locate.
In the case of the WBC programme, co-locating one RiR with a partner voluntary sector organisation in the case study locality facilitated in-depth qualitative case studies and practitioner observations. Use of this evidence further increased access and impact. After five months of investing time and capacity in building relations and networks, one RiR observed following a locality Redesign Implementation Meeting, how the chair applied the plural, personal pronoun ‘we’ to express that RiRs were part of the integrated team (Observation Note 021, 27.07.16).
Vertical and horizontal integration
Inspiring my management team – opening their eyes from their immediate business, giving them space to think / reflect review. Real value to my leads and teams to have their voices heard and made to feel they add value and influence the development of services.
Community engagement
Hopefully some of the evaluations that Carers Services have completed and shared with them have helped to develop their overall understanding of the ICO from a Carer’s perspective.
It also facilitated widening trusting relations between the voluntary sector and the ICO, opening up space for closer working on the WBC programme, and focusing on patient experience to drive changes.
Learning and improvement
Learning and improvement cycles describe a process of mutual learning:
Learning is a key part of improvement: it tends to happen first and improvement can follow.
Learning is reported to be more pronounced at individual (rather than organisational) level through direct contact with RiRs.
Organisational learning takes time to filter through individual learning, especially when coupled with evaluation findings (a latent process).
Learning can reaffirm assumptions leading into developing innovations or a more radical rethink when assumptions on expectations are not met.
Likeliness of emergent improvement can be enhanced through interaction with access and expectations cycles.
Concrete improvement outcomes are hard to attribute directly to RiR activities.
The RiR activities were recognised by survey respondents, in that RiRs:
provided additional resource (n=21 of 376 codings); not doing things for people but co-producing with them (see expectation management);
gave ad hoc advice (n=20; which increases access, see above);
provided different perspectives or challenge (n=25); which may have resulted in more or less direct learning, and hence relates to potential of improvement or disengagement;
brokered knowledge and intelligence and made links horizontally and vertically across the system while mobilising and connecting champions (n=23; see access, which may lead to improvements via the stakeholders engaged).
RiR attributes
Stakeholders provided numerous responses to the qualitative questions on the attributes required of an RiR, as well as perceived challenges and improvements. RiR learning and specific responses are summarised in a job profile (available on request from author). By overall ranking of the thematic content through the prevalence of codings, the following RiR job profile emerged: communication (n=28) and relationship skills (n=13); experience (n=30); training (n=12); knowledge (n=5); analytical (n=12) and judgemental (n=10) skills; mental effort (n=9); emotional effort (n=7); responsibility for improvement (n=4); and responsibility for ethics (n=3).
A good ability to connect with all personnel, to understand their roles and how to harness the information that they require so that they can do their job better. A good communicator, good energy, a belief in what they are doing, a broad understanding of the environment that they are called to research in and the problems that it is facing, so better equipped to engage and deliver relevant research.
More direct learning and improvement
In the stakeholder survey, evidence use was ranked as the area in which the RiRs had the greatest impact, 71% of respondents recognising some, moderate or a great impact (see impact domain results in Figure 1 above). This was supported by qualitative comments from the survey which suggest that implementing services ‘at pace’, finding dedicated analytical capacity, with ever-changing cost-saving and performance imperatives, was hard. Stakeholders appeared to have scarce resources to reflect about what the service changes were designed to achieve and were working as desired, focusing on ‘transactional’ rather than ‘transformational’ change (Observation note 050, Care Model Delivery Group; 03.10.17). This might also have been related to timing, as the RiRs started during a busy care model implementation phase and six months after the merger of the organisations.
Research has backed up with evidence some of my assumptions and understanding around how the organisation works and its culture. An external objective view which is evidence based, may have somewhat helped to influence the approach of the senior leadership of the Trust. (Middle manager)
This might be best illustrated by the impact of the substantive evaluation of the WBC programme (co-authored with partners, to be published elsewhere) which ultimately led to the WBC service being recommissioned in Spring 2018. Expectations and access were resolved early on, facilitated by dedicated managers and champions, who quickly grasped what the RiR model was trying to achieve, and who set up and drove a dedicated evaluation partnership.
(They) have provided consultancy on study design and their ongoing support will be critical in determining cost benefits in the second year of [the Well-Being Coordination] project starting in September. (WBC Project Manager)
More indirect learning and improvement
Several respondents pointed to the value of the RiR model in providing perspective based on intelligence gathered within the system: “helping me see a different perspective” (Executive); “They have been able to challenge our thinking which has led to some practice development” (Voluntary sector worker). Suggestive of reaching deeper to challenge underlying assumptions and norms, respondents noted that the RiRs not only provided insights on the “larger picture” (Programme Manager), or were able to get people “to be brave and think differently” (Operational Manager) They were also valued for highlighting “difficult issues which may not otherwise be addressed” (Middle Manager) and allowing “more reasoning around the why and how” (Programme Manager).
… I have appreciated the different thinking approaches and styles of working which have been helpful in meetings especially when we have needed to step back and consider the broader range of influences. They [the RiRs] have also helped me to understand the importance of research in practice and my contribution and role in enabling that. I think it has taken me a while to really understand and consider how best to maximise the RIR approach, because of the urgency and complexity of competing agendas and the need for more instant identification of key contributions. (Senior Manager)
Examples of more indirect learning, captured in our field notes, related to the impact of two practitioner surveys conducted by the RiRs. Collecting data and sharing the practitioner survey findings with locality teams and managers, helped build buy-in, networks and relationships from the bottom up (Access), which in turn led to a further collection of evidence on person-centred coordinated care. The findings from the work were developed into recommendations in the 12-month report. All of these key findings were picked up to various degrees in programmes of improvement and implementation work in the subsequent months.
The explicit use of analytical and emotional material presented in a novel way was instrumental in facilitating stakeholders’ understanding about the RiR model (aligning expectations) and its potential value to the organisation. This not only led to the ICO recommissioning the RiRs for two more years but also to work on the part of one of the Directors to develop the RiR model within other areas of the organisation (acute services). In terms of wider impact, RiRs and locality teams were co-hosting learning events for external visitors including NHS England, NHS Improvement, the International Foundation of Integrated Care, and academics (like those involved in social prescribing). The locality case study data also helped securing a nomination for the nationally renowned Health Services Journal Award 2018.
Discussion
Key findings
This paper adds to the recently emerging case-based evidence base around a new approach to mobilising actionable knowledge: the applied, action-oriented, translational model of Researchers-in-Residence.
Our study reports evidence of impact on individual stakeholders, the organisation and to a lesser degree the wider system. We identified three domains in which feedback and feedforward loops could create virtuous or negative cycles or impacts. For the model to succeed in terms of facilitating learning and improvement, critical investment into awareness or active resolution of expectations, and promotion of presence and relationships in key settings to enhance access has to be made, with an ongoing risk of disengagement. As trusting relationships are established, learning may be more likely to happen on the levels of individual capacity which may indirectly impact on organisational learning and actual service improvement. The above process is more straightforward when learning and co-produced output align with expectations, underlying models and assumptions, and tends to be more resource intensive and risky when challenging those.
RiR attributes and activities appear to be wide-ranging and beyond those that might traditionally be expected of an academic role. People, communication and negotiation skills are central to engagement, and critically feed into access and expectation cycles. Receptive context is also important in this, as well as finding the energy and leaders open to working with RiRs. This is set against a background in which resources are scarce and there is pressure to implement change at pace.
The key enabler for the above to happen was the building and maintaining of trusting relationships, which took time. This was facilitated by listening and giving without expecting immediate return, building general capacity for evaluation and research through ad hoc consultation, while being intentionally open for this openness and generosity to yield returns in future. We found substantive co-location to be an important factor, as this allowed relationship, shared understanding, and joint ownership to emerge.
Strengths and weaknesses
This is the first study that explores the RiR model from the perspective of a wide range of stakeholders, with whom they have engaged mainly around evaluation of existing service innovation rather than researcher-led service development. It not only assesses their perception of impact on them and their organisation but also their views on the key attributes required of the role when considering the embedded evaluation function. We reduced the likelihood of reporting bias by conducting an anonymous online survey.
Although development of our theoretical model was driven largely by respondents’ comments, the validity of the findings was increased through triangulation against our own field notes captured longitudinally during our research.
Nevertheless, this is one case study, focused on integrated care, and comparison with other sites would have strengthened the validity of the findings. As the paper builds on a single case study and from the stakeholder perspective, there is also an emphasis on learning and improvement for the organisation – rather than a sense of learning applicable beyond. The wider project always had an explicit dual aim of creating transferable knowledge, the outcome of which will be reported in future papers.
Another potential weakness relates to the embedded nature of the researchers, which puts RiRs at risk of ‘going native’, undermining their objectivity, that is, when using NHS/University email and respective lanyards selectively. Although building trusting relationships can yield access and rich insight often not readily available to traditional researchers, it could also taint or compromise the researchers’ interpretation of events, trading off insider status with objective, external judgement. Depending on epistemological stance (Greenhalgh, 2017) the ‘in-betweenness’ of the RiR model (Vindrola Padros et al, 2017) can be seen as a strength or a weakness, arguably increasing the risk of biased reporting. For example, in our case several evaluations of terminated service pilots we conducted were only reported internally. In addition, the participatory, action-orientated nature of the work means researchers inevitably have a stake in the outcome. If their future funding comes from the host organisation, there may also be a conflict of interest when reporting findings.
In this study, bias might have arisen, for example, through the choice of service innovations the RiRs were eventually tasked to evaluate, opting for projects with more perceived engagement and leadership. To mitigate this risk of bias, we tried to ensure rigour through having two RiRs with complementary skill sets (to enhance triangulation), as well as regular meetings off and on site with external academic supervisors to provide critical, objective feedback and oversight of the work, a model reportedly helpful elsewhere (Vindrola Padros et al, 2018).
How the key findings relate to other relevant studies:
The impacts of the RiR models, as reported here concurred with ‘expected impacts’ noted by (Vindrola Padros et al 2018). Our findings also resonate with other RiR case studies that noted the need to identify and work with ‘champions’ and ‘allies’ as an important strategy for gaining entry and engaging staff (Vindrola Padros et al, 2018). In our case the close working relationship with the central and critical QI champion (a co-author), and through gradually enrolling senior managers, aligns closely with learning from other participatory QI initiatives (Lalani et al, 2018). Beyond what existing evidence says about the importance of senior champions, we see middle managers (who were the biggest group of survey participants) as critical catalysts for indirect impact strategically and operationally. In line with existing case studies of co-produced work (Heaton et al, 2016), we would also add that clinical and frontline staff are key enablers (‘active agents’) for change. As with existing accounts from other RiRs (Eyre et al, 2017; Lalani et al, 2018), our case study suggests that this can be a resource-intensive, compromising and demanding task. The pace and timing of informal decision making also reduces the scope to seek and use evidence to influence decisions (Wilson et al, 2017), something that came out clearly as a theme in our study.
Policy, practice or research implications and/or recommendations
Protocol Flexibility / Governance: recommendations from other embedded studies (Vindrola Padros et al, 2018) suggest that terms of engagement should be agreed from the outset in order to manage expectations, and that regular iterations of feedback with teams and management groups should be scheduled. With regard to the former and in our context of service evaluation, we would add that absolute certainty in terms of engagement is impossible and even unhelpful with contexts changing. However, regular meetings are essential. Indeed, the RiRs felt most impactful when being part of the team, and formally embedded in existing governance arrangements, rather than as an adjunct to it.
Timing and boundaries: balance to be struck between serendipity, opportunism/ flexibility, intention and iterative review of strategy to determine where to create the most impact (linked to expectations, access).
Flexible and matched funding to enhance capacity: our project was core funded by a local charity which allowed some flexibility in the protocol. NIHR–CLAHRC funding also contributed to the time of senior supervisors, something that might be extended to roll out the programme across geographic footprints. Developing a sustainable source of evaluation capacity (that is, through an ongoing consultation service) across partners could serve as a pipeline for research collaboration.
Length of projects: we were also maximising the length of input and potential impact through part-time working (1.5FTE between RiRs), which in turn allowed time for other projects and co-bidding for additional funds.
Co-location and honorary status: we also think that it is necessary to be completely and sustainably immersed, and that the RiR contracts specify that the place of work is on site, holding honorary contracts to enhance insider status and data protection.
Job profile: (available on request from author) as seen from a stakeholder perspective. For the purpose of service evaluation and applied research, we would see the RiR as an academic post, rather than an NHS post. This is crucial to the impartiality of the role. Middle to senior researchers are probably required for the post to assure they are autonomous enough. We are looking to a current national study which seeks to address the gap in transferable learning and to develop practical guidelines and role specifications (Marshall et al, 2018).
NHS training and induction: job profile shows how important it is for impactful evaluation to bring and acquire experience of the health and social care systems (not necessarily clinical), something described as a ‘culture shock’ by other embedded researchers (Vindrola Padros et al, 2018). We would strongly endorse recommendations by RiR colleagues (Vindrola Padros et al, 2018) that suggest a clear assessment of the setup before allowing a minimum of three months of an introductory period. In our case this took 6–12 months.
(Peer-) support: having two RiRs with complementary skill sets, peer support, and autonomy, supported by respective clinical and policy Principal Investigators as well as local champions was crucial in both maintaining academic professional identity and integrity and sustaining the role.
Joint ownership: we would add that regular iterations of feedback and sense-checking is crucial (our RiRs did on average two presentations of emerging findings per month), in the sense of ensuring that there is actual co-production in the process of refining and enhancing insights and their potential impact.
Conclusion
Co-production of actionable knowledge appears to be real and relevant in this case study on the evaluation of integrated care model innovations, where the evidence base is inconclusive and where interventions are complex. Theory is emerging and needed to address the challenge around a causal understanding and attribution in a rapidly changing policy and practice context. The RiR model is one way to create relevant and rigorous evidence and to create ownership for change. In this, the RiR model could be considered a complex intervention in its own right. It appears that the same partnership principle driving integrated care, the pooling of resources for mutual benefit, extends to researchers fulfilling an important improvement function as part of the wider multidisciplinary team.
However, although creating a legacy of independent evaluation without ongoing academic input was a key aim and starting point, our evidence is inconclusive as to whether this might be sustained without continued presence of RiRs in the system. There is also the question of whether this example is just a way of creating a ‘learning organisation’. The expectations from both academia and practice seem critical in this. In our case, expectation around summative evaluation clashed with RiRs’ normative starting point that formative and summative are complementary, that implementation and effectiveness are inextricably linked, and that an openness to what ‘doesn’t work’ is critical for learning and actual improvement. Yet this takes time, resources and mutual commitment to reach a plateau (less investment into expectation and access), from which both academically rigorous output and long-term and more tangible improvement in practice can be created.
Acknowledgements
Thanks for the engagement of all our partners across sectors and systems, in particular those responding to the stakeholder survey. Specific thanks to Ann Wagner (Director of Strategy and Improvement), Dr Joanne Watson (Deputy Medical Director), and Dawn Butler (Deputy Director of Strategy, Performance and Planning) for executive endorsement and steer of this work, helping inform the business case for continued investment in the RiR model at TSDFT. Thanks also to Dr Iain Lang (PenCLAHRC) for comments on the draft manuscript.
This work was funded by the Torbay Medical Research Fund under Grant No 120, and the Torbay and South Devon NHS Foundation Trust (TSDFT). The research was supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
Conflict of interest statement
The authors declare that there is no conflict of interest.
References
Argyris, C, Schön, D, 1974, Theory in practice: increasing professional effectiveness, San Francisco, CA: Jossey-Bass
Clarke, D, Jones, F, Harris, R, Robert, G, 2017, What outcomes are associated with developing and implementing co-produced interventions in acute healthcare settings? A rapid evidence synthesis, BMJ Open 7, 7, e014650
Damery, SF, Flanagan, S, Combes, G, 2016, Does integrated care reduce hospital activity for patients with chronic diseases? An umbrella review of systematic reviews, BMJ Open 6, 11. doi:
Davies, H, Powell, A, Nutley, S, 2015, Mobilising knowledge to improve UK health care: learning from other countries and other sectors – a multimethod mapping study, Health Services and Delivery Research 3, 27, 1–190
Eyre, L, Farrelly, M, Marshall, M, 2017, What can a participatory approach to evaluation contribute to the field of integrated care? BMJ Quality and Safety 26, 7, 588–94. doi:
Greenhalgh, T, 2017, How to implement evidence-based healthcare, Oxford: Wiley
Guthrie B, Gillies J, Calderwood C, Smith G, Mercer S, 2017, Developing middle-ground research to support primary care transformation, British Journal of General Practice, 67, 664, 498–499. doi:
Heaton, J, Day, J, Britten, N, 2016, Collaborative research and the co-production of knowledge for practice: an illustrative case study, Implementation Science 11, 1, 20. doi:
Holmes, BJ, Best, A, Davies, H, Hunter, D, Kelly, MP, Marshall, M, Rycroft-Malone, J, 2017, Mobilising knowledge in complex health systems: a call to action, Evidence & Policy 13, 3, 539–60
Kreindler, SA, 2016, What if implementation is not the problem? Exploring the missing links between knowledge and action, International Journal of Health Planning and Management 31, 2, 208–26. doi:
Lalani, M, Hall, K, Skrypak, M, Laing, C, Welch, J, Toohey, P, Seaholme, S, Weijburg, T, Eyre, L, Marshall, M, 2018, Building motivation to participate in a quality improvement collaborative in NHS hospital trusts in Southeast England: a qualitative participatory evaluation, BMJ Open 8, 4, e020930. doi:
Lloyd, HM, Pearson, M, Sheaff, R, Asthana, S, Wheat, H, Sugavanam, TP, Britten, N, Valderas, J, Bainbridge, M, Witts, L, Westlake, D, Horrell, J, Byng, R, 2017, Collaborative action for person-centred coordinated care (P3C): an approach to support the development of a comprehensive system-wide solution to fragmented care, Health Research Policy and Systems 15, 1, 98. doi:
Lockett, A, El Enany, N, Currie, G, Oborn, E, Barrett, M, Racko, G, Bishop, S, Waring, J, 2014, Health services and delivery research, A formative evaluation of collaboration for leadership in applied health research and care (CLAHRC): institutional entrepreneurship for service innovation, Southampton, NIHR Journals Library
Marshall, M, 2014, Researchers-in-residence: a solution to the challenge of evidence-informed improvement? Primary Health Care Research and Development 15, 4, 337–8. doi:
Marshall, M, Eyre, L, Lalani, M, Khan, S, Mann, S, de Silva, D, Shapiro, J, 2016, Increasing the impact of health services research on service improvement: the researcher-in-residence model, Journal of the Royal Society of Medecine 106, 6, 220–25. doi:
Marshall, M, Mear, L, Ward, V, O’Brien, B, Davies, H, Warring, J, Fulop, N, 2018, Optimising the impact of health services research on the organisation and delivery of health services: a study of embedded models of knowledge co-production in the NHS (embedded), www.journalslibrary.nihr.ac.uk/programmes/hsdr/165221/#/
Marshall, M, Pagel, C, French, C, Utley, M, Allwood, D, Fulop, N, Pope, C, Banks, V, Goldmann, A, 2014, Moving improvement research closer to practice: the Researcher-in-Residence model, BMJ Quality and Safety 23, 10, 801–5. doi:
National Audit Office, 2017, Health and social care integration: report by the Comptroller and Auditor General, HC 1011 Session 2016–17
Palmer, VJ, Weavell, W, Callander, R., Piper, D, Richard, L, Maher, L, Boyd, H, Herrman, H, Furler, J, Gunn, J, Iedema, R, Robert, G, 2018, The Participatory Zeitgeist: an explanatory theoretical model of change in an era of coproduction and codesign in healthcare improvement, Medical Humanities
Reed, RL, McIntyre, E, Jackson-Bowers, E, Kalucy, L, 2017, Pathways to research impact in primary healthcare: what do Australian primary healthcare researchers believe works best to facilitate the use of their research findings? Health Research Policy and Systems 15, 1, 17. doi:
Thistlethwaite, P, 2011, Integrating health and social care in Torbay: improving care for Mrs Smith, www.kingsfund.org.uk/sites/default/files/integrating-health-social-care-torbay-case-study-kings-fund-march-2011.pdf
Van de Ven, AH, 2007, Engaged scholarship: a guide for organizational and social research, New York: Oxford University Press
Vindrola Padros, C, Eyre, L, Baxter, H, Cramer, H, George, B, Wye, L, Fulop, N, Utley, M, Phillips, N, Brindle, P, Marshall, M, 2018, Addressing the challenges of knowledge co-production in quality improvement: learning from the implementation of the researcher-in-residence model, BMJ Quality and Safety
Vindrola Padros, C, Pape, T, Utley, M, Fulop, NJ, 2017, The role of embedded research in quality improvement: a narrative review, BMJ Quality and Safety 26, 1, 70–80. doi:
Ward, V, House, A, Hamer, S, 2009, Knowledge brokering: the missing link in the evidence to action chain? Evidence & Policy 5, 3, 267–79
Wilding, H, 2010, Integrating care: from horizontal to vertical integration, Journal of Integrated Care 18, 3, 15–20. doi:
Wilson, PM, Farley, K, Bickerdike, L, Booth, A, Chambers, D, Lambert, M, Thompson, C, Turner, R, Watt, IS, 2017, Health services and delivery research, Effects of a demand-led evidence briefing service on the uptake and use of research evidence by commissioners of health services: a controlled before-and-after study, Southampton, NIHR Journals Library
