Abstract
Background:
Despite the known need for empirical research-to-policy studies, little is known about the factors and conditions needed to support meaningful evidence use or how to intervene to promote quality evidence use.
Aims and objectives:
To study research-policy processes empirically and descriptively, we conducted an ethnography that focused on the impact of the Research-to-Policy Collaboration (RPC) on legislator and researcher evidence use or policy engagement, including whether and how researchers and policymakers created and sustained meaningful relationships.
Methods:
The ethnography included participant observation as well as pre- and post- semi-structured interviews from policymakers (n=17), researchers (n=23), and RPC staff (n=5). The team attended relevant events as well as observed the formal and informal ways research is used in policymaking.
Findings:
In the paper, we describe how 1) legislative priorities were identified; 2) networks were established and maintained; 3) trainings evolved over time; 4) relationships between RPC staff, congressional staff, and researchers were facilitated; and 5) RPC followed up with policymakers and researchers.
Discussion and conclusions:
We 1) describe the experiences of participants and whether involvement in the intervention changed attitudes or behaviours about evidence use in policy; 2) describe the RPC process in practice, and how it was implemented and evolved over time; and 3) better understand the conditions supporting evidence use in policymaking. We conclude with the value of the RPC as a resource to fill a niche within the evidence and policy space, as well as suggestions for future research-to-policy programmes and practices.
Key messages
Describe whether a research-policy intervention changed attitudes and behaviours of policymakers and researchers.
Demonstrate how research-policy initiatives are operationalised.
Illustrate the conditions to support evidence use in policymaking through an empirical experiment.
Discuss lessons for future research-policy engagement.
Introduction
Using science to improve or create health policies is a multilevel process, which requires a good understanding of relationship building, behaviours, problems, actors, and results (Davies et al, 2012; Parkhurst, 2017). There is presumed agreement that improving evidence in policy is laudable, yet the processes by which this happens are rarely studied empirically. In reality, we know little about whether existing initiatives succeed in effective evidence uptake, or indeed what ‘success’ looks like (Karcher et al, 2021; Cairney et al, 2020). For example, a recent study found that over 400 organisations are engaging in research-to-policy work, yet only a few of these initiatives have been evaluated, with even fewer evaluations in the public domain (Hopkins et al, 2021). The problems are exacerbated by siloed projects and a lack of theoretical application (Hopkins et al, 2021). Therefore, there is a critical need to better understand the complexities and trade-offs in policymaking to implement evidence-informed policies to improve health equity.
Despite the known need for empirical research-to-policy studies, little is known about the factors and conditions needed to support meaningful evidence use or how to intervene to promote quality evidence use (Tseng, 2012). One factor which is consistently identified as a key facilitator of evidence use is the creation and maintenance of relationships between evidence producers and users (Ward et al, 2009; Bednarek et al, 2016). The promotion of joint working, collaborative research relationships and space for developing shared agendas is said to enable greater uptake of research evidence (Ward, 2009; Kothari and Wathen, 2017). These relationships are meant to deliver improved evidence use through improved understanding of the context within which decision makers operate, and enable better integration with other forms of knowledge (Kothari and Wathen, 2013; Evans and Scarborough, 2014). However, relationships of all kinds from consultative (for example, policymakers often turn to experts when addressing issues that are part of a political agenda) through the longer-term, are all said to present opportunities for improving the flow of knowledge between researchers and decision makers. In theory, these are opportunities for researchers to inform what policy solutions are considered and identify solutions with the greatest evidence of effectiveness (Kingdon, 2011; Cairney and Jones, 2016; Cairney et al, 2016). Another study found that research dissemination to policymakers (that is, improving the supply of evidence through briefs and reports) is only one part of the problem (Oliver et al, 2014; Cairney et al, 2016). The other barriers to increasing the use of evidence in policymaking is relationship building and systems change.
Although the literature clearly points towards relational work as central to evidence use, it is less clear about what this means in practice. Firstly, there are many kinds of relationships possible between professional colleagues. There is a difference between interacting with decision makers through formal, regular-though-rare opportunities like advisory committees, project-based interactions over weeks or months, and what some term ‘professional friendships’ – long-term, authentic and trusted contacts with whom joint working is genuinely possible (Day, 1994). Secondly, these types of relationships provide opportunities for interactions of different kinds; short and fleeting, or more sustained; instrumental or consultative through to discursive and exploratory.
Reflecting the growing interest in this area, an increasing number of interventions which attempt to provide opportunities for relationships between individuals or institutions (or both) are being implemented, for example, providing long-term funding for collaborative research teams (Kislov et al, 2018), and fellowship schemes which allow movement of academics (for example, the American Association for the Advancement of Science) or policymakers (for example, the Duke Foundation, or Cambridge University’s Centre for Science and Policy programme) between their home institutions. Existing evaluations tend to focus on relatively in-depth cases (Braga and Davis, 2014; Ward et al, 2021), and few are able to undertake good evaluations, whether internal or public, of knowledge-mobilisation or relational initiatives (Powell et al, 2017). More evidence is needed about the mechanisms which are said to generate relationships, and about the outcomes to which these relationships are intended to lead (Rickinson and Edwards, 2021). One such evaluation is the one we conducted on the Research-to-Policy Collaboration (RPC).
The Research-to-Policy Collaboration
The RPC intervention was initially developed by the National Prevention Science Coalition and taken on by Max Crowley and Taylor Scott as part of a randomised controlled trial (RCT) to test how and when relationships improve evidence use by policymakers. The RPC is an ongoing initiative that we evaluated as part of the RCT, focusing on its evolution. The RPC intervention’s primary aims were to increase the use of evidence in policymaking as well as researchers’ engagement in the policymaking process. Secondary aims were to increase the number of productive interactions and relationships between researchers and legislators. The underlying goal is to intentionally and strategically create opportunities for policymakers and researchers to build relationships around a shared area of interest (for example, child maltreatment); with the assumption that this shared interest will result in increased use of evidence in legislation (Crowley and Scott, 2017; Crowley et al, 2021). The RPC intervention involved identifying legislative priorities, developing networks of researchers to engage in identified legislative priorities, training researchers for policy engagement, and facilitating enduring collaborations between researchers and policymakers. Full details are available from Crowley et al (Crowley et al, 2021), but in brief the RPC process comprised:
Network identification: legislative offices (n=40 control; n=40 intervention) were chosen based on the intentionally broad topic area of ‘child maltreatment’. RPC staff and interns identified relevant researchers (n=30 control; n=30 intervention) by contacting researchers through child maltreatment listservs (for example, conference or coalition email groups that scholars previously signed up for); academic networking (for example, reaching out to known scholars); and ‘cold-calling’ scholars identified through internet searches. Once a legislative office was identified, an RPC Policy Associate, intern, or fellow met in-person with the legislative staffer who oversaw the child maltreatment portfolio, to gain a deeper insight into the office’s policy interests and future work. A second meeting was then scheduled to recruit the offices into the policy network and administer a survey with the staffers about their use of research evidence. As the intervention matured, the needs assessment and recruitment meetings were combined.
Training: researchers in the intervention group were invited to attend six web-based trainings focused on: 1) an introduction to RPC; 2) the legislative process; 3) relationship building; 4) role playing; 5) avoiding lobbying; and 6) preparing for the legislative meeting. While trainings were initially mandatory to participate in RPC Hill Day meetings, this requirement was later rescinded due to researcher availability and scheduling conflicts.
Facilitating collaborative relationships: policy networks were established at in-person Hill Day meetings, where relevant researchers and legislative staffers were introduced by RPC staff. Policymakers in the control group received a light-touch intervention, such as a follow-up resource by email (for example, policy brief) and were not invited to Hill Day meetings. Scheduling Hill Day meetings between researchers and policymakers was one of the more tedious jobs of the RPC intervention team, which required flexibility for researchers and RPC staff. Meetings were scheduled around policymaker schedules and researchers were matched based on availability and research area expertise.
Follow-up: the RPC intervention is based on a feedback loop model, which begins with RPC reaching out to policymakers and researchers, connecting policymakers and researchers with similar portfolios or scholarly work, respectively, and then continuing to create strategic connections between policymakers and researchers. The follow-up meetings were typically led by an RPC Policy Associate with researchers joining by virtual meeting (for example, Zoom).
Unusually for this type of activity, the RPC was evaluated via a RCT mixed-methods study to assess the impact of the RPC on evidence uptake in the 116th US Congress. The RCT reported mainly quantitative outcomes including citations of social science research in legislative documents (Crowley et al, 2021; Long et al, 2021). Alongside this, and to study research-policy processes empirically and descriptively, we conducted an ethnography, observing and interviewing RPC staff, congressional staffers, and researchers. The ethnographic component was particularly focused on the impact of the RPC on legislators and researchers’ uptake in evidence use or policy engagement, respectively, including whether and how researchers and policymakers created and sustained meaningful relationships. Specifically, we assessed how the RCT was implemented and whether participation in the RCT changed perspectives and behaviours related to evidence use in policy.
The ethnographic evaluation is a qualitative evaluation of the RPC, and the authors were not involved in the conceptualisation or implementation of the RPC intervention, only the evaluation. Our analysis focuses on the implementation of the RPC as an intervention, therefore this paper aims to 1) describe the experiences of participants and understand whether involvement in the RPC changed attitudes or behaviours about evidence use in policy; 2) describe the RPC process in practice, and how it was implemented and evolved over time; and 3) better understand the conditions supporting evidence use in policymaking. In particular, we sought to understand the role of the RPC in establishing relationships between researchers and policymakers, characterising these relationships, and understanding their importance in evidence use.
Methods
Data collection
We conducted an ethnographic study with legislative staffers in the US Congress, RPC staff, and researchers who were engaging in the research-to-policy space. Ethnography was the ideal approach to studying evidence use, given the nuanced nature of relationships and public policy. In addition to researchers being able to ask in-depth and follow-up questions, they can observe the experiences of participants over an extended period. The primary aim of the ethnography was to examine the effectiveness of the RPC model among legislative staff and researchers, as well as assess legislators’ use of research and researchers’ skill and engagement in policy. Data were collected through participant observation of events and email exchanges as well as pre- and post- semi-structured interviews from policymakers (n=17; 23 interviews conducted), researchers (n=23; 35 interviews conducted), and RPC staff (n=5; 14 interviews conducted). The first author immersed herself in the Washington, DC political scene, including moving to the capital area and attending relevant events, as well as observing Hill Day meetings (n=24) between staffers and researchers, RPC staff meetings (n=30) as they discussed project implementation, policy trainings (n=5) by RPC staff for researchers, Hill Day prep meetings (n=5) for researchers, and debriefing sessions (n=5) with researchers. In addition to formal meetings, she conducted participatory observation with researchers, policymakers, and RPC staff to gain a deeper understanding of the RPC in practice, and the formal and informal ways research is used in policymaking. Examples included participating in post-Hill Day dinners with researchers, congressional dinners with policymakers, and congressional briefings.
The interviews assessed motivations for utilising the RPC as well as participants’ experiences being part of the RPC network. In addition, we asked participants about the barriers and facilitators for using evidence in policymaking, perceptions of evidence and how this influences use, and types of evidence use supported by policymakers. This study was approved by Pennsylvania State University and University of Texas Medical Branch Institutional Review Boards. Protocols are publicly available and can be found at Transforming Evidence for Policy and Practice.
Interviews
Semi-structured interviews are a versatile method of qualitative data collection because they incorporate enough structure to ensure key research questions are addressed, while still including open-ended dialogue. This method provides the opportunity for participants to add new and unexpected context to the study, and for researchers to think critically about responses and probe accordingly (Galletta, 2013). Interviewees provide personal narratives that explain their experiences (for example, how policymakers use research). Knowledge is considered co-constructed between interviewee and interviewer; therefore, reflexive praxis was used to account for the researchers’ assumptions, attitudes, and ideas (Copland and Creese, 2015). Interviews with policymakers enable a holistic understanding of the experiences of participants and what individual (for example, assumptions, attitudes) and contextual (for example, environment, setting) factors influence the experiences of policymakers. Participants were asked if the interview could be confidentially recorded so that de-identified, complete, and verbatim transcriptions could comprise study data. All participants were reassured that their responses were confidential such that managers or bosses would not have access to identifiable responses or comments about participants’ experiences. If a participant did not want to be recorded (n=1), we took detailed notes corresponding with stenographic best practices for field researchers (Lee, 2004; Trochim et al, 2008). Interviews were conducted face-to-face, on the phone, or over video conferencing.
Observations
Participant observation is a research technique that involves understanding the experiences of participants by observing everyday procedures, such as interacting with co-workers or performing job duties. Observations included meetings with researchers, committee hearings, and informal and formal meetings between workers. Following ethnographic best practices, ‘open observations’ of key meetings and training events were conducted to enrich the interviews with more information about how participants interacted. The aim of the observations was to build trusting relationships with the participants to maximise the utility of the interviews, and to provide follow-up material with interviewees. The field notes were not coded alongside the transcripts; rather, they allowed the ethnographers to develop a more holistic reflective understanding of the policymaking process. The protocol for ethnographer participation and field notes is derived from a distillation of recommendations and identified best practices (Mulhall, 2003; Copland and Creese, 2015). We documented brief, chronological notes regarding what we saw, heard, and sensed regarding interactions, experiences, and our own internal reflections in order to obtain a deep intuitive sense of their experiences. Reflexive notes allowed us to capture potential sources of bias from prior experiences. At the end of each meeting day, we recorded reflections on broad patterns across multiple meetings (Mulhall, 2003).
Observations helped us develop a sense of the culture and tone of each group and to understand the micro-interactions underpinning evidence use implementation in policy. We assessed what was being discussed, how it was being discussed (including verbatim notes for semantic comparisons), and who was interacting with whom and how. The complexity of social events was captured comprehensively (Copland and Creese, 2015). Meetings were sampled purposively to ensure representation of interviewed participants and a range of diverse experiences and opinions. By sitting/standing behind or to one side of any work-related activities, we experienced the dialogue and how interactions unfolded. At meetings or events, we circulated unobtrusively with the intent to generate fruitful conversation and positive relationships with participants, to enable better interviews, and to develop a more holistic understanding of their experiences as participants. As no causal inferences are intended from this set of data, we were primarily concerned with inductively understanding the world of the policymakers, so the observations’ goal is to acclimatise us to their world and improve participants’ familiarity with us.
Data analysis
The qualitative data analyses took an inductive approach to identify key themes and discourses that emerged from narrative data, and we explored the different rationalities and rhetoric employed by participants. Open coding was done to identify themes regarding interactions, attitudes, and experiences, and codes inductively emerged. Of focus were participants’ rhetoric about values, attributions, interactions, and credibility. Coding of these was led entirely by participants’ language reflected in the data. This approach allowed us to explore social interactions and the use of evidence by policymakers; going beyond simple typologies to understand the relationship between actors, how language and communication differ between settings (for example, legislative committees), and how the presence of certain actors influences narratives (for example, high-ranking managers).
Interviews were used to understand if the RPC model enhanced the use of evidence in policy, the effectiveness of the model in increasing researchers’ skills and engagement, and how the model could be improved for future research-to-policy engagement. Observations also provided key insights regarding patterns of interactions and communication, or rhetoric used under certain circumstances. Interview transcripts were coded and analysed using Atlas.ti cloud software.
Findings
RPC was started in 2015 to facilitate the sharing of evidence between researchers and policymakers. The RPC trained researchers, conducted legislative needs assessments, and coordinated Hill Day meetings. RPC staff led the trainings and Hill Day meetings, with the objective of creating long-term relationships between staffers and researchers who shared similar areas of interest (for example, human trafficking). RPC conducted the trainings, Hill Day meetings, and follow-up during the 116th US Congress. Researchers were selected to be part of the network based on their expertise in child maltreatment, but political views were not collected. Policymakers were selected based on whether they had a child maltreatment policy portfolio, with 28 Democrats, 18 Republicans, and 1 Independent participant.
Overall, participants (that is, researchers and policymakers) enjoyed being part of the RPC network. Policymakers said it helped their access to evidence and they were generally appreciative of the Hill Day meetings. Researchers reported that it helped them learn how policymakers meet with constituents, advocacy groups, and scholars, as well as providing a different value to their research. While some policymakers noted that the RPC Hill Day meetings were confusing, they regarded RPC staff as “genuine” and “persistent”. Other policymakers said that the RPC intervention is “interesting” and that “there’s not enough of it in policy”. These sentiments were heard in many interviews with policymakers. Researchers described the intervention as “surprising” and, though they were generally in favour of the process, that improvements could be made in communication and relationship building. Researchers were “pleasantly surprised” by how engaged the legislative staffers were, how knowledgeable they were on the topics, and that they were willing to listen to and discuss expertise and science.
Trainings
Researchers in the intervention group were initially invited to attend web-based trainings conducted by the RPC Associate Director that focused on an introduction to RPC, the legislative process, relationship building, role playing, and avoiding lobbying as well as an in-person session about preparing for the legislative meeting. The meetings were envisioned as complementary to the Hill Day meetings to offer support for successful researcher-policymaker collaboration. The trainings were largely based on the experiences of RPC staff. While trainings were initially mandatory to participate in RPC Hill Day meetings, this requirement later evolved to be ‘highly encouraged’ due to researcher availability and scheduling conflicts.
The training sessions for the researchers also evolved throughout the intervention to become more light-touch as well as changed from in-depth live lectures to pre-recorded videos. Originally conceptualised as a six-part training, the RPC first condensed it to five online live lectures with one in-person prep session, and then later repackaged it to five pre-recorded training videos and one in-person prep session. The change in training, as described by RPC leaders, was a logistical choice when balancing the ‘tension for researchers on how much time they can give to participate in capacity- building trainings’. Additionally, availability of trainers and low participation rates for the live trainings influenced their decision to alter their modules. However, according to an RPC staff member, most of the researchers did not watch the pre-recorded trainings that were provided before their Hill Day meetings. Overall, researchers who attended the trainings reported feeling more confident translating complex ideas for a more public audience and having a better understanding of how RPC operates; however, researchers and RPC staff members reported that the in-person prep session before the Hill Day meetings provided the most benefit.
Hill Day meetings
Policymakers
Policymakers said they liked that RPC was “willing and able to bring rigorous data back” and that it was “a value that could be added” to their office. Positive reflections about the meetings included having access to an abundance of resources, reliable research and information, data-based scientific information to corroborate legislation, expanded networks with the research community, and academic resources that are difficult to access because of pay walls, such as peer-reviewed journal articles. Generally, policymakers appreciated the meetings and liked the idea of RPC. Reflections on how RPC could be improved upon included providing a list of topics in advance to ensure the correct staff person from the legislative office was present, especially in the event of last-minute scheduling changes; holding initial meetings over the phone or email; providing a clearer message about the purpose of the meeting and the expertise being offered, which would help distinguish between RPC and the Congressional Research Service; and inviting fewer researchers to the Hill Day meetings, which many described as “overwhelming” and “confusing”.
Researchers
I think that a very key component was having the staff physically there and having those relationships already initiated before we walked through the door and having the staff there to provide kind of the scaffolding around, you know, last time we spoke, you mentioned you were interested in this. And so these are folks who have that knowledge. I could tell that those sorts of reminders and that sort of scaffolding was really useful to the staffers for kind of orienting who we were, what we were doing there. And I found that I was… I didn’t really experience as much nervousness as I would have expected, because I knew that the RPC staff was going to walk in and introduce us and kind of set the stage, and if things started getting off track, I knew that they would kind of pull things back in and reroute us. And so that really made it that I could just focus on the content area and not worry so much about introducing myself or introducing the topic or kind of stating why I was there. That all felt like it was already established.
Researchers also appreciated the brief overviews and dossiers they received from RPC staff. One researcher said it gave her “clear info on who this person is” and helped her navigate “hot topics” or “do-not-touch topics”. Similar to policymakers, researchers felt that the number of people in the room was a concern. One person explained, “there were a lot of pieces and players. We had people on Skype. We had people in the room. And when you have that short period of time with a staffer you might not be able to communicate everything that you want”.
The mission and vision of the Hill Day meetings as opportunities to learn the policymaking process and link existing research to policy was met with gratitude; however, the logistics of the meetings were often described as unclear and lacking structure. Researchers stated there were no clear questions, direction, or topics; travel support was difficult and confusing, particularly for junior researchers with smaller allocation accounts or less experience asking for university financial support; expectations were high, such as requests to quickly make or change travel plans and turnaround policy products within a few days; and the lack of travel planning was especially difficult for female academics who are generally saddled with balancing home life and work life (Sutherland, 2018). Other logistical issues were often unavoidable, such as one researcher who could not make the meetings because the “metro caught on fire” or staffers who forgot about or cancelled meetings at the last minute. A participant agreed with the intent of RPC, but said “in the end, it didn’t get executed because mostly people didn’t want to talk about the things that I was prepared to talk about”. Another participant “felt like the staffers were confused about what we could help with… and I didn’t feel like they had any sense that we are focused on children and families”.
Policymaker preparedness and impact had wide-ranging responses from researchers, which was expected given that each office had staffers of different experience levels (for example, a chief policy associate, first-year policy fellow). Some researchers reported that the intent of the meetings and the reality of policymakers’ preparedness did not line up. One researcher was asked to go to the meetings only seven days in advance and felt that it was “kind of disruptive”, especially because the meetings “did not feel urgent or impactful”. However, other researchers described the experience as “exceptional”, particularly because of how enthusiastic the policymaker was or because policymakers had done research and were knowledgeable about the topics being discussed.
Follow-up
Originally, the RPC staff members were only meant to connect researchers and policymakers through the Hill Day event, with the idea that researchers and policymakers would then forge relationships based on shared interests (that is, knowledge-broker model). In practice however, the RPC team often became the knowledge brokers, instead of the researchers, as a result of RPC staff fortifying the relationships with the policymakers through in-person meetings, constant email communication, and being the go-between for sending policymakers needed evidence-informed policy products. One participant noted, “I feel like I have a relationship with the team members. I don’t know if I feel confident about sending stuff to the staffers right now”. In retrospect, this unintentional outcome may have hampered researchers’ ability to build relationships with policymakers, but it did increase the ability for RPC team members to schedule meetings (which researchers were part of) and provide evidence-informed research to offices. One policymaker noted that researchers from the Hill Day meeting had not followed up, but that “the RPC folks have”. Another policymaker stated: “An [RPC staff name redacted] and I have been chatting about some of the things that we talked about because we do think that there could be some promising ideas to develop from it”. One of the researchers said that relationship building with policymakers would need to be more proactive and persistent on her part to maintain the relationship, which was difficult since she was not the one scheduling initial meetings or meeting agendas.
In addition to managing follow-up meetings, RPC staff collected research materials that were sent to the staffers. Policymakers often reported being grateful for the follow-up materials, but suggested more up-to-date research, direction on how the research informs the policy issue, and specific policy proposals or research that is evaluating a particular policy. Preliminary findings also suggest that policymakers were willing to engage with researchers who may have different political values, but how or when that research was used needs further exploration. Researchers were also interested and willing to work with policymakers of different political parties than their personal political affiliation. Regarding follow-up materials, researchers provided this information to RPC for the policymakers, but often did not know what happened to the information and how the materials were used. One researcher commented, “I wonder if the work just sort of disappears into the ether of the Hill”. They said they expected to build relationships with staffers but felt that the research products that were sent to staffers became the focal point or that they were not in a position to continue facilitating meetings.
Discussion
The ethnographic evaluation aimed to identify how the RPC model impacted legislative staff and researchers’ use of evidence, and whether their programme created and sustained meaningful relationships between the two groups. We followed and questioned policymakers’ use of evidence and researchers’ engagement in the policy process through the three main points of contact with the RPC: 1) trainings; 2) Hill Day meetings; and 3) Hill Day meeting follow-up. Having examined these contact points, we now discuss the successes and limitations of the RPC model in supporting evidence use between policymakers and researchers.
The RPC process in practice
The RPC was developed in response to a well-documented facilitator of evidence use in policy, namely strong collaborative relationships. The RPC approach (that is, being policy responsive by selecting legislative priorities, providing training for researchers, facilitating collaborative relationships, and following up) was designed to maximise opportunities for these relationships to develop in a meaningful and sustainable way. The amount of work it took to develop and implement this approach should not be underestimated. The RPC Associate Director managed the day-to-day operations, including implementation approach, staff (that is, interns, postdocs), the RPC network, trainings, and overseeing the scheduling of Hill Day meetings. As RPC matured, the Associate Director relied on policy interns to assist with initial policymaker contact and Hill Day scheduling.
The RPC intervention’s primary aims were to increase the use of evidence in policymaking as well as researchers’ engagement in the policymaking process. To do this, the RPC aimed to increase the number of productive interactions and relationships between researchers and legislators. Overall, the RPC accomplished its objective of increasing researchers’ engagement in the policymaking process; however, the relationships established tended to be short-term and directly tied to single meetings. The interactions between researchers and policymakers within these relationships were somewhat instrumental and focused on the exchange of specific pieces of knowledge or information. By contrast, we did not find evidence of long-term relationships being established by RPC between researchers and policymakers. Rather, there was a move away from facilitating direct relationships and towards becoming an intermediary broker between researchers and staffers, by the RPC, alongside facilitating more consultative interactions between researchers and staffers. For instance, attitudes among the researchers showed a shift in their perception from ‘RPC helping build relationships’ to ‘helping create opportunities to share policy products’ (that is, policy briefs).
These shifts may be attributed to how RPC was implemented over time. Initially, the perceived role of the RPC staff was that of intermediary supports who facilitated meetings and trainings, matched researchers and legislative offices, brokered trust and created relationship-building opportunities between policymakers and researchers, and were a resource for non-biased information. Toward completion, the facilitation of meetings and matching experts with legislative offices created the foundation for a relationship between RPC staff and legislative staffers, as opposed to the intended target (that is, the researcher). That is, the RPC became the dissemination point of information instead of the connector (Boaz et al, 2021). Contributing to this shift were systemic factors, including time and resources of researchers, growing skills and awareness of RPC implementers, and intervention fatigue among policymakers.
In their goal to be relationship connectors, RPC staff successfully eased researcher discomfort and kept meetings on topic; however, the presence of RPC moderators paradoxically put researchers at ease but hampered relationship building as the moderators were seen as the knowledge brokers. Therefore, the goal of building long-term, structured relationships was limited as RPC became the go-betweens. This finding was evidenced in the development of skills among the staff and intern group in how they communicate, how they select researchers and evidence, and how they collate that evidence themselves for policymakers.
Perhaps because of the huge workload involved in identifying, training, and facilitating interactions for researchers, even in a relatively confined policy area, over time the RPC became more streamlined. For example, rather than sending out open-ended invites, the RPC recruited more directly. This may have sped the process up, but also may have reduced the diversity of participants. In addition, the RPC got better at identifying with whom to work (that is, those who were most likely to engage). However, RPC staff as brokers rather than conduits did not seem to effect the use of evidence by legislative staffers, but the sustainability of this approach should be further studied.
Another output to increase researcher skill and policy engagement, outside of the Hill Day meetings, was the policy trainings for researchers. Researchers who participated in RPC trainings reported being more prepared to engage with policymakers, relative to how they believed prior to training. That said, researchers who did not attend the trainings also felt prepared to engage with policymakers. This equivalence in preparedness may be attributed to self-selection bias of the trainings, which were initially required and then highly encouraged by RPC staff. Time restraints also became a major barrier for trainings, as well as for interactions between researchers and policymakers, particularly given that the two parties function within completely different timelines.
Overall, most researchers and staffers reported positive interactions with the RPC, finding the trainings useful and collaborative meetings helpful, respectively. Researchers tended to report more positive experiences than policymakers, perhaps speaking to the relative naivety of researchers about policy processes, and the relative fatigue of staffers in engaging with multiple academic brokers. Although the number of researchers who engaged with legislators increased, the overall attitudes about evidence use did not change among staffers. While attitudes regarding research did not change, the RPC certainly succeeded in providing opportunities for productive interactions, measured by researchers’ and policymakers’ attitudes regarding future engagement in the RPC network and researcher-policymaker meetings.
The implicit assumptions underpinning the model (see, for example, Crowley, 2018) were that 1) contact would lead to relationships; 2) research evidence should be the primary source of information for policymakers; 3) the provision of evidence would lead to greater uptake; 4) greater uptake of evidence means more research is cited in legislation; and 5) greater uptake of evidence leads to ‘better’ policy. These assumptions are very widely shared in the literature (Oliver et al, 2014) and the RPC explicitly aimed to use existing literature throughout its development. However, robust tests of these assumptions have rarely if ever been made, and as this evaluation shows, are not always borne out in practice. We found that merely providing opportunities for connection between researchers and policymakers was not sufficient to establish meaningful trusted relationships; and in fact, even provision of this level of facilitated opportunity was an extremely time-consuming task. It is interesting that the RPC itself has managed to establish relationships with staffers – becoming the go-between, rather than a creator of relationships. Of note, the relationship building in this study was limited to a few meetings, but a more robust model with multiple meetings or points of contact may have contributed to policymaker-researcher relationships.
There is inadequate data at this stage to establish whether RPC interactions had a long-term effect on relationship building between policymakers and researchers, but it did facilitate relationships between RPC staff and policymakers as well as a number of conversations between researchers and policymakers. As an intervention, RPC has evolved since our initial evaluation and further studies of its ongoing development will reveal whether and how long-term relationships are built through a knowledge-broker model. Regarding the one-off conversations, both staffers and researchers said they found them both useful and positive. It may be that these one-off conversations are adequate for evidence use in policy, or are in certain conditions. The literature reports that relationships are important, but there are many facets to and aspects of potential academic-policy relations which remain unparsed. It could be that having single points of contact, such as the RPC, is easier to manage for policymakers, rather than trying to maintain meaningful links with tens if not hundreds of individual researchers. It could also be true that short-term, instrumental interactions are appropriate and useful for some knowledge needs. Whether longer-term, more ‘authentic’ professional friendships are indeed more important for evidence use than one-off conversations remains unclear (Shearer et al, 2014; Oliver and Faul, 2018).
Conclusions
The RPC is a multifaceted intervention aiming to train researchers, provide opportunities for creating and sustaining meaningful relationships between researchers and legislative staffers, and enable evidence uptake into policy. In practice, we found that the initiative moved away from attempting to build relationships directly, and towards becoming a knowledge broker itself. Both researchers and policymakers stated clearly that RPC is a valued resource with the potential to fill a niche within the evidence and policy space. Educating policymakers continually about RPC, why they are different from the Congressional Research Service, and what they can tangibly offer to legislation is an important part of the model that can be improved upon. Moreover, RPC as a source for expedited evidence to inform policy strongly relies on continued engagement of RPC staff as moderators between researchers and policymakers.
In the future, the RPC intervention may benefit from recognising and embedding the shift of their framework away from structured relationship building between researchers and legislators to one that directly builds relationships between RPC staff and legislators. This shift will allow RPC to continue bridging research with policy, while also engaging in long-term relationships with legislative offices. Finally, training for researchers helps legitimise researchers’ roles and encourages them to engage in policymaking. The policy training for researchers should be further explored to improve future engagement of researchers in the policymaking process. Longitudinal follow-up is necessary to evaluate long-term impacts of the RPC intervention to examine how to ensure continuation of engagement after the termination of RPC’s involvement.
The RPC tended to build short-term, instrumental relationships rather than long-term collaborations between researchers and policymakers. More research should also study whether long-term relationships between staffers and researchers are more important than one-off conversations that are mediated through already-existing relationships, and into the range of relationships, which can be created and supported in the evidence-policy space.
Funding
Generous support for this work was provided by the William T. Grant Foundation.
Acknowledgements
The authors wish to thank Max Crowley and Taylor Scott for the opportunity to study the initial Research to Policy Collaboration intervention as well as the researchers and Congress members who participated in the study. We also thank editors and reviewers for very constructive points which have improved our understanding of the issues, as well as – we hope – the manuscript.
Research ethics
This work was reviewed and approved by Pennsylvania State University and the University of Texas Medical Branch’s Institutional Review Boards.
Contributor statement
The RPC was designed partly in response to a large systematic review by KO. Both SGW and KO planned and undertook data collection and analysis. SGW drafted the manuscript and both authors approved the final draft.
Conflict of interest
The authors declare that there is no conflict of interest.
References
Bednarek, A.T., Shouse, B., Hudson, C.G. and Goldburg, R. (2016) Science-policy intermediaries from a practitioner’s perspective: the Lenfest Ocean Program experience, Science and Public Policy, 43: 291–300. doi: 10.1093/scipol/scv008
Boaz, A., Borst, R., Kok, M. and O’Shea, A. (2021) How far does an emphasis on stakeholder engagement and co-production in research present a threat to academic identity and autonomy? A prospective study across five European countries, Research Evaluation, 30(3): 361–69. doi: 10.1093/reseval/rvab013
Braga, A. and Davis, E. (2014) Implementing science in police agencies: the embedded research model, Policing, 8: 294–306. doi: 10.1093/police/pau039
Cairney, P. and Jones, M.D. (2016) Kingdon’s multiple streams approach: what is the empirical impact of this universal theory? Policy Studies, 44(1): 37–58. doi: 10.1111/psj.12111
Cairney, P., Oliver, K. and Wellstead, A. (2016) To bridge the divide between evidence and policy: reduce ambiguity as much as uncertainty, Public Administration Review, 76(3): 399–402. doi: 10.1111/puar.12555
Cairney, P.A. (2020) Understanding Public Policy: Theories and Issues, London: Red Globe Press.
Copland, F. and Creese, A. (2015) Linguistic Ethnography: Collecting, Analysing and Presenting Data, London: Sage.
Crowley, D.M. and Scott, J.T. (2017) Bringing rigor to the use of evidence in policy making: translating early evidence, Public Administration Review, 77(5): 650–55. doi: 10.1111/puar.12830
Crowley, D.M. et al (2021) Lawmakers’ use of scientific evidence can be improved, Proceedings of the National Academy of Sciences, 118(9): e2012955118. doi: 10.1073/pnas.2012955118
Davies, H.T., Nutley, S. and Smith, P. (2012) What works? Evidence-based policy and practice in public services.
Day, C. (1994) Personal development planning: a different kind of competency, British Journal of In-Service Education, 20(3): 287–302. doi: 10.1080/0305763940200302
Evans, S. and Scarbrough, H. (2014) Supporting knowledge translation through collaborative translational research initiatives: ‘bridging’ versus ‘blurring’ boundary-spanning approaches in the UK CLAHRC initiative, Social Science and Medicine, 106: 119–27. doi: 10.1016/j.socscimed.2014.01.025
Galletta, A. (2013) Mastering the Semi-Structured Interview and Beyond: From Research Design to Analysis and Publication, Manhattan, NY: NYU Press.
Hopkins, A., Oliver, K., Boaz, A., Guillot-Wright, S. and Cairney, P. (2021) Are research-policy engagement activities informed by policy theory and evidence? Seven challenges to the UK impact agenda, Policy Design and Practice, 4(3): 341–56. doi: 10.1080/25741292.2021.1921373
Karcher, D.B., Cvitanovic, C., Colvin, R.M., Van Putten, I.E. and Reed, M.S. (2021) Is this what success looks like? Mismatches between the aims, claims, and evidence used to demonstrate impact from knowledge exchange processes at the interface of environmental science and policy, Environmental Science & Policy, 125: 202–18.
Kingdon, J.W. (2011) Agendas, alternatives, and public policies, 1976, 1977, 1978, 1979 [United States], Interuniversity Consortium for Political and Social Research, doi: 10.3886/ICPSR28024.v1.
Kislov, R., Wilson, P.M., Knowles, S. and Boaden, R. (2018) Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations, Implementation Science, 13: 111. doi: 10.1186/s13012-018-0805-y
Kothari, A. and Wathen, C.N. (2013) A critical second look at integrated knowledge translation, Health Policy, 109(2): 187–91. doi: 10.1016/j.healthpol.2012.11.004
Kothari, A. and Wathen, C.N. (2017) Integrated knowledge translation: digging deeper, moving forward, Epidemiology and Community Health, 71(6): 619–23. doi: 10.1136/jech-2016-208490
Lee, R.M. (2004) Recording technologies and the interview in sociology, 1920–2000, Sociology, 38(5): 869–89. doi: 10.1177/0038038504047177
Long, E.C., Smith, R.L., Scott, J.T., Gay, B., Giray, C., Storace, R., Guillot-Wright, S. and Crowley, D.M. (2021) A new measure to understand the role of science in US Congress: lessons learned from the Legislative Use of Research Survey (LURS), Evidence & Policy, 17(4): 689–707.
Mulhall, A. (2003) In the field: notes on observation in qualitative research, Journal of Advanced Nursing, 41(3): 306–13. doi: 10.1046/j.1365-2648.2003.02514.x
Oliver, K. and Faul, M. V. (2018) Networks and network analysis in evidence, policy and practice, Evidence & Policy, 14(3): 369–79. DOI: 10.1332/174426418X15314037224597
Oliver, K., and Cairney, P. (2019) The dos and don’ts of influencing policy: a systematic review of advice to academics, Palgrave Communications, 5(1): 1–11. doi: 10.1057/s41599-018-0199-0
Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14: 2. doi: 10.1186/1472-6963-14-2
Oliver, K., Hopkins, A., Boaz, A., Guillot-Wright, S. and Cairney, P. (2021) What works to promote research-policy engagement?, Evidence & Policy, doi: 10.1332/174426421X16420918447616.
Parkhurst, J. (2017) The Politics of Evidence: From Evidence-Based Policy to the Good Governance of Evidence, Abingdon: Routledge.
Powell, A., Davies, H. and Nutley, S. (2017) Facing the challenges of research‐informed knowledge mobilization: ‘Practising what we preach?’ Public Administration, 96(1): 36–52. doi: 10.1111/padm.12365
Rickinson, M. and Edwards, A. (2021) The relational features of evidence use, Cambridge Journal of Education, 51(4): 508–26. doi: 10.1080/0305764X.2020.1865877
Shearer, J. C., Dion, M. and Lavis, J. N. (2014) Exchanging and using research evidence in health policy networks: a statistical network analysis, Implementation Science, 9(1): 126. doi: 10.1186/s13012-014-0126-8.
Sutherland, K.A. (2018) Work-life balance: exploring the myths and realities of family, home, work, and life pressures for early career academics, in K.A. Sutherland, Early Career Academics in New Zealand: Challenges and Prospects in Comparative Perspective, Cham: Springer, pp 117–36.
Trochim, W.M., Marcus, S.E., Mâsse, L.C., Moser, R.P. and Weld, P.C. (2008) The evaluation of large research initiatives: a participatory integrative mixed-methods approach, American Journal of Evaluation, 29(1): 8–28. doi: 10.1177/1098214007309280
Tseng, V. (2012) The uses of research on policy and practice, Social Policy Report, 26(2). doi: 10.1002/j.2379-3988.2012.tb00071.x
Ward, V., House, A. and Hamer, S. (2009) Knowledge brokering: the missing link in the evidence to action chain? Evidence & Policy, 5(3): 267–79.
Ward, V., Tooman, T., Reid, B., Davies, H. and Marshall, M. (2021) Embedding researchers into organisations: a study of the features of embedded research initiatives, Evidence & Policy, 17(4): 593–614.
