Cutting through the noise during crisis by enhancing the relevance of research to policymakers

Authors:
Taylor ScottPennsylvania State University, USA

Search for other papers by Taylor Scott in
Current site
Google Scholar
Close
,
Jessica PugelPennsylvania State University, USA

Search for other papers by Jessica Pugel in
Current site
Google Scholar
Close
,
Mary FernandesGeorgia State University, USA

Search for other papers by Mary Fernandes in
Current site
Google Scholar
Close
,
Katherine CruzJohns Hopkins Bloomberg School of Public Health, USA

Search for other papers by Katherine Cruz in
Current site
Google Scholar
Close
,
Elizabeth C. LongPennsylvania State University, USA

Search for other papers by Elizabeth C. Long in
Current site
Google Scholar
Close
,
Cagla GirayWeber Shandwick, USA

Search for other papers by Cagla Giray in
Current site
Google Scholar
Close
,
Rachel StoracePennsylvania State University, USA

Search for other papers by Rachel Storace in
Current site
Google Scholar
Close
, and
D. Max CrowleyPennsylvania State University, USA

Search for other papers by D. Max Crowley in
Current site
Google Scholar
Close
Full Access
Get eTOC alerts
Rights and permissions Cite this article

Background

It is widely recognised that policymakers use research deemed relevant, yet little is understood about ways to enhance perceived relevance of research evidence. Observing policymakers’ access of research online provides a pragmatic way to investigate predictors of relevance.

Aims and objectives

This study investigates a range of relevance indicators including committee assignments, public statements, issue prevalence, or the policymaker’s name or district.

Methods

In a series of four rapid-cycle randomised control trials (RCTs), the present work systematically explores science communication strategies by studying indicators of perceived relevance. State legislators, state staffers, and federal staffers were emailed fact sheets on issues of COVID (Trial 1, N = 3403), exploitation (Trial 2, N = 6846), police violence (Trial 3, N = 3488), and domestic violence (Trial 4, N = 3888).

Findings

Across these trials, personalising the subject line to the legislator’s name or district and targeting recipients based on committee assignment consistently improved engagement. Mentions of subject matter in public statements was inconsistently associated, and state-level prevalence of the issue was largely not associated with email engagement behaviour.

Discussion and conclusions

Together, these results indicate a benefit of targeting legislators based on committee assignments and of personalising the subject line with legislator information. This work further operationalises practical indicators of personal relevance and demonstrates a novel method of how to test science communication strategies among policymakers. Building enduring capacity for testing science communication will improve tactics to cut through the noise during times of political crisis.

Abstract

Background

It is widely recognised that policymakers use research deemed relevant, yet little is understood about ways to enhance perceived relevance of research evidence. Observing policymakers’ access of research online provides a pragmatic way to investigate predictors of relevance.

Aims and objectives

This study investigates a range of relevance indicators including committee assignments, public statements, issue prevalence, or the policymaker’s name or district.

Methods

In a series of four rapid-cycle randomised control trials (RCTs), the present work systematically explores science communication strategies by studying indicators of perceived relevance. State legislators, state staffers, and federal staffers were emailed fact sheets on issues of COVID (Trial 1, N = 3403), exploitation (Trial 2, N = 6846), police violence (Trial 3, N = 3488), and domestic violence (Trial 4, N = 3888).

Findings

Across these trials, personalising the subject line to the legislator’s name or district and targeting recipients based on committee assignment consistently improved engagement. Mentions of subject matter in public statements was inconsistently associated, and state-level prevalence of the issue was largely not associated with email engagement behaviour.

Discussion and conclusions

Together, these results indicate a benefit of targeting legislators based on committee assignments and of personalising the subject line with legislator information. This work further operationalises practical indicators of personal relevance and demonstrates a novel method of how to test science communication strategies among policymakers. Building enduring capacity for testing science communication will improve tactics to cut through the noise during times of political crisis.

Key messages

  • Research translation efforts should be relevant; however, practical guidance is needed to operationalise the concept of relevance.

  • A series of four experimental trial tests found that personalisation consistently cued relevance.

  • Post-hoc analyses suggest that related committee assignment may also increase perceived relevance. In contrast, how vocal legislators were about the issue was an inconsistent marker of relevance.

  • Targeting and personalising research translation efforts are two practical applications of these study findings.

Given the magnitude of the COVID-19 pandemic, it has arguably never been more important for scientists to effectively communicate to policymakers their research about virus transmission, prevention, and the social and economic implications. However, studies of the use of research evidence (URE) have rarely prospectively examined communication strategies for improving the reach of scientific evidence. Scholars studying URE recognise the importance of policymakers’ access to timely and relevant research evidence when policy windows or opportunities open (Tseng, 2012; Mackie et al, 2015; Bogenschneider, 2020). However, little is known about what strategies may enhance the perceived relevance of research during the ‘right time’. Herein lies the value of strategic research communications that improve perceived relevance to increase policymakers’ access to timely research.

This study employs a rapid-cycle randomised controlled design to test a science communication strategy for increasing the likelihood that researchers’ email messages are opened. To our knowledge, the present set of studies (including Long et al, 2021) are the first to experimentally test science communication strategies among legislative audiences and to present a replicable and feasible methodology for investigating ways to communicate about research evidence to policymakers. Overall, little scholarly work examines the effects of email subject lines (Calfano, 2016), whereas several studies have investigated the impact of advocacy messages (Magee, 2020). We argue that policymakers open emails based on perceived relevance of the subject line. We defined and tested perceived relevance by tailoring email messages to mention the state or legislator’s name and by examining patterns of behaviour associated with legislators’ affiliations, public statements, and state-level epidemiological data.

Policymakers’ research access and use

Theory in this field underscores a common misconception of research use as a linear process from production to access (Contandriopoulos et al, 2010; Boaz and Davies, 2019). Unidirectional research dissemination efforts that ‘push’ information from researchers, rather than ‘pull’ scientific information based on practical needs may have key limitations (Contandriopoulos et al, 2010; Brown, 2012; Tseng, 2012; Macoubrie and Harrison, 2013). Although the field has come to recognise a social dimension of ‘knowledge adoption’ (Brown, 2012; Oliver et al, 2014; Boaz and Davies, 2019), a recent systematic review also suggests that the most frequently reported facilitator of policymakers’ URE involves dissemination and access of relevant research. Thus, there remains a need to improve dissemination strategies that correspond with the trusting interpersonal connections with researchers that is increasingly recognised as a best practice for research translation (Oliver et al, 2014; Boaz and Davies, 2019).

Studying science communication may improve interpersonal connections between researchers and policymakers by enhancing the perceived relevance of research to policymakers and adapting to their norms and priorities. The following study grows out of work done by the Research-to-Policy Collaboration (RPC), a replicable model for facilitating researcher-policymaker interactions (Crowley et al, 2018; Scott et al, 2019; Crowley et al, 2021a; 2021b). This model was augmented during the pandemic to support researchers’ virtual interactions with state and federal legislative offices via the dissemination of relevant fact sheets that were produced in response to policy issues of importance to legislative offices participating in researcher interactions. Subsequent sections describe the ways in which researchers can improve the way they communicate the relevance of their work and which legislators may find it most relevant.

Relevance of research

Immense demands for legislators’ attention contribute to information overload and heighten the need for researchers to cut through the noise (Brownson et al, 2006). By design, elected officials represent their voters or constituents and thus the local relevance of information guides their attention (Brownson et al, 2006). Information must also be timely, corresponding with rapidly evolving, narrow windows of opportunity for public policy change (Kingdon, 1984; Mackie et al, 2015).

Although national crises redirect policymakers’ attention on a routine basis, the COVID-19 pandemic has resulted in an unprecedented outpouring of public science communications, and an enormous demand for policymakers to quickly make decisions with the best available information. Furthermore, many social issues and inequities are compounded by the pandemic (for example, increases in violence; Crossman and Monk, 2020; Konrad, 2020). Especially in the US, the pandemic coincided with a reckoning for racial justice in response to protests of police killings, and thus there has been attention on specific sub-groups affected by multiple national crises (for example, Black, Indigenous, persons of colour). Hence, these issues were particularly timely at the time of this study since policymakers may prioritise temporally relevant science communications the most (Oliver et al, 2014).

In addition to timeliness and local relevance, legislators have their own ideology that may influence the likelihood of their accessing research evidence. A primary goal of science communication efforts should be to recognise the target audience’s needs and respond with corresponding information (Macoubrie and Harrison, 2013). Policymakers’ committee affiliations may be a useful heuristic for identifying interested legislators. More labour-intensive methods involve developing a deeper understanding of the policymakers’ interests, as conveyed either in public forums or private interactions with legislative personnel. A mid-way point between interaction and one-way dissemination might involve an enhanced dissemination that tailors science communication to policymakers’ interests.

Although a fair amount of work has been done to illustrate that relevance of research is important to policymakers, less work has been done to operationalise indicators of relevance. This work assesses perceived relevance according to whether an email ‘cuts through the noise’ and is opened despite the constant glut of messages received. Marketing researchers face a similar challenge and recognise that perceived relevance and a recognised sender improve the success of an email campaign (Georgieva, 2012; Marcelino, 2015). We manipulated subject lines to assess indicators of perceived relevance while holding constant the message sender. We investigate the conditions in which legislators access research through a dissemination to explore strategies for communicating relevance.

Enhancing perceived relevance of research by policymakers

We hypothesise several mechanisms by which policymakers may perceive research as relevant, including using the legislators’ name or state name, relevant committee affiliation, state-level impact of the mentioned policy issue, and the extent that legislators mention the policy issue in public statements (Table 1). Thus, these questions relate to both message construction and recipient identification.

Table 1:

Operationalisation of relevance

Relevance Theory/hypothesis Operationalisation
Personalised message Messages that contain the recipient’s name or state may indicate it was hand-selected for them, improving perceived relevance. Trial condition group based on subject line received (for example, personalising the state name, legislator name).
Targeted audience
Legislative power Legislators are well positioned to pass policies on issues that will be seen by a committee on which they serve. Information about a committee-related issue may be seen as more relevant. Serves on a committee related to the issue (yes/no).
Issue prevalence One goal of state legislatures is to solve problems affecting their constituents. Information about a given issue may seem more relevant to legislators in states that are more severely impacted by the issue. Quartiles of how much the issue impacts the state, based on related epidemiological data.
Demonstrated interest Individual legislators often seek to be a champion of a given issue, may produce more public statements about their championed issues, and may be more interested in information related to those issues. Information about an issue that legislators are vocal about may be seen as more relevant. Quartiles of how much the legislator posts public documents related to the issue.

Constructing a personalised message

We hypothesise that one heuristic policymakers may use to determine relevance of materials involves mention of the legislators’ name or state, which could indicate the information has been hand-selected for their purposes. Marketing professionals think personalising emails is useful for improving recipient interaction (Druckman and Green, 2013). Despite this, such tailoring for policy audience has yet to be rigorously tested and published in the scientific literature. Extant literature has found mixed results regarding the effect of personalising emails. For instance, one study found a ‘lacklustre’ effect of personalising subject lines (that is, including the person’s name in the subject line) of emails sent to individuals on a Planned Parenthood contact list (Calfano, 2016). Conversely, another study found that prospective members of a new professional organisation were more likely to become members with a personalised subject line (Druckman and Green, 2013). Both studies indicate a personalised subject line might affect the success of the email campaign; however, the salience of personalisation may depend on contextual factors such as subject matter or target audience. We hypothesise that reference to the recipient’s state or district will have a similar effect as mention of the legislators’ name since, by virtue of their respective roles, legislators represent respective geographies, and their staff represent their boss’s interests.

Identifying a legislative target audience

We also hypothesise that certain legislators will find the subject matter relevant regardless of message personalisation. Legislators are publicly assigned to different issue-focused committees, thus providing an easy tool to identify target legislators based on topic (Reid and Montilla, 2001; Goldstein et al, 2011). Another indicator of legislator interest is the local impact of a policy issue based on publicly-available, aggregate epidemiological data. Lastly, policymakers release thousands of public statements regarding their policy priorities. Such comments are increasingly made searchable in online data platforms that scrape text from web-based platforms (for example, Twitter, legislative archives).

Testing science communication strategies

In light of the need to demonstrate the value of scientific information to those who make decisions about practice, policy, and innovation, it is time to reconsider how research is shared via digital mediums with audiences outside institutional settings. Emails offer a highly feasible tool to study science communication and framing because minor differences in language (for example, manipulating a word in a subject line) can be studied and access and engagement are easily quantified through electronic tracking metrics. Legislator email attitudes were most recently examined in the early 2000s (Cooper, 2002; Richardson and Cooper, 2006), irrespective of research evidence use. Substantial resources are spent on understanding how to maximise recipient interaction with digital messages in the world of e-commerce (that is, sales within an online marketplace; Turban et al, 2017). In particular, efforts primarily centre on how to optimise the proportion of recipients who open emails (that is, open-rate) and click on a specific link embedded in an email (that is, click-rate). Multiple tools allow advertisers to test how changes in a message (for example, wording, colour, or font) impact open- and click-rates via randomised controlled trials of messaging campaigns – often referred to as A/B testing.

Randomised controlled trials are increasingly deployed outside of traditional product marketing. For instance, in the 2008 presidential campaign, these techniques were used in email-based fundraising efforts, in which basic email characteristics were randomised to determine what would lead to the highest campaign donation (Wattal et al, 2010). This strategy is now commonplace in election campaigns at federal, state, and local levels (Special Report, 2016). In contrast, the scientific community sends hundreds of thousands (if not millions) of messages to elected officials and their staff each year, in an attempt to translate research findings to policy (Jones-Jamtgaard and Lee, 2017). However, few researchers systematically consider the impact of their research dissemination efforts and even fewer actively test strategies for improving research translation. Thus, we aim to advance science communication strategies by studying perceived relevance indicators.

Methods

The present study used four randomised controlled trials (RCTs) to examine science communication strategies during 2020, a year marked by significant social crisis stemming from the COVID-19 pandemic and racial injustice. These issues informed the timeliness and relevance of the disseminated content (Table 1). Materials as part of this study were created in the RPC intervention (Crowley et al, 2021a; 2021b) by participating scholars who were responding to policymakers’ interests related to current political crises. Emails included novel research-based materials that had not been previously distributed. Replicating the test across policy issues enabled us to examine context dependencies. We use A/B testing in rapid-cycle randomised controlled trials (that is, iteratively investigating phenomena in a short time period such that the results of each trial inform the hypotheses of subsequent trials; Cody and Asher, 2014; Peek et al, 2014) to test the impact of tailored subject lines on open-rates. Subsequently, post-hoc analyses drew from data obtained in the RCTs to investigate correlates of legislative engagement with research messages.

Sample

An online Client Relationship Management (CRM) software, Quorum, developed specifically for legislative outreach, identified state legislators and staff, as well as federal legislative staff based on their committee assignments. Recipients of each trial were developed purposively based on message content, and then placed in groups using a random number generator in Excel, that was then sorted before dividing the sample into equal-sized groups (See CONSORT diagram in Figure 1).

Figure 1: A schematic with consort diagram of an online client relationship management software in four panels. Panel a, b, c, and d represent trial 1, 2 3, and 4 respectively. Trials include COVID-19, exploitation, police relations, and domestic violence.
Figure 1:

CONSORT Diagram, Panel A: Trial 1, Panel B: Trial 2, Panel C: Trial 3, Panel D: Trial 4

Citation: Evidence & Policy 2023; 10.1332/174426421X16535828173307

Subsequent to this randomised allocation, 3–9% of intended recipients did not receive the email because their email address was not available in the CRM or because they had left their position between the time of sample identification and intervention deployment. Recipients and senders were blind to study conditions and hypotheses. Trials varied in topic, including COVID-19 (Trial 1), exploitation (Trial 2), police relations (Trial 3), and domestic violence (Trial 4). Table 2 provides sample characteristics for each trial. The evolving hypotheses and rationale for testing each of these subject lines are documented alongside trial findings.

Table 2:

Sample characteristics

Characteristics Trial 1 Trial 2 Trial 3 Trial 4
Sample size 3403 6846 3488 3888
Dissemination date March 23, 2020 May 8, 2020 July 16, 2020 September 10, 2020
Sample characteristics State legislators and federal staff State legislators, their staff, and federal staff State legislators only State legislators and federal staff
Sampled by issue area Health, children and families Health, children, and families All Judiciary
Party
 Republican 1655 2993 1836 1718
 Democrat 1700 3750 1603 2132
 Independent 48 103 49 38
Contact type
 State legislator 2403 3177 3488 1656
 State staffer 0 2996 0 1526
 Federal staffer 891 633 0 697
 Missing role 109 40 0 9

Note Missing role was due to updated classifications after the date of distribution (for example, legislators who left office, staffers who moved offices or became legislators themselves).

Measures

Quorum was used to send the messages, monitor email open-rates, and provide aggregate data about legislative behaviour derived from publicly available sources. This study was determined exempt by the Pennsylvania State University’s Institutional Review Board contingent on the use of a disclosure statement about data collection and its use for study purposes.

Dependent variable: research access

Email opens were tracked for 14 days. Open-rates were measured employing industry standard CRM best practices that include an invisible one-pixel tracking image that renders in the recipient’s inbox when opened.

Experimental independent variable: personalization

Messages were tailored to include the legislator’s name or locale (that is, state or district) in the subject line (Table 3).

Table 3:

Rapid-cycle trial subject lines

Condition Trial 1 Trial 2 Trial 3 Trial 4
A: Control Mitigating the effects of COVID on the world’s families Ways that policy efforts can address exploitation Police-community relations fact sheet Policy can help victims disclose violence
B: Personalised Mitigating the effects of COVID on American families Ways that [legislator name] can address exploitation Police-community relations for [legislator name] [State name] can help victims disclose violence
C: Personalised Mitigating the effects of COVID on [state name] families Ways that [state OR district name] can address exploitation Police-community relations for [state name] Helping victims in [state name] disclose violence
D: Personalised Helping victims disclose violence in [state name]

Observational independent variables

State impact

To imitate practical conditions in which individuals identify states most impacted by a social condition, epidemiological data were obtained from publicly available sources (Supplemental Table 1). For each indicator, four ordinal groups were created based on quartiles of state issue relevance within the sample, including Not Relevant (minimum to lower quartile), Slightly Relevant (lower quartile to median), Relevant (median to upper quartile), and Very Relevant (upper quartile to maximum).

Public comments

In addition to providing email metrics for all emails sent through Quorum, the platform also synthesises publicly available data about legislators, including their public comments (for example, newsletters, press releases, dear colleague letters, floor or hearing statements, and social media). For each trial, five ordinal groups were created based on frequency distributions of search terms within the sample, guided by the Diffusion of Innovation theory groups (Rogers, 1995). These groups are the No Relevance (0 documents), Low Relevance (1 document), Medium Relevance (2–3 documents), Medium-High Relevance (4–5 documents), and High Relevance (6+ documents). Search terms are reported by trial in Supplemental Table 2. Comments were not coded further for other features, such as support for or opposition to the topic.

Committee assignments

Membership in issue-related committees are reported for each trial in Supplemental Table 3. Each legislator was assigned a dichotomous indicator of relevant committee membership.

Analyses

The non-tailored message conditions were compared to the tailored message conditions with two-tailed tests. Logistic regression was used to understand whether the email was opened at all; whereas negative binomial regression was used to analyse the total number of times the email was opened, which may have been particularly elevated when a message was forwarded. Negative binomial analyses were used because data did not meet the Poisson regression assumption for identical mean and variance (Gardner et al, 1995). All analyses used a two-tailed alpha of .05. Analysts used a de-identified copy of data in final analyses but were aware of hypotheses. Supplemental tables, data and syntax from this study are available on OSF at https://osf.io/xt832. Recipients opening more than 50 times (likely due to forwarding) were designated as outliers and excluded from analysis in the negative binomials but retained in logistic analyses.

Findings

Of all legislators and staff emailed, 20.3%–40.5% opened each email. Findings for each trial are presented in sequence, explaining how each trial was informed by the last, consistent with rapid-cycle testing. Descriptive statistics and results are presented in Table 4 and results pertaining to relevance indicators in Table 5.

Table 4:

RCT main effects analyses of personalised subject lines

Trial Open-rate OR (95% CI) IRR (95% CI)
Trial 1
Mitigating the effects of COVID-19 on the world’s families 25.3%
Mitigating the effects of COVID-19 on American families 23.3% 0.90 (0.74, 1.09) 1.01 (0.85, 1.20)
Mitigating the effects of COVID-19 on [recipient state name]families 28.3% 1.16 (0.97, 1.40) 1.21* (1.02, 1.44)
Trial 2
Ways that policy efforts can address exploitation 20.3%
Ways that [legislator name] can address exploitation 28.0% 1.51*** (1.32, 1.74) 1.37*** (1.20, 1.55)
Ways that [state/district name] can address exploitation 25.1% 1.31*** (1.14, 1.50) 1.22** (1.07, 1.39)
Trial 3
Police-community relations fact sheet 36.9%
Police-community relations for [legislator name] 40.5% 1.16 (0.98, 1.37) 1.09 (0.93, 1.27)
Police-community relations for [state name] 39.6% 1.12 (0.94, 1.32) 1.11 (0.95, 1.31)
Trial 4
Policy can help victims disclose violence 23.0%
[State name] can help victims disclose violence 25.1% 1.12 (0.91, 1.38) 1.20 (0.97, 1.50)
Helping victims in [state name] disclose violence 26.2% 1.19 (0.96, 1.46) 1.44*** (1.16, 1.78)
Helping victims disclose violence in [state name] 25.8% 1.16 (0.95, 1.43) 1.12 (0.90, 1.40)

Note: OR = Odds Ratio, IRR = Incident Rate Ratio

* p < .05, ** p < .01, *** p < .001.

Table 5:

Post-hoc analysis of how often emails were opened by relevance indicators

Subject matter Incident rate ratios (95% CI)
Personalisation Affiliations State-level prevalence Mentions in public statements
Trial 1: COVID-19 1.14** (1.03, 1.26) 1.17* (1.00, 1.35) 1.01 (0.94, 1.08) 0.92*** (0.89,0.96)
Trial 2: Exploitation 1.19*** (1.11, 1.28) 0.99 (0.88, 1.12) 1.06* (1.01, 1.11) 0.94 (0.88, 1.00)
Trial 3: Police violence 0.97 (0.88. 1.06) 1.28*** (1.11, 1.47) 0.98 (0.92, 1.04) 1.32*** (1.22, 1.43)
Trial 4: Victims of violence 1.19* (1.04, 1.36) 1.23** (1.06, 1.44) 1.00 (0.93, 1.08) 1.09* (1.01, 1.17)

Note * p < .05, ** p < .01, *** p < .001.

Trial 1

A newsletter related to COVID-19 was distributed and tested keywords for world, American, and the recipient’s state name. We hypothesised that American would perform better than world, and the state name would perform the best. The greatest open-rates were observed for messages that emphasise local (that is, state name) rather than national (that is, ‘America’) or global scale (that is, ‘world’; χ2(1) = 6.24, p = .044). Recipients of the email personalised with legislators’ state name opened the emails 15% more times than those who received American and 18% more than those who received world. World and American did not significantly differ (p = .91).

Trial 2

Since there was no significant difference between world and American, we sought to test an additional relevance indicator the legislators’ names by disseminating a newsletter related to exploitation and human trafficking. The greatest open-rates were observed for messages that were personalised using either the legislator name or the state/district name. This was true for both the odds of opening the email (χ2(1) = 31.02, p < .001) and the number of times emails were opened (χ2(1) = 19.75, p < .001). Recipients of emails personalised with the legislator’s name were 51% more likely to open and opened 37% more times than recipients of non-personalised emails. Recipients of emails personalised with the state name were 31% more likely to open and opened 22% more times than recipients of non-personalised emails.

Trial 3

We sought to replicate findings in another policy context by disseminating a fact sheet related to police-community relationships. Recipients of emails with a personalised subject line were no more or less likely to open (χ2(1) = 3.08, p = .08), nor open more times (χ2(1) = 1.89, p = .17), than recipients of non-personalised emails.

Trial 4

Trial 3 findings were inconsistent with prior trials and we sought to explain this inconsistency by testing the placement of the personalisation within the subject line. Specifically, Trial 3 subject lines were personalised at the end of the subject line. Trial 4 tested placement by disseminating a fact sheet related to domestic violence using the state name to personalise subject lines at the beginning, middle, end, or not at all. Results indicated a clinically significant, though not statistically significant, effect of personalising on the odds of opening the email (χ2(1) = 2.76, p = .10), such that the personalised messages were opened by 12%–19% more people than the non-personalised message. The message with the personalisation in the middle was opened 44% more times than the non-personalised line, which was statistically significant (p = .001). The front- and back-loaded personalised lines were opened a clinically-significant higher amount than the non-personalised one, too, though did not reach statistical significance (p = .09-.30, respectively).

Observational relevance analyses

Relevance indicators were modelled to inform the practice of outreach to policymakers based on publicly available information. Indicators were analysed together in a combined model to provide evidence on which indicators were most explanatory. Recipients were excluded from these analyses with observational data only if they were missing a data point relevant to that analysis (pairwise deletion). Correlations between relevance indicators across all trials were small (rs = -.17-.10; Supplemental Table 4), indicating orthogonality. Statistical patterns varied by subject matter for each trial (Table 5). In three out of four analyses, recipients who had relevant committee affiliations opened 17%–26% more times than those without a relevant affiliation. Only one analysis (Trial 2) indicated that state-level prevalence was associated with 6% more frequent email opens for those in states with a higher prevalence of human trafficking. In two out of four analyses, recipients in legislative offices in a higher quartile of public statement mentions of the issue opened the message more frequently, between 9%–31% more opens for every quartile. In contrast, those mentioning COVID-19 exhibited 8% fewer opens for every quartile of mentions of the pandemic. Logistic regressions revealed a similar pattern of findings except that recipients in states with a high COVID-19 prevalence rate were more likely to open the email (Trial 1; OR = 1.11, p = .01), though not necessarily more likely to open it multiple times or forward it according to the negative binomial.

Discussion

This work experimentally tests measurable and action-oriented indicators of policy relevance with a legislative audience, builds on prior work (Tseng, 2012; Oliver et al, 2014; Mackie et al, 2015; Bogenschneider, 2020), and further operationalises legislative relevance. Results suggest that personalised messages may increase perceived relevance and thus increase email opening, over and beyond other indicators of relevance. Using a legislator’s name and locale is consistent with Moray’s (1959) Cocktail Party Effect theory in which hearing one’s own name elicits attention. A policymaker’s locale is intertwined with their identity given their role to represent their constituents’ values (Goldschmidt, 2017). The effect of personalisation may depend on the placement of the manipulation, semantic differences in subject lines, or context and timeliness of the policy issue. Together, these findings suggest a need for routine evaluation of scientific communication practice and dissemination efforts.

Enduring capacity for testing science communication strategies is particularly important because the results of a single trial are difficult to interpret in isolation, but become more meaningful when interpreting results across policy issues and over time. Open-rate differences between trials illustrate how timeliness influences relevance (Kingdon, 1984). For instance, the relevance of police-community relationships was particularly high at the time of Trial 3 and resulted in the highest open-rates (36.9–40.5%) and the weakest effects of subject line in this study; this suggests that timeliness could downplay the effects of science communication strategies. In contrast, open-rates were low for Trials 2 and 4 (20.3–28.0%), indicating personalisation is not a panacea for improving access.

There may be contextual differences in which legislators perceive subject matter as relevant. Committee affiliation was a significant relevance indicator in three of four trials, whereas a lack of clear jurisdictional boundaries for human trafficking may have reduced the effect of affiliation in Trial 2. In contrast, state-level prevalence indicators were weak predictors for relevance except for the second trial related to exploitation, which was the only significant relevance indicator associated with increased human trafficking research access. Legislators were also 11% more likely to open the COVID-19 newsletter when there was greater prevalence of the virus in the state; however, they did not open the newsletter more times.

The conflicting results regarding legislators’ mentions of the policy issue could indicate either a demand for relevant information or an existing supply of information as policymakers attempt to persuade others. Vocal legislators may have already had information sources amid a surge in the ‘supply’ of information. The availability of neutral and credible information sources may also vary by policy issue, which could influence the ‘demand’ side of research access. Politicised issues like policing might heighten the demand for independent, scholarly research, but more work is needed to better understand political influences on the supply and demand for research evidence.

Limitations and future directions

Though this study makes a number of contributions to the field and illustrates the feasibility of routine evaluations of science communication strategies, there are minor analytic challenges to note for future replication. The sample between trials varied from both transition in and out of legislative roles and the selection of legislative personnel based on relevant committee affiliations. This ‘revolving door’ also makes it difficult to maintain data on recipients’ affiliations and roles at the time of the trial. Additionally, legislative public statements data vary based on both legislative staff capacity and state-level infrastructure for digitising public records (for example, floor statements and hearings); therefore, the salience of legislators’ statements as a relevance indicator is likely underestimated. Legislative statements were identified strictly by their mentioning of a topic, rather than by statements indicating support for or opposition to the topic. It is unclear how more in-depth coding would affect results.

The current study examined open-rates because the studied manipulation took place in the email subject line; therefore, click-rates were not evaluated. Future research should assess click-rates as an indicator of further access and engagement with research evidence. Open-rates are underestimated because emails are only counted as ‘opened’ if the recipient’s email server automatically downloads an embedded image, whereas firewalls may affect these data (Georgieva, 2012). Thus, a robust sample size is needed to compensate for this limitation as well as to detect small effects. Future trials should also consider evaluating the click-through rate (for example, opening the fact sheet), indicating that the message body was relevant and compelling enough for the recipient to take the token action of clicking for more information (Georgieva, 2012). Complexity or valence of the email responses should also be examined to understand how the information is being interpreted.

Future research dissemination efforts should incorporate a data-driven approach for optimising outreach. This work grows out of a set of studies that are the first to experimentally test emails for legislative science communication and to demonstrate the feasibility of evaluating messages through routine A/B testing. As the research community better understands science communication, the overall impact and value of the research enterprise may grow. Research institutions would benefit from tools that evaluate digital research access because effective dissemination strategies may vary by subject matter (for example, social science vs biomedical content), organisation (for example, perceived credibility; Stallings, 2009; Contandriopoulos et al, 2010), and target audience (for example, decision makers at local, state, or national levels; Majdzadeh et al, 2008). Moreover, there is more to learn about perceived relevance and credibility.

The present study focuses on access and reach of research-based information and is limited in scope with regard to actual use patterns. Future efforts should experimentally test the impact of science dissemination strategies on policymaker URE. The low cost of widespread dissemination may yield value despite small impact: even a 3% increase in opens can functionally result in hundreds more legislators viewing a research message. Future studies should evaluate the impact of dissemination, compare it to the impact of more interactive approaches, and evaluate the cost-effectiveness of multiple strategies.

Research institutions should adopt a cohesive and interactive research translation strategy. The current work builds on the RPC (Crowley et al, 2018; Scott et al, 2019), which facilitates rapid translation of relevant content by using interactive legislative needs assessments and connecting researchers with legislative officials. Then it disseminates written content to legislative offices, which encourages ongoing researcher-legislative interaction. A multi-prong research translation strategy helps scientists to respond to policymakers’ needs rather than ‘pushing’ their own interests (Tseng, 2012), and increases impact by multiple dissemination methods (Macoubrie and Harrison, 2013).

Conclusion

This study’s findings are consistent with the body of literature that indicates policymakers are more likely to open messages they deem as relevant. It further operationalises practical indicators of personal relevance based on their committee affiliations, state attributes, or personal interests. Due to variation in findings depending on subject matter and context, we urge the field to increase its capacity for testing science communication strategies whenever research is disseminated. This will enhance research translation practice by strengthening adaptation to different disciplines and chronological landscapes so that we can improve tactics that cut through the noise during times of political crisis. The rapid-cycle testing methodology can be used to further refine science communication practice as well as strategies that move scholarly work from the ivory tower toward social impact.

Funding

Generous support for this work was provided from the William T. Grant Foundation, the NICHD (P50HD089922), the National Science Foundation (GRANT 2030660), the Society for the Psychological Study of Social Issues, the Society for Research in Child Development, the Huck Institutes of the Life Sciences at Pennsylvania State University, and the Social Science Research Institute at Pennsylvania State University.

Acknowledgements

We are grateful for the participation and time investment of Jan Mooney and Abigail O. Akande. These researchers allowed us to send emails to thousands of people under their name eliciting hundreds of emails through which to sort, and expertly responded to requests from recipients. We would further like to acknowledge the precise and responsive work of the team who sent out these emails and fielded requests, and the professional and relevant contributions of the team that helped create the fact sheets. This study would not be possible without these capable teams working together seamlessly.

We would also like to extend a tremendous amount of gratitude to the researchers who synthesised studies and disseminated their work as part of this study. The content for the Trial 1 email was prepared by Cagla Giray, with contributions by Kara Ayers, John Dziak, Brittany Gay, Kevin Gee, Melanie Dyan Hetzel-Riggin, Sandra Lendyard, Lisa Merkel-Holguin, Soojin Oh Park, Galena Rhoades,Cathryn Richmond, Luke Russell, and Zena Shuber. The content for the Trial 2 email content was prepared by Toria Herd and Cagla Giray, with contributions by Chris Blank, Maya Boustani, Tommy Chou, Stacy Frazier, Mariam Garuba, Sarah Helseth, Kelli Hughes, Renata Konrad, and Katie Steck. The content for the Trial 3 email was created by Jan Mooney. The content for the Trial 4 email was created by Abigail Akande, Nicolyn Charlot, and Brittany Gay.

Research ethics statement

This study was determined exempt by the Pennsylvania State University’s Institutional Review Board contingent on the use of a disclosure statement about data collection and its use for study purposes.

Contributor statement

TS, EL, and MC conceptualised and designed the study. JP also contributed to study design. JP and MF led analyses and reported results. KC, CG, and RS contributed to the data capture.

Conflict of interest

The authors declare that there is no conflict of interest.

References

  • Boaz, A. and Davies, H. (2019) What Works Now? Evidence-Informed Policy and Practice, Bristol: Policy Press.

  • Bogenschneider, K. (2020) Positioning universities as honest knowledge brokers: best practices for communicating research to policymakers, Family Relations, 69(3): 62843. doi: 10.1111/fare.12339

    • Search Google Scholar
    • Export Citation
  • Brown, C. (2012) The policy-preferences model: a new perspective on how researchers can facilitate the take-up of evidence by educational policy makers, Evidence & Policy, 8(4): 45572.

    • Search Google Scholar
    • Export Citation
  • Brownson, R.C., Royer, C., Ewing, R. and McBride, T.D. (2006) Researchers and policymakers: travelers in parallel universes, American Journal of Preventive Medicine, 30(2): 16472. doi: 10.1016/j.amepre.2005.10.004

    • Search Google Scholar
    • Export Citation
  • Calfano, B. (2016) Power lines: unobtrusive assessment of email subject line impact on organization website use, Journal of Political Marketing, 8(3): 117.

    • Search Google Scholar
    • Export Citation
  • Cody, S. and Asher, A. (2014) Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-cycle Evaluation to Improve Program Development and Outcomes, Princeton: Mathematica Policy Research, https://www.brookings.edu/wp-content/uploads/2016/06/predictive_analytics_rapid_cycle_evaluation_cody_asher.pdf.

    • Search Google Scholar
    • Export Citation
  • Contandriopoulos, D., Lemire, M., Denis, J.L. and Tremblay, É. (2010) Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature, Milbank Quarterly, 88(4): 44483. doi: 10.1111/j.1468-0009.2010.00608.x

    • Search Google Scholar
    • Export Citation
  • Cooper, C.A. (2002) Email in the state legislature: evidence from three states, State and Local Government Review, 34(2): 12732. doi: 10.1177/0160323X0203400205

    • Search Google Scholar
    • Export Citation
  • Crossman, K. and Monk, K. (2020) Supporting families facing domestic violence during the COVID-19 pandemic: research-to-policy collaboration, https://www.research2policy.org/covid19-domestic-violence.

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, T. and Fishbein, D. (2018) Translating prevention research for evidence-based policymaking: results from the research-to-policy collaboration pilot, Prevention Science, 19(2): 26070. doi: 10.1007/s11121-017-0833-x

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, J.T., Long, E.C., Green, L., Giray, C., Gay, B., Israel, A., Storace, R., McCauley, M. and Donovan, M. (2021a) Cultivating researcher-policymaker partnerships: a randomized controlled trial of a model for training public psychologists, American Psychological Association, https://doi.apa.org/fulltext/2022-28577-008.html.

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, J.T. and Long, E.C. (2021b) Lawmakers’ use of scientific evidence can be improved, Proceedings of the National Academy of Sciences, 118(9): e2012955118.

    • Search Google Scholar
    • Export Citation
  • Druckman, J.N. and Green, D.P. (2013) Mobilizing group membership: the impact of personalization and social pressure emails, Sage Open, 3(2): 2158244013492781. doi: 10.1177/2158244013492781

    • Search Google Scholar
    • Export Citation
  • Economist (2016) Politics by numbers, 26 March, https://www.economist.com/special-report/2016/03/26/politics-by-numbers?fsrc=scn/fb/te/pe/ed/politicsbynumbers.

    • Search Google Scholar
    • Export Citation
  • Gardner, W., Mulvey, E.P. and Shaw, E.C. (1995) Regression analyses of counts and rates: poisson, overdispersed Poisson, and negative binomial models, American Psychological Association Bulletin, 118(3): 392. doi: 10.1037/0033-2909.118.3.392

    • Search Google Scholar
    • Export Citation
  • Georgieva, M. (2012) An introduction to email marketing, Winn Technology Group, http://cache.winntech.net/docs/ebooks/An-Introduction-to-Email-Marketing.pdf.

    • Search Google Scholar
    • Export Citation
  • Goldschmidt, K. (2017) Why you shouldn’t contact senators and representatives who don’t represent you, Congressional Management Foundation, https://www.congressfoundation.org/news/blog/1348.

    • Search Google Scholar
    • Export Citation
  • Goldstein, L.B. et al. (2011) American heart association and nonprofit advocacy: past, present, and future: a policy recommendation from the American heart association, Circulation, 123(7): 81632. doi: 10.1161/CIR.0b013e31820a5528

    • Search Google Scholar
    • Export Citation
  • Jones-Jamtgaard, K.N. and Lee, C.M. (2017) A quick guide to effective grassroots advocacy for scientists, Molecular Biology of the Cell, 28(16): 215558. doi: 10.1091/mbc.e17-03-0170

    • Search Google Scholar
    • Export Citation
  • Kingdon, J.W. (1984) Agendas, Alternatives, and Public Policies, Boston, MA: Little, Brown.

  • Konrad, R.A. (2020) Human trafficking and exploitation, Research-to-Policy Collaboration, https://www.research2policy.org/covid-19/human-trafficking-and-exploitation.

    • Search Google Scholar
    • Export Citation
  • Long, E.C., Pugel, J., Scott, J.T., Charlot, N., Giray, C., Fernandes, M.A. and Crowley, D.M. (2021) Rapid-cycle experimentation with state and federal policymakers for optimizing the reach of racial equity research, American Journal of Public Health, 111(10): 176871. doi: 10.2105/AJPH.2021.306404

    • Search Google Scholar
    • Export Citation
  • Mackie, T.I., Sheldrick, R.C., Hyde, J. and Leslie, L.K. (2015) Exploring the integration of systems and social sciences to study evidence use among child welfare policymakers, Child Welfare, 94(3): 3358.

    • Search Google Scholar
    • Export Citation
  • Macoubrie, J. and Harrison, C. (2013) Human services research dissemination: what works? Final report, Office of Planning, Research and Evaluation, https://www.acf.hhs.gov/sites/default/files/documents/opre/litreview.pdf.

    • Search Google Scholar
    • Export Citation
  • Magee, M.P. (2020) Experiments in advocacy: what works and why, AdvocacyLabs, 33, https://www.future-ed.org/wp-content/uploads/2020/07/Experiments-in-Advocacy.pdf.

    • Search Google Scholar
    • Export Citation
  • Majdzadeh, R., Sadighi, J., Nejat, S., Mahani, A.S. and Gholami, J. (2008) Knowledge translation for research utilization: design of a knowledge translation model at Tehran University of Medical Sciences, Journal of Continuing Education in the Health Professions, 28(4): 27077. doi: 10.1002/chp.193

    • Search Google Scholar
    • Export Citation
  • Marcelino, A.C.de.M. (2015) How to increase email marketing campaigns’ credibility, Silo.Tips, https://silo.tips/download/how-to-increase-marketing-campaigns-credibility.

    • Search Google Scholar
    • Export Citation
  • Moray, N. (1959) Attention in dichotic listening: affective cues and the influence of instructions, Quarterly Journal of Experimental Psychology, 11(1): 5660. doi: 10.1080/17470215908416289

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 2. doi: 10.1186/1472-6963-14-2

    • Search Google Scholar
    • Export Citation
  • Peek, C.J., Glasgow, R.E., Stange, K.C., Klesges, L.M., Purcell, E.P. and Kessler, R.S. (2014) The 5 Rs: an emerging bold standard for conducting relevant research in a changing world, Annals of Family Medicine, 12(5): 44755. doi: 10.1370/afm.1688

    • Search Google Scholar
    • Export Citation
  • Reid, E.J. and Montilla, M.D. (2001) Exploring Organizations and Advocacy: Strategies and Finances, Washington, DC: Urban Institute, https://www.urban.org/sites/default/files/publication/61251/310226-Exploring-Organizations-and-Advocacy.PDF.

    • Search Google Scholar
    • Export Citation
  • Richardson, L.E. and Cooper, C.A. (2006) Email communication and the policy process in the state legislature, Policy Studies Journal, 34(1): 11329. doi: 10.1111/j.1541-0072.2006.00148.x

    • Search Google Scholar
    • Export Citation
  • Rogers, E.M. (1995) Diffusion of innovations: modifications of a model for telecommunications, in M.W. Stoetzer and A. Mahler (eds) Die Diffusion Von Innovationen in der Telekommunikation, Berlin/Heidelberg: Springer, pp 2538, http://link.springer.com/10.1007/978-3-642-79868-9_2.

    • Search Google Scholar
    • Export Citation
  • Scott, J.T., Larson, J.C., Buckingham, S.L., Maton, K.I. and Crowley, D.M. (2019) Bridging the research–policy divide: pathways to engagement and skill development, American Journal of Orthopsychiatry, 89(4): 434. doi: 10.1037/ort0000389

    • Search Google Scholar
    • Export Citation
  • Stallings, T. (2009) Rethinking the relationship between subject line length and email performance: a new perspective on subject line design, Silo.Tips, https://silo.tips/download/rethinking-the-relationship-between-subject-line-length-and-performance-a-new-pe.

    • Search Google Scholar
    • Export Citation
  • Tseng, V. (2012) The uses of research in policy and practice, Social Policy Report, 26(2), https://wtgrantfoundation.org/library/uploads/2015/10/The-Uses-of-Research-in-Policy-and-Practice.pdf. doi: 10.1002/j.2379-3988.2012.tb00071.x

    • Search Google Scholar
    • Export Citation
  • Turban, E., Whiteside, J., King, D. and Outland, J. (2017) Innovative EC systems: from E-Government to E-Learning, knowledge management, E-Health, and C2C commerce, in E. Turban, J. Whiteside, D. King, and J. Outland (eds) Introduction to Electronic Commerce and Social Commerce, Berlin/Heidelberg: Springer, pp 13763.

    • Search Google Scholar
    • Export Citation
  • Wattal, S., Schuff, D., Mandviwalla, M. and Williams, C.B. (2010) Web 2.0 and politics: the 2008 US presidential election and an e-politics research agenda, Management Information Systems Quarterly, 34(4): 66988. doi: 10.2307/25750700

    • Search Google Scholar
    • Export Citation
  • View in gallery
    Figure 1:

    CONSORT Diagram, Panel A: Trial 1, Panel B: Trial 2, Panel C: Trial 3, Panel D: Trial 4

  • Boaz, A. and Davies, H. (2019) What Works Now? Evidence-Informed Policy and Practice, Bristol: Policy Press.

  • Bogenschneider, K. (2020) Positioning universities as honest knowledge brokers: best practices for communicating research to policymakers, Family Relations, 69(3): 62843. doi: 10.1111/fare.12339

    • Search Google Scholar
    • Export Citation
  • Brown, C. (2012) The policy-preferences model: a new perspective on how researchers can facilitate the take-up of evidence by educational policy makers, Evidence & Policy, 8(4): 45572.

    • Search Google Scholar
    • Export Citation
  • Brownson, R.C., Royer, C., Ewing, R. and McBride, T.D. (2006) Researchers and policymakers: travelers in parallel universes, American Journal of Preventive Medicine, 30(2): 16472. doi: 10.1016/j.amepre.2005.10.004

    • Search Google Scholar
    • Export Citation
  • Calfano, B. (2016) Power lines: unobtrusive assessment of email subject line impact on organization website use, Journal of Political Marketing, 8(3): 117.

    • Search Google Scholar
    • Export Citation
  • Cody, S. and Asher, A. (2014) Smarter, Better, Faster: The Potential for Predictive Analytics and Rapid-cycle Evaluation to Improve Program Development and Outcomes, Princeton: Mathematica Policy Research, https://www.brookings.edu/wp-content/uploads/2016/06/predictive_analytics_rapid_cycle_evaluation_cody_asher.pdf.

    • Search Google Scholar
    • Export Citation
  • Contandriopoulos, D., Lemire, M., Denis, J.L. and Tremblay, É. (2010) Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature, Milbank Quarterly, 88(4): 44483. doi: 10.1111/j.1468-0009.2010.00608.x

    • Search Google Scholar
    • Export Citation
  • Cooper, C.A. (2002) Email in the state legislature: evidence from three states, State and Local Government Review, 34(2): 12732. doi: 10.1177/0160323X0203400205

    • Search Google Scholar
    • Export Citation
  • Crossman, K. and Monk, K. (2020) Supporting families facing domestic violence during the COVID-19 pandemic: research-to-policy collaboration, https://www.research2policy.org/covid19-domestic-violence.

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, T. and Fishbein, D. (2018) Translating prevention research for evidence-based policymaking: results from the research-to-policy collaboration pilot, Prevention Science, 19(2): 26070. doi: 10.1007/s11121-017-0833-x

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, J.T., Long, E.C., Green, L., Giray, C., Gay, B., Israel, A., Storace, R., McCauley, M. and Donovan, M. (2021a) Cultivating researcher-policymaker partnerships: a randomized controlled trial of a model for training public psychologists, American Psychological Association, https://doi.apa.org/fulltext/2022-28577-008.html.

    • Search Google Scholar
    • Export Citation
  • Crowley, D.M., Scott, J.T. and Long, E.C. (2021b) Lawmakers’ use of scientific evidence can be improved, Proceedings of the National Academy of Sciences, 118(9): e2012955118.

    • Search Google Scholar
    • Export Citation
  • Druckman, J.N. and Green, D.P. (2013) Mobilizing group membership: the impact of personalization and social pressure emails, Sage Open, 3(2): 2158244013492781. doi: 10.1177/2158244013492781

    • Search Google Scholar
    • Export Citation
  • Economist (2016) Politics by numbers, 26 March, https://www.economist.com/special-report/2016/03/26/politics-by-numbers?fsrc=scn/fb/te/pe/ed/politicsbynumbers.

    • Search Google Scholar
    • Export Citation
  • Gardner, W., Mulvey, E.P. and Shaw, E.C. (1995) Regression analyses of counts and rates: poisson, overdispersed Poisson, and negative binomial models, American Psychological Association Bulletin, 118(3): 392. doi: 10.1037/0033-2909.118.3.392

    • Search Google Scholar
    • Export Citation
  • Georgieva, M. (2012) An introduction to email marketing, Winn Technology Group, http://cache.winntech.net/docs/ebooks/An-Introduction-to-Email-Marketing.pdf.

    • Search Google Scholar
    • Export Citation
  • Goldschmidt, K. (2017) Why you shouldn’t contact senators and representatives who don’t represent you, Congressional Management Foundation, https://www.congressfoundation.org/news/blog/1348.

    • Search Google Scholar
    • Export Citation
  • Goldstein, L.B. et al. (2011) American heart association and nonprofit advocacy: past, present, and future: a policy recommendation from the American heart association, Circulation, 123(7): 81632. doi: 10.1161/CIR.0b013e31820a5528

    • Search Google Scholar
    • Export Citation
  • Jones-Jamtgaard, K.N. and Lee, C.M. (2017) A quick guide to effective grassroots advocacy for scientists, Molecular Biology of the Cell, 28(16): 215558. doi: 10.1091/mbc.e17-03-0170

    • Search Google Scholar
    • Export Citation
  • Kingdon, J.W. (1984) Agendas, Alternatives, and Public Policies, Boston, MA: Little, Brown.

  • Konrad, R.A. (2020) Human trafficking and exploitation, Research-to-Policy Collaboration, https://www.research2policy.org/covid-19/human-trafficking-and-exploitation.

    • Search Google Scholar
    • Export Citation
  • Long, E.C., Pugel, J., Scott, J.T., Charlot, N., Giray, C., Fernandes, M.A. and Crowley, D.M. (2021) Rapid-cycle experimentation with state and federal policymakers for optimizing the reach of racial equity research, American Journal of Public Health, 111(10): 176871. doi: 10.2105/AJPH.2021.306404

    • Search Google Scholar
    • Export Citation
  • Mackie, T.I., Sheldrick, R.C., Hyde, J. and Leslie, L.K. (2015) Exploring the integration of systems and social sciences to study evidence use among child welfare policymakers, Child Welfare, 94(3): 3358.

    • Search Google Scholar
    • Export Citation
  • Macoubrie, J. and Harrison, C. (2013) Human services research dissemination: what works? Final report, Office of Planning, Research and Evaluation, https://www.acf.hhs.gov/sites/default/files/documents/opre/litreview.pdf.

    • Search Google Scholar
    • Export Citation
  • Magee, M.P. (2020) Experiments in advocacy: what works and why, AdvocacyLabs, 33, https://www.future-ed.org/wp-content/uploads/2020/07/Experiments-in-Advocacy.pdf.

    • Search Google Scholar
    • Export Citation
  • Majdzadeh, R., Sadighi, J., Nejat, S., Mahani, A.S. and Gholami, J. (2008) Knowledge translation for research utilization: design of a knowledge translation model at Tehran University of Medical Sciences, Journal of Continuing Education in the Health Professions, 28(4): 27077. doi: 10.1002/chp.193

    • Search Google Scholar
    • Export Citation
  • Marcelino, A.C.de.M. (2015) How to increase email marketing campaigns’ credibility, Silo.Tips, https://silo.tips/download/how-to-increase-marketing-campaigns-credibility.

    • Search Google Scholar
    • Export Citation
  • Moray, N. (1959) Attention in dichotic listening: affective cues and the influence of instructions, Quarterly Journal of Experimental Psychology, 11(1): 5660. doi: 10.1080/17470215908416289

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 2. doi: 10.1186/1472-6963-14-2

    • Search Google Scholar
    • Export Citation
  • Peek, C.J., Glasgow, R.E., Stange, K.C., Klesges, L.M., Purcell, E.P. and Kessler, R.S. (2014) The 5 Rs: an emerging bold standard for conducting relevant research in a changing world, Annals of Family Medicine, 12(5): 44755. doi: 10.1370/afm.1688

    • Search Google Scholar
    • Export Citation
  • Reid, E.J. and Montilla, M.D. (2001) Exploring Organizations and Advocacy: Strategies and Finances, Washington, DC: Urban Institute, https://www.urban.org/sites/default/files/publication/61251/310226-Exploring-Organizations-and-Advocacy.PDF.

    • Search Google Scholar
    • Export Citation
  • Richardson, L.E. and Cooper, C.A. (2006) Email communication and the policy process in the state legislature, Policy Studies Journal, 34(1): 11329. doi: 10.1111/j.1541-0072.2006.00148.x

    • Search Google Scholar
    • Export Citation
  • Rogers, E.M. (1995) Diffusion of innovations: modifications of a model for telecommunications, in M.W. Stoetzer and A. Mahler (eds) Die Diffusion Von Innovationen in der Telekommunikation, Berlin/Heidelberg: Springer, pp 2538, http://link.springer.com/10.1007/978-3-642-79868-9_2.

    • Search Google Scholar
    • Export Citation
  • Scott, J.T., Larson, J.C., Buckingham, S.L., Maton, K.I. and Crowley, D.M. (2019) Bridging the research–policy divide: pathways to engagement and skill development, American Journal of Orthopsychiatry, 89(4): 434. doi: 10.1037/ort0000389

    • Search Google Scholar
    • Export Citation
  • Stallings, T. (2009) Rethinking the relationship between subject line length and email performance: a new perspective on subject line design, Silo.Tips, https://silo.tips/download/rethinking-the-relationship-between-subject-line-length-and-performance-a-new-pe.

    • Search Google Scholar
    • Export Citation
  • Tseng, V. (2012) The uses of research in policy and practice, Social Policy Report, 26(2), https://wtgrantfoundation.org/library/uploads/2015/10/The-Uses-of-Research-in-Policy-and-Practice.pdf. doi: 10.1002/j.2379-3988.2012.tb00071.x

    • Search Google Scholar
    • Export Citation
  • Turban, E., Whiteside, J., King, D. and Outland, J. (2017) Innovative EC systems: from E-Government to E-Learning, knowledge management, E-Health, and C2C commerce, in E. Turban, J. Whiteside, D. King, and J. Outland (eds) Introduction to Electronic Commerce and Social Commerce, Berlin/Heidelberg: Springer, pp 13763.

    • Search Google Scholar
    • Export Citation
  • Wattal, S., Schuff, D., Mandviwalla, M. and Williams, C.B. (2010) Web 2.0 and politics: the 2008 US presidential election and an e-politics research agenda, Management Information Systems Quarterly, 34(4): 66988. doi: 10.2307/25750700

    • Search Google Scholar
    • Export Citation
Taylor ScottPennsylvania State University, USA

Search for other papers by Taylor Scott in
Current site
Google Scholar
Close
,
Jessica PugelPennsylvania State University, USA

Search for other papers by Jessica Pugel in
Current site
Google Scholar
Close
,
Mary FernandesGeorgia State University, USA

Search for other papers by Mary Fernandes in
Current site
Google Scholar
Close
,
Katherine CruzJohns Hopkins Bloomberg School of Public Health, USA

Search for other papers by Katherine Cruz in
Current site
Google Scholar
Close
,
Elizabeth C. LongPennsylvania State University, USA

Search for other papers by Elizabeth C. Long in
Current site
Google Scholar
Close
,
Cagla GirayWeber Shandwick, USA

Search for other papers by Cagla Giray in
Current site
Google Scholar
Close
,
Rachel StoracePennsylvania State University, USA

Search for other papers by Rachel Storace in
Current site
Google Scholar
Close
, and
D. Max CrowleyPennsylvania State University, USA

Search for other papers by D. Max Crowley in
Current site
Google Scholar
Close

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 468 468 0
Full Text Views 568 568 135
PDF Downloads 382 382 81

Altmetrics

Dimensions