Abstract
Background:
Despite significant progress in HIV treatment and prevention, the US remains far from its goal of ‘Ending the HIV Epidemic’ by 2030. Economic models using local data can synthesise the evidence to help policymakers allocate HIV resources efficiently, but persistent research-to-practice gaps remain. Little is known about how to facilitate the use of economic modelling data among local public health policymakers in real-world settings.
Aims and objectives:
To explore the dissemination of results from a locally-calibrated economic model for HIV prevention and treatment and identify the factors influencing potential uptake of the model for public health decision making at the local level.
Methods:
Four virtual focus groups with 26 local health department policymakers in Baltimore, Miami, Seattle, and New York City were held between July 2020 and May 2021. Qualitative content analysis of transcripts identified key themes around using the localised economic model in policy decisions.
Results:
Participants were interested in using local data in their decisions to allocate resources for HIV prevention/treatment. Six themes emerged: 1) importance of understanding local policy context; 2) health equity considerations; 3) using evidence to support current priorities; 4) difficulty of changing strategies, even incrementally; 5) bang for the incremental buck (efficiency) vs. previous impact; and 6) community values.
Conclusion and relevance:
To optimise acceptance and use of results from economic models, researchers should engage with local community members and public health decision makers early to understand budgetary and community priorities. Participants prioritised evidence that supports their existing strategies, considers budgets and funding streams, and improves health equity; however, real-world budget constraints and conflicting interests serve as barriers to implementing model recommendations and reaching national goals.
Background
In 2019, the federal government launched the ‘Ending the HIV Epidemic in the US’ (EHE) initiative, with the goal of reducing new HIV infections in the US by 90% by 2030 (Fauci et al, 2019). To do so, it proposes scaling up key HIV prevention and treatment strategies and engaging communities to improve health equity, mitigate stigma and discrimination, and build public trust (Fauci et al, 2019). We developed a simulation model that can estimate the public health and economic impacts of these strategies (Collins Jr and Sapiano, 2016), helping policymakers allocate EHE resources efficiently and equitably (Nosyk et al, 2020c; Quan et al, 2021). But to be useful to public health decision makers, the evidence produced by this model must be accepted and used. Population-based simulation models proved quite useful during the COVID-19 pandemic, as researchers tried to estimate the pandemic’s course under different scenarios and assumptions (Nosyk et al, 2020b). With increasing public awareness of the value of such modelling, we sought to explore the factors that might facilitate the use of model-generated evidence in in public health decision making.
We ground our study within the context of how decision makers use evidence in general, and economic evidence in particular. Weiss and colleagues (Weiss, 1986) described three primary ways that decision makers use research evidence: instrumentally, conceptually, and symbolically. Instrumental use means that the decision maker used the evidence directly, to propose changes to policy or practice. This use is quite rare. More commonly, evidence is used conceptually to inform the decision maker’s general point of view. Even more common is the symbolic, or political, use of research, in which the decision maker uses evidence to support policies or practices already espoused for other reasons.
Before it can be used, however, evidence from health research must be translated and disseminated to policymakers. Dissemination research is an underdeveloped field in implementation science (Purtle et al, 2022) and a recent review identified that few studies have explored what aspects of dissemination are most effective in translating health research (particularly modelling data) to policymakers (Ashcraft et al, 2020). Specifically, the Ashcraft et al review suggested that dissemination of social policy research is effective when it starts early, galvanises support, uses champions and brokers, considers contextual factors, is timely, relevant, and accessible, and when the researchers know the players and process (Ashcraft et al, 2020). Further, a recent randomised field experiment by Purtle et al found that including state-tailored economic evidence increased engagement with dissemination materials by state legislators, but only among Democrats, highlighting that tailored dissemination strategies according to political affiliation may also increase engagement (Purtle et al, 2022). These findings suggest that there are important local differences in how our model, which provides localised economic evidence, might be perceived and applied.
Cost-effectiveness analyses from local data can help determine the public health impact and value of implementing public health strategies to address the needs of priority areas and populations (Krebs and Nosyk, 2021). However, we know little about how local public health policymakers perceive and utilise evidence generated by a localised model, and about the best approaches and tools for communicating and disseminating technically complex data to optimise its use to support decision making.
In a recent review, Funk and colleagues identified four overlapping types of tools for translating health information for policymakers (Funk et al, 2022). They include visualisations and modelling, which can transform evidence into formats more accessible to policymakers; packaging and synthesis tools, such as briefs, to condense findings; communication and dissemination tools, such as oral presentations and newsletters; and information linkage and exchange, such as deliberative dialogues with community members and local decision makers, to encourage acceptance and application of findings.
In this study, we sought to explore the dissemination of results from a locally-calibrated economic model for HIV prevention and treatment, and aimed to identify the factors influencing potential uptake of the model for public health decision making at the local level.
Methods
Model description
Previously, members of the research team developed a dynamic, compartmental HIV transmission model calibrated to replicate city-level HIV micro-epidemics within six US cities prioritised in the EHE initiative (Atlanta, Baltimore, Miami, Seattle, Los Angeles, and New York City) (Nosyk et al, 2020c). In each city, the adult population was partitioned by sex, HIV risk group, race/ethnicity and sexual risk behavior. The model tracked individuals susceptible to HIV acquisition through the course of infection, diagnosis, treatment with antiretroviral therapy (ART) and ART dropout. To inform the 1,667 parameters needed to populate our model, we synthesised evidence from 59 peer-reviewed publications and 24 public health and surveillance reports and executed primary analyses using 11 data sets (Krebs et al, 2019). Of these 1,667 parameters, 1,517 (91%) were city-specific and 150 (9%) were common for all cities (Krebs et al, 2019).
The model was designed as a proof-of-concept, to demonstrate that a locally-oriented modelling framework could provide more tailored normative guidance, different recommendations across the US’ diverse urban settings, and ultimately greater public health value, than a nationally-oriented model. The work was inspired by a similar locally-oriented modelling effort in Kenya (Anderson et al, 2014), as well the investigators’ own experience supporting the HIV/AIDS response in British Columbia, Canada (Nosyk et al, 2015; 2014; Wang et al, 2017; Enns et al, 2019). The scientific case was laid out in a peer-reviewed publication in the early stages of the project (Panagiotoglou et al, 2018). The geographic orientation of the project ultimately aligned with the Ending the HIV Epidemic (EHE) Initiative, announced in 2019 (Fauci et al, 2019), and captured 12 of 48 counties targeted in the EHE initiative. Details on the development of the model’s evidence synthesis (Krebs et al, 2019), calibration and validation process (Zang et al, 2020) have been published elsewhere.
The model integrated 16 interventions, selected from the US Centers for Disease Control and Prevention ‘Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention’ and from the literature (Krebs et al, 2020). These interventions were selected based on the strength of their supporting evidence and promise in scalability and were organised across four domains: HIV prevention programmes (syringe service programmes, medications for opioid use disorder, and pre-exposure prophylaxis); HIV testing; ART engagement (ART initiation and retention); and ART re-engagement (re-initiation). The population-level impact of HIV prevention programmes were entered in the model by reducing the probability of HIV transmission, and HIV testing and care interventions affected model parameters by dictating the probability of transitioning between different health states (compartments) (Krebs et al, 2020).
The model determined city-specific cost-effective strategies of implementing or scaling up combinations of these 16 interventions, and projected long-term HIV incidence as well as the economic value of the optimal combination strategies (Krebs et al, 2020; Nosyk et al, 2020c). We assumed the impact for all interventions to be additive to existing service levels for each city (Nosyk et al, 2020a). Therefore, all scenarios considered only incremental investments, rather than disinvestments or reallocations of existing funds. We conducted the cost-effectiveness analysis from the healthcare sector perspective to present implementation strategies that inform healthcare policy and resource allocation decision making, and conformed to best practice guidelines of the Second Panel on Cost-Effectiveness in Health and Medicine (Sanders et al, 2016).
We determined each evidence-based intervention’s scale of delivery; population-level impact; period over which the intervention is scaled-up and sustained; and the costs of implementation, delivery and sustainment (Krebs et al, 2020). We synthesised evidence from 11 peer-reviewed publications, 12 public health and surveillance reports and three publicly-available data sets to model the impact of each of the interventions. Further, to derive implementation, delivery and sustainment costs for each intervention, we synthesised evidence from 25 peer-reviewed publications, 11 public health and surveillance reports and six publicly-available data sets (Krebs et al, 2020). The costs attributable to each intervention included costs of implementation, delivery and sustainment, and were specific to the setting in which the intervention was delivered. The sources and methods used to estimate the effectiveness, scale of delivery, and the costs of each individual intervention have been previously documented in detail (Krebs et al, 2020). As a baseline, the model assumed that the interventions would be implemented at existing proportions across risk and ethnic groups, representing the expected level of scale-up that could be achieved within current social and structural constraints on access to care.
Dissemination approach and evaluation
The research team grounded its approach in best practices for dissemination (Lavis et al, 2003; Gagliardi et al, 2015). Knowledge translation was a collaborative process between the research team and local health policy decisionmakers and community members, and was conducted in four steps: 1) development of report drafts; 2) initial presentation and report delivery; 3) focus group discussions; and 4) refinement of communication materials and supplemental analyses.
Initial development of city-specific reports
We presented results from the model through city-specific reports in two formats: a 35-page report and a two-page brief (Supplementary Appendix 12).
The complete report provided information on the inputs and dynamics of the model. It presented estimated HIV incidence in 2030 under four investment scenarios: 1) maintaining current service levels (status quo); 2) individual interventions scaled-up at previously documented scales of delivery; 3) the highest-valued combination of interventions delivered at existing scales of delivery; and 4) the highest-valued combination of interventions delivered at near-ideal levels of implementation to reach EHE targets. The reports illustrated incremental expenditures of the highest-valued combination strategy (scenario 3) from 2020 to 2030 from the healthcare sector perspective (in 2019 value using a 3% annual discount rate).
The two-page, city-specific briefs summarised the key findings on 1) projected HIV incidence under the status quo (that is, scenario 1); 2) projected HIV incidence under the highest-valued combination strategy (that is, scenario 3); 3) incremental costs of the highest-valued strategy; and 4) recommendations regarding the highest-valued combination of interventions identified to deliver the greatest health benefits while remaining cost-effective. The brief was formatted with charts, graphics, and bullet points. By design, it omitted details of the model’s development, calibration, and validation, focusing instead on telegraphing the key findings to busy policymakers.
Study design and recruitment
For each city, we identified a sample of state and county-level decision makers in HIV programme planning and local public health departments through the professional network of the study’s investigators, scientific advisory committee, and local HIV researchers. Participants were identified from department and organisation websites and later snowball sampling, as additional participants were identified by referrals from initial contacts within the research team’s network.
Participant recruitment began in January 2020, but was interrupted by the onset of the COVID-19 pandemic. Recruitment efforts resumed in July 2020 and continued through May 2021.
Ethics
This study was approved by the Simon Fraser University Office of Research Ethics and The Providence Health Care Research Ethics Board (H16-00652).
Virtual focus groups
Guided focus groups were conducted over videoconferencing between July 2020 and May 2021. We developed a moderator guide to 1) identify stakeholders’ local priorities and determine their prior experience with economic modeling; 2) identify factors that facilitate use of the model in public health decision making; and 3) obtain feedback on the content, structure, and appearance of the report and accompanying brief.
Prior to each focus group, the 35-page reports, two-page briefs, introductory slides, and informed consent form explaining the project were distributed to each participant for review. At the start of each focus group, the Principal Investigator (BN) delivered a 45-minute presentation of the model (rationale, evidence synthesis, calibration, and key findings). Local health policy decision makers and community members provided verbal consent for their participation and audio recording prior to the guided discussion.
Following the presentation, two collaborators (ZM and JW) facilitated a 45- to 60-minute group discussion. Based on participants’ feedback, we refined our communication materials, conducted supplemental analyses, and shared the materials with participants after the discussion.
Qualitative analysis
Each focus group was audio-recorded, transcribed, and de-identified. Three members of the research team (ZM, JW, and MA) coded the data independently using Dedoose software, a computer-based qualitative data analysis software program. The team conducted a thematic content analysis of transcripts, inductively identifying unique codes consistent with several theories of research utilisation in policymaking familiar to the research team (Lavis et al, 2002). The three coders then discussed the transcripts together. During the pilot coding process, additional codes were identified from initial review and discussion and added to the coding scheme, and some codes were collapsed or eliminated. Coding was carried out in multiple iterations to allow for the emergence of novel codes, and all transcribed text was coded and reviewed for patterns across the data set. The Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guideline for qualitative research was used for study design and analysis of our data (Tong et al, 2007).
In total, 20 a priori codes (Supplementary Appendix 23) were developed into a codebook using these concepts. All transcripts were triple-coded (ZM, JW, and MA), with group meetings to examine results and to resolve coding disagreements by consensus. Next, investigators created analytic memos for all codes, describing key themes and a list of relevant quotations from participants. Using the constant comparison method, in which newly collected data are compared with categories that emerge from previously collected data, all memos were thematically analysed for consistency and agreement. Any conflicts were resolved by the senior author (ZM). Durable patterns across the research team were included in the final results. Of note, all three members of the research team coded all transcripts in order to improve fidelity and identify and reach consensus regarding thematic quotes.
Data availability
Raw data (transcripts) and study materials can be found in a third-party repository at https://osf.io/y9ka6.
Results
A total of 26 stakeholders working on HIV programming within local health departments participated in the focus group discussions across four cities (Baltimore, Miami, New York City, and Seattle). The roles and educational levels of participants are shown in Table 1.
Characteristics of focus group participants (N=26)
| Characteristic | N |
|---|---|
| City a | |
| Baltimore | 5 |
| Miami | 8 |
| New York City | 9 |
| Seattle | 4 |
| Primary role | |
| Director or commissioner | 15 |
| Medical or public health professor and/or epidemiologist | 6 |
| Programme supervisors | 5 |
| Education | |
| MD, MPH | 6 |
| PhD, MPH | 6 |
| PhD or DrPH | 5 |
| MPH or other | 9 |
One focus group per city (total 4)
The analysis of the focus group discussions generated six themes that can influence uptake of the model’s findings: 1) it is important to understand the context of local policy; 2) health equity in HIV prevention and treatment should be addressed; 3) local health policy decision makers want evidence to support their current practices; 4) change is hard, even incremental change; 5) local policymakers often want to allocate additional dollars to what has worked in the past, even if the model identifies higher-value investments; and 6) models should clearly take into account community values.
Within each of these themes, the participants identified barriers, facilitators, and current practice, and gave their perspectives on ‘ideal conditions’ for research design and communication to inform policy decisions. Table 2 summarises these results with representative quotes. Table 3 offers actionable recommendations.
Key themes and representative quotes
| Theme | Representative quotes |
|---|---|
| The intersection of policy contexts with local data is important | In terms of how the money gets spent specifically, we’re not really supposed to be making those decisions on our own at the Health Department. It’s the community’s decision. And we have – and it takes a lot of effort to get that input from the community, but that’s what we’re supposed to be doing. And it’s more explicit in some funding streams than others. |
| At least for us, because we had the sense that… we are the heroin capital of the United States, there’s going to continue to be a need for them [syringe services programmes]. | |
| Health equity in HIV prevention and treatment should be addressed | Our biggest investments in Ending the HIV Epidemic (EHE) are probably going to be focused on increasing care in the most disadvantaged populations through low-barrier care and sort of enhanced access to PrEP in the most disadvantaged communities, and testing in the most disadvantaged |
| I think that, especially for me where I’ve focused a lot of my work on thinking about PrEP and HIV prevention for Black and Latina women, like the fact that even like the beginning of the slides it was like PrEP was even only considered as an intervention for men who have sex with men. But women make up 20 percent of our new diagnoses. Black and Latina women make up 90 percent of those new diagnoses. And we very much see it as like an element of justice to make sure that that is an intervention that’s available and accessible and affordable for everyone who wants it. | |
| Local health policy decision makers want evidence for current practices | But you sort of identify optimal interventions using the current slate of interventions that we have happening so that we know what’s working and what’s not. [But] limiting the PrEP intervention to just that for ‘high-risk MSM’ is problematic, but also not even including it as one of the optimal interventions within the combination would be in stark contrast to what we’re currently doing. |
| I want to be confident that we’re getting a good return on investment on not just the money that we invest, but people’s time, their energy, their emotional energy, all of those different types [of current practices]. | |
| Change is hard – even incremental change |
Putting aside where you put dollars, if we emphasise some approaches over others where we’re asking our communities and we’re asking ourselves to kind of turn in one way, and if we’re turning in one way, what are we not paying attention to? And I think there’s a real cost associated with that, as well. |
| It’ll land flat if we don’t acknowledge all of those things, but also kind of frame it in a way that that doesn’t mean you don’t do other things. We’re not going to stop, for example, giving away condoms. | |
| Local policymakers often want to allocate additional dollars to what has worked in the past, even if the model identifies higher-value investments | Is this against a backdrop? You mentioned syringe support. At one point, Baltimore is a city where it had a 66 percent prevalence among injection drug use, and so it was really the implementation with syringe support services that helped bring that down very briskly, and now we bounce anywhere between 6 percent and 12 percent or 8 percent, which is – and I didn’t see how that had been factored in. |
| This goes back to the differences between I think it was the 16 indicators used versus like what we’re doing in real time. I mean, if you overlay our local Ending the Epidemic strategies, which lists sort of our high-level interventions, I think some conspicuous differences would be HIV testing in non-clinical settings. We’re directing a lot of resources towards that. And one, because it’s proven to be an excellent return, but also given COVID-19, it’s been a way that we’ve continued trying to maintain our HIV testing during physical distancing orders. | |
| Models should clearly take into account community values | First, I see a lot of things missing from the cost attribution side, and I don’t know if – obviously, this is just a summary of a lot of things grouped together. But the types of things where I don’t necessarily know where they fall in here. I mean, the obvious one is social determinants, when we talk to the community about what do we need to prevent HIV infections. It’s all about social determinants all the time. And that’s not funded by anyone, and that’s not – at least not to the degree that it would need to be to address these issues. And there hasn’t been engagement from those partners at the federal level to talk about housing, to talk about job training and all the other things, to talk about food, access to food, things like that. |
| It’s a long process and it’s a community planning process, so it’s not always pretty and it’s not always efficient as community planning is. But at the end of the day when you come out the other side, you have community investment in what we put out there. And I think that alone is really important in supporting success of these programmes. |
Recommendations that researchers should consider for optimal use of local economic simulation modelling evidence for policy-decision making
| Recommendations |
|---|
| • Involve policymakers, community members, and other working partners early in the research process to integrate their values and priorities into local models; actively work to understand and develop specific local research questions meeting local needs |
| • Dedicate more resources to cultivating working relationships and informal networks with both local decision makers and local data stewards to facilitate evidence synthesis and knowledge translation and dissemination |
| • Emphasise the positive value of use of local data and engagement between community collaborators and modellers when disseminating local economic evidence |
| • Guide prospective data collection, or gain access to data that may not be publicly available, by making sure to account for equity-oriented strategies that address health disparities and the social determinants of health when interfacing with policymakers about the model’s possible conclusions and the data required to best inform local decisions |
| • Consider capacity vs. interest (for spending and investment) before and during the development of local economic models |
| • Emphasise practical considerations (such as real-world budget constraints, resource requirements for implementation and conflicting interests) over optimal or marginal benefits when developing local models and disseminating local evidence |
| • Identify current local priorities and successes, and clarify how the model can build on these successes |
Theme 1: The intersection of policy contexts with local data is important
Overall, participants saw the value of using local data in the model, rather than national data. Many participants also stressed the importance of understanding how different jurisdictions, agencies, and funders interact on a local level, given the breadth of interventions the model considered. While the model identified the optimal combination of strategies, participants noted that the optimal strategy may not be the most practical one, given jurisdictional silos and political considerations.
One practical constraint they noted was that local budgets and funding streams were often officially or unofficially earmarked for certain purposes. For example, a large source of funding was the federally-administered Ryan White HIV/AIDS programme (which provides grants to cities, states, and local organisations for HIV-related services), which didn’t fund certain prevention strategies, or, in some cases, the groups dispensing money had already decided how the funds should be spent.
Theme 2: Health equity in HIV prevention and treatment should be addressed
By design, the investigators initially modelled optimal interventions at existing scales of delivery, to demonstrate that additional efforts would be needed to reach vulnerable and minority populations. Participants appreciated the need to address health inequities by increasing the reach of certain interventions, and expressed interest in using the model to estimate the effects of targeted outreach. Policymakers did not want to ‘bake in’ existing inequities by assuming existing uptake of the interventions.
Participants were interested in how the model performed when looking at specific groups, which varied by city. These groups included subsets of racial minority and gender-diverse populations, such as Latina women and Black transgender individuals. Researchers noted that they planned to conduct these analyses and that equity was a focus of ongoing dissemination activities.
Theme 3: Local health policy decision makers want evidence to support their current practices
Consistent with the ‘symbolic’ use of research, participants thought the model was particularly useful when it supported what they already planned to do, and had less enthusiasm for using the model when it was not well aligned with current priorities. Participants voiced less interest in using the findings incrementally to identify underutilised strategies for additional investment. Some noted that ongoing public health activities were not assessed in the model, such as contact tracing efforts and campaigns to build public trust. Some policymakers appeared to be searching for support for their current plans rather than evidence to inform subsequent decisions.
Theme 4: Change is hard, even incremental change
Participants cited the inherent difficulty and ‘friction costs’ of changing direction or implementing new strategies, even if the model focuses on ‘the next dollar spent’. In other words, although the model did not emphasise changing current budget priorities, participants expressed concern when the model suggested additional spending on an area that was not currently prioritised. Some participants held the misperception that the model was estimating the value of current spending, instead of theoretical incremental budget increases, especially when the audience may not have discretionary control over the next dollar spent. Participants noted that it was challenging to think beyond their fixed budgets to hypothetical investment scenarios, much less act on the information presented. Most participants thought that the two-pager was generally helpful, but rarely described the next steps of using it (for example, I will take this to my boss).
Although the model used public sources to estimate the costs of implementing each strategy, participants questioned whether it captured the costs of building an infrastructure for these new strategies, including the real costs of implementing a new intervention as well as the hard-to-quantify costs of changing entrenched interests. Estimating these costs required numerous assumptions, particularly for the human resources needed to account for patient volumes across healthcare settings, physician practices, or HIV clinics (Krebs et al, 2020).
Theme 5: Local policymakers often want to allocate additional dollars to what has worked in the past, even if the model identifies higher-value investments
Participants questioned the results of the model when the highest-value strategy did not include allocating additional resources to previously successful interventions. The economic model took as its starting point the existing strategic investments and estimated the most efficient use of additional investments. But many community members and working partners had a history of implementing strategies that had worked up to that point, and expressed the desire to continue to invest in what had been working. In that sense, there was a disconnect between the experience of the policymakers and the model’s recommendations for incremental investments to reach national EHE goals.
This theme captures the question raised and discussed by several community collaborators, “Should cities invest in their successes or their gaps?” The model is oriented towards investing in the gaps to make the most of any additional dollar. However, the working partners appeared to place more value in general on investing in the successes (that is, community collaborators underestimated diminishing returns on additional investments in ongoing strategies).
Theme 6: Models should clearly take into account community values
Participants questioned whether and how community priorities and values were included in the model. Some policymakers were concerned that community members would be disheartened if the ideas or programmes they supported were not part of the model and further emphasised the importance and impact of engaging community collaborators, including members of the local Ryan White Planning Councils, which determine how programme resources are allocated and include people living with HIV accessing these services. They thought the lack of community involvement would be a barrier to using the model to guide policy, especially if it involved changing existing priorities. Moreover, it became clear that even working partners and decision makers in the same city had different priorities, likely due to their different backgrounds and organisations with whom they worked. This raises the question of how to translate these models for a variety of community members, as the approach may differ from translation to policymakers.
Discussion
The use of research to inform policy decisions has been examined across many fields and contexts (Lavis et al, 2003; Lavis, 2006; Mitton et al, 2007). However, most of this work has focused on how policymakers can use existing research to support their policies. Less literature explores the ways that researchers can most effectively address the needs and concerns of public health policymakers. In this study, we explored how the scientific community can help reduce the gaps between clinical and cost-effectiveness research and investment strategies in HIV/AIDS, and how this evidence can be used by decision makers to determine the local resources needed to reach national targets in HIV incidence reduction.
We identified a range of perspectives from public health policymakers about the usefulness of a locally-calibrated model for HIV prevention and treatment. A better understanding of how to translate HIV models is timely in light of the COVID-19 pandemic and EHE, which have led to new opportunities for funding and policies to combat the spread of HIV. Public health policymakers and community members might use the evidence from economic modelling to implement strategies that will deliver the greatest health benefits with finite resources.
Across focus groups, working partners and community collaborators varied in their experience working with simulation models and whether they thought the evidence would be useful when allocating specific budgets and funding streams. While the modelling results demonstrated that none of the cities would reach EHE targets by 2030 at existing scales of service delivery, many decision makers grappled with the concept of the ‘next incremental dollar spent’ or expressed a desire for economic evaluations of what they were already doing. Their insights and perspectives centred on six key areas, including wanting evidence for interventions currently prioritised by local health departments, budgetary considerations and limitations to real-world implementation, tensions between community values and the model’s inputs, and a strong interest in improving health equity. In their feedback about our reports, they appeared to value the face-to-face communication and the clarity of the results illustrated in the two-page briefs.
Working partners and community collaborators also expressed contrasting priorities by their affiliations with different communities of patients, providers, and policymakers. Further qualitative work by our group will elucidate these priorities and perceived unmet needs, as well as local workforce capacity to implement model recommendations in specific EHE jurisdictions. Some health policy leaders also highlighted opportunities to improve engagement with federal partners around services prioritised by community members – particularly services that address the social determinants of health and structural barriers to care (for example, housing and transportation services).
From the initial contact with community collaborators in these focus groups, further dissemination, community engagement, and analyses are currently underway. We are presenting the model findings to community planning council members developing a state-wide five-year Integrated HIV Prevention and Care Plan (Health Resources and Services Administration, 2021) for organisations funded by the Health Resources and Services Administration (HRSA) and the Centers for Disease Control and Prevention (CDC). These efforts are expected to form a key channel that enhances the uptake of the model in EHE jurisdictions. By disseminating these projections to local stakeholders within these networks, our work could help identify equity-oriented strategies, and the associated costs and resources needed to reach regional targets.
This collaborative process is consistent with Gagliardi’s model of integrated knowledge translation, which suggests researchers collaborate with policymakers to develop questions meeting their needs (Brownell and Roberto, 2015; Gagliardi et al, 2015). Early engagement with key stakeholders holds promise, but poses considerable logistical challenges for modellers, particularly those analysing multiple cities with diverse stakeholders. Time and opportunities for engagement are limited, given the competing priorities and daily pressures in public health departments. Without practical outputs to present, early contact may yield little immediate benefit to decision makers, and both personnel and priorities may shift in the model development process. In this application for instance, the EHE initiative was introduced near the end of the 48-month development process (Krebs et al, 2019), and recommendations for PrEP use also changed (Owens et al, 2019) in the time the work was developed.
The timeliness of the modelled results, and ability to continuously update estimates as new surveillance and other data become available, are other notable challenges in fully realising the potential benefits of a multi-jurisdictional modelling platform. Such platforms, and simulation modelling efforts to guide public health decision making more generally, are likely to proliferate in the coming years, as health decision makers grapple with a range of competing priorities and significant constraints on both human and fiscal resources. Going forward, similar modelling efforts should dedicate more resources to establishing working relationships with networks of both decision makers and local data stewards. These efforts should facilitate knowledge translation, ideally build communications from existing model infrastructure and results, and establish working relationships to facilitate both evidence synthesis and knowledge translation, by streamlining elements of the model’s development and dissemination cycle.
These kinds of relationships can also help modellers understand specific local questions and help policymakers understand the data needed to answer those questions. Public health decision makers and community partners often expressed a desire to see modelling results for specific local interventions, yet the data to inform model inputs for the implementation components of these interventions are often sparse or not publicly available (Krebs et al, 2020). Real-world implementation dynamics can be incorporated in models but require high-quality local data to improve the quality and relevance of results and conclusions (Krebs and Nosyk, 2021). This represents an important opportunity for collaboration between local policymakers and modellers. By helping policymakers better understand what conclusions the model could reach, as well as the inputs required to inform local decisions, modellers can help guide prospective data collection, or gain access to data that may not be publicly available.
Further, given sparse evidence on costs for implementation and sustainment strategies for scaling-up interventions across different systems, settings, population subgroups, and levels of implementation (Krebs and Nosyk, 2021), this study’s findings highlight the potential for greater community collaborator involvement in determining the context for hypothetical investment scenarios. In addition to increased researcher-stakeholder collaboration, there remains an opportunity to continue to ‘push’ evidence to policymakers, since effective dissemination strategies may require less resources or complement capacity- and relationship-building strategies (Purtle et al, 2022).
Refinement of communication materials and supplemental analyses
We used focus group findings to refine the final city-specific reports and to guide subsequent analyses that focused on the impact of equity-focused HIV strategies (Quan et al, 2021). These analyses demonstrated that a strategy explicitly focused on reducing disparities in access across racial/ethnic groups would further reduce HIV incidence and produce greater health benefits at a lower overall cost within each of these settings (Quan et al, 2021).
In this study, the initial analyses based on the status quo prompted explicit discussions on inequities in access to care, and the efforts local public health departments were taking to reduce them. The modelling team deliberately used status quo levels of access to care across racial/ethnic groups to highlight how these inequities would stand in the way of reaching EHE goals (Nosyk et al, 2020a) and their limiting effects. Other modellers might consider how their work can advance equity issues by quantifying the effects of longstanding disparities in access and treatment.
The exercise also highlighted several differences between cost-effectiveness studies and the forms of evidence local stakeholders use in their decision making. Cost-effectiveness analysis typically considers incremental investments to new or existing interventions. The desire to understand the value of current budget items, and even recommendations for potential disinvestment, represents a different, but nevertheless important perspective for scientists to consider. Decision makers operating within funding silos often cannot consider broader societal perspectives given their own budget constraints. To address these issues, we conducted a budgetary impact analysis of our results, to disaggregate and attribute the incremental costs of intervention strategies to key funding agencies (Enns et al, 2021). We then estimated the budgetary expansion required to fund optimal strategies for each city, compared to current funding levels, thereby grounding our findings in the real-world policy context.
Limitations
Not all relevant local decision makers were represented in the focus groups, nor were community representatives, and so our results may not reflect the full breadth of opinion on the use of the modelling framework. While some of the findings may be specific to a localised model for HIV treatment and prevention, the translation and communication challenges are likely present in other simulation models of cost-effective public health interventions. While the two-page brief was considered helpful, it was delivered in the context of the larger report and presentation. Whether it would be useful as a standalone piece, or for subsequent dissemination by community collaborators and working partners, is worthy of further study. Additionally, newer communication tools, including intuitive model diagrams, ‘model stories’, and interactive interfaces for end users might improve dissemination efforts in this field and should be explored in future work (Nosyk et al, 2020b). Lastly, while this study was conducted in direct engagement with working partners and community members from different US regions, only participants from four cities were included, so these findings may not be generalisable to other cities. Of note, lack of generalisability is only a minor limitation, given that generalisability is not a primary goal of qualitative research.
Conclusions
We identified several key themes that highlighted the needs of policymakers, and provide recommendations to facilitate the use of models and similar tools in local policy contexts. To optimise acceptance of local economic modelling and enhance the relevance of simulation model findings to policymakers, scientists should engage policymakers and other community stakeholders at multiple stages of the research to develop an early understanding of the local landscape for both budget and public health policy decisions. When sharing findings, researchers should consider emphasising practical and feasible considerations (such as real-world budget constraints, resource requirements for implementation, and conflicting community interests), and investigating the impact of other locally-tailored interventions prioritised by stakeholders, in addition to describing optimal benefits. Lastly, helping policymakers understand data requirements for models can prompt prospective data collection for local interventions and improve coordination of HIV prevention and care activities through using implementation data to identify service gaps and to estimate potential long-term impacts of the next dollar spent.
Notes
Corresponding author.
Supplementary Appendix 1: https://osf.io/y9ka6.
Supplementary Appendix 2: https://osf.io/q7avb.
Funding
This work was supported by the National Institutes on Drug Abuse under NIDA grant [no. R01DA041747; 3R01DA041747-05S1]. Cherish Grant support: https://urldefense.com/v3/__https:/pubmed.ncbi.nlm.nih.gov/?sort=date&term=P30*DA040500*2FDA*2FNIDA*NIH*HHS*2FUnited*States*5BGrant*Number*5D&sort_order=desc__;KyUlKyslKyUrJQ!!LLK065n_VXAQ!mrr-7mCYTSGnK7LOEtPvdCmtU5R6J5y5JVucGYORG43hQbt8G3Se3pPadimOJgTFL6p4ZIzcRBKjSiVCruOFqxXv$ P30 DA040500/DA/NIDA NIH HHS/United States.
Research ethics
This study was approved by the Simon Fraser University Office of Research Ethics and the Providence Health Care Research Ethics Board (H16-00652).
Contributor statement
MP, EB, EK, XJ, and BN developed the localised economic model. One- to two-page summary reports were prepared by ZM and JW. JW, ZM, BN developed the moderator questions for the focus group with revisions provided by MP, EB, EK, and XJ. JW, ZM, and BN attended and/or moderated the focus groups. MA lead the qualitative data analysis including development of a codebook. MA, ZM, and JW coded the data and identified themes. MA, ZM, and JW conducted data analysis and interpretation, with contributions from BN, MP, EB, EK, and XJ. MA with significant edits and contributions from ZM and JW wrote the first and subsequent drafts of the manuscript, with comments and additions from BN, MP, EB, EK, and XJ. All authors approved of the final version of the manuscript.
Conflict of interest
The authors declare that there is no conflict of interest.
References
Anderson, S.J. et al. (2014) Maximising the effect of combination HIV prevention through prioritisation of the people and places in greatest need: a modelling study, The Lancet, 384: 249–56, doi: 10.1016/S0140-6736(14)61053–9.
Ashcraft, L.E., Quinn, D.A. and Brownson, R.C. (2020) Strategies for effective dissemination of research to United States policymakers: a systematic review, Implementation Science, 15(1): 89, doi: 10.1186/s13012-020-01046-3.
Brownell, K.D. and Roberto, C.A. (2015) Strategic science with policy impact, The Lancet, 385(9986): 2445–46, doi: 10.1016/S0140-6736(14)62397-7.
Collins, C.B. Jr, Sapiano, T.N. (2016) Lessons learned from dissemination of evidence-based interventions for HIV prevention, American Journal of Preventive Medicine, 51(4): S140–S147, doi: 10.1016/j.amepre.2016.05.017.
Enns, B., Krebs, E. and Nosyk, B. (2021) Budget impact of local strategies to end the HIV epidemic on federal, state and public health department funding, Society of Medical Decision Making Oral Abstract 2021 Meeting, https://smdm.org/meeting/page/oral-abstract-2021-meeting/43rd-annual-north-american-meeting.
Enns, B., Min, J.E., Panagiotoglou, D., Montaner, J.S. and Nosyk, B. (2019) Geographic variation in the costs of medical care for people living with HIV in British Columbia, Canada, BMC Health Services Research, 19(1): 1–11, doi: 10.1186/s12913-019-4391-8.
Fauci, A.S., Redfield, R.R., Sigounas, G., Weahkee, M.D. and Giroir, B.P. (2019) Ending the HIV epidemic: a plan for the United States, JAMA, 321(9): 844–45, doi: 10.1001/jama.2019.1343.
Funk, T., Sharma, T., Chapman, E. and Kuchenmüller, T. (2022) Translating health information into policy-making: a pragmatic framework, Health Policy, 126(1): 16–23, doi: 10.1016/j.healthpol.2021.10.001.
Gagliardi, A.R., Berta, W., Kothari, A., Boyko, J. and Urquhart, R. (2015) Integrated knowledge translation (IKT) in health care: a scoping review, Implementation Science, 11: 38, doi: 10.1186/s13012-016-0399-1.
Health Resources and Services Administration (2021) Integrated HIV prevention and care plan guidance, including the statewide coordinated statement of need, CY 2022–2026, Division of HIV/AIDS prevention, National Center for HIV/AIDS, viral hepatitis, STD, and TB prevention, https://ryanwhite.hrsa.gov/sites/default/files/ryanwhite/grants/integrated-hiv-dear-college-6-30-21.pdf.
Krebs, E. and Nosyk, B. (2021) Cost-effectiveness analysis in implementation science: a research agenda and call for wider application, Current HIV/AIDS Reports, 18(3): 176–85, doi: 10.1007/s11904-021-00550-5.
Krebs, E. et al. (2019) Developing a dynamic HIV transmission model for 6 US cities: an evidence synthesis, PloS One, 14(5): e0217559, doi: 10.1371/journal.pone.0217559.
Krebs, E. et al. (2020) The impact of localized implementation: determining the cost-effectiveness of HIV prevention and care interventions across six US cities, AIDS, 34(3): 447–458, doi: 10.1097/QAD.0000000000002455.
Lavis, J.N. (2006) Research, public policymaking, and knowledge‐translation processes: Canadian efforts to build bridges, Journal of Continuing Education in the Health Professions, 26(1): 37–45, doi: 10.1002/chp.49.
Lavis, J.N., Robertson, D., Woodside, J.M., McLeod, C.B., Abelson, J. and Knowledge Transfer Study Group (2003) How can research organizations more effectively transfer research knowledge to decision makers?, Milbank Quarterly, 81(2): 221–48, doi: 10.1111/1468-0009.t01-1-00052.
Lavis, J.N., Ross, S.E., Hurley, J.E., Hohenadel, J., Stoddart, G.L., Woodward, C. and Abelson, J. (2002) Examining the role of health services research in public policymaking, Milbank Quarterly, 80(1): 125–54, doi: 10.1111/1468-0009.00005.
Mitton, C., Adair, C.E., McKenzie, E., Patten, S.B. and Perry, B.W. (2007) Knowledge transfer and exchange: review and synthesis of the literature, Milbank Quarterly, 85(4): 729–68, doi: 10.1111/j.1468-0009.2007.00506.x.
Nosyk, B. et al. (2020a) ‘Ending the epidemic’ will not happen without addressing racial/ethnic disparities in the United States human immunodeficiency virus epidemic, Clinical Infectious Diseases, 71(11): 2968–71, doi: 10.1093/cid/ciaa566.
Nosyk, B. et al. (2020b) Dissemination science to advance the use of simulation modeling: our obligation moving forward, Medical Decision Making, 40(6490): 718–21, doi: 10.1177/0272989X20945308.
Nosyk, B. et al. (2020c) Ending the HIV epidemic in the USA: an economic modelling study in six cities, Lancet HIV, 7(7): e491–e503, doi: 10.1016/S2352-3018(20)30033-3.
Nosyk, B., Krebs, E., Eyawo, O., Min, J., Barrios, R. and Montaner, J. (2014) Cost-effectiveness analysis along the continuum of HIV care: how can we optimize the effect of HIV treatment as prevention programs?, Current HIV/AIDS Reports, 11(4): 468–78, doi: 10.1007/s11904-014-0227-7.
Nosyk, B., Min, J.E., Lima, V.D., Hogg, R.S., Montaner, J.S.G. and STOP HIV/AIDS study group (2015) Cost-effectiveness of population-level expansion of highly active antiretroviral treatment for HIV in British Columbia, Canada: a modelling study, Lancet HIV, 2(9): e393–400, doi: 10.1016/S2352-3018(15)00127-7.
Owens, D.K. et al. (2019) Preexposure prophylaxis for the prevention of HIV infection: US Preventive Services Task Force recommendation statement, JAMA, 321(22): 2203–13, doi: 10.1001/jama.2019.639.
Panagiotoglou, D. et al. (2018) Building the case for localized approaches to HIV: structural conditions and health system capacity to address the HIV/AIDS epidemic in six US cities, AIDS and Behavior, 22(9): 3071–82, doi: 10.1007/s10461-018-2166-6.
Purtle, J., Nelson, K.L., Gebrekristos, L., Lê-Scherban, F. and Gollust, S.E. (2022) Partisan differences in the effects of economic evidence and local data on legislator engagement with dissemination materials about behavioral health: a dissemination trial, Implementation Science, 17: 38, doi: 10.1186/s13012-022-01214-7.
Quan, A. et al. (2021) Improving health equity and ending the HIV epidemic in the United States: a distributional cost-effectiveness analysis in six cities, Lancet HIV, 8(9): e581–e590, doi: 10.1016/S2352-3018(21)00147-8.
Sanders, G.D. et al. (2016) Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine, JAMA, 316(10): 1093–1103, doi: 10.1001/jama.2016.12195.
Tong, A., Sainsbury, P. and Craig, J. (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups, International Journal for Quality in Health Care, 19(6): 349–57, doi: 10.1093/intqhc/mzm042.
Wang, L., Min, J.E., Zang, X., Sereda, P., Harrigan, R.P., Montaner, J.S.G. and Nosyk, B. (2017) Characterizing human immunodeficiency virus antiretroviral therapy interruption and resulting disease progression using population-level data in British Columbia, 1996–2015, Clinical Infectious Diseases, 65(9): 1496–1503, doi: 10.1093/cid/cix570.
Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 426–31, doi: 10.2307/3109916.
Zang, X. et al. (2020) Development and calibration of a dynamic HIV transmission model for 6 US cities, Medical Decision Making, 40(1): 3–16, doi: 10.1177/0272989X19889356.
