To what extent does evidence support decision making during infectious disease outbreaks? A scoping literature review

Authors:
Andreea SalajanEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Andreea Salajan in
Current site
Google Scholar
Close
,
Svetla TsolovaEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Svetla Tsolova in
Current site
Google Scholar
Close
,
Massimo CiottiEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Massimo Ciotti in
Current site
Google Scholar
Close
, and
Jonathan E. SukEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Jonathan E. Suk in
Current site
Google Scholar
Close
Open access
Get eTOC alerts
Rights and permissions Cite this article

Background:

Infectious disease outbreaks require decision makers to make rapid decisions under time pressure and situations of scientific uncertainty, and yet the role of evidence usage in these contexts is poorly understood.

Aims and objectives:

To define and contextualise the role of scientific evidence in the governance of infectious disease outbreaks and to identify recommendations for overcoming common barriers to evidence-informed decision making.

Methods:

A scoping review and an expert workshop to provide additional input into recommendations on enhancing evidence uptake during infectious disease outbreaks taking place in European settings.

Findings:

Forty-nine records reporting on multiple decision-making processes during infectious disease outbreaks of the past ten years were included in the study. Decision makers prioritise expert advice, epidemiological data and mathematical modelling data for risk characterisation and management, but tend to be challenged by scientific uncertainties, which allow for conflicting interpretations of evidence and for public criticism and contestation of decision-making processes. There are concrete opportunities for optimising evidence usage to improve public health policy and practice through investment in decision-making competencies, relationship building, and promoting transparent decision-making processes.

Discussion and conclusions:

It is not necessarily a disregard of evidence that puts a strain on decision making in health crises, but rather competing interests and the lack of clear, unambiguous and rapidly available evidence for risk characterisation and effectiveness of response measures.

The relationship between science and public health decision making is relatively understudied but is deserving of greater attention, so as to ensure that the pursuit of evidence for decision making does not challenge timely and effective crisis management.

Abstract

Background:

Infectious disease outbreaks require decision makers to make rapid decisions under time pressure and situations of scientific uncertainty, and yet the role of evidence usage in these contexts is poorly understood.

Aims and objectives:

To define and contextualise the role of scientific evidence in the governance of infectious disease outbreaks and to identify recommendations for overcoming common barriers to evidence-informed decision making.

Methods:

A scoping review and an expert workshop to provide additional input into recommendations on enhancing evidence uptake during infectious disease outbreaks taking place in European settings.

Findings:

Forty-nine records reporting on multiple decision-making processes during infectious disease outbreaks of the past ten years were included in the study. Decision makers prioritise expert advice, epidemiological data and mathematical modelling data for risk characterisation and management, but tend to be challenged by scientific uncertainties, which allow for conflicting interpretations of evidence and for public criticism and contestation of decision-making processes. There are concrete opportunities for optimising evidence usage to improve public health policy and practice through investment in decision-making competencies, relationship building, and promoting transparent decision-making processes.

Discussion and conclusions:

It is not necessarily a disregard of evidence that puts a strain on decision making in health crises, but rather competing interests and the lack of clear, unambiguous and rapidly available evidence for risk characterisation and effectiveness of response measures.

The relationship between science and public health decision making is relatively understudied but is deserving of greater attention, so as to ensure that the pursuit of evidence for decision making does not challenge timely and effective crisis management.

Key messages

  • Challenges to evidence-informed decision making during infectious disease outbreaks are numerous but understudied

  • Scientific uncertainty often challenges decision making and facilitates the contestation of expertise

  • Political, economic and media pressure impact technical decision making during outbreaks

  • Knowledge transfer can be enhanced by collaborative risk governance networks and processes

Introduction

Public health emergencies (PHEs) are situations that require immediate response actions. Decisions made during an emergency may have longstanding implications for global health. Moreover, public health emergencies, as evidenced by recent outbreaks such as COVID-19, pandemic influenza, Zika and Ebola virus disease (EVD) outbreaks in West Africa and the Democratic Republic of the Congo, tend to have highly uncertain trajectories and can be disruptive to society (UNDG, 2015; ECDC, 2018; WHO, 2019). Although the public health response to emergencies is expected to be both timely and informed by the best available evidence (Palmer et al, 2013), the usage of evidence during health emergencies is poorly understood. In other branches of public health, an emphasis on the usage of evidence in decision-making processes has gained substantial traction in recent years. The concept of evidence-based medicine (EBM) is strongly entrenched; sustained by financial and political support and pursued by academia and governments, evidence-based policy more generally is an established domain within public health and beyond (Young et al, 2002; Oliver et al, 2014).

To our knowledge, there are no comprehensive reviews of the available literature on evidence-informed decision making during health emergencies. Here, we present the findings from a scoping review designed to map the existing peer-reviewed and grey literature on evidence-informed decision making during one particular type of health emergency, infectious disease outbreaks. This scoping review focused on decision-making processes taking place at organisational, international, national, sub-national or local levels during infectious disease outbreaks. Due to the varying decision-making contexts and structures across countries and public health systems, this review identifies generally applicable insights from the literature and also identifies barriers to the use of evidence in decision making. Finally, based upon this review’s findings as well as the insights from an expert workshop, we present potential solutions for enhancing evidence-informed decision making in the context of infectious disease outbreaks.

Methods

This scoping review identified peer-reviewed and grey literature on the topic of evidence-informed decision making during infectious disease outbreaks. We focused on identifying reported infectious disease outbreaks rather than emergencies, given that there was a wider degree of subjectivity with the latter term. The geographical region of primary interest was Europe (EU Member States, European Economic Area countries and EU pre-accession countries). Europe, which has a high population density, is consistently identified as both an area of elevated risk for emerging infectious disease events and as a global region where there is a positive reporting bias of infectious disease events (that is, due to high efforts in disease reporting) (Allen et al, 2017). In addition, many European countries share similar governance and socioeconomic contexts.

This scoping review was guided by a methodological framework for scoping studies (Arksey and O’Malley, 2005) and by relevant items from the guidance for conducting scoping reviews developed by the Joanna Briggs Institute (Peters et al, 2015). As recommended, a study protocol was developed in advance to guide the process (Salajan et al, 2020a).

Research questions

This scoping review focused on addressing two specific questions (Salajan et al, 2020a):

(RQ 1) How is evidence used to inform decision making during infectious disease outbreaks in European and analogous settings?

(RQ 2) What can be done to enhance the use of evidence in decision making during infectious disease outbreaks?

Search strategy

From an initial broad search of the literature in May 2018, selected keywords and index terms were extracted from relevant papers to develop a search strategy for the electronic databases Scopus, Embase and Google Scholar in June 2018. In addition, we searched Google’s Advanced Search Interface for relevant grey literature. A technical report on best practice recommendations for conducting after-action reviews (AARs) (ECDC, 2018) was furthermore scanned for eligible reports. By applying pearl growing (or citation tracking) as a secondary search method (Greenhalgh and Peacock, 2005), we searched the reference lists of shortlisted articles and reports and then searched for articles that have cited the shortlisted papers in the electronic databases and checked them for eligibility. All searches were limited to abstracts published in English and articles published in English or German, in or after 2009, when the first influenza pandemic of the 21st century was declared. A logic grid with selected keywords and index terms is provided, together with the search strings for each electronic database as supplementary material (Salajan et al, 2020b).

Inclusion and exclusion criteria

Eligibility of both peer-reviewed and grey literature was determined according to a set of pre-defined inclusion and exclusion criteria, and supported by the guiding questions developed in advance for the data extraction process (Salajan et al, 2020a). One person sifted all records, and a second researcher reviewed the final set of sifted studies prior to proceeding to the analysis of findings. The study selection process was an iterative process guided by the broad, exploratory questions of this scoping review, and further narrowed down by the eligibility criteria that helped us identify case studies and evaluations of infectious disease outbreaks in the European or similar contexts, which address or explore how evidence is used in decision-making processes.

All types of literature and methodologies, except for literature reviews, student theses and news items, were eligible for inclusion if published in English or German between 2009 and June 2018, when the last searches were conducted. The geographical region of interest was Europe (EU Member States, European Economic Area countries and EU pre-accession countries); however, for data saturation, reports from the US, Canada and Australia were also included, as these countries are broadly comparable to EU Member States in terms of socioeconomic development and systems of government.

The focus of the present paper was on infectious disease outbreaks, and other types of health emergencies or crises were thus excluded. Articles reporting on decision making in non-emergency contexts, such as longer-term policy-making processes, were excluded. We included case studies, evaluation reports, review articles and commentaries on previous disease outbreak events at the international, national or sub-national level. Decision making in clinical settings, decision making by emergency medical responders, decision making for capacity building and resource management, and decision making for individual health decisions were excluded, as these aspects of decision making were beyond the scope of this study. The search strategy yielded several articles on the topic of policy making for non-communicable diseases as well as for seasonal influenza and HIV. However, in order to focus on crisis decision making in acute disease outbreak settings rather than the longer-term process of public health policy making, these articles were excluded.

Literature on the general process of evidence-informed policy making within public health or elsewhere, without any reference to a disease outbreak context, was excluded. We also excluded papers introducing theoretical models and frameworks for preparedness planning, evidence generation, surveillance or decision making, unless supported by empirical data from infectious disease outbreaks. In addition, studies and reports covering disease outbreaks from a strictly veterinary or environmental perspective were excluded, as were risk assessments and surveillance reports if they did not reflect upon evidence use in decision making.

Data management and extraction

All records were imported to and managed with EndNote. Duplicates were removed both electronically supported by EndNote and manually. Study characteristics and other relevant data related to the research questions of this review were extracted using a uniform data extraction sheet, which had been developed in advance (Figure 1; Salajan et al, 2020a). Formal quality assessment was not undertaken, as the aim of this scoping review was to summarise and interpret key findings emerging from the body of literature, rather than to assess or compare the methodological quality of the included studies and grey literature.

Figure 1:
Figure 1:

PRISMA flow diagram

Citation: Evidence & Policy 16, 3; 10.1332/174426420X15808913064302

Incorporating input from expert consultation

An Expert Workshop was hosted by ECDC on ‘The use of evidence in decision making during public health emergencies’ on 5–6 December 2018, which brought together public health scientists, public practitioners with experience in managing health crises, and social and political scientists (ECDC, 2019). A preliminary version of this study was presented at the workshop, alongside other relevant work and case studies from selected practitioners. In break-out sessions, participants were then asked, based on the presentations during the workshop, to identify a series of key recommendations. In the Recommendations section and Table 2, the recommendations highlighted by this workshop (ECDC, 2019) that had also been identified in this scoping review are presented.

Results

Summary of results

A total number of 2,842 items were found via electronic searches and by pearl growing methods. An additional 12 items were identified from an ECDC technical report on best practice recommendations for conducting AARs (ECDC, 2018). After removal of duplicates, titles and abstracts of 2,474 unique records were screened for eligibility on the basis of the study selection criteria. Following the initial screening, we retained 110 items for full-text eligibility assessment in the next step. Based on the full-text assessment of eligibility and relevance, we included 49 records in our analysis (Figure 1; Salajan et al, 2020c).

The majority of the included articles were published in peer-reviewed journals (n=37) and twelve evaluation reports were retrieved from non-peer-reviewed sources. Most of the peer-reviewed articles utilised qualitative study designs such as interviews, observations, qualitative document analyses; or were reflections, reviews and opinion pieces (n=34). The remaining peer-reviewed articles utilised quantitative (n=1) or mixed methods (n=2).

Various complex decision-making processes at international, national, sub-national and local level were studied – 26 articles and reports covered decision-making processes in Europe, while the rest focused on the global perspective or non-European contexts. All 49 records covered infectious disease outbreaks, however, the 2009 influenza pandemic was by far the most common health emergency setting (n=39). Other examples included outbreaks of SARS, Ebola, E. coli, Q fever and Schmallenberg virus.

The findings were grouped into three main themes, each of which is defined by several sub-themes as shown in Table 1 and illustrated by various examples from the reviewed literature, as presented below in greater detail. It should be noted that, in this scoping review, no quality assessment of the reviewed literature was undertaken. The examples and conclusions presented under each sub-theme reflect the views and opinions of the authors of the selected papers and not those of the authors of this study.

Table 1:

Themes and sub-themes related to evidence-informed decision making in infectious disease outbreaks

Themes Sub-themes
Expertise and decision-making processes in infectious disease outbreaks •Defining evidence for decisionmaking in the context of disease outbreaks
•The role of public health experts
•Feasibility of scientific advice
The impact of uncertainty on decision making in infectious disease outbreaks •Conceptualisations of science and uncertainty
•Coping mechanisms: precautionary measures and adherence to preparedness plans
Extra-scientific factors influencing decision-making processes in infectious disease outbreaks •Institutional and legal structures
•Political influences and power dynamics
•Conflicts of interest
•Media influence
•Experiences and lessons learned from past disease outbreaks

Expertise and decision-making processes during infectious disease outbreaks

Defining evidence for decision making in the context of disease outbreaks

While there was no standard definition of ‘good evidence’ for decision making, data-driven decision making was prioritised in different disease outbreak settings. Real-time epidemiological data and data from mathematical models was utilised for taking decisions on response measures (Association of State and Territorial Health Officials, 2010; Flynn, 2010; Krause et al, 2010; Rizzo et al, 2010; Weeramanthri et al, 2010; Directorate for Civil Protection and Emergency Planning, 2011; Hanquet et al, 2011; Van Kerkhove and Ferguson, 2012; Van Der Sande et al, 2013; Baekkeskov, 2014; Huizer et al, 2015; Abeysinghe, 2017). The literature reports that several European countries relied on modelling data to inform response decisions during the influenza pandemic (Hine, 2010; Rizzo et al, 2010; Greco et al, 2011; Hanquet et al, 2011; Barker, 2012; Chambers et al, 2012; Van Kerkhove and Ferguson, 2012; Mansnerus, 2013; Van Der Sande et al, 2013). Some noted, however, that a high demand for robust science for decision making resulted in an over-reliance on modelling data without fully understanding its limitations (Hine, 2010; Barker, 2012; Chambers et al, 2012; Mansnerus, 2013). Meanwhile, a Canadian case study among mathematical modellers and senior public health professionals (Driedger et al, 2014), and a survey among health officials at local public health departments in California, argued that mathematical models were potentially underutilised during the influenza pandemic in those settings (Kayman et al, 2015).

As priority was given to evidence derived from the biomedical sciences in the response to the 2009 influenza pandemic, experiences from Europe, Canada, and from the WHO suggested that input from behavioural, social, and communication sciences was rarely considered for decision making (Health Quality Council of Alberta, 2010; Hine, 2010; WHO, 2011; House of Commons, 2011; Crosier et al, 2015). For example, in-depth interviews with social scientists and senior policy and communication officials in England, Italy and Hungary revealed that communication research was either unavailable or underutilised in the planning and implementation of communication strategies in response to the influenza pandemic (Crosier et al, 2015). In a series of papers focused on the global governance of pandemic influenza, it was argued that the governance of pandemic influenza has been heavily influenced by evidence-based medicine and has been reinforced by particular groups of actors, including policy makers and the medical community. As such, biomedical knowledge and practice continue to dominate (pandemic) influenza-related policy (Kamradt-Scott 2012; 2013).

The role of public health experts

The important role of public health experts or scientific advisors in the response to different communicable disease threats was a recurring theme in the literature. Comparative case studies analysing H1N1 influenza response decisions in the Netherlands, Denmark, Sweden, and the US found that government-appointed public health experts steered policy decisions during the pandemic (Baekkeskov and Rubin, 2014; Baekkeskov 2016a; 2016b; Baekkeskov and Öberg, 2017). Varying national norms and beliefs among scientists were argued to have led to varying Dutch and Danish vaccination policy decisions (Baekkeskov, 2016b). One research article studied ‘sense-making’ among public health officials from organisations in France, Canada, the US, and at the supranational level found that interviewees did not always draw similar conclusions about the state of the influenza pandemic, despite having had access to the same sources of validated information (Keller et al, 2012). The article emphasised the complexity of on-going sense-making processes among public health officials, who relied on multiple sources of information and personal experience to inform their decisions as the pandemic unfolded (Keller et al, 2012).

Similar to national public health experts, international public health organisations or agencies issued advice, guidance and recommendations during the 2009 H1N1 pandemic (Greco et al, 2011; Hanquet et al, 2011; Health Protection Agency, 2010; Baekkeskov, 2014). In fact, data from the H1N1 pandemic suggested that some international expert institutions even struggled to respond to the large volume of information requests from both their stakeholders and the mass media (Greco et al, 2011; Baekkeskov, 2014). Although expert advice was in high demand, the guidance provided by international expert organisations was not implemented consistently at country level (Greco et al, 2011; Hanquet et al, 2011). In a review of the ECDC response to the influenza pandemic of 2009/2010, it is noted that the ECDC published a guidance document which clearly described the scientific evidence against the effectiveness of containment measures outside confined settings. However, it is noted that there was no evidence that this guidance had been taken into account by most Member States (Greco et al, 2011).

Additional reflections on vaccination policy decisions for pandemic influenza in Europe were gathered at a ‘lessons learnt’ workshop organised by the Belgian Medicine Agency and the Belgian Inter-Ministers Influenza Cell and held in Brussels (Hanquet et al, 2011). Representatives from the European Commission (EC), the European Medicine Agency (EMA), the WHO, and the ECDC were joined by representatives from seven European countries (Belgium, Germany, Hungary, Italy, the Netherlands, Sweden, and the UK) at the workshop (Hanquet et al, 2011). According to the conference report, the international organisations (WHO, ECDC and EC) issued evidence-based recommendations to help standardise national policies in Europe but, nonetheless, the policy outcomes varied among EU Member States (Hanquet et al, 2011). However, in an official assessment of the pandemic response at the EC and Member State level serviced by the EC, it was reported that several Member States expected policy guidance, technical advice and EU-wide coordination from the EC and its expert agencies (ECDC and EMA) during the pandemic (Health Protection Agency, 2010).

Feasibility of implementing scientific advice

Concerns about the feasibility and compatibility of scientific advice with the practical reality of policy making – meaning its practical, ethical, legal and social implications – were often weighed against its public health benefits in different situations (Hine, 2010; Klaiman et al, 2011; Stein et al, 2011; WHO, 2011; Van Kerkhove and Ferguson, 2012; Van Der Sande et al, 2013; Kamradt-Scott, 2013; Roodenrijs et al, 2014; Kamradt-Scott, 2016; Kayman et al, 2015). In a survey among California public health officials involved in school closure decisions during the 2009 influenza pandemic, approximately 90% of survey participants stated that considerations of accountability, administrative feasibility, proportionality and legality played a role in their decisions (Kayman et al, 2015). A US study drawing on several types of data – primary documentation, interviews, and a two-day workshop in July 2009 at the University of California Berkeley with officials within infectious disease response organisations in Canada, France, the US and, at the supranational level, public health officials involved in the 2009 pandemic – found that ‘contextualised’ information was perceived as important during the pandemic: study participants reported seeking out additional sources of information, such as from informal networks, to access richer or off-the-record information about the context in which the outbreak took place, even if it increased their workload (Keller et al. 2012).

In the Netherlands, evidence-based advice during the H1N1 pandemic was first evaluated for feasibility, practicality and compatibility by an administrative body, before it was presented to the ministries in charge (Stein et al, 2011). Another Dutch study investigated the information needs for decision making during infectious disease outbreaks (Huizer et al, 2015). From interviews with public health officials involved in risk assessment and policy making, the authors concluded that political, economic and technical factors, as well as societal context and the feasibility of scientific advice, should be considered for decision making (Huizer et al, 2015). Yet another case study compared the actual risk management of Q fever and Schmallenberg virus outbreaks in the Netherlands with the hypothesised application of the International Risk Governance Council (IRGC)-framework to assess its acceptability and added value (Roodenrijs et al, 2014). To strengthen disease control of ambiguous infectious disease risks through integrated risk governance, it was suggested that concern assessments and careful stakeholder involvement in risk management, in addition to an explicit risk characterisation, could be incorporated to improve the perceived effectiveness and public acceptance (Roodenrijs et al, 2014).

The impact of uncertainty on decision making in infectious disease outbreaks

Conceptualisations of science and uncertainty

A number of case studies illustrated that underlying conceptualisations and understandings of science and uncertainty played a role during decision-making processes in the context of infectious disease outbreaks (Upshur, 2012; Rosella et al, 2013; Müller-Seitz and Macpherson, 2013; Roodenrijs et al, 2014; Baekkeskov and Rubin, 2014; Huizer et al, 2015; Baekkeskov 2016a; 2016b; Abeysinghe 2017; 2014). One Canadian study explored the underlying ideologies for evidence use by decision makers during the 2009 H1N1 influenza pandemic (Rosella et al, 2013). Through policy analysis and interviews with authorities – including government officials, medical health officers, H1N1 scientific advisory group members and clinical experts – from five Canadian provinces, the research article explored the underlying decision-making processes for four response policies covering the topics of vaccination, school closures and personal protective equipment (Rosella et al, 2013). Authorities in favour of strictly evidence-based ideologies perceived greater challenges when evidence was uncertain or unavailable, compared to those who prioritised contextual or policy considerations in their decisions (Rosella et al, 2013).

During disease outbreaks, where information is incomplete and the available evidence base allowed for conflicting interpretations (Keller et al, 2012; Baekkeskov, 2016b), public health agencies have acted from varying epistemological perspectives and traditions. Drawing upon the distinction between ‘normal’ and ‘post-normal’ science (Funtowicz and Ravetz, 1993), a series of papers suggested that, in the midst of the 2009 influenza pandemic, ‘normal’ science was applied by the WHO while ‘post-normal’ science was applied by the Council of Europe (Abeysinghe 2017; 2014). This led to diverging risk characterisations, which in turn led to a definitional and policy contestation between the two institutions (Abeysinghe 2017; 2014).

Coping mechanisms: precautionary measures and adherence to preparedness plans

In the UK, the planning for a pandemic was well developed, and the National Framework for responding to an influenza pandemic was key to facilitating a rapid response and reducing pressure on the decision-making process (Hine, 2010). Because the plan had been developed with avian H5N1 influenza in mind, the ‘reasonable worst-case scenario’ planning assumptions were insufficiently flexible and needed to be adapted to the circumstances at hand (Hine, 2010). Although the UK-wide strategy was deemed proportionate and effective by the independent review of the response of the UK governments to the H1N1 pandemic (Hine, 2010), the tendency to assume the worst-case scenario in the absence of information in an emergency setting was recognised (Hine, 2010). The review therefore recommended that outbreak response decision making should be proportionate to the perceived level of risk (Hine, 2010).

The Parliamentary Assembly of the Council of Europe criticised the WHO’s handling of the influenza pandemic in a report issued in June 2010, in which the proportionality and appropriateness of response measures were discussed (Flynn, 2010). In the document, the Assembly stated that far-reaching response measures had been justified on the grounds of the precautionary principle and the uncertainty surrounding the risk when, in fact, empirical evidence was pointing to a milder-than-expected pandemic (Flynn, 2010). Furthermore, the Assembly advised all public health authorities to review their use of the precautionary principle, as it can arguably contribute to anxiety and unease among the general population and be further exaggerated by the mass media (Flynn, 2010). Insights from Germany reflected upon how scientific uncertainty was handled by the German authorities during the influenza pandemic (Feufel et al, 2010). According to one analysis of the country’s response to the 2009 pandemic, measures such as the vaccination purchases were not commensurate with the available information at the time, but were made defensively by decision makers in order to shield themselves from future criticism (Feufel et al, 2010).

Extra-scientific factors influencing decision-making processes in infectious disease outbreaks

Institutional and legal structures

In the reviewed literature, national and international institutional and legal structures were listed as barriers because they were found to have hampered coordination and collaboration for effective decision making in disease outbreaks (Association of State and Territorial Health Officials, 2010; Feufel et al, 2010; Hine, 2010; Krause et al, 2010; Public Health Agency of Canada, 2010; Directorate for Civil Protection and Emergency Planning, 2011; Klaiman et al, 2011; Keller et al, 2012; Chambers et al, 2012; Rosella et al, 2013; Burkle and Hanfling, 2015). For example, the UK’s and Norway’s official H1N1 pandemic review reports found complex institutional and bureaucratic structures to have inhibited information flow and to have consequently slowed down the response (Hine, 2010; Directorate for Civil Protection and Emergency Planning, 2011). Another example from a review of key policy barriers faced by US decision makers at different levels of governance during the influenza pandemic highlighted how structural and legal policy barriers made it difficult to coordinate decision-making processes between federal and state levels of policy making (Association of State and Territorial Health Officials, 2010). The influence of legal structures was addressed in a conference report from a workshop on ‘Lessons learnt from the pandemic A(H1N1) influenza vaccination policies’ held in Brussels (Hanquet et al, 2011). In the document, it was concluded that legally binding advance purchase agreements with vaccine manufacturing companies had a great impact on decision making in European countries (Hanquet et al, 2011).

Additional challenges were attributed to the confusion over roles and responsibilities for decision making, and mechanisms for international collaboration in crisis contexts (Health Protection Agency, 2010; Stein et al, 2011; WHO, 2011; Müller-Seitz and Macpherson, 2013). The issue was addressed in the WHO’s ‘Report of the review committee on the functioning of the international health regulations (2005) in relation to pandemic (H1N1) 2009’ (WHO, 2011) and in the EC’s ‘Assessment report on the EU-wide response to pandemic (H1N1) 2009’ (Health Protection Agency, 2010). In the EC assessment report, a need to better clarify and communicate the roles and functions of the Commission and its agencies to Member States ahead of a disease outbreak was identified (Health Protection Agency, 2010). At country level, findings from Germany, Belgium and the Netherlands underline the importance of clearly defining roles and responsibilities for decision making in advance (Stein et al, 2011; ECDC, 2015; Müller-Seitz and Macpherson, 2013; Broekema et al, 2018). It was additionally argued that better collaboration with decision makers at the local level is needed to improve national outbreak responses in the future (Directorate for Civil Protection and Emergency Planning, 2011; Chambers et al, 2012).

Political influences and power dynamics

The literature suggests that certain risk management decisions during disease outbreaks were influenced by internal and external political factors. In spite of what turned out to be a milder pandemic scenario than the UK had been preparing for, it was decided to adhere to the pandemic influenza plan (Hine, 2010; Chambers et al, 2012). In the absence of clear audit trails behind the decisions made during the pandemic, the decision to follow pre-established policies in the UK was understood by the authors of these reports to have been influenced by political considerations (Hine, 2010; Chambers et al, 2012). An article drawing upon multiple case studies, interviews and workshops with public health officials involved in the H1N1 pandemic response across the globe reported similar observations: many preparedness plans had been developed for a different influenza strain, H5N1, and were deemed inappropriate for the novel influenza virus that emerged in 2009 (Keller et al, 2012). Some interviewees argued they came under political pressure to follow the plans as written, and had to explain why responses diverged from the pre-made plans (Keller et al, 2012).

Another example of how political motives might have influenced response decisions is available through an analysis of public discourses on pandemic influenza vaccination policy in Denmark and Sweden (Baekkeskov and Öberg, 2017). The authors found that expertise and apparent consensus among experts with high status became governance tools through which policy alternatives were marginalised (Baekkeskov and Öberg, 2017). The authors concluded that, by following expert advice, politicians shielded themselves from criticism and blame that could endanger their future electability (Baekkeskov and Öberg, 2017). Other studies within the context of pandemic influenza also discussed the possibility of how blame avoidance or fear of taking political risks might influence decision makers’ actions in uncertain, emergency contexts (Baekkeskov and Rubin, 2014; Baekkeskov, 2016a). In a review of the events of the 2009 influenza pandemic in Germany, it was pointed out that political decision-making processes, as well as the underlying personal or political agendas of scientific advisors, remain mostly unknown (Stollorz, 2013).

In international organisations, the balancing act between science and politics put a strain on internal response capabilities (Cohen and Enserink, 2009; Baekkeskov, 2014; Kamradt-Scott, 2016). During the 2014 Ebola outbreak in West Africa, the WHO faced a political challenge as the organisation struggled to act upon its mandate without damaging the relationship with its Member States (Kamradt-Scott, 2016). In the critical review of the EVD outbreak and other global health emergencies, the author elaborated on how, due to criticism of ‘crying wolf’ over its response to the 2009 influenza pandemic, the WHO was reluctant to declare another Public Health Emergency of International Concern (PHEIC) (Kamradt-Scott, 2016). Political considerations seem also to have influenced response decisions in a different setting, as stated by a case study on the 2009 influenza pandemic response from the perspective of an international public health agency (Baekkeskov, 2014). Here, the authors draw upon real-time observations and in-depth interviews to argue that demonstrating capability and establishing a favourable reputation among particular audiences were factors that affected the agency’s response (Baekkeskov, 2014).

Discussion

This scoping review set out to identify the mechanisms through which evidence feeds into decision-making processes during infectious disease outbreaks. Our analysis of 49 records, many of which were empirical case studies and official evaluation reports, revealed that science is paramount to the management of infectious disease outbreaks. Another key finding is that evidence is one among many competing influences that define decision-making processes in an outbreak. Scientific uncertainty, institutional and legal structures, the media, political factors, and private sector influences can play a role in decision-making processes and evidence use. While we cannot make definite conclusions about the effect of evidence on decision outcomes, or untangle its impact from that of other competing influences based on the available literature, our observations are nonetheless in line with what is currently known about public health policy making: science is in practice just one piece of the policy-making puzzle (Bowen and Zwi, 2005; Bowen et al, 2009; Orton et al, 2011; Humphreys and Piot, 2012; Liverani et al, 2013). Nevertheless, infectious disease outbreaks present unique opportunities for the use of scientific input in decision-making processes, as high priority is given to research on the premise that policies and practices in support of health worldwide are expected to be grounded in the best scientific knowledge (WHO, 2012).

When evidence-driven decisions are to be made in infectious disease outbreaks, some degree of scientific uncertainty can be expected. Our findings reveal that it was not a disregard of evidence that put a strain on decision making in various contexts, but rather the lack of clear, unambiguous and rapidly available evidence for risk characterisation and effectiveness of response measures. The relationship between science and public health decision making should thus be carefully monitored, to ensure that the pursuit of evidence for decision making does not challenge timely and effective crisis management. By reviewing and, when applicable, comparing decision or policy outcomes among emergency settings, the aims and benefits of evidence-informed decision making within public health could be reinforced.

To facilitate effective knowledge transfer between scientific experts and decision makers, it is necessary to take decision makers’ context-specific information needs into account when developing evidence-based recommendations or advice. When and how the feasibility of scientific advice or the potential political, ethical, legal, and social considerations will be accounted for during the decision-making process should be agreed upon in advance, to manage expectations among stakeholders about the nature and intended purpose of scientific advice. Although excluded from this literature review, several decision support tools, frameworks and evaluation tools for evidence-informed decision making were identified during our search. Evaluations of the potential added value and applicability of such tools for evidence-informed decision making in highly dynamic and uncertain contexts, such as infectious disease outbreaks, could be a useful direction for future research. Other opportunities might emerge from harnessing the crisis management literature to better understand decision processes during crisis response.

When defining evidence for decision making in crisis contexts, it was suggested in some of the reviewed articles to re-evaluate the status quo, which currently prioritises biomedical research above other disciplines, such as social and political science. This is likely rooted in traditionalist approaches that advocate the use of epidemiology and simultaneously devalue the potential contribution of more interpretative research methods in public health (Baum, 1995). Public health problems can have complex social, economic, political, biological and environmental causes (Baum, 1995), and infectious disease outbreaks are known to be driven by multiple underlying factors, many of which are outside the scope of public health (Semenza et al, 2016). Positive experiences with multi-stakeholder collaboration gained from initiatives, such as the One Health approach (Brumme and Lee, 2012; Mackenzie et al, 2014), could be drawn upon to inspire relationship- and trust-building strategies among stakeholders involved in the development of scientific advice. Similarly, collaboration strategies could be targeted towards non-scientific stakeholders who play a role during infectious disease outbreaks, such as the media and local communities. Nevertheless, such initiatives require careful consideration of feasibility concerns and potential risks, such as the loss of consensus among experts.

Our analysis, complemented by existing literature on the role of political and institutional influences on evidence use for health policy (Liverani et al, 2013), illustrates that the process of evidence-informed decision making cannot be isolated from the political environments in which those decisions are taken. As science and politics are intertwined, it is necessary to study the politics of evidence-informed decision making in order to enhance the impact of evidence in the process. Institutional barriers and confusion over mandates, staff roles and responsibilities in an outbreak can hamper a timely crisis response by inhibiting information flow, collaboration and coordination between key scientific actors and decision-making bodies, as demonstrated in this review. Overcoming these barriers at different levels of risk governance will require strong leadership to implement targeted solutions based on response evaluations. It would be worthwhile exploring the types of institutional structures and risk governance systems that best facilitate knowledge translation and evidence use for outbreak decision making during public health emergencies, and to share best practices where applicable. Preparedness activities such as simulation exercises can help strengthen response activities by testing collaboration, roles and responsibilities at various levels of governance in advance, and by strengthening relationships between scientific and decision-making counterparts (ECDC, 2014).

The case studies and evaluation reports included in our analysis comprised a heterogeneous sample of decision-making processes and authorities. Hence, we could not compare decision-making processes in different settings. In some contexts, decision-making authority lies within government-appointed public health experts and public health agencies, while in other settings political authorities rely on expert advice but are officially the ones in charge. At EU-level, the Health Security Committee (HSC) is an advisory group for health security in Europe, for which scientific advice, crisis management and coordination are considered priority issues (European Parliament and the Council of the European Union, 2013). During the pandemic H1N1 in 2009, the HSC played a key role in coordinating the response in the EU, and was a platform for sharing information on countermeasures and for coordinating communication activities. However, the terms of reference and legal basis of the HSC were strengthened following the H1N1 pandemic with the introduction of Decision 1082/2013/EU (74).

Recommendations

Based on the findings from this scoping review and with additional input from an expert workshop that linked social scientists with public health practitioners to examine the use of evidence in decision making during public health emergencies (ECDC, 2019), we have summarised three main recommendations and various examples that could potentially improve evidence-informed public health decision making in response to future infectious disease outbreaks (Table 2).

Table 2:

Recommendations for enhancing evidence-informed decision making during infectious disease outbreaks.

Recommendations Examples References
Develop decision-making competencies Training of decision makers and public health experts:
a.Develop a basis for evidence use among decision makers by promoting scientific literacy a. Hine, 2010; Van Kerkhove and Ferguson, 2012; Driedger et al, 2014; Burkle and Hanfling, 2015
b.Develop political literacy and understanding of the policy-making process among scientific experts b. ECDC (European Centre for Disease Prevention and Control), 2019
c.Develop knowledge translation and communication strategies for information exchange between scientific experts and decision makers c. Public Health Agency of Canada, 2010; Chambers et al, 2012; Rosella et al, 2013; Driedger et al, 2014; Burkle and Hanfling, 2015; Crosier et al, 2015; Huizer et al, 2015; Kilianski and Evans, 2015
Build relationships and trust for collaboration Build relationships and trust between key actors:
a.Enhance existing and build new networks to stimulate knowledge synergy and ensure the diversity of expertise a. Flynn, 2010; Hine, 2010; House of Commons, 2011; Kamradt-Scott, 2012; Keller et al, 2012; Driedger et al, 2014; Crosier et al, 2015; Broekema et al, 2018
b.Consider networking events (best practice workshops, post-event evaluations, simulation exercises) at regional, national and international level b. Kayman et al, 2015
c.Evaluate the desired level of dialogue between scientific experts and decision makers c. ECDC (European Centre for Disease Prevention and Control), 2019
d.Consider alternative risk governance structures and collaborative frameworks d. Public Health Agency of Canada, 2010; Weeramanthri et al, 2010; Chambers et al, 2012; Kamradt-Scott, 2012; Müller-Seitz, 2013; Driedger et al, 2014; Roodenrijs et al,2014; Burkle, 2015; Kamradt-Scott, 2016
e.Consider local and community engagement for preparedness and response e. Public Health Agency of Canada, 2010; Krause et al, 2010; Directorate for Civil Protection and Emergency Planning (DSB), 2011; House of Commons, 2011; Klaiman et al, 2011; Chambers et al, 2012; Roodenrijs et al, 2014; Huizer et al, 2015; Kayman and Logar, 2015
Promote transparency Promote transparent decision-making processes and evidence use:
a.Establish, divide and test strategic, operational and analytical tasks and mandates among key actors in advance a. Association of State and Territorial Health Officials (ASTHO), 2010; Health Quality Council of Alberta, 2010; Health Protection Agency, 2010; Hine, 2010; Krause et al, 2010; Directorate for Civil Protection and Emergency Planning (DSB), 2011; Greco et al, 2011; Klaiman et al, 2011; Upshur, 2012; Rosella et al, 2013; ECDC (European Centre for Disease Prevention and Control), 2015; Broekema et al, 2018
b.Improve knowledge resource availability through research and evaluation mechanisms b. Association of State and Territorial Health Officials (ASTHO), 2010; Health Quality Council of Alberta, 2010; Hanquet et al, 2011; WHO, 2011; Kamradt-Scott, 2012; Keller et al, 2012; Rosella et al 2013; Roodenrijs et al, 2014; ECDC (European Centre for Disease Prevention and Control), 2015; Huizer et al, 2015
c.Implement transparent risk communication strategies c. Feufel et al, 2010; Flynn, 2010; Hine, 2010; Krause et al, 2010; Public Health Agency of Canada, 2010; Directorate for Civil Protection and Emergency Planning (DSB), 2011; Hanquet et al, 2011; Klaiman et al, 2011; WHO, 2011; Van Kerkhove and Ferguson, 2012; Burkle and Hanfling, 2015; Huizer et al, 2015; Kayman and Logar, 2015; Kilianski and Evans, 2015
d.Reveal conflicts of interest d. Feufel et al, 2010; Flynn, 2010; House of Commons, 2011; WHO, 2011; Stollorz, 2013; Forcades Vila, 2015

These recommendations are in line with what are considered to be effective mechanisms for engagement between scientists and policy makers, according to a recent systematic review of advice to academics for influencing policy, which identified the engagement with and the understanding of the policy process, and the relationship building with policy makers, as top pieces of advice (Oliver and Cairney, 2019). Nevertheless, the authors remind us that there is no clear path for scientists to take when it comes to engaging with policy, but instead they must navigate several dilemmas and accept trade-offs (Oliver and Cairney, 2019). At the EU level, a Knowledge Management for policy (KPM) initiative has been introduced by the EC’s Joint Research Centre (JRC), with the aim of maximising the value and impact of research for evidence-informed policy making in the EU (Topp et al, 2018). To achieve this aim the JRC identifies eight KPM skills: synthesising research, managing expert communities, understanding policy and science, interpersonal skills, engaging with citizens and stakeholders, communicating scientific knowledge, monitoring and evaluation, and advising policy makers (Topp et al, 2018).

In practice, the realities of the politics and power dynamics underpinning decision making and knowledge production are too messy and complex to be solved by the aforementioned recommendations alone. The recommendations could be viewed as a useful starting point from which current public health emergency preparedness strategies should be re-evaluated, in favour of more collaborative approaches and increased transparency for the evidence-informed management of infectious disease outbreaks and other types of PHEs.

Limitations

This scoping review did not include a quality appraisal of the reviewed literature, and thus no synthesis of the results as it is standard for systematic literature reviews (Khan et al, 2003). In the absence of quality appraisals, this scoping review does not assess potential sources of bias from within the appraised studies. However, it is possible to speculate that the overall literature assessed in this study is subject to a reporting bias. As decision making during infectious disease outbreaks can have political and economic ramifications, it is possible that many reports that would have been of interest to this study have not surfaced to the public domain.

Following the established framework (Arksey and O’Malley, 2005) and guidelines for conducting scoping reviews (Peters et al, 2015), the process applied in this review was both structured and rigorous. Although the terminology used to describe the key concepts of this paper was specified in advance (Salajan et al, 2020a), some definitions, for example, ‘decision making’ and ‘policy making’, were used interchangeably, because there were no standardised definitions among the reviewed literature. Finally, the 49 references included in this review comprise a non-representative sample of the literature on evidence-informed decision making during disease outbreaks. Literature from non-English speaking countries was underrepresented, and the geographical focus on European and analogous settings minimises its generalisability to other contexts. In addition, the influenza pandemic was overrepresented among the disease outbreaks, and literature from areas other than infectious disease outbreaks might have added valuable insight into crisis decision making, but was not included in this review. In spite of these shortcomings, the present scoping review highlights a diverse sample of decision-making processes and settings. Our findings are to a large extent consistent across studies and existing literature on the topic. Almost half of the reviewed literature studied decision making from a global or non-European perspective, and this heterogeneity may increase the generalisability of the findings. By reviewing case studies and critical evaluations of past disease outbreaks, the present review collects valuable insight on crisis decision making, with implications for public health research and practice.

Conclusion

Focusing on infectious disease outbreaks in European and analogous settings, this scoping review highlights the challenges of decision making during highly dynamic and uncertain situations. This review contextualises the role of evidence among several competing influences that were present in decision-making processes in response to infectious disease outbreaks. Evidence use might depend upon factors such as uncertainty, feasibility, institutional structures and political pressures, but the fast-paced, non-linear nature of disease outbreaks challenges our ability to fully understand decision-making processes in those settings. Perspectives from crisis management research, political and social science, in addition to case studies and evaluations of decision processes and outcomes would allow for a broader and more in-depth understanding of the issue. The findings from this study make a strong case for investment in public health emergency preparedness strategies to ensure that the capacities and capabilities for evidence-informed decision making are developed well ahead of the next emergency. Priority actions for overcoming the challenges to evidence-informed decision making include capacity, skills and relationship building among key actors involved in the outbreak response, in addition to the promotion of transparent decision-making processes.

Funding

This work was conducted by staff at the European Centre for Disease Prevention and Control and did not involve any external funding.

Acknowledgements

We would like to thank all of the experts that participated in the ECDC workshop on ‘The use of evidence in decision making during public health emergencies’, which took place in Stockholm, Sweden on 5–6 December 2018. In addition, we would like to thank Karl Ekdahl, Mike Catchpole, Jeanine Pommier, Judit Takacs, and two anonymous reviewers for their constructive feedback on earlier versions of this manuscript.

Conflict of interest

The Authors declare that there is no conflict of interest.

References

  • Abeysinghe, S. (2014) An uncertain risk: the World Health Organization’s account of H1N1, Science in Context, 27(3): 51129. doi: 10.1017/S0269889714000167

    • Search Google Scholar
    • Export Citation
  • Abeysinghe, S. (2017) Contesting a pandemic: the WHO and the Council of Europe, Science as Culture, 26(2): 16184. doi: 10.1080/09505431.2016.1212825

    • Search Google Scholar
    • Export Citation
  • Allen, T., Murray, K.A., Zambrana-Torrelio, C., Morse, S.S., Rondinini, C., Di Marco, M., Breit, N., Olival, K.J. and Daszak, P. (2017) Global hotspots and correlates of emerging zoonotic diseases, Nature Communications, 8: 1142. doi: 10.1038/s41467-017-01293-x

    • Search Google Scholar
    • Export Citation
  • Arksey, H. and O’Malley, L. (2005) Scoping studies: towards a methodological framework, International Journal of Social Research Methodology, 8(1): 1932. doi: 10.1080/1364557032000119616

    • Search Google Scholar
    • Export Citation
  • Association of State and Territorial Health Officials (ASTHO) (2010) Assessing Policy Barriers to Effective Public Health Response in the H1N1 Influenza Pandemic. Project Report to the Centers for Disease Control and Prevention, Arlington,VA: ASTHO.

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2014) Reputation-seeking by a government agency in Europe: direct evidence from responses to the 2009 H1N1 “swine” influenza pandemic, Administration & Society, 49(2): 16389. doi: 10.1177/0095399714528177

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2016a) Explaining science-led policy-making: pandemic deaths, epistemic deliberation and ideational trajectories, Policy Sciences, 49(4): 395419. doi: 10.1007/s11077-016-9264-y

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2016b) Same threat, different responses: experts steering politicians and stakeholders in 2009 H1N1 vaccination policy-making, Public Administration, 94(2): 299315. doi: 10.1111/padm.12244

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. and Öberg, P. (2017) Freezing deliberation through public expert advice, Journal of European Public Policy, 24(7): 100626. doi: 10.1080/13501763.2016.1170192

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. and Rubin, O. (2014) Why pandemic response is unique: powerful experts and hands-off political leaders, Disaster Prevention and Management: An International Journal, 23(1): 8193. doi: 10.1108/DPM-05-2012-0060

    • Search Google Scholar
    • Export Citation
  • Barker, K. (2012) Influenza preparedness and the bureaucratic reflex: anticipating and generating the 2009 H1N1 event, Health & Place, 18(4): 7019.

    • Search Google Scholar
    • Export Citation
  • Baum, F. (1995) Researching public health: behind the qualitative-quantitative methodological debate, Social Science & Medicine, 40(4): 45968.

    • Search Google Scholar
    • Export Citation
  • Bowen, S. and Zwi, A. B. (2005) Pathways to “evidence-informed” policy and practice: a framework for action’, PLOS Medicine, 2(7): e166. doi: 10.1371/journal.pmed.0020166

    • Search Google Scholar
    • Export Citation
  • Bowen, S., Erickson, T., Martens, P. J. and Crockett, S. (2009) More than “using research”: the real challenges in promoting evidence-informed decision-making, Healthcare Policy, 4(3): 87102.

    • Search Google Scholar
    • Export Citation
  • Broekema, W., van Eijk, C. and Torenvlied, R. (2018) The role of external experts in crisis situations: a research synthesis of 114 post-crisis evaluation reports in the Netherlands, International Journal of Disaster Risk Reduction, 31: 209. doi: 10.1016/j.ijdrr.2018.03.021

    • Search Google Scholar
    • Export Citation
  • Brumme, Z. L. and Lee, K. (2012) Operationalizing the one health approach: the global governance challenges, Health Policy and Planning, 28(7): 77885.

    • Search Google Scholar
    • Export Citation
  • Burkle, F. M. and Hanfling, D. (2015) ‘Political leadership in the time of crises: primum non nocere’, PLoS Currents Disasters, May 29, Edition 1. doi: 10.1371/currents.dis.fd8aaf6707cd5dd252e33c771d08b949

    • Search Google Scholar
    • Export Citation
  • Chambers, J., Barker, K. and Rouse, A. (2012) Reflections on the UK’s approach to the 2009 swine flu pandemic: conflicts between national government and the local management of the public health response, Health & Place, 18(4): 73745.

    • Search Google Scholar
    • Export Citation
  • Cohen, J. and Enserink, M. (2009), After delays, WHO agrees: the 2009 pandemic has begun, Science, 324: 14967. doi: 10.1126/science.324_1496

    • Search Google Scholar
    • Export Citation
  • Crosier, A., McVey, D. and French, J. (2015) ‘“By failing to prepare you are preparing to fail”: lessons from the 2009 H1N1 “swine flu” pandemic’, European Journal of Public Health, 25(1): 1359. doi: 10.1093/eurpub/cku131

    • Search Google Scholar
    • Export Citation
  • Directorate for Civil Protection and Emergency Planning (DSB) (2011) New Influenza A Virus (H1N1): A Summary of a Study on the National Response in Norway, Tønsberg: DSB.

    • Search Google Scholar
    • Export Citation
  • Driedger, S. M., Cooper, E. J. and Moghadas, S. M. (2014) Developing model-based public health policy through knowledge translation: the need for a “communities of practice”, Public Health, 128(6): 5617. doi: 10.1016/j.puhe.2013.10.009

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2014) Handbook on Simulation Exercises in EU Public Health Settings: How to Develop Simulation Exercises Within the Framework of Public Health Response to Communicable Diseases, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2015) Ebola Emergency Preparedness in EU Member States: Conclusions From Peer-review Visits to Belgium, Portugal and Romania, Stockholm: ECDC. https://www.ecdc.europa.eu/sites/default/files/media/en/publications/Publications/ebola-emergency-preparedness-belgium-portugal-romania.pdf

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2015) Public Health Emergency Preparedness for Cases of Viral Haemorrhagic Fever (Ebola) in Belgium: A Peer Review, 16–19 March 2015, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2018) Best Practice Recommendations for Conducting After-action Reviews to Enhance Public Health Preparedness, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2018) Ebola Virus Disease Outbreak in North Kivu and Ituri Provinces, Democratic Republic of the Congo, Second Update, 21 December 2018, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2019) The Use of Evidence in Decision-making During Public Health Emergencies, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • European Parliament and the Council of the European Union (2013) Decision No 1082/2013/EU of the European Parliament and of the Council of 22 October 2013 on serious cross-border threats to health and repealing Decision No 2119/98/EC Text with EEA relevance, No 2119/98/EC, Official Journal of the European Union, pp 115.

  • Feufel, M.A., Antes, G. and Gigerenzer, G. (2010) Competence in dealing with uncertainty: lessons to learn from the influenza pandemic (H1N1) 2009, Bundesgesundheitsblatt, Gesundheitsforschung, Gesundheitsschutz, 53(12): 12839. doi: 10.1007/s00103-010-1165-1

    • Search Google Scholar
    • Export Citation
  • Flynn, P. (2010) The handling of the H1N1 pandemic: more transparency needed, in P. Assembly (ed), Strasbourg: Council of Europe, p 21.

  • Forcades Vila, T. (2015) Flu vaccination: the gap between evidence and public policy, International Journal of Health Services, 45(3): 45370. doi: 10.1177/0020731415585976

    • Search Google Scholar
    • Export Citation
  • Funtowicz, S.O. and Ravetz, J.R. (1993) Science for the post-normal age, Futures, 25(7): 73955. doi: 10.1016/0016-3287(93)90022-L

  • Greco, D., Stern, E. and Marks, G. (2011) Review of ECDC’s Response to the Influenza Pandemic 2009–2010, Stockholm: ECDC.

  • Greenhalgh, T. and Peacock, R. (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources, BMJ, 331(7524): 10645. doi: 10.1136/bmj.38636.593461.68

    • Search Google Scholar
    • Export Citation
  • Hanquet, G., Van Damme, P., Brasseur, D., De Cuyper, X., Gregor, S., Holmberg, M., Martin, R., Molnár, Z., Pompa, M. G., Snacken, R. et al. (2011) Lessons learnt from pandemic A(H1N1) 2009 influenza vaccination: highlights of a European workshop in Brussels, 22 March 2010, Vaccine, 29(3): 3707. doi: 10.1016/j.vaccine.2010.10.079

    • Search Google Scholar
    • Export Citation
  • Health Protection Agency (2010) Assessment Report on the EU-wide Response to Pandemic (H1N1) 2009, Salisbury: Health Protection Agency.

  • Health Quality Council of Alberta (2010) Review of Alberta’s Response to the 2009 H1N1 Influenza Pandemic, Edmonton: AB Government of Alberta.

    • Search Google Scholar
    • Export Citation
  • Hine, D. D. (2010) The 2009 Influenza Pandemic: An Independent Review of the UK Response to the 2009 Influenza Pandemic, London: Cabinet Office.

    • Search Google Scholar
    • Export Citation
  • House of Commons (2011) Scientific Advice and Evidence in Emergencies: Third Report of Session 2010–11, Vol. 1: Report, Together with Formal Minutes, Oral and Written Evidence, London: The Stationery Office.

    • Search Google Scholar
    • Export Citation
  • Huizer, Y.L., Kraaij-Dirkzwager, M.M., Timen, A., Schuitmaker, T.J. and Steenbergen, J. E.V. (2015) Context analysis for epidemic control in the Netherlands, Health Policy, 119(1): 6673. doi: 10.1016/j.healthpol.2014.10.004

    • Search Google Scholar
    • Export Citation
  • Humphreys, K. and Piot, P. (2012) Scientific evidence alone is not sufficient basis for health policy, BMJ, 344: e1316. doi: 10.1136/bmj.e1316

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2012) Evidence-based medicine and the governance of pandemic influenza, Global Public Health, 7(Suppl. 2): S11126.

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2013) The politics of medicine and the global governance of pandemic influenza, International Journal of Health Services, 43(1): 10521. doi: 10.2190/HS.43.1.h

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2016) WHO’s to blame? The World Health Organization and the 2014 Ebola outbreak in West Africa, Third World Quarterly, 37(3): 40118. doi: 10.1080/01436597.2015.1112232

    • Search Google Scholar
    • Export Citation
  • Kayman, H., Logar, T. (2015) A framework for training public health practitioners in crisis decision-making, Disaster Medicine and Public Health Preparedness, 10(1): 16573. doi: 10.1017/dmp.2015.149

    • Search Google Scholar
    • Export Citation
  • Kayman, H., Salter, S., Mittal, M., Scott, W., Santos, N., Tran, D. and Ma, R. (2015) School closure decisions made by local health department officials during the 2009 H1N1 influenza outbreak, Disaster Medicine and Public Health Preparedness, 9(4): 46471. doi: 10.1017/dmp.2015.72

    • Search Google Scholar
    • Export Citation
  • Keller, A.C., Ansell, C.K., Reingold, A.L., Bourrier, M., Hunter, M.D., Burrowes, S. and MacPhail, T.M. (2012) Improving pandemic response: a sensemaking perspective on the spring 2009 H1N1 pandemic, Risk, Hazards & Crisis in Public Policy, 3(2): 137. doi: 10.1080/19475705.2011.638027

    • Search Google Scholar
    • Export Citation
  • Khan, K.S., Kunz, R., Kleijnen, J. and Antes, G. (2003) Five steps to conducting a systematic review, Journal of the Royal Society of Medicine, 96(3): 11821. doi: 10.1177/014107680309600304

    • Search Google Scholar
    • Export Citation
  • Kilianski, A., Evans, N.G. (2015) Effectively communicating the uncertainties surrounding Ebola virus transmission, PLoS Pathogens, 11(10). doi: 10.1371/journal.ppat.1005097

    • Search Google Scholar
    • Export Citation
  • Klaiman, T., Kraemer, J.D. and Stoto, M.A. (2011) Variability in school closure decisions in response to 2009 H1N1: a qualitative systems improvement analysis, BMC Public Health, 11: 73. doi: 10.1186/1471-2458-11-73

    • Search Google Scholar
    • Export Citation
  • Krause, G., Gilsdorf, A., Becker, J., Bradt, K., Dreweck, C., Gärtner, B., Löwer, J., Marcic, A., Nicoll, A. and Pott, E. (2010) Erster Erfahrungsaustausch zur H1N1-Pandemie in Deutschland 2009/2010, Bundesgesundheitsblatt-Gesundheitsforschung-Gesundheitsschutz, 53(5): 5109. doi: 10.1007/s00103-010-1074-3

    • Search Google Scholar
    • Export Citation
  • Liverani, M., Hawkins, B. and Parkhurst, J. O. (2013) Political and institutional influences on the use of evidence in public health policy: a systematic review, PLoS One, 8(10): e77404. doi: 10.1371/journal.pone.0077404

    • Search Google Scholar
    • Export Citation
  • Mackenzie, J.S., McKinnon, M. and Jeggo, M. (2014) One health: from concept to practice, in A. Yamada, L.H. Kahn, B. Kaplan, T. P. Monath, J. Woodall and L. Conti (eds) Confronting Emerging Zoonoses: The One Health Paradigm, Tokyo: Springer Japan, pp. 16389.

    • Search Google Scholar
    • Export Citation
  • Mansnerus, E. (2013) Using model-based evidence in the governance of pandemics, Sociology of Health and Illness, 35(2): 28091. doi: 10.1111/j.1467-9566.2012.01540.x

    • Search Google Scholar
    • Export Citation
  • Müller-Seitz, G. and Macpherson, A. (2013) Learning during crisis as a “war for meaning”: the case of the German Escherichia coli outbreak in 2011, Management Learning, 45(5): 593608.

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Cairney, P. (2019) The dos and don’ts of influencing policy: a systematic review of advice to academics, Palgrave Communications, 5(1): 21. doi: 10.1057/s41599-019-0232-y

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Lorenc, T. and Innvær, S. (2014) New directions in evidence-based policy research: a critical analysis of the literature, Health Research Policy and Systems, 12(1). doi: 10.1186/1478-4505-12-34

    • Search Google Scholar
    • Export Citation
  • Orton, L., Lloyd-Williams, F., Taylor-Robinson, D., O’Flaherty, M. and Capewell, S. (2011) The use of research evidence in public health decision making processes: systematic review, PLOS ONE, 6(7): e21704.

    • Search Google Scholar
    • Export Citation
  • Palmer, S., Jansen, A., Leitmeyer, K., Murdoch, H. and Forland, F. (2013) Evidence-Based Medicine applied to the control of communicable disease incidents when evidence is scarce and the time is limited, Eurosurveillance, 18(25). doi: 10.2807/1560-7917.ES2013.18.25.20507

    • Search Google Scholar
    • Export Citation
  • Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D. and Soares, C. B. (2015) Guidance for conducting systematic scoping reviews, International Journal of Evidence-Based Healthcare, 13(3): 1416. doi: 10.1097/XEB.0000000000000050

    • Search Google Scholar
    • Export Citation
  • Public Health Agency of Canada (2010) Lessons Learned Review: Public Health Agency of Canada and Health Canada Response to the 2009 H1N1 Pandemic, Ottawa: Public Health Agency of Canada. https://www.canada.ca/content/dam/phac-aspc/migration/phac-aspc/about_apropos/evaluation/reports-rapports/2010-2011/h1n1/pdf/h1n1-eng.pdf

    • Search Google Scholar
    • Export Citation
  • Rizzo, C., Rota, M. C., Bella, A., Giannitelli, S., De Santis, S. D., Nacca, G., Pompa, M. G., Vellucci, L., Salmaso, S. and Declich, S. (2010) Response to the 2009 influenza A (H1N1) pandemic in Italy, Eurosurveillance, 15(49): 6. doi: 10.2807/ese.15.49.19744-en

    • Search Google Scholar
    • Export Citation
  • Roodenrijs, J.C. M., Kraaij-Dirkzwager, M. M., Van Den Kerkhof, J. H. T. C. and Runhaar, H. A. C. (2014) Risk governance for infectious diseases: exploring the feasibility and added value of the IRGC-framework for Dutch infectious disease control, Journal of Risk Research, 17(9): 116182. doi: 10.1080/13669877.2013.875935

    • Search Google Scholar
    • Export Citation
  • Rosella, L. C., Wilson, K., Crowcroft, N. S., Chu, A., Upshur, R., Willison, D., Deeks, S. L., Schwartz, B., Tustin, J., Sider, D. and Goel, V. (2013) Pandemic H1N1 in Canada and the use of evidence in developing public health policies: a policy analysis, Social Science and Medicine, 83: 19. doi: 10.1016/j.socscimed.2013.02.009

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020a) Scoping review protocol: exploring the role of evidence in decision-making during public health emergencies, Online Supplemental File 12: Scoping review search strategy and results. Study Protocol. Figshare, https://figshare.com/articles/Online_Supplemental_File_2_Search_Strategy_and_Results/11744316.

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020b): Online Supplemental File 2: Scoping review search strategy and results. Study Protocol. Figshare. https://figshare.com/articles/Online_Supplemental_File_2_Search_Strategy_and_Results/11744316

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020c): Online Supplemental 3: Table S1. Overview of the included peer-reviewed and grey literature. Figshare. Journal Contribution https://figshare.com/articles/Table_S1_Overview_of_the_included_peer-reviewed_and_grey_literature/11744340/1.

    • Search Google Scholar
    • Export Citation
  • Semenza, J. C., Lindgren, E., Balkanyi, L., Espinosa, L., Almqvist, M. S., Penttinen, P. and Rocklöv, J. (2016) Determinants and drivers of infectious disease threat events in Europe, Emerging Infectious Diseases, 22(4): 5819. doi: 10.3201/eid2204.151073

    • Search Google Scholar
    • Export Citation
  • Stein, M. L., van Vliet, J. A. and Timen, A. (2011) Chronological Overview of the 2009/2010 H1N1 Influenza Pandemic and the Response of the Centre for Infectious Disease Control of the National Institute for Public Health and Environment (RIVM), Bilthoven: National Institute for Public Health and Environment (RIVM). https://www.rivm.nl/bibliotheek/rapporten/215011006.pdf

    • Search Google Scholar
    • Export Citation
  • Stollorz, V. (2013) The pandemic of the experts in the mass media: how to create trust in public communication by acknowledging nescience and uncertainty, Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 56(1): 1107. doi: 10.1007/s00103-012-1581-5

    • Search Google Scholar
    • Export Citation
  • Topp, L., Mair, D., Smillie, L. and Cairney, P. (2018) Knowledge management for policy impact: the case of the European Commission’s Joint Research Centre, Palgrave Communications, 4(1): 87. doi: 10.1057/s41599-018-0143-3

    • Search Google Scholar
    • Export Citation
  • UNDG (United Nations Development Group) (2015) Socio-economic Impact of Ebola Virus Disease in West African Countries: A Call for National and Regional Containment, Recovery and Prevention, Dakar: UNDG https://www.undp.org/content/dam/rba/docs/Reports/ebola-west-africa.pdf.

    • Search Google Scholar
    • Export Citation
  • Upshur, R. E. (2012) Evidence and ethics in public health: the experience of SARS in Canada, New South Wales Public Health Bulletin, 23(5–6): 10810. doi: 10.1071/NB11044

    • Search Google Scholar
    • Export Citation
  • Van Der Sande, M. A. B., Jacobi, A., Meijer, A., Wallinga, J., Van Der Hoek, W. and Van Der Lubben, M. (2013) The 2009 influenza A (H1N1) pandemic: management and vaccination strategies in the Netherlands, Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 56(1): 6775. doi: 10.1007/s00103-012-1582-4

    • Search Google Scholar
    • Export Citation
  • Van Kerkhove, M. D. and Ferguson, N. M. (2012) Epidemic and intervention modelling: a scientific rationale for policy decisions? Lessons from the 2009 influenza pandemic, Bulletin of the World Health Organization, 90(4): 30610. doi: 10.2471/BLT.11.097949

    • Search Google Scholar
    • Export Citation
  • Weeramanthri, T. S., Robertson, A. G., Dowse, G. K., Effler, P. V., Leclercq, M. G., Burtenshaw, J. D., Oldham, S. J., Smith, D. W., Gatti, K. J. and Gladstones, H. M. (2010) Response to pandemic (H1N1) 2009 influenza in Australia: lessons from a State health department perspective, Australian Health Review, 34(4): 47786. doi: 10.1071/AH10901

    • Search Google Scholar
    • Export Citation
  • WHO (2011) Implementation of the International Health Regulations (2005). Report of the Review Committee on the Functioning of the International Health Regulations (2005) in Relation to Pandemic (H1N1) 2009, Geneva: World Health Organization.

    • Search Google Scholar
    • Export Citation
  • WHO (2012) The WHO Strategy on Research for Health, Geneva: World Health Organization.

  • WHO (2019) 24: Situation Report on the Ebola outbreak in North Kivu, WHO Regional Office for Africa, Brazzaville: World Health Organization. https://apps.who.int/iris/bitstream/handle/10665/279604/SITREP_EVD_DRC_20190116-eng.pdf?ua=1

    • Search Google Scholar
    • Export Citation
  • Young, K., Ashby, D., Boaz, A. and Grayson, L. (2002) Social science and the evidence-based policy movement, Social Policy and Society, 1(3): 21524. doi: 10.1017/S1474746402003068

    • Search Google Scholar
    • Export Citation
  • Abeysinghe, S. (2014) An uncertain risk: the World Health Organization’s account of H1N1, Science in Context, 27(3): 51129. doi: 10.1017/S0269889714000167

    • Search Google Scholar
    • Export Citation
  • Abeysinghe, S. (2017) Contesting a pandemic: the WHO and the Council of Europe, Science as Culture, 26(2): 16184. doi: 10.1080/09505431.2016.1212825

    • Search Google Scholar
    • Export Citation
  • Allen, T., Murray, K.A., Zambrana-Torrelio, C., Morse, S.S., Rondinini, C., Di Marco, M., Breit, N., Olival, K.J. and Daszak, P. (2017) Global hotspots and correlates of emerging zoonotic diseases, Nature Communications, 8: 1142. doi: 10.1038/s41467-017-01293-x

    • Search Google Scholar
    • Export Citation
  • Arksey, H. and O’Malley, L. (2005) Scoping studies: towards a methodological framework, International Journal of Social Research Methodology, 8(1): 1932. doi: 10.1080/1364557032000119616

    • Search Google Scholar
    • Export Citation
  • Association of State and Territorial Health Officials (ASTHO) (2010) Assessing Policy Barriers to Effective Public Health Response in the H1N1 Influenza Pandemic. Project Report to the Centers for Disease Control and Prevention, Arlington,VA: ASTHO.

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2014) Reputation-seeking by a government agency in Europe: direct evidence from responses to the 2009 H1N1 “swine” influenza pandemic, Administration & Society, 49(2): 16389. doi: 10.1177/0095399714528177

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2016a) Explaining science-led policy-making: pandemic deaths, epistemic deliberation and ideational trajectories, Policy Sciences, 49(4): 395419. doi: 10.1007/s11077-016-9264-y

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. (2016b) Same threat, different responses: experts steering politicians and stakeholders in 2009 H1N1 vaccination policy-making, Public Administration, 94(2): 299315. doi: 10.1111/padm.12244

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. and Öberg, P. (2017) Freezing deliberation through public expert advice, Journal of European Public Policy, 24(7): 100626. doi: 10.1080/13501763.2016.1170192

    • Search Google Scholar
    • Export Citation
  • Baekkeskov, E. and Rubin, O. (2014) Why pandemic response is unique: powerful experts and hands-off political leaders, Disaster Prevention and Management: An International Journal, 23(1): 8193. doi: 10.1108/DPM-05-2012-0060

    • Search Google Scholar
    • Export Citation
  • Barker, K. (2012) Influenza preparedness and the bureaucratic reflex: anticipating and generating the 2009 H1N1 event, Health & Place, 18(4): 7019.

    • Search Google Scholar
    • Export Citation
  • Baum, F. (1995) Researching public health: behind the qualitative-quantitative methodological debate, Social Science & Medicine, 40(4): 45968.

    • Search Google Scholar
    • Export Citation
  • Bowen, S. and Zwi, A. B. (2005) Pathways to “evidence-informed” policy and practice: a framework for action’, PLOS Medicine, 2(7): e166. doi: 10.1371/journal.pmed.0020166

    • Search Google Scholar
    • Export Citation
  • Bowen, S., Erickson, T., Martens, P. J. and Crockett, S. (2009) More than “using research”: the real challenges in promoting evidence-informed decision-making, Healthcare Policy, 4(3): 87102.

    • Search Google Scholar
    • Export Citation
  • Broekema, W., van Eijk, C. and Torenvlied, R. (2018) The role of external experts in crisis situations: a research synthesis of 114 post-crisis evaluation reports in the Netherlands, International Journal of Disaster Risk Reduction, 31: 209. doi: 10.1016/j.ijdrr.2018.03.021

    • Search Google Scholar
    • Export Citation
  • Brumme, Z. L. and Lee, K. (2012) Operationalizing the one health approach: the global governance challenges, Health Policy and Planning, 28(7): 77885.

    • Search Google Scholar
    • Export Citation
  • Burkle, F. M. and Hanfling, D. (2015) ‘Political leadership in the time of crises: primum non nocere’, PLoS Currents Disasters, May 29, Edition 1. doi: 10.1371/currents.dis.fd8aaf6707cd5dd252e33c771d08b949

    • Search Google Scholar
    • Export Citation
  • Chambers, J., Barker, K. and Rouse, A. (2012) Reflections on the UK’s approach to the 2009 swine flu pandemic: conflicts between national government and the local management of the public health response, Health & Place, 18(4): 73745.

    • Search Google Scholar
    • Export Citation
  • Cohen, J. and Enserink, M. (2009), After delays, WHO agrees: the 2009 pandemic has begun, Science, 324: 14967. doi: 10.1126/science.324_1496

    • Search Google Scholar
    • Export Citation
  • Crosier, A., McVey, D. and French, J. (2015) ‘“By failing to prepare you are preparing to fail”: lessons from the 2009 H1N1 “swine flu” pandemic’, European Journal of Public Health, 25(1): 1359. doi: 10.1093/eurpub/cku131

    • Search Google Scholar
    • Export Citation
  • Directorate for Civil Protection and Emergency Planning (DSB) (2011) New Influenza A Virus (H1N1): A Summary of a Study on the National Response in Norway, Tønsberg: DSB.

    • Search Google Scholar
    • Export Citation
  • Driedger, S. M., Cooper, E. J. and Moghadas, S. M. (2014) Developing model-based public health policy through knowledge translation: the need for a “communities of practice”, Public Health, 128(6): 5617. doi: 10.1016/j.puhe.2013.10.009

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2014) Handbook on Simulation Exercises in EU Public Health Settings: How to Develop Simulation Exercises Within the Framework of Public Health Response to Communicable Diseases, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2015) Ebola Emergency Preparedness in EU Member States: Conclusions From Peer-review Visits to Belgium, Portugal and Romania, Stockholm: ECDC. https://www.ecdc.europa.eu/sites/default/files/media/en/publications/Publications/ebola-emergency-preparedness-belgium-portugal-romania.pdf

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2015) Public Health Emergency Preparedness for Cases of Viral Haemorrhagic Fever (Ebola) in Belgium: A Peer Review, 16–19 March 2015, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2018) Best Practice Recommendations for Conducting After-action Reviews to Enhance Public Health Preparedness, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2018) Ebola Virus Disease Outbreak in North Kivu and Ituri Provinces, Democratic Republic of the Congo, Second Update, 21 December 2018, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • ECDC (European Centre for Disease Prevention and Control) (2019) The Use of Evidence in Decision-making During Public Health Emergencies, Stockholm: ECDC.

    • Search Google Scholar
    • Export Citation
  • European Parliament and the Council of the European Union (2013) Decision No 1082/2013/EU of the European Parliament and of the Council of 22 October 2013 on serious cross-border threats to health and repealing Decision No 2119/98/EC Text with EEA relevance, No 2119/98/EC, Official Journal of the European Union, pp 115.

  • Feufel, M.A., Antes, G. and Gigerenzer, G. (2010) Competence in dealing with uncertainty: lessons to learn from the influenza pandemic (H1N1) 2009, Bundesgesundheitsblatt, Gesundheitsforschung, Gesundheitsschutz, 53(12): 12839. doi: 10.1007/s00103-010-1165-1

    • Search Google Scholar
    • Export Citation
  • Flynn, P. (2010) The handling of the H1N1 pandemic: more transparency needed, in P. Assembly (ed), Strasbourg: Council of Europe, p 21.

  • Forcades Vila, T. (2015) Flu vaccination: the gap between evidence and public policy, International Journal of Health Services, 45(3): 45370. doi: 10.1177/0020731415585976

    • Search Google Scholar
    • Export Citation
  • Funtowicz, S.O. and Ravetz, J.R. (1993) Science for the post-normal age, Futures, 25(7): 73955. doi: 10.1016/0016-3287(93)90022-L

  • Greco, D., Stern, E. and Marks, G. (2011) Review of ECDC’s Response to the Influenza Pandemic 2009–2010, Stockholm: ECDC.

  • Greenhalgh, T. and Peacock, R. (2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources, BMJ, 331(7524): 10645. doi: 10.1136/bmj.38636.593461.68

    • Search Google Scholar
    • Export Citation
  • Hanquet, G., Van Damme, P., Brasseur, D., De Cuyper, X., Gregor, S., Holmberg, M., Martin, R., Molnár, Z., Pompa, M. G., Snacken, R. et al. (2011) Lessons learnt from pandemic A(H1N1) 2009 influenza vaccination: highlights of a European workshop in Brussels, 22 March 2010, Vaccine, 29(3): 3707. doi: 10.1016/j.vaccine.2010.10.079

    • Search Google Scholar
    • Export Citation
  • Health Protection Agency (2010) Assessment Report on the EU-wide Response to Pandemic (H1N1) 2009, Salisbury: Health Protection Agency.

  • Health Quality Council of Alberta (2010) Review of Alberta’s Response to the 2009 H1N1 Influenza Pandemic, Edmonton: AB Government of Alberta.

    • Search Google Scholar
    • Export Citation
  • Hine, D. D. (2010) The 2009 Influenza Pandemic: An Independent Review of the UK Response to the 2009 Influenza Pandemic, London: Cabinet Office.

    • Search Google Scholar
    • Export Citation
  • House of Commons (2011) Scientific Advice and Evidence in Emergencies: Third Report of Session 2010–11, Vol. 1: Report, Together with Formal Minutes, Oral and Written Evidence, London: The Stationery Office.

    • Search Google Scholar
    • Export Citation
  • Huizer, Y.L., Kraaij-Dirkzwager, M.M., Timen, A., Schuitmaker, T.J. and Steenbergen, J. E.V. (2015) Context analysis for epidemic control in the Netherlands, Health Policy, 119(1): 6673. doi: 10.1016/j.healthpol.2014.10.004

    • Search Google Scholar
    • Export Citation
  • Humphreys, K. and Piot, P. (2012) Scientific evidence alone is not sufficient basis for health policy, BMJ, 344: e1316. doi: 10.1136/bmj.e1316

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2012) Evidence-based medicine and the governance of pandemic influenza, Global Public Health, 7(Suppl. 2): S11126.

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2013) The politics of medicine and the global governance of pandemic influenza, International Journal of Health Services, 43(1): 10521. doi: 10.2190/HS.43.1.h

    • Search Google Scholar
    • Export Citation
  • Kamradt-Scott, A. (2016) WHO’s to blame? The World Health Organization and the 2014 Ebola outbreak in West Africa, Third World Quarterly, 37(3): 40118. doi: 10.1080/01436597.2015.1112232

    • Search Google Scholar
    • Export Citation
  • Kayman, H., Logar, T. (2015) A framework for training public health practitioners in crisis decision-making, Disaster Medicine and Public Health Preparedness, 10(1): 16573. doi: 10.1017/dmp.2015.149

    • Search Google Scholar
    • Export Citation
  • Kayman, H., Salter, S., Mittal, M., Scott, W., Santos, N., Tran, D. and Ma, R. (2015) School closure decisions made by local health department officials during the 2009 H1N1 influenza outbreak, Disaster Medicine and Public Health Preparedness, 9(4): 46471. doi: 10.1017/dmp.2015.72

    • Search Google Scholar
    • Export Citation
  • Keller, A.C., Ansell, C.K., Reingold, A.L., Bourrier, M., Hunter, M.D., Burrowes, S. and MacPhail, T.M. (2012) Improving pandemic response: a sensemaking perspective on the spring 2009 H1N1 pandemic, Risk, Hazards & Crisis in Public Policy, 3(2): 137. doi: 10.1080/19475705.2011.638027

    • Search Google Scholar
    • Export Citation
  • Khan, K.S., Kunz, R., Kleijnen, J. and Antes, G. (2003) Five steps to conducting a systematic review, Journal of the Royal Society of Medicine, 96(3): 11821. doi: 10.1177/014107680309600304

    • Search Google Scholar
    • Export Citation
  • Kilianski, A., Evans, N.G. (2015) Effectively communicating the uncertainties surrounding Ebola virus transmission, PLoS Pathogens, 11(10). doi: 10.1371/journal.ppat.1005097

    • Search Google Scholar
    • Export Citation
  • Klaiman, T., Kraemer, J.D. and Stoto, M.A. (2011) Variability in school closure decisions in response to 2009 H1N1: a qualitative systems improvement analysis, BMC Public Health, 11: 73. doi: 10.1186/1471-2458-11-73

    • Search Google Scholar
    • Export Citation
  • Krause, G., Gilsdorf, A., Becker, J., Bradt, K., Dreweck, C., Gärtner, B., Löwer, J., Marcic, A., Nicoll, A. and Pott, E. (2010) Erster Erfahrungsaustausch zur H1N1-Pandemie in Deutschland 2009/2010, Bundesgesundheitsblatt-Gesundheitsforschung-Gesundheitsschutz, 53(5): 5109. doi: 10.1007/s00103-010-1074-3

    • Search Google Scholar
    • Export Citation
  • Liverani, M., Hawkins, B. and Parkhurst, J. O. (2013) Political and institutional influences on the use of evidence in public health policy: a systematic review, PLoS One, 8(10): e77404. doi: 10.1371/journal.pone.0077404

    • Search Google Scholar
    • Export Citation
  • Mackenzie, J.S., McKinnon, M. and Jeggo, M. (2014) One health: from concept to practice, in A. Yamada, L.H. Kahn, B. Kaplan, T. P. Monath, J. Woodall and L. Conti (eds) Confronting Emerging Zoonoses: The One Health Paradigm, Tokyo: Springer Japan, pp. 16389.

    • Search Google Scholar
    • Export Citation
  • Mansnerus, E. (2013) Using model-based evidence in the governance of pandemics, Sociology of Health and Illness, 35(2): 28091. doi: 10.1111/j.1467-9566.2012.01540.x

    • Search Google Scholar
    • Export Citation
  • Müller-Seitz, G. and Macpherson, A. (2013) Learning during crisis as a “war for meaning”: the case of the German Escherichia coli outbreak in 2011, Management Learning, 45(5): 593608.

    • Search Google Scholar
    • Export Citation
  • Oliver, K. and Cairney, P. (2019) The dos and don’ts of influencing policy: a systematic review of advice to academics, Palgrave Communications, 5(1): 21. doi: 10.1057/s41599-019-0232-y

    • Search Google Scholar
    • Export Citation
  • Oliver, K., Lorenc, T. and Innvær, S. (2014) New directions in evidence-based policy research: a critical analysis of the literature, Health Research Policy and Systems, 12(1). doi: 10.1186/1478-4505-12-34

    • Search Google Scholar
    • Export Citation
  • Orton, L., Lloyd-Williams, F., Taylor-Robinson, D., O’Flaherty, M. and Capewell, S. (2011) The use of research evidence in public health decision making processes: systematic review, PLOS ONE, 6(7): e21704.

    • Search Google Scholar
    • Export Citation
  • Palmer, S., Jansen, A., Leitmeyer, K., Murdoch, H. and Forland, F. (2013) Evidence-Based Medicine applied to the control of communicable disease incidents when evidence is scarce and the time is limited, Eurosurveillance, 18(25). doi: 10.2807/1560-7917.ES2013.18.25.20507

    • Search Google Scholar
    • Export Citation
  • Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D. and Soares, C. B. (2015) Guidance for conducting systematic scoping reviews, International Journal of Evidence-Based Healthcare, 13(3): 1416. doi: 10.1097/XEB.0000000000000050

    • Search Google Scholar
    • Export Citation
  • Public Health Agency of Canada (2010) Lessons Learned Review: Public Health Agency of Canada and Health Canada Response to the 2009 H1N1 Pandemic, Ottawa: Public Health Agency of Canada. https://www.canada.ca/content/dam/phac-aspc/migration/phac-aspc/about_apropos/evaluation/reports-rapports/2010-2011/h1n1/pdf/h1n1-eng.pdf

    • Search Google Scholar
    • Export Citation
  • Rizzo, C., Rota, M. C., Bella, A., Giannitelli, S., De Santis, S. D., Nacca, G., Pompa, M. G., Vellucci, L., Salmaso, S. and Declich, S. (2010) Response to the 2009 influenza A (H1N1) pandemic in Italy, Eurosurveillance, 15(49): 6. doi: 10.2807/ese.15.49.19744-en

    • Search Google Scholar
    • Export Citation
  • Roodenrijs, J.C. M., Kraaij-Dirkzwager, M. M., Van Den Kerkhof, J. H. T. C. and Runhaar, H. A. C. (2014) Risk governance for infectious diseases: exploring the feasibility and added value of the IRGC-framework for Dutch infectious disease control, Journal of Risk Research, 17(9): 116182. doi: 10.1080/13669877.2013.875935

    • Search Google Scholar
    • Export Citation
  • Rosella, L. C., Wilson, K., Crowcroft, N. S., Chu, A., Upshur, R., Willison, D., Deeks, S. L., Schwartz, B., Tustin, J., Sider, D. and Goel, V. (2013) Pandemic H1N1 in Canada and the use of evidence in developing public health policies: a policy analysis, Social Science and Medicine, 83: 19. doi: 10.1016/j.socscimed.2013.02.009

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020a) Scoping review protocol: exploring the role of evidence in decision-making during public health emergencies, Online Supplemental File 12: Scoping review search strategy and results. Study Protocol. Figshare, https://figshare.com/articles/Online_Supplemental_File_2_Search_Strategy_and_Results/11744316.

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020b): Online Supplemental File 2: Scoping review search strategy and results. Study Protocol. Figshare. https://figshare.com/articles/Online_Supplemental_File_2_Search_Strategy_and_Results/11744316

    • Search Google Scholar
    • Export Citation
  • Salajan, A., Tsolova, S., Ciotti, M. and Suk, J.E. (2020c): Online Supplemental 3: Table S1. Overview of the included peer-reviewed and grey literature. Figshare. Journal Contribution https://figshare.com/articles/Table_S1_Overview_of_the_included_peer-reviewed_and_grey_literature/11744340/1.

    • Search Google Scholar
    • Export Citation
  • Semenza, J. C., Lindgren, E., Balkanyi, L., Espinosa, L., Almqvist, M. S., Penttinen, P. and Rocklöv, J. (2016) Determinants and drivers of infectious disease threat events in Europe, Emerging Infectious Diseases, 22(4): 5819. doi: 10.3201/eid2204.151073

    • Search Google Scholar
    • Export Citation
  • Stein, M. L., van Vliet, J. A. and Timen, A. (2011) Chronological Overview of the 2009/2010 H1N1 Influenza Pandemic and the Response of the Centre for Infectious Disease Control of the National Institute for Public Health and Environment (RIVM), Bilthoven: National Institute for Public Health and Environment (RIVM). https://www.rivm.nl/bibliotheek/rapporten/215011006.pdf

    • Search Google Scholar
    • Export Citation
  • Stollorz, V. (2013) The pandemic of the experts in the mass media: how to create trust in public communication by acknowledging nescience and uncertainty, Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 56(1): 1107. doi: 10.1007/s00103-012-1581-5

    • Search Google Scholar
    • Export Citation
  • Topp, L., Mair, D., Smillie, L. and Cairney, P. (2018) Knowledge management for policy impact: the case of the European Commission’s Joint Research Centre, Palgrave Communications, 4(1): 87. doi: 10.1057/s41599-018-0143-3

    • Search Google Scholar
    • Export Citation
  • UNDG (United Nations Development Group) (2015) Socio-economic Impact of Ebola Virus Disease in West African Countries: A Call for National and Regional Containment, Recovery and Prevention, Dakar: UNDG https://www.undp.org/content/dam/rba/docs/Reports/ebola-west-africa.pdf.

    • Search Google Scholar
    • Export Citation
  • Upshur, R. E. (2012) Evidence and ethics in public health: the experience of SARS in Canada, New South Wales Public Health Bulletin, 23(5–6): 10810. doi: 10.1071/NB11044

    • Search Google Scholar
    • Export Citation
  • Van Der Sande, M. A. B., Jacobi, A., Meijer, A., Wallinga, J., Van Der Hoek, W. and Van Der Lubben, M. (2013) The 2009 influenza A (H1N1) pandemic: management and vaccination strategies in the Netherlands, Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 56(1): 6775. doi: 10.1007/s00103-012-1582-4

    • Search Google Scholar
    • Export Citation
  • Van Kerkhove, M. D. and Ferguson, N. M. (2012) Epidemic and intervention modelling: a scientific rationale for policy decisions? Lessons from the 2009 influenza pandemic, Bulletin of the World Health Organization, 90(4): 30610. doi: 10.2471/BLT.11.097949

    • Search Google Scholar
    • Export Citation
  • Weeramanthri, T. S., Robertson, A. G., Dowse, G. K., Effler, P. V., Leclercq, M. G., Burtenshaw, J. D., Oldham, S. J., Smith, D. W., Gatti, K. J. and Gladstones, H. M. (2010) Response to pandemic (H1N1) 2009 influenza in Australia: lessons from a State health department perspective, Australian Health Review, 34(4): 47786. doi: 10.1071/AH10901

    • Search Google Scholar
    • Export Citation
  • WHO (2011) Implementation of the International Health Regulations (2005). Report of the Review Committee on the Functioning of the International Health Regulations (2005) in Relation to Pandemic (H1N1) 2009, Geneva: World Health Organization.

    • Search Google Scholar
    • Export Citation
  • WHO (2012) The WHO Strategy on Research for Health, Geneva: World Health Organization.

  • WHO (2019) 24: Situation Report on the Ebola outbreak in North Kivu, WHO Regional Office for Africa, Brazzaville: World Health Organization. https://apps.who.int/iris/bitstream/handle/10665/279604/SITREP_EVD_DRC_20190116-eng.pdf?ua=1

    • Search Google Scholar
    • Export Citation
  • Young, K., Ashby, D., Boaz, A. and Grayson, L. (2002) Social science and the evidence-based policy movement, Social Policy and Society, 1(3): 21524. doi: 10.1017/S1474746402003068

    • Search Google Scholar
    • Export Citation
Andreea SalajanEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Andreea Salajan in
Current site
Google Scholar
PubMed
Close
,
Svetla TsolovaEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Svetla Tsolova in
Current site
Google Scholar
PubMed
Close
,
Massimo CiottiEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Massimo Ciotti in
Current site
Google Scholar
PubMed
Close
, and
Jonathan E. SukEuropean Centre for Disease Prevention and Control, Sweden

Search for other papers by Jonathan E. Suk in
Current site
Google Scholar
PubMed
Close

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 825 825 123
PDF Downloads 426 426 79

Altmetrics

Dimensions