Key messages

• COVID-19 necessitates new ways of thinking about emergent evidence-making.

• Emergencies open up questions about what constitutes ‘evidence-enough’ for rapid decisions in-the-now.

• Rapid assessment and outbreak science models offer indications of how evidence-making might be done differently.

• Adaptive evidence-making is needed not only during emergencies but also for intervening in the everyday.

As the COVID-19 pandemic is making abundantly clear, the relationship between evidence and policy decision making during times of emergency is crucial. With COVID-19, there is the imperative to act, swiftly, to contain the virus and slow the spread. But in these unprecedented conditions, faced with a novel coronavirus, it is all the more apparent that evidence is uncertain, emergent and evolving. In the absence of empirical data we become even more reliant on other ways of generating and synthesising evidence, forecasting the possible effects of interventions rather than trialling their situated outcomes. We become aware that we cannot act with the same sense of certainty about ‘what works’. Moreover, there is limited time for ‘business-as-usual’ processes of research translation, as new evidence of ‘unprecedented’ events comes to hand. Interventions are required in the immediate present, and at scale. Policy decisions are rapidly made, rarely definitive, and revised almost daily in the context of heightened social, economic and political concern. Emergencies speed up and challenge the ideal processes of evidence-based policy and practice such that evidence, translation, decision making, and intervention implementation proceed simultaneously, in greatly compressed timeframes, and in situations with complex, intersecting social, economic and political pressures.

COVID-19 opens up urgent questions about the nature of evidence and evidence-based decision making in health, especially what constitutes ‘evidence-enough’ when intervening in the face of the unknown. Although the pandemic throws these questions into sharp relief, we suggest that these are concerns not only for crisis periods but also for evidence-making and intervening in the everyday. Decision makers routinely deal with degrees of uncertainty and are often aware that although evidence is necessary for action, intended outcomes are without guarantee. Thus, building on rapid assessment and outbreak science approaches, we suggest that the unfolding COVID-19 pandemic challenges us to further consider the need for adaptive modes of evidence-making and intervention in health.

Towards adaptive evidence-making

The imagined ideals of evidence-based policy and practice in health are especially challenged in situations of urgency. This is the case given the simultaneous co-production of evidence and response in emergency situations. In the field of health there remains a preference for scientific evidence derived via controlled intervention designs which seek to isolate an intervention’s causal effects from the context of its implementation, and systematic reviews of these, to judge the effectiveness of interventions. Even in situations where evidence from controlled intervention studies is not available, evidence-based health policy and practice generally proceeds on the assumption that evidence must first be generated so that it can subsequently be translated into decisions about implementing interventions at scale (NICE, 2014). Evidence, policy and implementation are idealised, at least in theory, as separate and distinct sites, with the ‘gaps’ between them needing to be ‘bridged’ via the careful translation of evidence ‘into’ practice.

An established body of work demonstrates that the ideals of evidence-based policy and intervention are much messier and more complex in practice (Oliver et al, 2014). This ideal fits awkwardly with the fluid and ambiguous practices of policy work (Colebatch et al, 2010), and there are continuing disagreements about what constitutes ‘good’ evidence (Green, 2000; Cairney, 2017). While there is a push to develop approaches that smooth the bumpy interface between evidence, decision making and implementation, as well as to rethink concepts like ‘translation’ (Greenhalgh and Wieringa, 2011), the cyclical resurgence of calls for ‘robust’ evidence for policy (Haynes et al, 2012) highlights the ongoing challenge of broadening definitions of ‘what counts’ as evidence and how that evidence is used (Lancaster et al, 2017b).

While the ideal of evidence-based policy and practice might be cast as ‘illusory’ (Smith-Merry, 2020: 3), in the health field ‘there appears to be a reluctance to let go’ (Smith, 2013: 4). Public health emergencies profoundly challenge these aspirations to ‘hold on’ and bring to light the urgent need for different ways of thinking. The COVID-19 pandemic, as with other viral outbreaks and public health emergencies, requires evidence-making and decision making able to deal with an evolving situation where evidence and response are engaged in a simultaneous process of co-production. A recent review on evidence-informed decision making in infectious disease outbreaks noted it is ‘not a disregard of evidence that put a strain on decision making in various contexts, but rather the lack of clear, unambiguous and rapidly available evidence for risk characterisation and effectiveness of response measures’ (Salajan et al, 2020: 14). As such, it was suggested that there might be a need to ‘re-evaluate the status quo’ (Salajan et al, 2020: 14) and bring knowledge to bear in different ways. In effect, crisis contexts highlight the need to think differently about evidence-making and decision making, acknowledging that unambiguous evidence in evolving situations of complexity is an unachievable ideal, and that the effectiveness of response measures is situated and emergent.

Rapid assessment models have been developed to attempt to speed up – and importantly bring together – the processes of evidence-making and decision making in situations of complex public health and humanitarian emergencies (Manderson and Aaby, 1992; Rhodes et al, 1999; WHO, 1999; Trotter et al, 2001; Fitch et al, 2004). Since the 1990s, these methods have been used successfully around the world to generate data for decision making in the management of HIV/AIDS, substance use, family planning, malaria, tuberculosis, diarrheal disease, dengue fever, water sanitation, and disaster intervention. These methods orientate towards intervention development, using multiple data sources, the continual triangulation of data, cyclical processes of inductive hypothesis formation and testing, an investigative orientation akin to ‘the detective novel’, and active participation of local communities and non-scientific experts in research and interventions (Rhodes et al, 1999: 66). The focus is explicitly on undertaking assessments in order to respond, developing interventions and not merely knowledge, especially when inadequate data are available. A systematic review conducted following the 2013–2016 Ebola outbreak in West Africa highlights the value of such methods for identifying and addressing context-specific issues, acting as a guide for resource allocation, and providing data to plan long-term assistance (Johnson and Vindrola-Padros, 2017). Outbreak science has also been proposed to improve epidemic preparedness and emergency response. Outbreak science focuses on turning data into ‘actionable information for decisions’ by maximising how models, clinical knowledge, laboratory results, data science and statistics can be put to use to support decisions regarding resource allocation and the implementation of interventions both during, and between, disease outbreaks (Rivers et al, 2019). As with rapid assessment, it has been said that outbreak science requires ‘a paradigm shift’ (Rivers et al, 2019: 3) in the ways that evidence makers and decision makers work together.

We suggest that rapid assessment and outbreak science models offer indications of how evidence-making might be done differently, including via an emphasis on methodological pluralism, intervention pragmatism, and understanding health in its local social, cultural and economic situation. These models seek to adapt the ‘ideal’ of evidence-based approaches by mixing different forms of evidence, reconfiguring notions of ‘validity’ and ‘reliability’, and by giving value to iteration, and to emergence over succession. What differentiates these models from traditional approaches to generating evidence for decision making is not merely that they move quickly from ‘assessment’ to ‘response’, but that they implicitly recognise ‘the process of assessment as the beginning of the response itself’ (Greig and Kershnar, 2000: 25, emphasis original), thus breaking down an imagined separation between evidence and intervention, knowledge and response, knowledge producers and users. Such approaches open up evidence and expertise by making evidence and intervention through dialogue between scientists, decision makers and communities, enabling responses in-the-now; entangling present, situated, social and political concerns into the co-production of evidence and decisions. They are evidence-making practices orientated towards being responsive to an evolving situation of need. Following this path, we suggest that the unprecedented challenges of COVID-19 do not simply require us to speed-up existing evidence-based approaches, but rather necessitate different ways of thinking about how a more adaptive evidence-making can not only contribute to addressing uncertainties, but do so in a manner responsive to an emergent and evolving situation. We emphasise that this requires an openness to forms of evidence and expertise often absented from ‘business-as-usual’ evidence-based intervention.

Evidence-enough

In emergencies, generating and translating evidence to inform action is conceived as a race against time. In the case of COVID-19, epidemic preparedness analyses were quickly troubled, with the implementation of containment strategies both uncertain and unpredictable, including in countries with state of the art health care (Dalglish, 2020). Early in the pandemic, the Global Research Collaboration for Infectious Disease Preparedness (2020) noted that there are ‘many unknowns’ about COVID-19 and its optimal management, but nonetheless specified that there was an urgent need to develop ‘safe and effective countermeasures’. The need for immediate intervention means that decisions have to be made in the absence of knowledge and are often subject to revision. This prompts questioning of what constitutes evidence, and, more specifically, ‘evidence-enough’, for decision making in-the-now.

The events which unfolded on the Diamond Princess in February 2020 are a prime case. As the site for one of the first outbreaks of COVID-19 outside of China, the ship gained notoriety for poorly enforced and inadequate quarantine conditions. Infectious disease specialist Professor Kentaro Iwata posted a YouTube video giving a detailed account of the ‘completely chaotic’ conditions on board, outlining suggestions for immediate changes to prevent transmission (Iwata, 2020). The Diamond Princess subsequently became a case for epidemiological investigation (Zhang et al, 2020). But the question we raise is how observational evidence produced by experts in the moment, and disseminated via non-traditional avenues, might constitute ‘evidence-enough’ for iteratively altering intervention strategies in emergencies. Retrospectively, we can see how Professor Iwata’s account might have been taken up as evidence and used rapidly in a situation of uncertainty and emergent decision making. Another case example relates to the calculation of COVID-19 deaths in Wuhan. As many people died at home, without being diagnosed with, or treated for, COVID-19, official estimates of the death toll were brought into question. In the face of uncertainty, social media was mobilised to calculate the number of deaths based on cremation capacities of funeral homes, and the number of urns distributed to local families, triangulating these data with available official counts to generate estimates (Radio Free Asia, 2020). These examples point to the strength of different forms of observation being brought into triangulation as decision making happens in a compressed timescale, rather than preferencing data generation and intervention designs that proceed successively over longer periods. More broadly, and in the everyday, these case examples draw attention to the multiple forms of expertise, modes of observation and evidencing which are already present, and which might have been noticed at the time, or retrospectively, and brought into inventive triangulation, were it not for sometimes blinkered and delimiting preferences for particular evidence-making practices.

During times of crisis, when there is an imperative to act, the question becomes what can be done in light of the limitations of what can be known for now. As governments have scrambled to respond effectively and in time, it has become clear that absent evidence and indecision are as consequential as apparently certain knowledge and action. The urgency of the crisis has meant that expertise has been brought to bear in new ways. The thing we call ‘evidence’ is being enacted in a different mode, authorised, validated, and made legitimate through non-traditional platforms and practices, with judgements about its relevance for policy entangled within the evolving situation and urgent matters of concern. Dr Richard Horton, Editor in Chief of The Lancet, has taken to the media to strongly critique the inaction of the UK government (Horton, 2020b) and opened up the Direct Messaging function of his Twitter account to receive, and rapidly disseminate, ‘evidence from the NHS COVID-19 frontline’ (Twitter, 2020a) stepping out from behind the usual authorising practices of ‘objective evidence’ and peer review. Dr Michael Ryan, WHO Executive Director of Health Emergencies, has implored, ‘Be fast. Have no regrets.… Speed trumps perfection’ (News.com.au, 2020). In the absence of empirical evidence of transmission dynamics and intervention effects, modelled projections fill gaps in what can be known, generating multiple scenarios to inform decisions (for example, Ferguson et al, 2020). In this context, researchers have also questioned the enduring value of concepts such as internal validity, and suggested that there is a need to think with external validity to learn from different diseases in different times (Twitter, 2020c). How historical and anecdotal evidence from the US urban experience of the 1918–1919 influenza pandemic might be used to inform the implementation of non-pharmaceutical interventions now, despite the vast differences across time and socioeconomic contexts, has been considered (Ebrahim et al, 2020). What evidence exists is also being translated at breakneck speed. As member of the Oxford COVID-19 Evidence Service Professor Trisha Greenhalgh tweeted on March 21, ‘Please can we all now stop saying “it takes 17 years to get research evidence into practice and policy”. This week, I’ve seen observational studies and rapid reviews done in days, which have changed policy in minutes and practice in hours’ (Twitter, 2020b).

Policy debates surrounding asymptomatic transmission, and in particular the question of facemasks, offer another case example of how evidence is being enacted in a different mode. Given inadequate supplies of certified masks for healthcare workers, members of the community were initially discouraged from wearing facemasks. Amidst scientific debates about the effectiveness of various types of masks, the WHO issued advice stating that ‘there is not enough evidence for or against the use of masks (medical or other) for healthy individuals in the wider community’ (WHO, 2020). Faced with ‘mixed’ evidence, the UK government prevaricated saying it would be ‘guided by the scientists’ (BBC News, 2020b). Despite systematic reviews concluding that there was insufficient evidence to strongly support the widespread use of facemasks, researchers pushed back on these policy decisions. Invoking the ‘precautionary principle’ researchers argued that limited anecdotal, indirect and circumstantial findings ought not to be ignored, given the seriousness of COVID-19 and the consequences of inaction (Greenhalgh et al, 2020). They argued that ‘in the face of a pandemic the search for perfect evidence may be the enemy of good policy’ (Greenhalgh et al, 2020). Here, the assessment of ‘evidence-enough’ for action was made in context, in relation to a hinterland of other matters of political, social and economic concern (including the desire to reduce the period of lockdown). For such conclusions to be drawn in the pages of a peer-reviewed clinical journal such as the BMJ is also notable. But this invocation need not only be conceptualised as a weighing of the evidence, for and against, in an interim of uncertainty. As we have argued elsewhere (Rosengarten et al, 2020), masks may also be regarded not as a finite solution but as a possible in the midst of numerous uncertainties; a pragmatic response to a situation where evidence of situated effect can only come after (and potentially too late). In these unprecedented times, there is an opportunity to rethink what constitutes ‘evidence-enough’ to inform decision making, and how expertise and response might be brought to bear in novel ways. Following the path of rapid assessment and outbreak science approaches, we can consider anew how prioritising ‘what can be done’ might drive approaches to emergent and adaptive evidencing and create possibilities for responsive and iterative intervention.

Entangled evidence and interventions

Emergent and adaptive evidence-making is not simply a question of speed, or merely about shifting the boundaries of what constitutes ‘evidence-enough’ to act fast in the moment. The COVID-19 pandemic also demonstrates the need for a reflexive, responsive and adaptive science for decision making that recognises that evidence and interventions are each and all entangled, emerging through the situated relational dynamics involved in their accomplishment, constituted within the particular social and political practices and processes of their implementations (Rhodes and Lancaster, 2019; Rosengarten and Savransky, 2019). Evidence-based approaches in health tend to hold onto the idea of isolating an intervention’s effects from its context and, moreover, attempt to trial individual interventions successively to know ‘what works’ (Savransky and Rosengarten, 2016; Rhodes and Lancaster, 2019). In an evolving situation, as we are witnessing with COVID-19, it becomes clear to see that interventions are not isolatable not only from their situated contexts of implementation but also from each other; it is not practical to successively trial the effectiveness of individual interventions (especially social distancing and other non-pharmaceutical interventions such as business closures, lockdowns and travel bans). Take for example mathematical modelling released in the UK on March 16 outlining the impact of non-pharmaceutical interventions to reduce COVID-19 mortality and healthcare demand. In that report, the uncertainty of assumptions about intervention effectiveness was signalled: ‘Disentangling the relative effectiveness of different interventions from the experience of countries to date is challenging because many have implemented multiple (or all) of these measures with varying degrees of success’ (Ferguson et al, 2020: 14). The caveat noted here reveals the limitations of the assumptions underpinning evidence-based approaches in times of emergency, but also more generally. As the pandemic progresses and country-specific data become available, there have been attempts to ‘untangle’ the effects of various social distancing measures (Gibney, 2020). However, attempts to evaluate these data to know ‘what works’ fail to appreciate the situated nature of an effect, or how such an appreciation might alter assumptions about generalisation that continue to underpin evidence-based approaches (Rosengarten and Savransky, 2019). The entanglement of interventions in their situations of implementation is ever present (Rhodes and Lancaster, 2019). However, emergencies such as the COVID-19 pandemic make this more visible, as emergencies fuse this ever present complexity with reduced time. This raises the question of how complexity can be navigated, through time and iteratively, when there is the imperative to act fast.

Conclusion

The urgency of the COVID-19 pandemic prompts consideration of how decisions might be made, in ways that are responsive, in light of the limitations of what can be known. It throws into sharp relief the need for more nuanced understandings of what constitutes ‘evidence-enough’, and how it might be taken up rapidly via novel means of translation as decisions are iteratively made and revised. This requires acknowledging the entanglements of evidence and interventions in their social worlds, and highlights the need for dialogue between evidence makers, decision makers and publics.

Dr Tedros Adhanom Ghebreyesus, Director-General of the WHO, has said, “We need our collective knowledge, insight and experience to answer the questions we don’t have answers to, and to identify the questions we may not even realize we need to ask”. This requires a different kind of thinking about the relationship between evidence and decision making. Rapid assessment and outbreak science models have foreshadowed how evidence-making might be done differently for intervention during times of emergency. Generating reflexive and adaptive approaches to evidence-making and rapid intervention, in dialogue with decision makers and communities, during times of emergency outbreak is essential when little is known and when evidence and action are proceeding simultaneously. These lessons are not only for crises, however. Too often, as emergencies subside, we return to the ideal of evidence-based approaches. But we suggest that these moments illuminate the possibilities and affordances of an emergent and adaptive evidence-making, not only during health emergencies but for intervening in the everyday (Rhodes and Lancaster, 2019). By treating evidence, decision making and interventions as co-affective rather than as if independent, we might attend to outcomes beyond those anticipated in an evidence-based approach. The crisis around COVID-19 evidence provides an opportunity for reimagining a different sort of relation between science, policy and intervention and, by consequence, enables more reflective approaches to knowing and doing interventions in health.

Acknowledgements

The Centre for Social Research in Health at the University of New South Wales (UNSW) is supported by a grant from the Australian Government Department of Health. We are grateful for support from the UNSW SHARP (Professor Tim Rhodes) and Scientia Fellowship (Dr Kari Lancaster) programmes.

Contributors

KL led the conceptualisation and writing of the first and subsequent drafts of the manuscript, with comments, analysis and conceptual contributions from TR and MR.

Research ethics

The authors of this paper have declared that research ethics approval was not required since the paper does not present or draw directly on data/findings from empirical research.

Conflict of interest

The authors declare that there is no conflict of interest.