Abstract
Background:
While the last decade has seen the increasing refinement of methods for evidence collection and synthesis for clinical guidelines or health policy decision-making, no similar methodological advances can be observed for ethics guidelines. Accordingly, the evidence base of ethics recommendations often remains opaque. The ‘REIGN’ framework fills this gap by addressing how evidence can (and possibly should) be used to develop ethics guidance.
Methods:
A review of the academic and grey literature was conducted. To this end, PubMed and the websites of selected institutions engaged in ethics guideline development and/or health technology assessment were searched. The literature found was read and summarised. Through further conceptual analysis of the arguments, terminology and ideas provided in the literature the REIGN framework was developed.
Findings:
The framework consists of two parts. First, it provides a definition of evidence that is productive for the field of ethics while incorporating key ideas behind the evidence-based medicine movement. It also introduces ‘normative evidence’ in contrast to empirical evidence. Second, it identifies five ‘evidential support components’ (ESCs) as aspects of developing normative recommendations in the health context that can/should be substantiated by evidence. It also provides guidance regarding possible sources of evidence as well as quality appraisal of normative evidence.
Conclusions:
By structuring the dispersed discourses on the topic, the REIGN framework allows ethics guideline developers to think more coherently through the questions of whether, for what area and in what manner evidence should be sought.
Background
Policy makers in healthcare and research know that existing regulation is constantly outpaced by new developments like innovative health technology, societal changes or ecological challenges. Accordingly, there is a continuous need to adapt governance frameworks to new circumstances. In doing so, decision-making in health policy has increasingly relied on evidence and, to a lesser degree, expert consensus in recent decades (Bosch-Capblanch et al, 2012), Evidence-based medicine (EBM) (Sackett et al, 2000; Kunz et al, 2007) or healthcare (EBHC) (Muir Gray, 2001) provides an epistemological and methodological framework for fine-tuning these development processes for guidance documents. Within the EBM framework, guidelines or recommendations are to be informed by relevant data that are to be comprehensively collected and analysed. Much work has been undertaken to strengthen and refine the methods used to collect, assess, synthesise and report data (Page et al, 2021; Higgins et al, 2023) and to define how they should inform decisions (Schünemann et al, 2013).
The focus has thereby lain on rather technical criteria that are commonly consulted in decision-making in healthcare: intervention effectiveness, adverse effects and costs of health technologies/interventions. However, most challenges in healthcare are laced with ethical issues or even mainly composed of such. These issues have to be and are also addressed by guidelines. While most organisations charged with developing guidelines or other forms of guidance in healthcare have clearly delineated their processes for arriving at recommendations in domains such as effectiveness or cost-effectiveness, it is remarkable that ethical issues are seldom addressed in manuals for guideline development or health technology assessment (HTA). Not surprisingly, where ‘ethics’ is addressed in, for example, HTA reports, research has shown that it is addressed unsystematically. Literature reviews are sometimes conducted, but the methods for searching, identifying and analysing the literature are often unclear and underdeveloped. Furthermore, the findings tend to be superficial (DeJean et al, 2009; Polus et al, 2019; Kahrass et al, 2025).
It is to be expected that many institutions responsible for or supporting decision-making in healthcare struggle with the question of how to address ethical issues in healthcare. While institutions and singular authors have begun discussing and proposing possible frameworks and methods (Burls et al, 2011; Reiter-Theil et al, 2011; Mertz and Strech, 2014; Kristensen et al, 2017; Lysdahl et al, 2016; IQWiG, 2023), particularly the role of evidence in ethics guideline development is insufficiently addressed thus far. It is yet unclear how the call for rooting recommendations firmly in evidence, as emphasised by the EBM movement, can be translated into the development of ethics guidelines.
For that reason, the World Health Organization (WHO) in 2017 commissioned REIGN (Use of Research Evidence to Inform Guidance regarding Normative-ethical Topics) – a project to support the establishment of more transparent and systematic processes and methods for the use of research evidence in developing ethics guidelines. The REIGN framework as the main outcome of this work is presented in a 150-page discussion paper. This article provides a short version of the REIGN framework to allow researchers access to the main ideas and thereby hopefully improve the development processes of ethics guidelines.1
The REIGN framework aims to sensitise developers of ethics guidelines to the relevant questions they need to be asking with regard to evidence and can hopefully be the starting point for further refinement and development of methodological guidance in the context of ethics. For this reason, the framework should not be misunderstood as a ‘guideline for guideline development’. Rather, the framework provides first some clarifications to allow productive applications of concepts from the EBM movement to the ethics context. Second, it elaborates on the questions for what, from where and which type of evidence could and possibly should be used in ethics guideline development.
Methods
The main method for framework development was theoretical in nature. This means that we engaged in a ‘hermeneutic circle’ (iterative process of reading and thinking through a text against the backdrop of the goal of the research, previously read texts and their contexts, as well as our own background knowledge, and so on), which resulted in a deeper understanding of evidence, evidence-based medicine, the structure and aims of ethics guidelines, and further relevant aspects. This made it also possible to recognise gaps and problems that arise when EBM concepts are (too) simply adopted for the development of ethics guidelines. New concepts, theses and arguments have been formulated to close the gaps and solve the problems apparent in ethics guideline development and surrounding discourses. These concepts and arguments are initially to be understood only as an offer to guideline developers in thinking through some vexing problems and to enrich further more extensive discourses about the role of (research) evidence for the development of ethics guidelines.
Literature review
However, hermeneutic processes and hypothesising does not take place in a vacuum. It is based on the reading and thinking through of existing literature. Therefore, a systematic literature search of the academic literature as well as policy publications, adapted to the needs of conceptual work, was conducted as a basis for the hermeneutical process and idea generation. To this end, 18 countries (three from each WHO region) were chosen and relevant institutions’ websites were searched for methodological guidance/manuals on ethics guideline development. We considered relevant institutions to be (a) public health agencies, (b) organisations producing HTA reports, (c) clinical and public health guideline development agencies/groups, and (d) national bioethics committees with a focus on organisations acting on the national (not provincial) level. Organisations hinting at internal documents were contacted via email and asked to share these. In addition, relevant international organisations’ websites that develop clinical practice guidelines (CPGs), HTAs or other types of guidelines were screened for published documents.
In addition, BELIT, EthicsWeb and PubMed were searched for relevant publications. We identified four discourses as relevant for our endeavour: (a) general reflections on ‘evidence’ and empirical data in (medical) ethics (‘evidence-based ethics’ and ‘empirical ethics’); (b) methods/procedures for developing ethics guidelines or guidelines for normative-ethical topics; (c) integration/consideration of ethical issues, aspects, or arguments in HTA reports and clinical or public health guidelines; and (d) methods/concepts of (systematic) literature reviews in ethics as a way of synthesising evidence for ethics guidance. In including publications the authors decided to be rather inclusive as the publications were not used to answer a pre-specified question, but providing background knowledge for the analytical work necessary for framework development.2
Framework development
In total, 17 policy documents and 96 academic publications of potential relevance were identified. All were read by at least one author and key ideas summarised constituting the starting ground for further conceptual work. For this, we mainly employed classical philosophical tools like critical reading, hypothesising and thinking through the implications of certain assertions/theses, structuring and evaluating arguments, examining the consistency of terminology and ideas proposed, discussing among the author team and with further experts in the field, and so on. In addition, the framework was applied to two WHO ethics guidelines (‘case studies’). The framework was further sent out for extensive peer review to four experts. New insights gained from application of the framework and critical feedback from peer review led to refinements of the framework itself.
Theoretical premises
The framework was built on certain theoretical premises regarding decision-making in guideline development that in part model processes at WHO, but were also considered most sensible. It starts from the assumption that evidence cannot by itself determine what should be done, but only inform such decisions. For example, evidence reports can only show that a drug is very expensive while producing small effects. However, no action follows directly from that finding. People are needed to weigh costs against benefits – given, among other things, a certain budget and a disease distribution/prevalence – and to come to the judgement that the drug is (not) cost-effective which implies it should (not) be paid by relevant institutions. The need for human agency is based on the normativity of any ‘should we’ question, which will inevitably be answered in guidelines and which always necessitate judgement. It needs human experts to assess the evidence, who, based on such an assessment, come to decisions. This is also true for guidelines targeting more technical questions (for example, which treatment should we administer) as they also exhibit normativity, although possibly more implicitly so (Molewijk et al, 2008; Strech, 2008). Deliberation and consensus processes are therefore arguably inevitable in developing guidelines, which becomes particularly pertinent in ethics guideline development. Accordingly, the role of evidence in guideline development was from the start presumed to be limited to one of information and bias reduction: Evidence can have an information function by adding relevant (theoretical) background information, bringing into the discussion arguments not yet considered, clarifying points raised and generally broadening the perspectives of those involved in ethics guideline development. In addition, people might have preferences (or biases) towards certain ethical issues or certain arguments. They might therefore prefer to focus guidelines on one particular topic or to provide one specific recommendation (Brock, 1987). Where decisions have to be justified against the backdrop of evidence, it becomes more difficult to push in one direction for purely personal preferences; therefore, evidence can also have a bias reduction function.
Positionality statement
The framework was developed by the two authors. CK is a social scientist working on diverse bioethical topics using mostly empirical methods. She has worked with WHO on several ethics guidelines conducting systematic reviews and serving as part of the guideline development group. MM has a background in philosophy and sociology and a focus on the methodology of bioethics, in particular on the relationship between empirical research and normative-ethical analyses as well as systematic reviews of normative literature. He has also contributed to several HTA reports.
No author had any authority over the other, the work was conducted on equal footing. Differences in the interpretation of the literature or in the design of components of the framework were resolved through joint argumentative discussion and agreement on the interpretation, formulation in text and structure of the framework. While research question and methods were defined jointly with WHO, WHO had no influence on how the research was implemented, the findings or the formulation of the final report.
Findings
The literature search and the critical reading and thinking through of the key ideas contained in the literature made two things clear. First, there is as yet no comprehensive or consistent framework, guideline or theory that addresses the specific question of how research evidence can and should be taken into account in the development of ethics guidelines. Therefore, the development of a framework designed for this purpose is necessary (it is not enough, nor is it possible, to combine existing frameworks or guidelines to generate a suitable framework or guideline for the development of ethics guidelines). Second, the term ‘evidence’ itself requires critical reflection and appropriate interpretation when used in the context of ethics guidelines.
Providing a working definition of evidence for the context of ethics
The understanding of ‘evidence’ is coloured by the context in which it was first developed, namely the EBM tradition. Questions that were in the focus for healthcare decision-makers working in the EBM tradition were those of interventions’ (cost-)effectiveness. Evidence is therefore generally understood rather narrowly as systematically aggregated empirical (mainly quantitative) data which are assessed for quality and relevance and help determine stochastically if an effect or a phenomenon exists.
Such an understanding is of little help for those grappling with the concept of ‘evidence’ in the context of ethics. However, no word has a ‘natural meaning’ and there are no compelling reasons to understand evidence so narrowly in every context (Sehon and Stanley, 2003; Kulkarni, 2005). Indeed, increasingly academics are trying to broaden the concept to employ it productively for their contexts (Young et al, 2002; Rycroft-Malone et al, 2004; Stoklosa, 2013). As part of the REIGN framework, a definition is proposed that lifts the characteristic definitional elements of evidence in the EBM tradition to a more abstract level, thereby broadening the definition to allow productive application in other contexts. Relevant elements of the concept of evidence that have to be taken on when developing a broader definition are:
Evidence is normatively laden in that it is supposed to support decisions about what should be done. Information or data points accordingly become evidence due to the context of decision-making.
Due to its function in decision-making, the concept of evidence stipulates a method to grade data points as better/worse or more/less reliable for decision-making.
Evidence = a piece or body of information that is, by varying degrees, qualified within an existing knowledge system to either provide or support a central premise of a possible rational argument for holding a statement (conclusion) true, plausible or right (or false, implausible, or wrong) in a context of decision-making or directing actions.
In addition, for the context of ethics guideline development, it is important to differentiate between ‘normative evidence’ (NE) and ‘empirical evidence’ (EE) as they will have to be handled differently. Whether evidence should be considered normative or empirical (the differentiation criterion) will be determined primarily by the kind of information sought.
We stipulate the following understanding: Normative evidence is information about values, ethical principles, norms, rules, criteria or arguments with normative-ethical conclusions (for example, arguments prescribing, permitting or forbidding an action, possibly under certain conditions/safeguards). Normative evidence will therefore often have the form ‘X should be done/is valuable (because of Y)’. Empirical evidence, on the other hand, comprises information on, for example, whether a phenomenon or effect exists, in which way it exists, how it is perceived by those affected, how it is related to other phenomena or effects (correlation) or what caused the phenomenon or effect to come into being. This information can be quantitative or qualitative. This is not to say that empirical evidence does not exhibit implicit normativity as its generation follows certain normative criteria of good scientific practice (epistemic normativity) and can potentially be used to argue for a particular action (consequentialist normativity). However, the information gathered is not in itself normative, but descriptive (it addresses what is, not what should be).3 Generally, in ethics guideline development at least, both types of evidence are needed to draw legitimate conclusions.
Evidential support components
During the theoretical development of the framework, five so-called ‘evidential support components’ (ESCs) emerged. These also form the core of the framework. Each ESC acknowledges a question that will have to be answered by guideline developers (GDs) through deliberation and consensus seeking (even though these questions are not always explicitly addressed). The ESCs also identify data sets that could and possibly should inform the decision-making processes and would form part of the justificatory system on which the (quality of the) guideline is based. Spelling out the ESCs will aid guideline developers in deciding more explicitly for what and what level of evidence will be sought to support their deliberations. The framework differentiates five such ESCs: value base; conceptual disambiguation; need for action; strategies for addressing needs; and (hypothetical) arguments for actions (see Figure 1). Before the ESCs are explained in detail, it is important to point out that further evidence collection for each ESC, while possible, will not always be reasonable. In practice, the importance of supporting decisions associated with the various ESCs with evidence will vary. It will be the task of the GDs to judge how important further evidence will be. To support GDs in making these decisions, the authors developed a checklist (see Figure 2) that emphasises the consideration of three relevant dimensions: (a) Relevancy: Are the questions addressed within an ESC already answered for external reasons? (If, for example, the institution for which the guideline is developed has already fixed certain questions associated with ESCs). (b) Knowledge base: Is the knowledge base for decision about ESCs already sufficient? It might, for example, be the case that academic work addressing the questions of interest already exists providing all the information needed. (c) Proportionality: Are the financial, time and other costs associated with evidence collection, analysis and appraisal justified given the expected benefits of additional evidence? For example, if guidelines are urgently needed in a crisis, engaging in an extensive evidence collection process might not be justifiable. When considering proportionality, it should also be considered that there are various methods of collecting evidence that can be implemented more or less comprehensively. Less comprehensive methods might be justifiable given scarce resources, but these choices should be made transparently.
/img/Blank.svg)
Overview of the REIGN framework
Citation: Evidence & Policy 2025; 10.1332/17442648Y2025D000000049
Checklist for assessing the need for (further) evidence collection
Citation: Evidence & Policy 2025; 10.1332/17442648Y2025D000000049
Although, for reasons of clarity, the five ESCs have been presented as separate complexes of questions, the ESCs are, of course, often intertwined. The order of presentation follows a linear process, but in practice, the GDs will likely go back and forth between the ESCs (iterative process).
ESC 1: Value base
In identifying and resolving ethical issues GDs will refer to ethical theory, frameworks, principles, norms, values or even (legal) rights. It therefore makes sense to explicitly reflect and specify upfront, for example, which principles will guide the formulation and justification of recommendations in the guideline.4 Such a choice can be considered comparable to deciding upon a specific ethical approach as a method for integrating/evaluating ethical issues in HTA (Saarni et al, 2011; Lysdahl et al, 2016). Settling on a ‘value base’ is also comparable to employing a ‘normative corridor’ or ‘ethical corridor’ (Reiter-Theil et al, 2011). The idea of an ‘ethical corridor’ is that certain ethical positions can be excluded in the analysis or rather synthesis of the (normative) evidence because they are regarded as untenable by the institution that develops the guideline. The ‘value base’ might in some circumstances already be determined by existing institutional codes of ethics, professional standards or even legal obligations, but it might also be open to GDs to decide upon. Where the value base is defined upfront, it should still be made transparent to allow readers to evaluate and possibly challenge existing moral orientation.
Evidence can be supporting those decisions, namely normative evidence about which values, norms, principles or rights are seen as central for a certain context or topic of interest. For the context of public health, for example, Lee (2012) has conducted a systematic review of public health ethics frameworks providing an overview of various principles considered relevant in the context of public health. It might also be necessary to engage with communities targeted by the policy to elicit and consider their value system. Again, such information does not close the discussion what the value base should be, but opens it up and informs it.
ESC 2: Conceptual disambiguation
GDs in ethics face the challenge that ethical discourses are sometimes conceptually convoluted, which they will have to manage. Two issues stand out: On the one hand, different labels with very diverse normative connotations might be in use (for example, abortion, termination of pregnancy or foeticide). In addition, terms might be understood and used differently in different academic disciplines or cultural contexts or for personal interests. An example would be ‘incidental findings’ that can be understood as encompassing any health-relevant diagnostic finding that was not intended by the diagnostic means used or as (also) including negligent and false positive findings (Schmücker, 2016). To ensure productive discussion among those developing guidelines, but also to prevent misunderstanding among the readership, GDs will have to fix terms and definitions employed within the guideline.
There are various data that can support GDs in making these decisions. First, GDs might find an overview of terms and/or definitions used in the public and academic discourse with regard to the issue of interest helpful. Examples of that type of information are two systematic reviews that provide overviews of definitions/models of personalised medicine (Schleidgen et al, 2013) and health literacy (Sørensen et al, 2012). In addition, an analysis of the normative implications of the various terms or definitions could be helpful. Without an overview of those implications which provide reasons for choosing one term/definition over the other, this choice will be a difficult one to make and not considering relevant implication can likely hinder positive reception of the guideline.
ESC 3: Need for action
Another important question that the GDs will have to address is what ethical issues to focus on. In some circumstances this issue will have been answered externally (for example, if an organisation commissions a group to develop guidance for one specific intervention or situation). However, more often guidance will be sought for a particular context (for example, tuberculosis treatment and vaccination) which is wrought with many ethical issues that cannot all be addressed, but will have to be prioritised. In making these decisions GDs will have to consider (a) what ethical issues arise in a given context and (b) the reasons for prioritising certain issues over others.
Information that can support these decisions are overviews of ethical issues pertaining to the topic of interest. Such overviews have been generated for diverse contexts, for example, the care for patients with amyotrophic lateral sclerosis (Seitzer et al, 2016) or big data in biomedical contexts (Mittelstadt and Floridi, 2016).
With regards to reasons for or against the prioritisation of certain issues, one important argument is urgency in terms of prevalence or severity of consequences of respective issues. Empirical evidence regarding prevalence of (known) ethical challenges might accordingly be sought. Furthermore, it could be insightful to probe stakeholders’ or affected people’s perceptions regarding the significance of identified issues. Another important argument for focusing on certain issues is lack of guidance in this regard. Where abundant guidance already exists, it might not be too helpful to create an additional guidance document focusing on the exact same points, but look for areas where ethical orientation is insufficient (Neitzke et al, 2015). For this, an overview of the content of relevant guidance documents might be helpful. Additional evidence looking into further reasons for prioritising some ethical issues over others might complement the evidence base.
ESC 4: Strategies for addressing need
When ethical issues have been prioritised, the following questions will inevitably centre around what can be done to prevent, mitigate, manage or resolve identified problems or challenges that were deemed relevant.
ESC 4 therefore highlights the need to identify and describe different strategies (course of actions, changes to social practices, and so on) for solving identified problems so that the GD can decide which strategies should be explicitly considered in developing recommendations. Sometimes it might be common sense what options or solutions are attainable, so there might be no need to seek evidence. In contrast, the more complex the social practice or ethical problem is, the more complex and diverse possible strategies can be. When discussing adequate public health responses to, for example, a high prevalence of obesity, it might be worthwhile to know the variety of obesity prevention programmes developed and implemented as relevant background information for sensible recommendations. When identifying approaches, however, not just singular programmes but also more systemic approaches (including those often discussed in terms of ‘nudging’ [Bowie, 2009]) should possibly be considered. When several ethical problems or challenges have been identified, it will be necessary to identify solutions for each issue, while additional evidence generation will not necessarily always be demanded.
ESC 5: (Hypothetical) arguments for action
Lastly, GDs will have to decide what actions they recommend. To come to a conclusion, they will have to consider the relevant arguments for and against the strategies identified as part of ESC 4. These arguments must be categorised and evaluated against the background of the decisions in ESC 1, 2 and 3: The value base (ESC 1), that is, the fundamental ethical principles, norms and values for the ethics guideline, is decisive for assessing whether the normative premises of the arguments are acceptable or incompatible with the basic normative assumptions. The concepts used (ESC 2) and, in some cases, the arguments for the need for action (ESC 3) play a role with regard to the relevance and pertinence of the arguments found for the selected solution strategies (ESC 4).
It should be emphasised that many different types of arguments might be considered in ESC 5: ‘Classical’ ethical arguments like ‘A should (not) be done because it does (not) adequately consider principle B (for example, justice)’, but also economic or organisational arguments (for example, discussing barriers to implementation), social arguments (for example, addressing acceptability of proposed strategies or effects on power relations in relevant social groups) or legal arguments (for example, arguing that a certain action might be illegal, or clarification of whether and for whom claims for reimbursement exist). Such arguments (and the aspects of a health technology addressed by them) can be significant against the background of ethical principles, norms and values; they may, for example, have something to do ethically with obligations of beneficence, non-maleficence, the preservation of patient autonomy or justice. Frameworks for decision-criteria in health policy contexts might provide good overviews of the different types of arguments that will have to be considered, but should be chosen in accordance with the guidance topic (Goetghebeur et al, 2008; Tromp and Baltussen, 2012; Rehfuess et al, 2019). Arguments in the context of ESC 5 can also refer to larger societal consequences, in particular also to possible unintended consequences. However, the extent to which these should be taken into account also depends on the decisions made in ESC 1 and ESC 3 and on the focus of recommendations; those who primarily want to address doctors or a specific organisation may not need to address major societal transformations through technology in their recommendations.
Clearly, overviews of the arguments provided in the literature for or against a certain action will be helpful in this regard as they supply GDs with (initial) normative evidence. One example of where such an overview of relevant arguments was attempted is a systematic review addressing whether research participants should have access to trial drugs after the end of the trial (Sofaer and Strech, 2011). Provided that the respective normative arguments are also generally based on empirical premises – the truth of which may be questionable – the soundness, strength or plausibility of these arguments will have to be substantiated with empirical evidence. This is necessary to allow an estimate of how likely proclaimed effects will actually materialise. It should be kept in mind, however, that it might not always be possible to find salient empirical evidence. Sometimes, empirical research to produce relevant evidence is not even ethically admissible (for example, testing the psychological harm caused by active stigmatisation of obese people as a weight reduction strategy). Often even without such empirical evidence, the plausibility of an argument can be assessed while it remains important for GDs to be transparent about potential gaps in evidence.
Strategies for evidence collection for mainly normative evidence
Having identified the questions for which evidence could be collected, the question is how this should be done. As mentioned earlier, the last decades have seen much discussion and refinement of methods for collecting, analysing and synthesising empirical evidence for decision-making. Where empirical evidence is required to support ethics guideline development, we refer GDs to the relevant publications (Schünemann et al, 2013; WHO, 2014b; Page et al, 2021; Higgins et al, 2023). The following discussion will focus on normative evidence.
In contrast to empirical evidence, normative evidence can be gathered from a more diverse set of sources. Clearly, the academic literature or academic experts will be important sources for normative evidence. However, particularly when it comes to identifying relevant ethical issues, it might also be worthwhile to gather information from other stakeholders (for example, patient organisations, health professionals, governance bodies, and so on). Their perspectives might add important insights because, for example, the situation of people living with particular diseases or in certain contexts might not be adequately presented or values of populations concerned with the policy not expressed within the academic discourses. For example, a systematic review on ethical issues in public health surveillance found that the literature did rarely address issues of low-income countries like the need to prioritise in situations of limited resources (Klingler et al, 2017). It is important to reflect on possible biases and limited representation of certain perspectives in the academic discourse due to imbalances in power and money. On the other hand, information gathered from other sources, specifically stakeholder groups, should not be received uncritically due to possible conflicts of interest. It might therefore often be reasonable to access several sources of normative evidence to gain a comprehensive understanding. Table 1 provides an overview of different sources of evidence and strategies or methods to tap into those sources. Who the relevant stakeholders are will have to be determined in accordance with the topic and context, but should include all groups concerned with the planned policy.
Sources of normative evidence and factors possibly affecting saturation of evidence body
| Sources | Evidence collection strategy | Up-/downgrading implementation factors (from the perspective of thematic/argumentative saturation) | |
|---|---|---|---|
| External sources | Academic literature | Systematic review |
|
| Unsystematic or narrative literature review | |||
| Several single papers | |||
| Single paper (n=1) | |||
| Academic experts | Consensus process |
|
|
| Workshop | |||
| Commissioned theory application |
|
||
| Consultation (written or verbal) |
|
||
| Additional stakeholders | Interviews/focus groups |
|
|
| Opinion survey | |||
| Consensus process | |||
| Workshop | |||
| Consultation (written or verbal) |
|
||
| Internal source | GD group members | View of a single (or various) member(s) of the GD group |
|
| Consensus of all members of the GD group |
|
||
When choosing whether and how to collect evidence, GDs need to think about time and financial resources available for engaging in further evidence collection, but also how likely it is that the different methods for information collection will provide a high-quality, reliable evidence base. No final answer can be given to that question as hardly any research is conducted on those questions. For example, the ‘gold standard’ for collecting evidence on empirical questions are systematic reviews and meta-analyses. This method has also been translated to and used in the normative realm (Mertz et al, 2016; Kahrass et al, 2023), but other methods are more common – particularly the application of theory/theories or sets of principles to identify ethical issues or arguments. There has – so far – been no research conducted to compare the (quality of the) outcomes of employing these different approaches, which makes it difficult for GDs to decide how to go about evidence collection.
Certain points should nevertheless be kept in mind. First, there is a difference between the quality of (a) the individual information unit (for example, the plausibility of the individual argument) and (b) the body of evidence (for example, all the arguments collected).
To (a): As part of the process of collecting empirical evidence, generally a quality appraisal of individual studies or information units is conducted. It would be desirable to do the same as part of the process of normative evidence collection, however, there exists so far hardly any guidance to do this (Mertz, 2019) and tools developed require a high level of interpretative judgement (Scott et al, 2017). The reason is that it is not trivial to assess the quality of, for example, an argument. To do so, comprehensive knowledge of formal and informal logic and argumentation theory is necessary. It is not clear whether such knowledge can be broken down in checklists for those collecting evidence (as exist for empirical evidence), but it would be a worthwhile endeavour to generate such support tools.
To (b): Apart from the quality of individual information units, the body of evidence will be most helpful for GDs if it includes all relevant (and sufficiently qualified) argument/issues/values, and so on, meaning that thematic or argumentative saturation is reached. Such a criterion has been also discussed under other labels (for example, completeness) in the literature (Scott et al, 2017). Clearly, some methods will be more likely per se to create a saturated body of evidence (for example, systematic review versus reading a single paper). However, this will also depend on the way the method is implemented. For example, it will make a difference whether a stakeholder discussion group consists of a diverse set of stakeholders presenting various perspectives or is a rather homogeneous group; or whether the group follows formal procedures that ensure everybody is heard or more informal procedures where speaking time is allotted according to personal characteristics (such as command of used language or shyness). Table 1 provides an overview of implementation factors that might affect the comprehensiveness of the resulting body of evidence. In addition, the methods might be a better or worse fit to the topic and/or context. For example, if guidelines are to be formulated for a new technology or a newly emerged public health threat, it is possible that no paper has yet been written on the topic of interest (or one sufficiently similar), rendering every strategy that relies on searching the academic literature for that specific topic almost useless. For these reasons, we do not propose a hierarchy of strategies, but the GDs have to rely on their judgement to prioritise strategies for evidence collection in their context/for their topic.
In summary, the GDs must make decisions regarding evidence collection that have to be made under circumstances of uncertainty, but not ignorance. The points raised should be taken into consideration and decisions be made transparently so that future GDs can learn from them.
Discussion and conclusions
The approach presented attempts to offer a possible framework for a more ‘evidence-based’ development of ethics guidelines and similar ‘normative products’, for example, in the HTA field. For this purpose, the concept of evidence could not be uncritically adopted from the existing EBM tradition. Rather, the role of evidence in the context of ethics guidelines had to be critically examined, and the term adapted. In particular, it had to be considered whether a correlate on the normative or evaluative side can be assumed in addition to the empirical information that has so far been captured by the evidence label. Therefore, the REIGN framework proposes to distinguish between empirical evidence and normative evidence, both of which are (ideally) needed to develop and, most importantly, justify a recommendation in a guideline. Especially for the generation of normative evidence, the REIGN framework shows possible sources, reflects on the challenges of assessing quality, but also indicates (current) limitations. In addition, it suggests where in the development and justification of a recommendation what evidence is (ideally) needed and why – specified by a complex of questions or so-called evidential support components (ESCs), which draw on normative and empirical evidence in varying proportions, and which are all necessary to justify the final ethics recommendations. Since evidence is always related to a knowledge system, it is important to consider the associated limits of normative evidence in a pluralistic society and to expand knowledge systems, for example, through discourse (interdisciplinary and between experts and laypersons) and stakeholder participation.
The REIGN framework thereby fills a gap that existing frameworks supporting evidence collection for policy decision-making have not addressed so far. Most frameworks explicitly or implicitly postulate certain values and norms and deduce from those the necessity to seek and use certain types of empirical evidence. The WHO, for example, postulates values like equity and human rights and demands to (also) consider certain equity indicators in making decisions (WHO, 2014a). Most HTA approaches assume – based on accepted ethical norms of beneficence, non-maleficence and fairness – empirical information about effectiveness, safety and economic efficiency to become crucial for assessing a particular health technology (Kristensen et al, 2017; Lysdahl et al, 2016). Some frameworks like those mentioned in what follows also address the ‘ethical aspects’ of a given technology or intervention, thereby implying that additional aspects, values or norms not captured by the effectiveness, safety, and so on, dimensions might become relevant in policy making. Those, however, often only insufficiently address the question of which type of evidence is to be searched with what methods and used in what way. None explicitly address how to generate and/or use ‘evidence’ for the (underlying) normative sentences that are inevitably part of a recommendation. INTEGRATE-HTA provides a helpful overview of different approaches to solve ethical issues that might be associated with a given technology and used in HTA or other evidence-to-policy products. While these approaches generally provide theory, principles or questions (for example, the principles of beneficence, non-maleficence, autonomy and justice) to identify and possibly solve ethical issues, they do not or only superficially discuss the question of evidence in this regard. The Socratic approach (Hofmann et al, 2014), for example, only mentions systematic literature searches as methods for evidence collection with little guidance on how to adapt workflows to the ethics domain. The REIGN framework provides a more in-depth discussion and therefore orientation regarding the role evidence can play in arriving at recommendations. In addition, the REIGN framework does not presuppose that the issue at hand, the terms or the ethical principles are fixed upfront, but opens up a broader view on the use of evidence that transcends (probable) consequences of a technology (assessment) which are in the focus of most other frameworks. Overall, therefore, a strength of the REIGN framework is to provide a more comprehensive and systematic view of the role of evidence in the development and justification of ethics guidelines.
Limitations
However, the REIGN framework and its applicability clearly have limitations. Regarding the status quo of current practice, only a few countries and their respective institutions (for example, public health institutions, HTA agencies, and so on) could be screened for approaches to integrate evidence in ethics guideline development. Likewise, regarding the status quo of the academic discourse, only a limited review of such approaches was possible; still, the risk that crucial literature with this particular focus has been completely overlooked can still be considered low against the backdrop of the search strategy and the expert knowledge of the authors.
The framework itself is currently still rather general and cannot answer every ‘when and how’ question in the development of guidelines or recommendations. The framework can, however, provide a basis for structuring deliberations (for example, which sources for evidence should be considered, how can we think about quality of information points). Furthermore, the framework is based on certain theoretical assumptions that not everybody might share. This is not problematic in itself, as it applies to all comparable frameworks. Nevertheless, some more daring assumptions, for example, about the nature of evidence and the possibility of ‘normative evidence’, or the role and relation between empirical and normative evidence when it comes to developing and justifying recommendations, can certainly be disputed. Also, implying that systematic reviews in ethics have a certain relevance – or even the mere consideration of this method for ‘doing ethics’ – can be contested (Birchley and Ives, 2022). As the REIGN framework at the moment remains the work of only two researchers with their specific academic vitae, it is very important that the framework is discussed interdisciplinarily (by experts from guideline development, evidence-to-policy-approaches, ethics, healthcare policy, and so on), and that more concrete guidance for future practice is developed on this basis.
Implications for research and practice
Given the mentioned limitations and due to the fact that several conceptual questions are still debated and unsettled in the relevant scientific communities, some open questions remain that have to be addressed in further research.
Some might argue that a more ‘classical’ philosophical approach might be more appropriate when it comes to ethics guideline development. This would mean working with ethics theories and deducing arguments to arrive at recommendations. In our terminology, this would also be an approach based on evidence as empirical information will be taken into consideration, however, one that probably collects and uses evidence less systematically and transparently. Where such an approach is favoured, procedural questions remain nonetheless unanswered, for example, whether dedicated ethics experts are needed. Most importantly, it remains unclear whether such an approach achieves better or optimal results compared to an approach using ‘evidence-to-policy’ frameworks (such as REIGN). While relying on ethics theories and so on for developing ethics guidelines is definitely simpler and less costly, it is also less systematic and transparent, and resulting recommendations probably risk being considered as mere products of ‘ideology’, expressed, inter alia, by the ethics theories used.5
Developing a framework and reviewing it from the theoretical side is one thing; testing whether it can be applied in practice, where implementation problems arise, and whether it leads to the desired effects is another. Accordingly, it will be of importance to evaluate how REIGN fares in practice – particularly in comparison to more ‘classical’ approaches to ethics guideline development. However, it is not at all clear how to evaluate the REIGN framework and what endpoints would be important. It would be a good first step to see how REIGN is perceived in terms of usability by guideline developers. It would also have to be empirically investigated how the effectiveness of ethics guidelines is impacted by the usage of REIGN. Does it impact how well a guideline is disseminated, how high the acceptance of the recommendations is among the addressees, whether a specific practice changes according to the guideline, and – particularly difficult to assess – whether the practice becomes more ethical as a result?
Either way, to make evaluation possible, the guideline development process should be transparently reported in resulting guidelines (or accompanying documentation). This will also be important to allow readers to assess the quality of the development process (and possibly resulting recommendations and conclusions). Such reporting should at least reflect which evidence was sought and why. It can be structured in accordance with the ESCs mentioned earlier. Guideline developers should list for each ESC which (normative or empirical) information or sources of information were used, how these sources were accessed – and assessed – and what limitations are associated with the evidence base available.
However, also the process of guidelines development warrants further investigation and should accordingly be transparently reported. Since the GD have a very powerful role and can determine what evidence is sought in the first place, but are also largely responsible for the weighing of arguments and final formulation of recommendations, it needs to be examined more closely what the composition of a GD group should ideally look like and how the process should be structured. For example, while the WHO Handbook for Guideline Development (WHO, 2014b) proposes which groups should be represented in the GD group (for example, technical experts, affected communities, methodologists), it is currently unclear whether this applies to ethics guidelines in the same way, or whether there could be adjustments, especially with regard to the consideration of ethical expertise. Since there is presumably a certain danger, especially in the case of ethical issues, that subgroups will want to assert their interests (whether for political-ideological, moral or economic reasons), it is important to design the process in such a way that those rational reasons for which there is good empirical and, above all, normative evidence cannot be undermined merely by the power that some subgroups may have.
What also needs to be considered is that while the GD group commissions evidence collection, reviews the evidence and formulates final recommendations, there will also be people charged with collecting and analysing the evidence. While the necessary expertise will depend on the methods chosen for evidence collection, those commissioned should have sufficient proficiency regarding ‘ethics’ to be able to identify important sources and to assess the relevance of identified information for planned recommendations. Possible criteria include a certain degree of familiarity with ethics (topics, analysis, argumentation, and so on), understood as a normative enterprise (that is, not seen solely from a more sociological, descriptive perspective); but such criteria might need further refinement.
For a reliable evidence base, conducting systematic reviews should be actively considered and discussed during development. Systematic reviews can be crucial when there is much discussion about the ‘right’ course of action, as they help mitigate possible risks of bias when only a few sides are ‘heard’ during development or when a topic is discussed by various (sub)disciplines. Systematic reviews will be less valuable in certain contexts, for example, where the topic is comparably new and only a few academic publications on the topic of interest are to be expected. However, conducting systematic reviews is always also a question of available time and human resources. Therefore, how important or valuable a review could be for the final recommendation must be discussed (for example, for each ESC). This method cannot be considered the first choice a priori. Therefore, the pros and cons should be discussed beforehand, and it should be determined whether other strategies, which may be less reliable than systematic reviews, also suffice. If systematic reviews in ethics are recognised as a useful method (depending on the question and objective), there are still further questions regarding how to conduct such reviews, for example, on suitable search strategies, ways of synthesising the qualitative findings, and especially regarding any kind of quality appraisal as part of the method.
This underlines that there remain many open questions and the REIGN framework thus cannot offer a cookbook approach to ethics guideline development. However, it can support GDs in thinking more systematically and deciding more transparently about the role of evidence in guideline development processes.
Notes
A pre-edited version of the discussion paper can be accessed here: https://www.mhh.de/reign-framework.
A list of the policy documents found and considered can be found in Appendix A, a list of the considered academic literature in Appendix B of the REIGN discussion paper (see note 1).
Much data considered evidence will take on a mixed form in between pure empirical or normative evidence (for example, normative evidence that is already contextualised or incorporates empirical data in an argument). However, when developing a methodology for handling diverse forms of information, it is helpful to have clearly delineated these ‘ideal’ types of evidence.
The value base does not have to be consistent in the sense that principles can never come into conflict. It is to be expected that certain values identified as relevant can likely compete in certain situations (for example, certain health benefits might only be realisable when restricting someone else’s freedom). These situations might indeed be identified as relevant ethical issues given the context at hand as part of ESC 3.
Which has been an experience shared by us and our colleagues when we had to explain our philosophical methods to guidelines review committees used to reviewing clinical guidelines.
Funding
This work was supported by the World Health Organization (WHO).
Contributor statement
CK and MM jointly conceptualised the study, conducted the literature review, analysed the data, developed the framework, wrote parts of the first draft and revised subsequent versions of the manuscript.
Data availability statement
The authors are willing to share the data on which the analysis was based. Data and materials can also be found in the discussion paper providing the basis for this manuscript: https://www.mhh.de/reign-framework.
Research ethics statement
The authors of this article have declared that research ethics approval was not required since the article does not present or draw directly on data/findings from empirical research.
Conflict of interest
The authors declare that there is no conflict of interest.
References
Birchley, G. and Ives, J. (2022) Fallacious, misleading and unhelpful: the case for removing ‘systematic review’ from bioethics nomenclature, Bioethics, 36(6): 635–47, doi: 10.1111/bioe.13024
Bosch-Capblanch, X., Lavis, J.N., Lewin, S., Atun, R., Røttingen, J.A., Dröschel, D., et al (2012) Guidance for evidence-informed policies about health systems: rationale for and challenges of guidance development, PLOS Medicine, 9(3): e1001185, doi: 10.1371/journal.pmed.1001185
Bowie, N.E. (2009) How empirical research in human cognition does and does not affect philosophical ethics, Journal of Business Ethics, 88: 635–43, doi: 10.1007/s10551-009-0309-4
Brock, D.W. (1987) Truth or consequences: the role of philosophers in policy-making, Ethics, 97(4): 786–91, doi: 10.1086/292891
Burls, A., Caron, L., Cleret de Langavand, G., Dondorp, W., Harstall, C., Pathak-Sen, E., et al (2011) Tackling ethical issues in health technology assessment: a proposed framework, International Journal of Technology Assessment in Health Care, 27(3): 230–37, doi: 10.1017/S0266462311000250
DeJean, D., GIacomini, M., Schwartz, L. and Miller, F.A. (2009) Ethics in Canadian health technology assessment: a descriptive review, International Journal of Technology Assessment in Health Care, 25(4): 463–69, doi: 10.1017/S0266462309990390
Goetghebeur, M.M., Wagner, M., Khoury, H., Levitt, R.J., Erickson, L.J. and Rindress, D. (2008) Evidence and value: impact on DecisionMaking – the EVIDEM framework and potential applications, BMC Health Services Research, 8(1): 270, doi: 10.1186/1472-6963-8-270
Higgins, J., Thomas, J., Chandler, J., Cumpstom, M., Li, T., Page, M.J., et al (eds) (2023) Cochrane handbook for systematic reviews of interventions version 6.4, Cochrane, https://training.cochrane.org/handbook.
Hofmann, B., Droste, S., Oortwijn, W., CLeemput, I. and Sacchini, D. (2014) Harmonization of ethics in health technology assessment: a revision of the Socratic approach, International Journal of Technology Assessment in Health Care, 30(1): 3–9, doi: 10.1017/S0266462313000688
IQWiG (2023) Allgemeine Methoden. Version 7.0, https://www.iqwig.de/methoden/allgemeine-methoden_version-7-0.pdf.
Kahrass, H., Borry, P., Gastmans, C., Ives, J., van der Graaf, R., Strech, D., et al (2023) RESERVE – REporting of SystEmatic ReViews in Ethics: development, explanations and examples, Bioethics Open Research, 1: 4, doi: 10.12688/bioethopenres.17486.1
Kahrass, H., Schnarr, A., Faggion, C.M. and Mertz, M. (2025) Ethics domains in full HTA reports: an attempt to begin mapping the field, International Journal of Technology Assessment in Health Care.
Klingler, C., Silva, D.S., Schuermann, C., Reis, A.A. and Strech, D. (2017) Ethical issues in public health surveillance: a systematic qualitative review, BMC Public Health, 17(1): 295, doi: 10.1186/s12889-017-4200-4
Kristensen, F.B., Lampe, K., Wild, C., Cerbo, M., Goettsch, W. and Becla, L. (2017) The HTA Core Model® - 10 years of developing an international framework to share multidimensional value assessment, Value in Health, 20(2): 244–50, doi: 10.1016/j.jval.2016.12.010
Kulkarni, A.V. (2005) The challenges of evidence-based medicine: a philosophical perspective, Medicine, Health Care, and Philosophy, 8(2): 255–60, doi: 10.1007/s11019-004-7345-8
Kunz, R., Ollenschläger, G., Raspe, H., Jonitz, G. and Donner-Banzhoff, N. (eds) (2007) Lehrbuch Evidenz-basierte Medizin in Klinik und Praxis, Cologne: Deutscher Ärzte-Verlag.
Lee, L.M. (2012) Public health ethics theory: review and path to convergence, The Journal of Law, Medicine & Ethics, 40(1): 85–98, doi: 10.1111/j.1748-720X.2012.00648.x
Lysdahl, K., Mozygemba, K., Burns, J., Chilcott, J.B., Brönneke, J.B. and Hofmann, B. (2016) Guidance for assessing effectiveness, economic aspects, ethical aspects, socio-cultural aspects and legal aspects in complex technologies, https://validatehta.eu/wp-content/uploads/2022/07/3.-Guidance-for-assessing-effectiveness-economic-aspects-ethical-aspects-socio-cultural-aspects-and-legal-aspects-in-complex-technologies.pdf.
Mertz, M. (2011) Evidenzbasierte Klinische Ethik. Philosophische Untersuchungen zur Verwendung von Empirie und Evidenz in der (Medizin-)Ethik, München/Ravensburg: GRIN Verlag.
Mertz, M. (2019) How to tackle the conundrum of quality appraisal in systematic reviews of normative literature/information? Analysing the problems of three possible strategies (translation of a German paper), BMC Medical Ethics, 20(1): 81, doi: 10.1186/s12910-019-0423-5
Mertz, M. and Strech, D. (2014) Systematic and transparent inclusion of ethical issues and recommendations in clinical practice guidelines: a six-step approach, Implementation Science, 9(1): 184, doi: 10.1186/s13012-014-0184-y
Mertz, M., Kahrass, H. and Strech, D. (2016) Current state of ethics literature synthesis: a systematic review of reviews, BMC Medicine, 14(1): 152, doi: 10.1186/s12916-016-0688-1
Mittelstadt, B.D. and Floridi, L. (2016) The ethics of big data: current and foreseeable issues in biomedical contexts, Science and Engineering Ethics, 22(2): 303–41, doi: 10.1007/s11948-015-9652-2
Molewijk, B., Stiggelbout, A.M., Otten, W., Dupuis, H.M. and Kievit, J. (2008) First the facts, then the values? Implicit normativity in evidence-based decision aids for shared decision-making, Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen, 102(7): 415–20, doi: 10.1016/j.zefq.2008.08.014
Muir Gray, J. (2001) Evidence-Based Healthcare: How to Make Health Policy and Management Decisions, Edinburgh and New York: Churchill Livingston.
Neitzke, G., Riedel, A., Brombacher, L., Heinemann, W., Herrmann, A. and AG Ethikberatung im Gesundheitswesen (2015) Empfehlungen zur erstellung von ethik-leitlinien in einrichtungen des gesundheitswesens, Ethik in der Medizin, 27(3): 241–8, doi: 10.1007/s00481-015-0354-8
Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D., et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, BMJ (Clinical Research Ed.), 372: 71, doi: 10.1136/bmj.n71
Polus, S., Mathes, T., Klingler, C., Messer, M., Gerhardus, A., Stegbauer, C., et al (2019) Health technology assessment of public health interventions published 2012 to 2016: an analysis of characteristics and comparison of methods, International Journal of Technology Assessment in Health Care, 35(4): 280–90, doi: 10.1017/S0266462319000515
Rehfuess, E.A., Stratil, J.M., Scheel, I.B., Portela, A., Norris, S.L. and Baltussen, R. (2019) The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective, BMJ Global Health, 4(Suppl 1): e000844, doi: 10.1136/bmjgh-2018-000844
Reiter-Theil, S., Mertz, M., Schürmann, J., Stingelin Giles, N. and Meyer-Zehnder, B. (2011) Evidence – competence – discourse: the theoretical framework of the multi-centre clinical ethics support project METAP, Bioethics, 25(7): 403–12, doi: 10.1111/j.1467-8519.2011.01915.x
Rycroft-Malone, J., Seers, K., Titchen, A., Harvey, G., Kitson, A. and McCormack, B. (2004) What counts as evidence in evidence-based practice?, Journal of Advanced Nursing, 47(1): 81–90, doi:10.1111/j.1365-2648.2004.03068.x
Saarni, S.I., Braunack-Mayer, A., Hofmann, B. and van der Wilt, G.J. (2011) Different methods for ethical analysis in health technology assessment: an empirical study, International Journal of Technology Assessment in Health Care, 27(4): 305–12, doi: 10.1017/S0266462311000444
Sackett, D., Straus, S., Richardson, W., Rosenberg, W. and Haynes R.B. (2000) Evidence-Based Medicine: How to Practice and Teach EBM, 2nd edn, Edinburgh: Churchill Livingstone.
Schleidgen, S., Klingler, C., Bertram, T., Rogowski, W. and Marckmann, G. (2013) What is personalized medicine: sharpening a vague term based on a systematic literature review, BMC Medical Ethics, 14(1): 55, doi: 10.1186/1472-6939-14-55
Schmücker, R. (2016) Incidental findings: definition of the concept, in S. Weckbach (eds) Incidental Radiological Findings, Cham: Springer, pp 3–7.
Schünemann, H., Brozek, J., Guyatt, G. and Oxman, A. (2013) GRADE handbook for grading quality of evidence and strength of recommendations, The GRADE Working Group, https://gdt.gradepro.org/app/handbook/handbook.html.
Scott, A.M., Hofmann, B., Gutiérrez-Ibarluzea, I., Lysdahl, K.B., Sandman, L. and Bombard, Y. (2017) Q-SEA – a tool for quality assessment of ethics analyses conducted as part of health technology assessments, GMS Health Technology Assessment, 13: Doc02, doi: 10.3205/hta000128
Sehon, S.R. and Stanley, D.E. (2003) A philosophical analysis of the evidence-based medicine debate, BMC Health Services Research, 3: 14, doi: 10.1186/1472-6963-3-14
Seitzer, F., Kahrass, H., Neitzke, G. and Strech, D. (2016) The full spectrum of ethical issues in the care of patients with ALS: a systematic qualitative review, Journal of Neurology, 263(2): 201–9, doi:10.1007/s00415-015-7867-4
Sofaer, N. and Strech, D. (2011) Reasons why post-trial access to trial drugs should, or need not be ensured to research participants: a systematic review, Public Health Ethics, 4(2): 160–84, doi: 10.1093/phe/phr013
Sørensen, K., Van den Broucke, S., Fullam, J., Doyle, G., Pelikan, J., Slonska, Z., et al (2012) Health literacy and public health: a systematic review and integration of definitions and models, BMC Public Health, 12(1): 80, doi:10.1186/1471-2458-12-80
Stoklosa, A. (2013) The Concept of Evidence in Health Technology Assessment (HTA), Dissertation at the Institute for the History and Philosophy of Science and Technology, University of Toronto, https://tspace.library.utoronto.ca/handle/1807/69007.
Strech, D. (2008) Evidenz und Ethik. Kritische Analysen zur Evidenz-basierten Medizin und empirischen Ethik, Berlin: Lit Verlag.
Tromp, N. and Baltussen, R. (2012) Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers, BMC Health Services Research, 12(1): 454, doi: 10.1186/1472-6963-12-454
World Health Organization (2014a) Incorporating ethics, human rights, gender and social determinants into guidelines, in WHO Handbook of Guideline Development, 2nd edn, Geneva: World Health Organization, pp 45–56.
World Health Organization (2014b) WHO handbook for guideline development, https://iris.who.int/handle/10665/145714.
Young, K., Ashby, D., Boaz, A. and Grayson, L. (2002) Social science and the evidence-based policy movement, Social Policy and Society, 1(3): 215–24, doi: 10.1017/s1474746402003068
