1. Evidence-informed in their aims and needs analysis

Evidence-informed policy and practice is where relevant research findings are used in an appropriate and useful way to inform decision making. Evidence claims may be justified in some contexts but may be applied to decisions where they have no or limited relevance; for example, evidence about what works on average may be ineffective or even harmful in specific circumstances (and vice versa).

Matching the needs of the decision maker to the questions asked and the contexts in which they apply involves some engagement between decision making and research production. This can be conceived of as an evidence ecosystem that exists within a wider system of various stakeholders influencing research production and research use (Best and Holmes 2010; Gough et al, 2019). The main components of such an evidence ecosystem include decision making, research production and some engagement between such decision making and research production. All of this occurs within wider systems and contexts. An awareness of the components and functioning of such an evidence ecosystem (as in Figure 1) can help KBIs and other actors to plan and assess their work.

View Full Size

Evidence ecosystems (developed from Gough et al, 2011)

Citation: Evidence & Policy 18, 4; 10.1332/174426421X16353477842207

• Download Figure
• Download figure as PowerPoint slide

KBIs aim to facilitate the functioning of such evidence ecosystems in some way. An obvious starting point is therefore to assess the functioning of the evidence ecosystem that they plan to work or are currently working within. What is the pre-existing nature of research production, engagement with that research by users, and actual use of evidence in decision making? An assessment of this context can inform the choice of strategies for enabling such use of research evidence. KBIs can thus be seen as interventions into pre-existing evidence ecosystems to improve such systems in some way.

So, to what extent do KBIs systematically undertake such appraisal of the nature and extent of any deficiency in the relationship between the use and production of research evidence in their field? And having made such an appraisal, what are their strategies for intervening to improve the functioning of that evidence ecosystem?

What Works Centres (WWCs) are one type of KBI and evidence use infrastructure. In a study of UK WWCs (Gough et al, 2018), the most common aims identified were:

• primary research base: development of primary research (for example, Education Endowment Foundation (EEF));

• co-production: by researchers and users of primary and secondary research (for example, What Works Scotland);

• synthesis: clarifying the knowledge base (all of the UK WWCs);

• user access to research: communication of the evidence to professional practitioners (all of the UK WWCs);

• supporting evidence uptake: enabling the consideration and uptake of research (all of the UK WWCs);

• evidence-informed guidance: developing guidelines/recommendations for practice (for example, National Institute for Health and Care Excellence (NICE));

• enabling implementation: of decisions that have been informed by research evidence, including the use of strategies informed by the behavioural needs of users (for example, EEF, and Early Intervention Foundation (EIF)).

The centres thought that there were important needs to be fulfilled that they could serve. They thought that it was important for decision makers to have access to research evidence or guidance informed by research. The logic is that it is more efficient for a national service to identify relevant research evidence than for individual policymakers and practitioners to do so. Nevertheless, it was not always clear why providing access was often the only or predominant aim. There were other aspects of the evidence ecosystem that could be attended to. At the time of the study, some of the centres did take a more holistic approach to appraising, or enabling, all parts of their evidence ecosystems, and the wider systems within which these existed (including political dynamics), though these might not be included in their public descriptions of their work. There was also some explicit discussion of how different KBIs might relate to and interact with each other.

There are also differences in the type of policy and practice issues and related research evidence that KBIs engage with. Many KBIs focus on the identification and implementation of effective interventions or ‘what works’. For some of these the emphasis is on manualised programmes for intervention with a concern for fidelity of application. Others emphasise effective strategies and mechanisms that can be applied differently in different contexts (Gough, 2021).

There has been development over time in the aims and methods of the UK WWCs. Most started with a focus on the synthesis and communication of evidence, and then have developed an increased focus on user engagement and implementation. EEF in particular has invested in developing and, most importantly, evaluating a number of different strategies for enabling the use of evidence (Sharples et al, 2019), including how schools use research as a result of engaging with the Research Schools Network (Gu et al, 2020) and the scale-up of research-informed practice in the use of teaching assistants in schools (Maxwell et al, 2019). Another example is the Early Intervention Foundation’s (EIF) work on ‘supporting evidence use in policy and practice’ (Waddell, 2021), which includes better understanding of the behavioural needs of users (Waddell and Sharples, 2020). Bache (2020) has also written about the role of evidence in the work of the What Works Centre for Wellbeing.

A similar principle also applies to expert scientific advisory committees. They are explicit in that they provide science advice to parliaments and government departments. What is less clear is the rationale for developing this type of structure, as opposed to other ways that government could access research evidence, such as academic societies and government research analysts (Gough, 2020).

In sum, there are opportunities for KBIs to be more explicit about their analysis of the ecosystem in which they are intervening, what is most needed to improve the functioning of this system, why they have chosen their specific strategy, and how their contribution fits into this wider picture. Questions to consider are the extent to which KBIs are evidence-informed in their aims and needs analysis in terms of:

• analysis of the evidence ecosystem: how has the KBI assessed the pre-existing relationships between the use of research and its production and the ways in which it proposes to enhance this? The aims may be evident from a KBI’s name, but is there a justification of why a particular approach has been chosen over others?

• specific aims: which particular parts of the pre-existing evidence ecosystem does a KBI wish to change? What does it wish to change? What type of user issues and what types of research evidence and evidence claim does it focus on?

• users and beneficiaries: who will use and/or benefit from the KBI’s work?

• KBI development over time: what changes are there in the focus of their work over time, and the reasons for this (including changes in the wider evidence ecosystem or their position within it)?

• collaboration within the evidence ecosystem: what interactions are there with the other actors (including overlaps with the aims and work of other KBIs and collaboration with them)?

2. Evidence-informed in their methods and theory of change

In addition to, and highly related to the aims of the KBIs, are the methods and theories of change by which these aims will be achieved. If the aim is, for example, to synthesise and communicate evidence, KBIs will likely state the methods they use to achieve this. The study of WWCs (Gough et al, 2018) and another study of evidence web portals (Gough and White, 2018) found considerable variation in the nature and the extent of their description of KBIs’ methods of work in terms of:

• the use of standardised specific methods, guidance that allows flexibility, or individual project-specific methods;

• explaining and justifying the choice of specific methods;

• the quality of reporting of those methods.

KBIs are increasingly developing Theories of Change, and in doing so are becoming more explicit about how their methods will be effective in achieving their fundamental aims (for example, Blache, 2020; Gough, 2021; Waddell, 2021). But there are still instances of KBIs seeming to assume that an approach will be effective and useful without being explicit about why that might be so.

This is well illustrated by the aim of communicating research findings, which is often adopted as a default approach to supporting user engagement and decision making (Davies et al, 2015). This is despite the evidence from ‘research on research use’ that the communication of research findings, on its own, is not associated with increased use of those findings (Langer et al, 2016). EEF has shown this through its multi-armed randomised controlled trial of different ways to communicate research on literacy to teachers, where no evidence was found that any of these strategies were on their own effective (Lord et al, 2017). The communication of evidence seems to be a necessary but not a sufficient condition for the use of evidence.

There are behavioural factors to consider, such as the capacity (personal attributes), opportunity (environmental attributes), and motivations (psychological processes) that enable the use of evidence (Michie et al, 2011). Research use activities can often be driven by a desire by researchers for their findings to have impact, rather than by user demand for research on particular topics and perspectives, or more nuanced interactions between evidence and policy (Boswell and Smith, 2018; Langer et al, 2016). This can be addressed by KBIs as in, for example, the previously mentioned EIF project that designed KM strategies based on an understanding of the behavioural needs of research users (Waddell and Sharples, 2020).

There are also strong examples of KBIs integrating user perspectives into their work. The National Institute for Health and Care Excellence (NICE), for example, has a stakeholder-driven process for identifying health and social care practice questions, commissioning systematic reviews to address these issues (including the cost/benefits of different actions), and then stakeholder-driven interpretation of this to make recommendations for practice. The process is based on an explicit logic that is supported by research on stakeholder engagement, synthesis of evidence, social values, and the importance of contextual information (NICE, 2020; Gough, 2021). Similar user-informed approaches to developing guidance have been adopted by the WWC for Crime Reduction. Other centres, such as the Wales Centre for Public Policy, are demand-driven by government requests for evidence to inform policy decisions.

Some KBI strategies put an emphasis on building relationships between researchers and the potential users of research as in, for example, the secondment of researchers to government departments. However, ‘research on research’ indicates that such relationships are also a necessary, but not sufficient, condition. Relationships can have an effect on research use so long as they are accompanied by efforts to increase the capacity, opportunity and motivation for the evidence to be used in practice (Langer et al, 2016). Encouragingly, UK WWCs are currently collaborating in a project on how they can build the capacity of users to make, and act on, evidence-informed decisions, with a particular focus on effective implementation.

Similar questions about the nature of the brokerage activity can be asked of expert scientific advisory committees. There is not always clarity about how they identify and select experts to be members (including skills, topic areas, relationships with and perspectives shared with government), or the functioning of the committees and how they make decisions (Gough, 2020; Geddes, 2020). As the methods and processes are not explicit, the theories of change about their outcomes (and how this would differ from other approaches to providing science advice) are also not clear.

In sum, KBIs could build further confidence in their value and impact by demonstrating that their ways of operating are supported by or based on evidence on research use. Are KBIs evidence-informed in their methods and theories of change in terms of:

• overt consideration: (1) of both the demand (‘pull’) as well as the production (‘push’) components of the evidence ecosystem; (2) of the engagement of the planned users and beneficiaries in the work and their role and power in such decision making; (3) of the capacity, opportunity and motivation of decision makers to use research evidence in their work; (4) of potential negative effects and risks from the KBI’s work and how these will be avoided or ameliorated; and (5) of sustainability of the aims, methods and theories of change and capacity of the KBI to achieve this over time?

• theory of change: what specific methods are being used and what is the causal chain by which these are thought to achieve the interim and ultimate aims of the KBI?

• fitness for purpose and effectiveness: what is the basis for believing that the methods and theory of change are appropriate and effective and that this is supported by ‘research on research use’?

3. Credible standards for making justifiable evidence claims

KBIs aim to increase the use of research evidence by decision makers. In communicating selected research evidence, they are making claims about the trustworthiness and relevance of research evidence, and so the criteria that they use for making such evidence claims are key. The strength of evidence required to inform a decision may depend, of course, on the importance of the decision and the opportunity costs of making a decision one way or another. A short-term response to an immediate crisis is not the same as a long-term policy strategy. Whatever the nature of the decision, there is the danger that if the evidence claims are based on weak or inconsistent standards (and so not justifiable), then the users of research may be misled. Similar arguments can be made for the use of evidence provided by expert witnesses in courts (Ward, 2015) and scientific advisory committees. The evidence claims need to be both justifiable from the research and relevant to the issue at hand.

An evidence claim may not be justified for many reasons including:

• representativeness of the evidence base: the claim is made on the basis of research findings that are not representative of all the relevant and trustworthy studies on an issue;

• quality and relevance: the research is not of sufficient quality (methodologically trustworthy) and relevance to be relied upon;

• extent of evidence: the research is of sufficient quality and relevance but is not sufficient in extent to make the specific evidence claim (for example, trustworthy relevant evidence may be available on a large population but it might not be able to make justifiable claims about some sub-populations);

• interpretation and application: the research findings are not being applied appropriately to the issue under consideration.

When making a claim about what is known about a particular research question, it is of course important to consider all the relevant evidence, rather than individual studies which may not be representative of all the relevant and trustworthy evidence. The appraisal of evidence therefore requires an assessment of: (1) the ways that the evidence has been identified and brought together; (2) the quality and relevance of the studies included in such reviews, including ethical issues; and (3) the nature and extent of the totality of all the relevant evidence (Gough, 2021).

There can be dangers in only focusing on individual studies on their own. An example is the pressure that academics can be under for their research to have an impact. Their individual studies may be of high quality, but a decision maker would be better informed by knowing and being able to take account of all relevant justifiable research claims.

One way to examine the evidence standards of KBIs is to examine the evidence claims in their summaries of ‘evidence-informed’ policy and practice interventions, as in evidence portals and toolkits. The recommendations of portals may have widespread effect. Results for America, for example, has produced an Economic Mobility Catalog, which is a web-based resource for local governments to identify strategies that are effective in driving upward social mobility. The ratings of ‘good enough’ evidence are based on the ratings provided by a number of evidence portals that may have different evidence standards.

The previously mentioned small international survey of 15 national or international evidence portals found that only six of the portals used a formal (systematic) method of identifying and synthesising evidence for informing users of research about effective interventions (Gough and White, 2018). In some cases, there was variation in the standards used within a single centre.

Two of the portals used expert reviews, whereby a researcher uses their knowledge of a field to provide an overview of what is known. Such reviews may be excellent but rely on the knowledge of the expert, which may not be systematic in the depth and detail that they are able to identify, evaluate and synthesise knowledge from different studies. A similar approach is used by expert scientific advisory committees, who receive an interpretation of the evidence base that depends upon who is selected to be on the committee.

Five of the portals in the survey based their evidence claims on the basis of one or two good studies. The danger with such an evidence standard is that it is not considering the whole knowledge base and there may be many other good-quality studies that found an opposite result. It is interesting that most of the portals using the ‘one or two good studies’ criteria were focused on the effectiveness of intervention programmes (a specific combination of intervention components) rather than evidence of particular intervention strategies. Specific programmes are useful for indicating efficacy where there is intervention fidelity, but may be less adaptable to use in contexts which may differ significantly from those in which the programmes were developed. Table 1 shows that, for the five portals providing evidence on programmes, just one or two studies were enough for the evidence portals to inform users that the programmes were effective. The evidence standards of at least one of these portals have developed since the survey, but it is clear that standards of some KBIs for making an evidence claim of effectiveness can be quite low.

Specific standards will depend on the research questions being asked and the evidence claims made in response to these (Gough, 2021). The nature of the evidence and the standards for making evidence claims vary between, for example, research evidence on the effectiveness of an intervention, and the evidence in support of a causal model by which it had its effect.

The term ‘evidence standards’ itself can be problematic in that it is used to describe a range of different approaches to supporting or appraising research to make justifiable evidence claims. These approaches, can include (Gough, 2021):

• methods standards criteria (methodological criteria for making an evidence claim);

• methods guidance (advice as to appropriate research methods to make justifiable evidence claims);

• internal quality assurance (processes for ensuring that research methods are performed appropriately);

• reporting standards (criteria for transparent reporting of the execution of research);

• methods appraisal (procedures for checking and reporting on the relevance and trustworthiness of research studies and the basis of their evidence claims);

• stage of development, appraisal of effectiveness, and implementation of interventions (the extent that a certain policy or practice intervention has research evidence to justify its effectiveness and use).

All of these may be specified in extensive or minimal detail. There is thus much potential for confusion about what ‘evidence standards’ means, as well as the basis for making judgements within each of these different types of standards. Clarity about these issues is an important area for KBIs to be clear, consistent and coherent.

In sum, inadequate or inconsistent evidence standards could lead to audiences misinterpreting or placing too much trust in the findings and guidance presented. So, are KBIs evidence-informed in their use of credible evidence standards for making evidence claims in terms of:

• transparency: do the KBIs fully and explicitly report their specific methods and criteria for making evidence claims? Are these simple lists or are they manuals providing detailed explanations of the nature and basis of such judgments?

• consistency: are the KBIs consistent in their methods and criteria for making different evidence claims in different outputs?

• clarity: are KBIs clear about the nature of the evidence claim and how it is relevant and fits the needs of those to whom the claim is being communicated?

4. Evaluation of KBIs and contributing to the ‘use of research’ knowledge base

How do KBIs assess whether their aims have been achieved? Do these results inform their own development and research evidence on ‘research use’? Are we clear about ‘What works in What Works’ (ESRC, 2016)?

For KBIs to be evidence-informed, it would be expected that they would evaluate their progress in meeting their aims and modify their activity in response to their evaluations (as above on the KBIs’ theories of change). Such evaluation would also allow KBIs to provide research findings to contribute to the scientific knowledge of ‘research on research use’.

KBIs are naturally focused on undertaking the activities that they have been funded for. There may therefore be few resources available for them to commission external independent or internal self-evaluations. There are of course exceptions, with some KBIs formally evaluating most of their activities.

Where evaluation does take place, a distinction can be made between monitoring of work activity, measuring the achievement of desired outcomes (KBI goals), and the processes by which these are achieved. Monitoring activity can be relatively straightforward, such as recording numbers of meetings or products produced. For measuring the extent of desired outcomes, a distinction can be made between interim and final outcome measures. Interim measures can be testing stages in a hypothesised theory of change and the processes involved.

Assessing detailed theories of change are rare and interim measures of assessing change can be very simplistic, such as web analytics of visits to KBIs’ web pages. These may indicate that users have at least had some contact with KBIs’ resources, though this does not necessarily mean that this has then informed decision making and policy and practice. ‘Use of research’ means that research evidence was considered, though it may not always be easily apparent what role the research had in the decision-making process.

The case of expert scientific advisory committees for government is relevant, as there do not seem to be clear methods by which they are evaluated. There has been a focus on how government uses advice to respond to health emergencies such as the BSE crisis (Hinchliffe, 2001) and now the COVID-19 pandemic, with the latter subject to an inquiry by the UK Parliament’s Science and Technology Committee (Gough, 2020). There is also research on the use of evidence by legislatures (Kenny et al, 2017; Geddes, 2020).

Final outcome measures are often weakly specified. If the overall aim of a KBI is to increase the use of evidence, then any data showing that use has increased may be a measure of success (though in such ‘natural experiments’ the data is correlational and one cannot be sure what the causes of the changes are). This does not necessarily mean that the research has been used wisely or appropriately – just that it has been used. Even where the research has been ‘wisely used’, it may have led a decision maker to stop a planned action and so the influence of the KBI may be difficult to measure.

A more detailed way of appraising outcomes is to assess the effects on the intended beneficiaries of a KBI’s work. KBIs occasionally do measure changes in achievements of their ultimate beneficiaries (for example, pupil attainment), although this is rare (for example, Sibieta and Sianesi, 2019). For expert scientific advisory committees, final outcome measures could be based on whether the advice was acted upon and by the nature of the outcomes ultimately achieved.

In sum, external or self-evaluation is important in determining whether and to what extent KBIs are meeting their objectives, and how they or others can better meet such objectives in the future. Are KBIs explicit about how they are being evidence-informed in the evaluation of their work, and the contribution that this could make to the knowledge base (on research use), in terms of:

• rigorous evaluation: to indicate how they are meeting their aims (and other positive and negative effects) through the planned interim and final outcomes and appraisal of the adequacy of their theory of change?

• KBI development: using their evaluations to adjust and develop their work over time?

• evidence of effect: providing justifiable and relevant evidence claims about their positive contribution to the users and/or planned ultimate beneficiaries of their work?

• evidence standards: for making any such evidence claims (including the methods used to assess change and the use of subjective or objective measures of change)?

• research on research use: justifiable evidence claims about the KBIs’ work that contributes to the knowledge base on ‘research on research use’?