Abstract
Background:
‘Co-production’ is one of the key concepts in evidence-informed policy and practice – in terms of both its theoretical importance and its practical applications − being consistently discussed as the most effective strategy for mobilising evidence in policy and practice contexts. The concept of co-production was developed (almost) independently across multiple disciplines and has been employed in various policy and practice fields including environment, sustainability, and health.
Aims and objectives:
This paper surveys the literature to identify different meanings of co-production across different disciplinary bodies of knowledge. Such exploration is aimed at identifying the key points of convergence and divergence across different disciplinary and theoretical traditions.
Methods:
We performed a systematic search of Web of Science via a query designed to capture literature likely focusing on co-production, and then manually examined each document for relevance. Citation network analysis was then used to ‘map’ this literature by grouping papers into clusters based on the density of citation links between papers. The top-cited papers within each cluster were thematically analysed.
Findings:
This research identified five meanings of co-production, understood as a science-politics relationship, as knowledge democracy, as transdisciplinarity, as boundary management, and as an evidence-use intervention.
Discussion and conclusions:
Even though different clusters of scholarship exploring co-production are closely connected, this concept is mobilised to capture phenomena at different levels of abstraction – from post-structuralist theories of knowledge and power to specific strategies to be employed by researchers and policymakers.
1 Introduction
Co-production has emerged as one of the key concepts in understanding knowledge-policy interactions. As policy challenges become increasingly ‘wicked’, there is a greater need for usable knowledge: that is, credible, legitimate, and relevant to decision making (Cash et al, 2003). There is a growing consensus that this type of knowledge is not produced by researchers alone, but rather requires collective knowledge making across different groups of stakeholders (for example, Sarewitz and Pielke, 2007). ‘Co-production’ has been used to capture this process across multiple different disciplinary and policy areas. As argued by Miller and Wyborn (2020), this term was developed simultaneously across three academic communities: public administration (in particular the work of Eleanor Ostrom (1996)); science and technology studies (STS) (stemming from the work of Sheila Jasanoff (2004)); and sustainability science. Arguably, other disciplines played a role in these developments, most importantly health sciences through the work on evidence use, but also public participation and patient movements (Epstein, 1996).
The success of ‘co-production’ as a term used across such a variety of fields might be due to its flexibility and definitional ambiguity. This concept is often used without specific definitions (for example, Metz et al, 2019), which is problematic considering its multi-disciplinary but also multi-theoretical origins. Against this backdrop, two questions emerge. First, what types of meanings of co-production can we identify across different disciplinary communities? And second, what theoretical insights regarding co-production emerge from cross-disciplinary comparison? By answering these questions, this paper explores how the concept of co-production ‘travels’ across different communities, as well as the implications of its malleability. Therefore, unlike the existing reviews (for example, Gagliardi et al, 2015; Bremer and Meisch, 2017; Miller and Wyborn, 2020), we explicitly (and systematically – by combining a scoping review with citation network analysis) compare different disciplinary approaches, in order to identify various theoretical ‘cores’ of co-production across bodies of literature.
Such focus on co-production as a travelling concept allows us to build and expand on the existing literature by analysing co-production as a multi-level problem involving not only different strategies but also different – and sometimes contradictory – meanings. Consequently, we were able to identify five different ways of thinking about co-production – and conceptualising it as a phenomenon occurring at different levels of socio-political systems, institutions, and practices. This approach allows us to build on the existing scholarship by identifying tensions between these different theoretical and practical approaches to co-production. Therefore, this paper contributes to the interdisciplinary scholarship on co-production in two ways. Our first contribution is empirical: we begin by mapping the literature and then identify its theoretical underpinnings, existing definitions, target stakeholders, and strategies of co-production. The second contribution is theoretical: we synthesise the findings in order to argue that co-production can be understood as a spectrum of practices between macro-knowledge systems and micro-interactions aimed at increased knowledge utilisation.
2 Methods
This study aimed to determine the range and nature of the use of the term ‘knowledge co-production’ across diverse literatures, in order to clarify its conceptualisation. As such, an adapted scoping review methodology was employed (Arksey and O’Malley, 2005; Levac et al, 2010), supplemented by citation network analysis. See Table 1 for a checklist of the reporting items for this scoping review (Tricco et al, 2018). As ‘co-production’, and variants such as ‘co-creation’, are terms that are used in a number of different contexts, this represents a challenge for systematically capturing relevant literature and screening it for relevance. Our approach consists of systematically retrieving a large body of literature likely to be relevant, and then examining this for relevance via the manual screening of titles and abstracts. From this set of manually examined records, we then examine the structure of citation between the papers via network analysis to identify ‘clusters’ of densely interconnected papers. It is well established that papers that are interconnected densely with citation links are likely to be focusing on the same or similar topics, while sparse connections between clusters reflects different research focuses (Klavans and Boyack, 2017). By the use of citation network analysis, we were also able to examine links across all the references from papers deemed relevant, allowing us to detect commonly cited documents, either missed by our query or that were not indexed, that were likely relevant.
Preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (PRISMA-ScR) checklist (Tricco et al, 2018)
| Section | Item | Prisma-ScR checklist item | Reported in section |
|---|---|---|---|
| Title | |||
| Title | 1 | Identify the report as a scoping review. | see Abstract |
| Abstract | |||
| Structured summary | 2 | Provide a structured summary that includes (as applicable): background, objectives, eligibility criteria, sources of evidence, charting methods, results, and conclusions that relate to the review questions and objectives. | see Abstract |
| Introduction | |||
| Rationale | 3 | Describe the rationale for the review in the context of what is already known. Explain why the review questions/objectives lend themselves to a scoping review approach. | see Introduction and Methods first paragraph |
| Objectives | 4 | Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (for example, population or participants, concepts, and context) or other relevant key elements used to conceptualise the review questions and/or objectives. | see Introduction |
| Methods | |||
| Protocol and registration | 5 | Indicate whether a review protocol exists; state if and where it can be accessed (for example, a Web address); and if available, provide registration information, including the registration number. | N/A |
| Eligibility criteria | 6 | Specify characteristics of the sources of evidence used as eligibility criteria (for example, years considered, language, and publication status), and provide a rationale. | see Methods section 2.1 |
| Information sources | 7 | Describe all information sources in the search (for example, databases with dates of coverage and contact with authors to identify additional sources), as well as the date the most recent search was executed. | see Methods sections 2.1, 2.3, and 2.4 |
| Search | 8 | Present the full electronic search strategy for at least one database, including any limits used, such that it could be repeated. | see Methods section 2.1 |
| Selection of sources of evidence | 9 | State the process for selecting sources of evidence (that is, screening and eligibility) included in the scoping review. | see Methods section 2.2 |
| Data charting process | 10 | Describe the methods of charting data from the included sources of evidence (for example, calibrated forms or forms that have been tested by the team before their use, and whether data charting was done independently or in duplicate) and any processes for obtaining and confirming data from investigators. | see Methods section 2.5 |
| Data items | 11 | List and define all variables for which data were sought and any assumptions and simplifications made. | see Methods section 2.5 |
| Critical appraisal of individual sources of evidence | 12 | If done, provide a rationale for conducting a critical appraisal of included sources of evidence; describe the methods used and how this information was used in any data synthesis (if appropriate). | N/A |
| Synthesis of results | 13 | Describe the methods of handling and summarising the data that were charted. | see Methods section 2.5 |
| Results | |||
| Selection of sources of evidence | 14 | Give numbers of sources of evidence screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally using a flow diagram. | see Methods section 2.4 |
| Characteristics of sources of evidence | 15 | For each source of evidence, present characteristics for which data were charted and provide the citations. | N/A |
| Critical appraisal within sources of evidence | 16 | If done, present data on critical appraisal of included sources of evidence (see item 12). | N/A |
| Results of individual sources of evidence | 17 | For each included source of evidence, present the relevant data that were charted that relate to the review questions and objectives. | see synthesis in Findings |
| Synthesis of results | 18 | Summarise and/or present the charting results as they relate to the review questions and objectives. | see Findings |
| Discussion | |||
| Summary of evidence | 19 | Summarise the main results (including an overview of concepts, themes, and types of evidence available), link to the review questions and objectives, and consider the relevance to key groups. | see Discussion |
| Limitations | 20 | Discuss the limitations of the scoping review process. | see Methods |
| Conclusions | 21 | Provide a general interpretation of the results with respect to the review questions and objectives, as well as potential implications and/or next steps. | see Conclusion |
| Funding | |||
| Funding | 22 | Describe sources of funding for the included sources of evidence, as well as sources of funding for the scoping review. Describe the role of the funders of the scoping review. | see Funding section |
2.1 Search strategy
Given the exploratory nature of the study, the search string employed was designed to capture the breadth of relevant literature. In May 2020, we performed a systematic search on Web of Science (WoS) for terms appearing in titles, abstracts, and keywords of documents indexed in the Core Collection via TS=((‘co-produc*’ OR ‘coproduc*’ OR ‘co-creation’ OR ‘cocreation’) AND (‘knowledge’ OR ‘evidence’)). We restricted the results to the following indexes: Science Citation Index Expanded, 1900-present; Social Sciences Citation Index, 1900-present; Arts & Humanities Citation Index, 1975-present; Book Citation Index–Science, 2005-present; Book Citation Index–Social Sciences & Humanities, 2005-present; and the Emerging Sources Citation Index, 2015-present. Results were restricted to include only documents written in the English language, and further restricted to include only articles published as journal papers or as book chapters, leaving a total of 3,795 documents.
2.2 Study selection
Search results were downloaded from WoS and uploaded to Mendeley for title and abstract review. The research team met to agree inclusion and exclusion criteria before a pilot run of results screening. For the pilot, two research assistants each reviewed 200 results and then held another meeting in which the research team further clarified and refined the inclusion and exclusion criteria. Due to the challenge of balancing comprehensiveness and feasibility inherent to scoping reviews (Levac et al, 2010), the 3,795 initial search results were divided between the two research assistants. All articles were screened by their title, and if necessary their abstract, and included or excluded based on the criteria identified in Table 2. Once all results had been initially screened, those that had been flagged as questionable were reviewed by the study lead (JBG), and the research assistants reviewed 10% of each other’s included studies. Through this process, 529 papers were identified for inclusion in the review. Considering this large number of relevant studies, additional analysis of the results was conducted through network analysis before proceeding to full-text review and data extraction.
Inclusion and exclusion criteria
| Inclusion criteria | Exclusion criteria |
|---|---|
| • Papers discussing co-production of knowledge, evidence, or science in policymaking • Primary research articles or reviews: journal articles and book chapters • Published in the English language |
• Co-production of value, innovation • Co-production of public services (without any knowledge components) • Co-production for clinical decision making • Lack of references to knowledge, evidence, science, or policy • Non-primary research or review: editorials, proceedings, or other |
2.3 Network analysis
The 529 records (506 primary search articles; 23 reviews) that met the inclusion criteria were then analysed via citation network analysis. All bibliometric data were downloaded, including their full reference lists. The full reference lists of all retrieved papers were extracted and numerical identifiers were linked to each unique reference. These data were stored as an ‘edge-list’ that records the references from a citing paper i in the ‘Source’ column to a cited paper j in the ‘Target’ column. A ‘node attribute list’ was used to identify papers, which includes the numerical identifier for all papers and relevant bibliometric details, such as title of paper and WoS accession number. These data were then cleaned to merge references duplicated by variants in format or due to referencing errors. From these data, a directed citation network was constructed, with papers as nodes and citations links between papers as directed edges between nodes. All data underpinning the following analysis are openly available via the Zenodo platform (Bandola-Gill et al, 2021).
All network analyses and visualisation were performed in Gephi 0.9.2 (Bastian et al, 2009). Two key measures were used. First, to establish the number of citations a paper receives from other papers within this network, in-degree was calculated for all nodes. Second, we clustered the network by modularity maximisation (Newman and Girvan, 2004) via the Leiden algorithm (Traag et al, 2019), which partitions a network into clusters of nodes that are densely interconnected by edges but sparsely connected to nodes in other clusters. Previous research has established that clustering direct citation networks in this manner establishes papers focusing on the same or similar research topics (Klavans and Boyack, 2017).
2.4 Quantitative summary
For a quantitative summary of records identified at each step of our review see Figure 1. Our initial search yielded 3,795 results. Following title and abstract review, 529 relevant studies were identified. Figure 2 shows the distribution of papers that met the inclusion criteria by year of publication.
Flowchart of review stages and numbers of records identified
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Flowchart of review stages and numbers of records identified
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Flowchart of review stages and numbers of records identified
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Knowledge co-production articles and reviews retrieved from the Web of Science, 1996 to May 2020
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Knowledge co-production articles and reviews retrieved from the Web of Science, 1996 to May 2020
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Knowledge co-production articles and reviews retrieved from the Web of Science, 1996 to May 2020
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
These 529 papers were then analysed via network analysis. First, a citation network was constructed consisting only of these 529 papers and the citation links between them. In total, 440 of these papers were directly interconnected via 1,578 citation links in a large weakly connected component (a connected network), while 89 were unconnected to this structure. Second, due to the existence of unconnected papers, and because we suspected many of these might be interconnected to this structure by shared references to papers not recovered by our search, we analysed the full citation network. This includes the 529 papers retrieved and their full bibliographies – with nodes representing either fully retrieved papers or non-retrieved unique references. In total, this results in a network of 23,338 nodes connected together by 33,112 citation links. Thus, the average reference list size of this set of 529 papers is ~63, but an average of only ~1.4 references are shared by these papers. This is perhaps unsurprising; knowledge co-production is a concept deployed in many different empirical research areas, and many of these references will be directed at papers in that specific empirical area. Due to the large size of this network, we restricted the number of non-retrieved documents in the network to include only documents cited by at least three of the 529 included papers. A cut-off of an in-degree of >2 for non-retrieved documents was also used by Batagelj et al (2017) in their citation network analysis of the history of peer-review research. Referred to as the ‘boundary problem’, citation networks constructed from a set of documents from any specific literature search tend to contain many references to literature not retrieved, and many of these non-retrieved documents are low-cited and not clearly related to the topic of interest. Thus, a boundary has to be erected to ensure that a citation network is sufficiently focused on the intended research topic. By this, the network is reduced to a total of 1,897 nodes and 9,759 edges. While some of these non-retrieved references will not be directly focusing on knowledge co-production, their inclusion in the network aids community detection, as shared references pull papers more clearly into distinct research areas. By this, 525 of the 529 retrieved papers now reside within a large weakly connected component – with only four papers unattached to this structure (that is, ‘isolates’).
Removing the isolates from the following analysis leaves a connected network composed of 1,893 nodes connected by 9,759 edges. We applied modularity maximisation to this network via the Leiden algorithm (Resolution: 0.8; Iterations: 100; Restarts: 100), which detected five clusters with a modularity value (Q) of 0.53 for the entire network. This means this network can be reasonably well partitioned into clusters in which papers within a given cluster direct the majority of their references to other papers in that same cluster. Figure 3 shows this network.
Citation network of knowledge co-production papers retrieved from WoS (n=525), and any unique reference cited by at least three of these papers (n=1,368). Total network size n=1893; m=9759. Nodes are shaded and labelled by membership of one of five clusters detected via modularity maximisation (Q=0.53) via the Leiden Algorithm. Nodes are sized relative to in-degree (within network citations) – larger nodes reflect higher numbers of within network citations. The structural position of nodes is determined by the Lin-log ForceAtlas2 algorithm (Jacomy et al, 2014).
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Citation network of knowledge co-production papers retrieved from WoS (n=525), and any unique reference cited by at least three of these papers (n=1,368). Total network size n=1893; m=9759. Nodes are shaded and labelled by membership of one of five clusters detected via modularity maximisation (Q=0.53) via the Leiden Algorithm. Nodes are sized relative to in-degree (within network citations) – larger nodes reflect higher numbers of within network citations. The structural position of nodes is determined by the Lin-log ForceAtlas2 algorithm (Jacomy et al, 2014).
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Citation network of knowledge co-production papers retrieved from WoS (n=525), and any unique reference cited by at least three of these papers (n=1,368). Total network size n=1893; m=9759. Nodes are shaded and labelled by membership of one of five clusters detected via modularity maximisation (Q=0.53) via the Leiden Algorithm. Nodes are sized relative to in-degree (within network citations) – larger nodes reflect higher numbers of within network citations. The structural position of nodes is determined by the Lin-log ForceAtlas2 algorithm (Jacomy et al, 2014).
Citation: Evidence & Policy 19, 2; 10.1332/174426421X16420955772641
Clusters were ordered by their size, and labelled from A to E. As Table 3 shows, clusters range in size from 443 nodes to 295, and each cluster contains both manually included papers and non-retrieved documents cited by at least three of the papers that met inclusion criteria.
Cluster membership
| Cluster | Total manually retrieved | Non-retrieved references cited >2 by included papers | Total nodes (documents) within cluster | Total Edges (citation links) with cluster |
|---|---|---|---|---|
| A | 121 | 332 | 443 | 1587 |
| B | 109 | 315 | 424 | 1363 |
| C | 106 | 284 | 390 | 1358 |
| D | 100 | 241 | 341 | 1643 |
| E | 89 | 206 | 295 | 849 |
To better understand the connections within and between clusters, we analysed how references were distributed. In Table 4, we see that papers within a cluster are far more likely to reference other studies within that cluster than to reference a paper in a different cluster – an average of 70% of all references are directed at other papers in the same cluster.
Percentage of references within and between five knowledge co-production clusters
| Clusters | Total | |||||
|---|---|---|---|---|---|---|
| A | B | C | D | E | ||
| A | 69.6% | 8.2% | 8.9% | 10.9% | 2.4% | 100.0% |
| B | 7.8% | 67.9% | 10.0% | 11.7% | 2.5% | 100.0% |
| C | 8.2% | 8.4% | 69.8% | 10.5% | 3.1% | 100.0% |
| D | 12.0% | 11.4% | 8.4% | 65.8% | 2.3% | 100.0% |
| E | 5.0% | 3.3% | 4.5% | 4.7% | 82.6% | 100.0% |
2.5 Cluster classification: qualitative analysis
Qualitative thematic analysis of the ten publications most cited within each cluster by other papers within that cluster was performed to classify clusters into distinct groups of literature within the conceptualisation of knowledge co-production. Here, we use citations not as a proxy for quality, which is a highly contentious and problematic assumption (Leng and Leng, 2020), but simply to highlight those papers that are commonly referred to within a set of literature. As each cluster has the heavy tailed right skewed distribution of citation that is typical of all citation networks (Albarrán et al, 2011), only a small number of papers are commonly cited by others. In some clusters, some of the most cited papers would not meet our inclusion criteria in terms of having sufficient focus on knowledge co-production; however, they represent precursors of relevant literature on knowledge co-production, demonstrating the progression of ideas leading to use of the concept.
The full review of the ten papers from each of the five clusters identified by the network analysis was conducted using a data extraction chart created in Excel. The categories in this chart were designed to explore the use and conceptualisation of the term ‘knowledge co-production’ in the literature. Categories included: the policy topic, definition(s) of co-production, relevant theories, strategies of co-production, actors involved in co-production, and outcomes or impacts as described in studies. Following data extraction, team members met to discuss and compare each of the clusters they had individually reviewed, identifying distinctions among each of the sub-networks of studies within the overall network of literature on knowledge co-production. The description of the five meanings of co-production (as identified in each cluster) presented below was aimed at capturing the theoretical underpinnings of these approaches as well as the associated strategies (identified in data extraction). The description was aimed at consolidating the key themes identified in all ten papers in each cluster – as such the writing-up process was iterative and involved moving between: 1) the in-depth description of each cluster and comparison across the clusters to highlight the differences and similarities, and 2) the full text of the key papers and the data extraction table.
Finally, our methodological approach has some limitations – the main one stemming from limited search terms. There are forms of collaboration that might fall within the scope of ‘co-production’ practices (for example research-practice partnerships, community-engaged research, participatory action research, culturally relevant evaluation, and so on) which would not be captured by our approach. Nevertheless, the focus on ‘co-production’ as the key search term was a necessary trade-off, needed for capturing cross-disciplinary scholarship and its diverse theoretical approaches (per the research questions, outlined in the Introduction) while making the search manageable.
3 Findings: Five approaches to co-production
3.1 Mapping the field
As with any interdisciplinary topic, co-production is discussed using a variety of theoretical and empirical tools. In our research, we have identified five substantive clusters of scholarship which approach co-production differently (see Figure 3, Table 5). The clearest differentiation between clusters could be identified, perhaps unsurprisingly, across disciplinary lines, in particular environmental and sustainability science (clusters B, C, D), health (E), and STS (A). As we will show in this section, these differences reflect not only the substantive focus of the papers but also the theoretical, strategic, and conceptual underpinnings of the literature.
Different meanings of co-production
| Co-production is… | Key challenge | Theoretical underpinnings | Key discipline |
|---|---|---|---|
| Science-politics relationship | Understanding changing cultural, epistemic, and institutional orders | Jasanoff (2004) | STS |
| Knowledge democracy | Opening up knowledge systems | Jasanoff (2004); social learning; knowledge systems |
Environmental sciences (environmental management) |
| Transdisciplinary research | New forms of knowledge production | Mode-2 (Nowotny et al, 2001) Interdisciplinary research |
Sustainability science/research |
| Boundary management | Matching the supply of and demand for science in policy | Boundary work Gieryn (1983) Boundary organisations Guston (2001) |
Environmental sciences (climate change) |
| Evidence use intervention | Improving the uptake of evidence in policy | Knowledge utilisation Weiss (1979) Evidence-based policymaking |
Health sciences |
Even though we have identified a wide variety of ways in which the term co-production is mobilised across different strands of the literature, there are some important commonalities which point to the fact that, despite its being a ‘conceptually stretched’ term, the five approaches to co-production share a number of assumptions. The first assumption is about the motivation behind the turn to co-production. The papers across different disciplinary fields are relatively uniform in their perception of the growing complexity of societal problems and, consequently, the need for new approaches to research-driven interactions. This concern with the growing complexity of ‘wicked problems’ (Rittel and Weber, 1973) might explain the dominance of specific fields (health and the environment) as the areas in which this conceptual innovation was initiated. Therefore, the starting point across the different clusters of scholarship is the notion of co-production as a strategy aimed at addressing intractable policy challenges.
Secondly, the papers share an assumption that co-production involves interaction between researchers and non-academics (broadly speaking). The starting point of the exploration is an acknowledgement that knowledge required for addressing complex challenges is not produced solely within traditional structures, but rather is developed across different communities and knowledge systems. This leads to a question of ‘intensity’ of co-production: what degree of collaboration merits the term? Or in other words: where does ‘regular’ research end and ‘co-production’ begin? For example, Reed (2008), citing Rowe and Frewer (2000), identified progressive levels of stakeholder engagement (communicate, consult, or participate) and specific methods that can be used at each. Similarly, Martin (2010) presented a spectrum of low to active engagement by practitioners in the co-production of social research, distinguishing between five types: 1) practitioners as informants; 2) practitioners as recipients; 3) practitioners as endorsers; 4) practitioners as commissioners; and 5) practitioners as co-researchers. Brandt et al (2013) included an assessment of the intensity of practitioners’ involvement in transdisciplinary projects on a scale of: 1) Information (one-way communication); 2) Consultation (closer communication, including response); 3) Collaboration (participants having notable influence); and 4) Empowerment (practitioners having decision-making authority). They found varying levels of intensity across the 104 case studies they analysed, though very few achieved empowerment. Therefore, there seems to be agreement not only that co-production is a range of practices at varied levels of engagement, but also, more implicitly, that the deeper forms of co-production are more strongly desired (even if they remain aspirational).
3.2 Five meanings of co-production
Co-production as a relationship between science and politics
Representations of the natural world attain stability and persuasive power, in my view, not through forcible detachment from context, but through constant, mutually sustaining interactions between our senses of the is and the ought: of how things are and how they should be. The epistemic claims of environmental science are most trusted when they engage with practices that confer normative authority – not only scientific practices such as peer review (Merton’s ‘organized skepticism’) but also the cultural practices of democratic politics and the law. (Jasanoff, 2010: 236)
This theorisation of co-production assumes that science is inseparable from society, as they both draw on the same cultural, ethical, and institutional canons (Jasanoff, 2010: 237). Therefore, assuming the spheres of science and society to be closely intertwined, research in this cluster explores how they were strategically separated, by mobilising Latour’s (1987; 1993) idea of ‘purification’ separating nature and human culture.
Another STS framework popular within this cluster is Funtowicz and Ravetz’s (1993) post-normal science. The authors employ the Kuhnian notion of paradigms in science to highlight the development of a new format of science production: progressing from ‘normal’ to ‘post-normal’. Funtowicz and Ravetz (1993) argue that, given the increasing complexity of policy challenges (for example, that of environmental issues), the traditional ‘normal’ models of science are inadequate as problems become both more complex and more urgent. Post-normal problems require public participation both in decision making and in quality assessment of the results of scientific knowledge production (Funtowicz and Ravetz, 1993: 740).
Consequently, the key papers drawing on this framework of co-production explore the relationship between science and decision making as a matter of social and epistemic orders and the relationships between them. As such, this is a cluster with the least extensive focus on practical strategies. Nevertheless, this theorisation, through its strong focus on a relationship between science and society, concentrates on non-traditional actors (from the perspective of both science and policy), such as lay experts (for example, Callon, 1999).
Co-production as knowledge democracy
These abstract conceptualisations are translated into more practical considerations within Cluster B. Co-production here is a mutual and collaborative knowledge generation with a particularly strong emphasis (especially compared to other clusters) on the integration of local and indigenous knowledge to improve governance and management (Berkes, 1999; 2009; Fazey et al, 2013). Therefore, we summarise this approach, following Cornell et al (2013), as co-production as knowledge democracy, since this cluster of scholarship focuses most explicitly on increasing the diversity of forms of knowing and the interaction between them.
Here again, studies mobilise Jasanoff’s (2004) definition of co-production, in terms of processes of mutual reshaping of knowledge making and decision making. However, this group also discuss co-production as a practical strategy for supporting ‘social learning’ (Berkes, 2009) across different stakeholder groups, or focus on co-production as a matter of concrete strategies aimed at collaborative knowledge production and ‘adaptive’ co-management (Cash et al, 2006; Armitage et al, 2009) or governance (Folke et al, 2005). As summarised by Armitage et al (2009: 95), this strategy ‘explicitly links learning (experiential and experimental) and collaboration to facilitate effective governance’.
The key underpinning assumption of this cluster is the multiplicity of different forms of knowledge interacting with each other within the governing process (Cash et al, 2006). For example, Raymond et al (2010) identify a number of epistemological tensions between different forms of knowledge across varied dimensions, including local vs generalised knowledge, informal vs formal, novice vs expert, tacit vs implicit vs explicit, and traditional vs local vs scientific. Therefore, the (co-)production of knowledge for governance requires acknowledgement of the lack of universality of knowledge and its flexible assessment criteria (Raymond et al, 2010).
Co-production has been understood as a problem of knowledge systems (Cornell et al, 2013; Tengö et al, 2014) and concerns over keeping them ‘open’ in order to solve complex challenges of sustainability. For example, Tengö et al (2014: 584) propose a framework for a multiple evidence base (MEB) approach that ‘emphasises the complementarity of knowledge systems and the values of letting each knowledge system speak for itself, within its own context, without assigning one dominant knowledge system with the role of external validator’.
Different knowledge systems include: local knowledge, indigenous knowledge, practitioner knowledge, traditional knowledge, transdisciplinary knowledge, social science knowledge, natural science knowledge, and technical knowledge (Tengö et al, 2014). Therefore, the key challenge with this form of co-production lies in integrating different ways of knowing and bridging the different epistemological and philosophical assumptions behind them so as to identify different types of knowledge, engage them, evaluate them, and then apply the integrated knowledge (Raymond et al, 2010: 1771). Significantly, this is the only form that highlights community-driven approaches, for example by including living documents and ‘community tours’ (Armitage et al, 2011).
Co-production as transdisciplinarity
Cluster C, with considerable consistency across papers, sees co-production as a component of transdisciplinary research. Transdisciplinarity is defined as: ‘an extended knowledge production process including a variety of actors and with an open perception of the relevance of different forms of information produced by the scientific and lay community’ (Mobjörk, 2010: 866).
This cluster is underpinned by an assumption that monodisciplinary scientific knowledge is not sufficient for solving increasingly complex problems (Hirsch Hadorn et al, 2006; Pohl et al, 2010; Jahn et al, 2012; Lang et al, 2012; Brandt et al, 2013; Polk, 2015). Unsurprisingly, the key theoretical idea underpinning this cluster was one of Mode-2 science: Gibbons et al’s (1994) and Nowotny et al’s (2001) books being two of the most cited publications within this cluster. This framework assumes that, due to the growing uncertainty and complexity of social problems, traditional, disciplinary science (Mode-1) is no longer sufficient to respond to these challenges. What is needed is knowledge that is produced ‘in the context of application’, rather than within the university context. This new Mode-2 science is inherently transdisciplinary, and is assessed by the criterion of usability (or ‘extended’ quality control) rather than by purely academic excellence.
Building on these theoretical debates, the perspective of co-production as transdisciplinarity conceptualises it as a problem-driven activity, requiring a new approach to knowledge production that is not only interdisciplinary (crossing different disciplines) but transdisciplinary (crossing different institutional settings in which different forms of knowledge are produced). A key aim is to ‘extend’ knowledge production by collaborating with practitioners (Mobjörk, 2010). Consequently, this approach focuses on the problem of integration of different bodies of knowledge (Jahn et al, 2012; Brandt et al, 2013). In part, this is achieved through interactive, participatory, and collaborative approaches, characterised by direct involvement and continuous exchange that bridges or blurs the boundaries between academic and non-academic actors, or scientists and stakeholders (Pohl et al, 2010; Lang et al, 2012; Polk, 2015). To achieve this, authors discuss key principles such as inclusion, collaboration, integration, usability, and reflexivity (Polk, 2015).
Understanding co-production through the prism of transdisciplinarity positions it more strongly on the academic side of science-policy interactions. This might lead to an issue with the scale of participation in knowledge production, captured by Mobjörk (2010) through a focus on consulting and participatory transdisciplinarity, which explores the question of whether knowledge production is aimed at capturing the views and perspectives of stakeholders, or at actually co-producing knowledge through participation of stakeholders in the process of knowledge production. Another challenge has to do with the transformative potential of transdisciplinary knowledge and its actual impact on solving grand challenges (Wiek et al, 2012; Polk, 2014) – including that of its own evaluations as well as competing evaluative frameworks of science and practice (Wiek et al, 2012).
Given the focus on transforming traditional knowledge production, this cluster explores not only different strategies but also their consequences for knowledge integration. The papers are consistent in identifying specific strategies at all stages of research. The first stage involves building a team and collectively framing the problem, followed by collaborative research, and finally applying and implementing the knowledge (Jahn et al, 2012; Lang et al, 2012; Brandt et al, 2013; Polk, 2015). Lang et al (2012) identify an extensive list of challenges and corresponding coping strategies at each of these phases, the latter of which include, for example, joint leadership and stakeholder mapping in phase 1; demonstration projects, structured knowledge integration methods, designing projects with appropriate levels of participation, and conflict reconciliation mechanisms in phase 2; and comparative studies for generalisability, collaborative and reflexive discourse, and advanced evaluation methodologies in phase 3.
Co-production as boundary management
Cluster D also focuses on the usability of knowledge but approaches it from a slightly different perspective. Here, the main focus is on structures supporting increased use of scientific knowledge in policy setting, stemming from the observed gap between production of evidence and its use in policymaking (Kirchhoff et al, 2013). The literature within this cluster focuses on the multiplicity of boundaries, both within the Science and Technology systems and between science and policy (Cash et al, 2003; Cash et al, 2006). An important theoretical perspective within this cluster is one on boundary work (Gieryn, 1983; Guston, 2001). Like Cluster C, this conceptualisation of co-production is concerned with knowledge produced across different institutional and social orders. However, unlike Cluster C’s strong focus on new types of scientific practices, this conceptualisation of co-production is more oriented towards the practice of bridging different communities. Within this cluster, co-production has been understood as ‘the process of producing usable, or actionable, science through collaboration between scientists and those who use science to make policy and management decisions’ (Meadow et al, 2015: 179).
Providing information that is ‘readily usable’ for decision making must therefore navigate and bridge any differences that might exist between what scientists might think is useful, and what is actually usable in practice. This entails establishing a shared vision of what knowledge is usable in a given decision process. We can think of the production and uptake of scientific knowledge as a pull-push process in which different conditions, mechanisms and institutions shape ultimate usability.
Co-production is therefore seen as a process oriented towards mediating these epistemological and practical differences in knowledge production across the domains.
This goal has been approached from the perspective of science policy as a mechanism for steering knowledge production towards applicability (Guston, 2001; Sarewitz and Pielke, 2007; Dilling and Lemos, 2011; Kirchhoff et al, 2013; Meadow et al, 2015). The authors within this cluster focus on the supply and demand sides of knowledge-for-policy, exploring the contexts both of production and of use of knowledge (Sarewitz and Pielke, 2007; McNie, 2007; Dilling and Lemos, 2011). This conceptualisation of co-production inherently focuses on mitigating the differences between different groups to increase the applicability of science. Consequently, the focus on the boundary between science and policy places this conceptualisation as a strategy of linking, brokering, and translating knowledge to produce the ‘right information’ (McNie, 2007).
These considerations have been translated into specific forms of practices, focusing predominantly on boundary organisations as institutionalised approaches to co-production (Guston, 2001; McNie, 2007; Dilling and Lemos, 2011; Lemos et al, 2012). A number of papers in this cluster include in-depth discussion of recommendations for successful operationalisation of knowledge co-production. For example, Lemos and Morehouse (2005) emphasise interactive research strategies with high levels of stakeholder participation and iterativity, including all-team meetings, researcher retreats, co-organisation of cross-disciplinary workshops, and joint authoring of publications.
Co-production as a research use intervention
The final strand of literature (Cluster E) differs significantly from the other four, both in terms of its substantive focus on health research and its theoretical framing of co-production in the context of evidence-based policy and practice. This cluster reflects an evolution of the literature on evidence use in policymaking, culminating in conceptualisations of the co-production, or joint production, of knowledge primarily through concepts of knowledge exchange (Graham et al, 2006; Contandriopoulos et al, 2010) and integrated knowledge translation (IKT) (Kothari and Wathen, 2013; Gagliardi et al, 2015). This cluster is most explicitly linked to service delivery (Bovaird, 2007; Contandriopoulos et al, 2010) and is the only one drawing on Eleanor Ostrom’s classic definition of co-production as: ‘the process through which inputs used to provide a good or service are contributed by individuals who are not in the same organization’ (Ostrom, 1996: 1073).
This cluster’s theoretical approach draws predominantly on the conceptual literature on the use of evidence, grounded in Weiss’s (1979) work on types of evidence with which to conceptualise knowledge exchange, translation, or brokering (Nutley et al, 2007; Contandriopoulos et al, 2010). The focus here is on co-production as an ‘intervention’ aimed at increasing the use of evidence in policy and public services, indicating strategic and deliberate efforts to disseminate knowledge (as opposed to everyday flows of knowledge which happen in different settings) (Contandriopoulos et al, 2010), and at creating closer partnerships between policymakers and researchers by encouraging ‘linking and exchange’ to improve partnerships between these culturally and institutionally different groups (Lomas, 2000). These authors also identify a number of benefits of IKT, ultimately contributing to policy-relevant research questions and adaptable and useful findings that improve research uptake into policy and practice.
Theoretically, this cluster differs considerably from the other four (see Table 5). The problematisation of co-production within this cluster has been most explicitly expressed in terms of what Barry (2020) called the ‘negative ontology’ of problems. In particular, this cluster has the strongest focus on ‘gaps’ (Van De Ven and Johnson, 2006) or ‘barriers’ and ‘facilitators’ (Oliver et al, 2014): challenges to be addressed and problems to be overcome when working collaboratively across institutional boundaries. These conceptualisations are focused on knowledge generation for use in decision making, through mutual learning and bringing together the expertise of knowledge producers and users, including greater openness to a wider range of data sources. As such, this cluster concentrates on ongoing partnerships, relationships, and collaboration between researchers and decision makers (such as policymakers, managers, clinicians), as well as other stakeholders (such as practitioners, patients, the public, and intermediaries including knowledge brokers). Strategies for successful co-production in this cluster are relatively specific and include early engagement of research users, identification of partners with existing links to facilitate interaction, clear expectations surrounding roles, mechanisms for dialogue, information sharing, frequent meetings, financial incentives, and collaboration at all stages of the research process, including development of an action plan for integration of recommendations (Kothari and Wathen, 2013; Gagliardi et al, 2015).
4 Discussion: what is co-production?
This paper has aimed to explore the meanings of co-production across different disciplinary and theoretical approaches. As outlined in the paper’s introduction, co-production is gaining prominence, both as an analytical term and as a practical strategy applied to science-policy interface(s). And yet, its meaning is often undefined, besides being conceptually stretched across different disciplines and applied in different contexts (for example, environment and health). The paper employed scoping review methods and citation network analysis to identify specific clusters of scholarship exploring the co-production of knowledge and policy. We identified the highest-cited papers within each cluster to discern the conceptual ‘core’ of each of these communities. The reason for this strategy was conceptual; even though these clusters of scholarship are closely interconnected and in practice papers draw on papers from multiple clusters, focusing on the core texts within each cluster allowed us to concentrate on more subtle definitional differences.
This strategy enabled us to identify five different meanings of co-production across the clusters: co-production as a science-politics relationship; as knowledge democracy; as transdisciplinarity; as boundary management; and as an evidence use intervention. These definitions, and associated bodies of scholarship, are not completely separate but rather are closely connected. In general, all five conceptualisations of co-production have in fact captured, at least to a degree, forms of knowledge practices carried out between researchers and non-academic actors in response to the emerging complexity of global challenges.
At the same time, the definitions of co-production differed significantly, as they seemed to capture practices at different ends of the spectrum of co-production. On one hand, the concept was used to express changes happening at a systemic level, that is, changing relationships between science and society or the interplay between different knowledge systems, or even the emergence of ‘new’ science. On the other hand, co-production referred to specific approaches to producing ‘usable knowledge’ in collaboration with stakeholders, often to maximise its utility. The key insight into co-production identified in this review is the fact that it is a multi-level phenomenon which can occur on the level of a socio-political system (Clusters A and B), the level of institutions (Clusters C and parts of D), and the level of situated practices (Clusters D and E). Arguably these various approaches are complementary as they are different enactments of the same phenomenon, namely, changing social and epistemic orders. And all are necessary, as the notion of changes in practices often requires broader, systemic changes (since practices are reflections of values, ideologies, and ideas; see Swidler, 2001). Nevertheless there are important tensions between these five meanings, leading to two key implications for the broader scholarship.
First, this research points to tensions and contradictions between different disciplinary and theoretical approaches to co-production, as they present processes leading to various effects. This is particularly evident when contrasting ‘knowledge democracy’ and ‘evidence use intervention’ approaches to co-production. The former is focusing on expansion of different forms of knowing and extending what is considered ‘useful’ knowledge for solving policy challenges. The latter is working within the realm of dominant ways of knowing (such as scientific or research-based evidence). Furthermore, focusing on co-production as a practice-level phenomenon might lead to overlooking the broader setting in which these processes take place. Hence, the multi-level approach to co-production implies an exploration not only of the effectiveness of different types of initiatives but also of broader socio-political, epistemic, and cultural changes that enable or constrain these projects.
The second implication of the multi-level understanding of co-production lies in its fluctuating epistemic and political qualities. As argued by Bremer and Meisch (2017) and Lövbrand (2011), different approaches to co-production could be categorised as ‘descriptive’ (focused on analytical interpretation of the process) and ‘normative’, or ‘utilitarian’ (aimed at achieving specific goals, such as problem solving). Focusing exclusively on the latter carries some important risks: for example, as highlighted by Lövbrand (2007), it might lead to issues of narrowing down policy options, rather than opening up knowledge systems. The mapping of five types of co-production might help to identify different theoretical approaches, going beyond their utility – and importantly, the approaches going beyond specific projects were those that focused on lay, local, and indigenous knowledges. Accordingly, seeing co-production exclusively as a strategy aimed at research utilisation might not only lead to reproduction of existing power relations (Turnhout et al, 2020) but also eventually compromise its transformative promise.
5 Conclusion and implications for practice
This paper has explored questions about the meaning of knowledge co-production and theoretical insights based on variations therein. Five meanings were identified through citation network analysis and in-depth analysis of papers within five key clusters of scholarship. The goal of this mapping exercise was not to summarise the state of research on co-production but rather to identify the key pockets of scholarship and their conceptual and theoretical underpinnings. In this final section, we identify the key implications for scholarship and practice stemming from this research.
Foremost, our research has identified a need for more interdisciplinary exploration and cross-fertilisation between different scholarly communities working on co-production. There is a clear need for more transparency in defining co-production as well as for epistemic openness to different perspectives (see also Bremer and Meisch, 2017), in particular those appearing at different levels of the science-policy spectrum. This paper points to the value of such work, not only for identifying disciplinary blind spots but also for challenging different standpoints on what co-production is and how it ought to be practiced. As we argued throughout the paper, even though different bodies of literature have identifiable common ground, they are, at least at times, contradictory to each other (for example, the knowledge democracy and evidence use intervention approaches outlined in the preceding section). More interaction between different communities, for example health and environmental/sustainability researchers (Table 5), might offer important insights into these contradictions as well as encourage mutual learning.
This leads to the implication of this study for researchers and policymakers/ practitioners who are (or wish to be) engaged in co-produced projects – one pertaining to the definitional clarity of what form of co-production is being carried out. This is important not just for ‘getting everyone on the same page’ but also for stimulating reflection over who is invited and what strategies are employed. Therefore, co-production practice might require balancing approaches across the five meanings to achieve both instrumental effects (for example via evidence use intervention or boundary management) as well as the ‘opening up’ of different options through deliberation (for example via knowledge democracy).
Finally, this research has implications for research funders. When funding co-production projects, we would encourage decision makers to keep the definition open and appreciate the process of co-production, rather than just focusing on its effects (or ‘impacts’, see Bandola-Gill, 2019), in order to assure epistemic diversity in the problem-solving processes. As shown in this paper, not all meanings of co-production perform equally well at inclusivity of different forms of knowing, which has important consequences for possible ‘utility traps’ identified in this paper, whereas focus only on specific effects of co-production might lead to ignoring forms that are oriented towards more systematic re-thinking of who is a knowledge producer (such as science-politics relationship, knowledge democracy, or transdisciplinarity).
Funding
This review was funded by the University of Edinburgh Strategic Research Support Fund. JBG is supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme under grant agreement No 71512. RIL is supported by the ESRC grant number ES/V011790/1.
Research ethics statement
The authors of this paper have declared that research ethics approval was not required since the paper does not present or draw directly on data/findings from empirical research.
Contributor statement
JBG designed the study, conducted parts of data extraction and analysis and wrote the first draft of the manuscript. MA conducted parts of data extraction and analysis and contributed to the manuscript. RL conducted citation network analysis and contributed to the manuscript (particularly the Methods section).
Acknowledgements
We would like to thank Dr Cleo Davies for her contributions to data collection, and two anonymous reviewers for their helpful feedback. The manuscript also benefitted from the feedback of the participants of the 2021 ECPR Joint Session on evidence-based policymaking.
Conflict of interest
The authors declare that there is no conflict of interest.
Bibliography
Albarrán, P., Crespo, J.A., Ortuño, I. and Ruiz-Castillo, J. (2011) The skewness of science in 219 sub-fields and a number of aggregates, Scientometrics, 88(2): 385–97.
Arksey, H. and O’Malley, L. (2005) Scoping studies: towards a methodological framework, International Journal of Social Research Methodology, 8(1): 19–32. doi: 10.1080/1364557032000119616
Armitage, D., Berkes, F., Dale, A., Kocho-Schellenberg, E. and Patton, E. (2011) Co-management and the Co-production of knowledge: learning to adapt in Canada’s Arctic, Global Environmental Change, 21(3): 995–1004. doi: 10.1016/j.gloenvcha.2011.04.006
Armitage, D.R. et al. (2009) Adaptive co‐management for Social–ecological complexity, Frontiers in Ecology and the Environment, 7(2): 95–102. doi: 10.1890/070089
Bandola-Gill, J. (2019) Between relevance and excellence? Research impact agenda and the production of policy knowledge, Science and Public Policy, 46(6): 895–905. doi: 10.1093/scipol/scz037
Bandola-Gill, J., Arthur, M. and Leng, R.I. (2021) Citation network of the knowledge co-production literature. Supplementary data, Zenodo, doi: 10.5281/zenodo.5762451.
Barry, A. (2020) What is an environmental problem?, Theory, Culture and Society, 38(2): 93–117. doi: 10.1177/0263276420958043
Bastian, M., Heymann, S. and Jacomy, M. (2009) Gephi: an open source software for exploring and manipulating networks, International AAAI Conference on Weblogs and Social Media, 3(1): 361-2, https://ojs.aaai.org/index.php/ICWSM/article/view/13937.
Batagelj, V., Ferligoj, A. and Squazzoni, F. (2017) The emergence of a field: a network analysis of research on peer review, Scientometrics, 113(1): 503–32. doi: 10.1007/s11192-017-2522-8
Berkes, F. (1999) Sacred Ecology, New York: Routledge.
Berkes, F. (2009) Evolution of Co-management: role of knowledge generation, bridging organizations and social learning, Journal of Environmental Management, 90(5): 1692–1702. doi: 10.1016/j.jenvman.2008.12.001
Bovaird, T. (2007) Beyond engagement and participation: user and community coproduction of public services, Public Administration Review, 67(5): 846–60. doi: 10.1111/j.1540-6210.2007.00773.x
Brandt, P., Ernst, A., Gralla, F., Luederitz, C., Lang, D.J., Newig, J., Reinert, F., Abson, D.J. and von Wehrden, H. (2013) A review of transdisciplinary research in sustainability science, Ecological Economics, 92: 1–15. doi: 10.1016/j.ecolecon.2013.04.008
Bremer, S. and Meisch, S. (2017) Co‐production in climate change research: reviewing different perspectives, Wiley Interdisciplinary Reviews: Climate Change, 8(6): e482. doi: 10.1002/wcc.482
Callon, M. (1999) The role of lay people in the production and dissemination of scientific knowledge, Science, Technology and Society, 4(1): 81–94. doi: 10.1177/097172189900400106
Cash, D.W., Borck, J.C. and Patt, A. G. (2006) Countering the loading-dock approach to linking science and decision making: comparative analysis of El Niño/Southern Oscillation (ENSO) forecasting systems, Science, Technology, and Human Values, 31(4): 465–94. doi: 10.1177/0162243906287547
Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J. and Mitchell, R.B. (2003) Knowledge systems for sustainable development, Proceedings of the National Academy of Sciences, 100(14): 8086–8091. doi: 10.1073/pnas.1231332100
Contandriopoulos, D., Lemire, M., Denis, J.L. and Tremblay, É. (2010) Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature, Milbank Quarterly, 88(4): 444–83. doi: 10.1111/j.1468-0009.2010.00608.x
Cornell, S. et al. (2013) Opening up knowledge systems for better responses to global environmental change, Environmental Science and Policy, 28: 60–70. doi: 10.1016/j.envsci.2012.11.008
Dilling, L. and Lemos, M.C. (2011) Creating usable science: opportunities and constraints for climate knowledge use and their implications for science policy, Global Environmental Change, 21(2): 680–89. doi: 10.1016/j.gloenvcha.2010.11.006
Epstein, S. (1996) Impure Science: AIDS, Activism, and the Politics of Knowledge, Berkeley, CA: University of California Press.
Fazey, I. et al. (2013) Knowledge exchange: a review and research agenda for environmental management, Environmental Conservation, 40(1): 19–36. doi: 10.1017/S037689291200029X
Folke, C., Hahn, T., Olsson, P. and Norberg, J. (2005) Adaptive governance of social-ecological systems, Annual Review of Environment and Resources, 30: 441–73. doi: 10.1146/annurev.energy.30.050504.144511
Funtowicz, S.O. and Ravetz, J.R. (1993) Science for the Post-normal age, Futures, 25(7): 739–55. doi: 10.1016/0016-3287(93)90022-L
Gagliardi, A.R., Berta, W., Kothari, A., Boyko, J. and Urquhart, R. (2015) Integrated knowledge translation (IKT) in health care: a scoping review, Implementation Science, 11(1): 1–12. doi: 10.1186/s13012-015-0367-1
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P. and Trow, M. (1994) The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies, London: Sage.
Gieryn, T.F. (1983) Boundary-work and the demarcation of science from non-science: strains and interests in professional ideologies of scientists, American Sociological Review, 48(6): 781–95. doi: 10.2307/2095325
Graham, I.D., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W. and Robinson, N. (2006) Lost in knowledge translation: time for a map? Journal of Continuing Education in the Health Professions, 26(1): 13–24. doi: 10.1002/chp.47
Guston, D.H. (2001) Boundary organizations in environmental policy and science: an introduction, Science, Technology, and Human Values, 26(4): 399–408. doi: 10.1177/016224390102600401
Hirsch Hadorn, G., Bradley, D., Pohl, C., Rist, S. and Wiesmann, U. (2006) Implications of transdisciplinarity for sustainability research, Ecological Economics, 60(1): 119–28. doi: 10.1016/j.ecolecon.2005.12.002
Jacomy, M., Venturini, T, Heymann, S., and Bastian, M. (2014) ForceAtlas2, a continuous graph layout algorithm for handy network visualization designed for the Gephi software, PLoS One, 9(6): e98679. doi: 10.1371/journal.pone.0098679
Jahn, T., Bergmann, M. and Keil, F. (2012) Transdisciplinarity: between mainstreaming and marginalization, Ecological Economics, 79(C): 1–10. doi: 10.1016/j.ecolecon.2012.04.017
Jasanoff, S. (1990) The Fifth Branch: Science Advisers as Policymakers, Cambridge, MA: Harvard University Press.
Jasanoff, S. (2005) Designs On Nature, Princeton, NJ: Princeton University Press.
Jasanoff, S. (2010) A new climate for society, Theory, Culture and Society, 27(2–3): 233–53. doi: 10.1177/0263276409361497
Jasanoff, S. (ed) (2004) States of Knowledge: The Co-production of Science and the Social Order, Milton Park: Routledge.
Kirchhoff, C.J., Lemos, M.C. and Dessai, S. (2013) Actionable knowledge for environmental decision making: broadening the usability of climate science, Annual Review of Environment and Resources, 38: 393–414. doi: 10.1146/annurev- environ-022112-112828
Klavans, R. and Boyack, K.W. (2017) Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?, Journal of the Association for Information Science and Technology, 68(4): 984–98. doi: 10.1002/asi.23734
Kothari, A. and Wathen, C.N. (2013) A critical second look at integrated knowledge translation, Health Policy, 109(2): 187–91. doi: 10.1016/j.healthpol.2012.11.004
Lang, D.J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., Swilling, M. and Thomas, C.J. (2012) Transdisciplinary research in sustainability science: practice, principles, and challenges, Sustainability Science, 7(1): 25–43. doi: 10.1007/s11625-011-0149-x
Latour, B. (1987) Science in Action: How to Follow Scientists and Engineers Through Society, Cambridge, MA: Harvard University Press.
Latour, B. (1993) We Have Never Been Modern, Cambridge, MA: Harvard University Press.
Lemos, M.C., Kirchhoff, C.J. and Ramprasad, V. (2012) Narrowing the climate information usability gap, Nature Climate Change, 2(11): 789–94. doi: 10.1038/nclimate1614
Lemos, M.C. and Morehouse, B.J. (2005) The co-production of science and policy in integrated climate assessments, Global Environmental Change, 15(1): 57–68. doi: 10.1016/j.gloenvcha.2004.09.004
Leng, G. and Leng, R.I. (2020) The Matter of Facts: Skepticism, Persuasion, and Evidence in Science. Cambridge, MA: MIT Press.
Levac, D., Colquhoun, H. and O’Brien, K.K. (2010) Scoping studies: advancing the methodology, Implementation Science, 5(1): 1–19. doi: 10.1186/1748-5908-5-69
Lomas, J. (2000) Using ‘linkage and exchange’ to move research into policy at a Canadian foundation, Health Affairs, 19(3): 236–40. doi: 10.1377/hlthaff.19.3.236
Lövbrand, E. (2007) Pure science or policy involvement? Ambiguous boundary-work for Swedish carbon cycle science, Environmental Science and Policy, 10(1): 39–47.
Lövbrand, E. (2011) Co-producing European climate science and policy: a cautionary note on the making of useful knowledge, Science and Public Policy, 38(3): 225–36.
Martin, S. (2010) Co-production of social research: strategies for engaged scholarship, Public Money and Management, 30(4): 211–18. doi: 10.1080/09540962.2010.492180
McNie, E.C. (2007) Reconciling the supply of scientific information with user demands: an analysis of the problem and review of the literature, Environmental Science and Policy, 10(1): 17–38. doi: 10.1016/j.envsci.2006.10.004
Meadow, A.M., Ferguson, D.B., Guido, Z., Horangic, A., Owen, G. and Wall, T. (2015) Moving toward the deliberate coproduction of climate science knowledge, Weather, Climate, and Society, 7(2): 179–91. doi: 10.1175/WCAS-D-14-00050.1
Metz, A., Boaz, A. and Robert, G. (2019) Co-creative approaches to knowledge production: what next for bridging the research to practice gap?, Evidence & Policy, 15(3): 331–37.
Miller, C.A. and Wyborn, C. (2020) Co-production in global sustainability: histories and theories, Environmental Science and Policy, 113: 88–95. doi: 10.1016/j.envsci.2018.01.016
Mobjörk, M. (2010) Consulting versus participatory transdisciplinarity: a refined classification of transdisciplinary research, Futures, 42(8): 866–73.
Newman, M.E. and Girvan, M. (2004) Finding and evaluating community structure in networks, Physical Review E, 69(2Pt2): 026113. doi: 10.1103/PhysRevE.69.026113
Nowotny, H., Scott, P.B. and Gibbons, M.T. (2001) Re-thinking Science: Knowledge and the Public in an Age of Uncertainty, New York: John Wiley & Sons.
Nutley, S.M., Walter, I. and Davies, H.T. (2007) Using Evidence: How Research Can Inform Public Services, Bristol: Policy Press.
Oliver, K., Innvar, S., Lorenc, T., Woodman, J. and Thomas, J. (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers, BMC Health Services Research, 14(1): 1–12. doi: 10.1186/1472-6963-14-1
Ostrom, E. (1996) Crossing the great divide: coproduction, synergy, and development, World Development, 24(6): 1073–87. doi: 10.1016/0305-750X(96)00023-X
Pohl, C. et al. (2010) Researchers’ roles in knowledge Co-production: experience from sustainability research in Kenya, Switzerland, Bolivia and Nepal, Science and Public Policy, 37(4): 267–81. doi: 10.3152/030234210X496628
Polk, M. (2014) Achieving the promise of transdisciplinarity: a critical exploration of the relationship between transdisciplinary research and societal problem solving, Sustainability Science, 9(4): 439–51. doi: 10.1007/s11625-014-0247-7
Polk, M. (2015) Transdisciplinary co-production: designing and testing a transdisciplinary research framework for societal problem solving, Futures, 65: 110–22. doi: 10.1016/j.futures.2014.11.001
Raymond, C.M., Fazey, I., Reed, M.S., Stringer, L.C., Robinson, G.M. and Evely, A.C. (2010) Integrating local and scientific knowledge for environmental management, Journal of Environmental Management, 91(8): 1766–77. doi: 10.1016/j.jenvman.2010.03.023
Reed, M.S. (2008) Stakeholder participation for environmental management: a literature review, Biological Conservation, 141(10): 2417–31. doi: 10.1016/j.biocon.2008.07.014
Rittel, H.W. and Weber, M.M. (1973) Dilemmas in a general theory of planning, Policy Sciences, 4(2): 155–69. doi: 10.1007/BF01405730
Rowe, G. and Frewer, L. (2000) Public participation methods: a framework for evaluation in science, Technology and Human Values, 25(1): 3–29. doi: 10.1177/016224390002500101
Sarewitz, D. and Pielke, R.A. (2007) The neglected heart of science policy: reconciling supply of and demand for science, Environmental Science and Policy, 10(1): 5–16. doi: 10.1016/j.envsci.2006.10.001
Swidler, A. (2001) What anchors cultural practices, in T.R. Schatzki, K. Knorr-Cetina, and E. von Savigny (eds) The Practice Turn in Contemporary Theory, Milton Park: Routledge, pp 74–92.
Tengö, M., Brondizio, E.S., Elmqvist, T., Malmer, P. and Spierenburg, M. (2014) Connecting diverse knowledge systems for enhanced ecosystem governance: the multiple evidence base approach, Ambio 43(5): 579–91.
Traag, V.A., Waltman, L. and van Eck, N.J. (2019) From Louvain to Leiden: guaranteeing well-connected communities, Scientific Reports, 9(1): 1–12. doi: 10.1038/s41598-019-41695-z
Tricco, A.C. et al. (2018) PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation, Annals of Internal Medicine, 169(7): 467–73. doi: 10.7326/M18-0850
Turnhout, E., Metze, T., Wyborn, C., Klenk, N. and Louder, E. (2020) The politics of co-production: participation, power, and transformation, Current Opinion in Environmental Sustainability, 42: 15–21. doi: 10.1016/j.cosust.2019.11.009
Van de Ven, A.H. and Johnson, P.E. (2006) Knowledge for theory and practice, Academy of Management Review, 31(4): 802–21. doi: 10.5465/amr.2006.22527385
Weiss, C.H. (1979) The many meanings of research utilization, Public Administration Review, 39(5): 426–431. doi: 10.2307/3109916
Wiek, A., Ness, B., Schweizer-Ries, P., Brand, F.S. and Farioli, F. (2012) From complex systems analysis to transformational change: a comparative appraisal of sustainability science projects, Sustainability Science, 7(1): 5–24. doi: 10.1007/s11625-011-0148-y
