Abstract
There has been much debate about the contribution of ‘design thinking’ to the fields of public policy and governance. This article makes an empirical contribution to this debate by examining the Organised Crime Field Lab – an environment for experimenting with, learning about and innovating in collaborative governance. The study involved working with 18 different multi-agency collaborations involving over 160 professionals as they developed novel approaches to fighting organised crime. Combining quasi-experimental and action research methods, our analysis offers valuable insights into how an environment can be designed that creates the conditions to support collaborations in overcoming the most common challenges in their design process. In particular, we find that a specially designed environment including a structured but flexible problem-solving space, an inclusive facilitative process and a custom-made accountability structure can support collaborative design processes.
Introduction
Research on how the public sector can become more innovative and deliver public value more effectively and efficiently has proliferated over the past two decades (Borins, 2014). Most of this research focuses on innovations in policies and instruments. Moore and Hartley (2008) argue that an important but still understudied category is innovation in governance. Institutional arrangements for nominating complex public problems, developing multi-agency responses and evaluating societal outcomes can themselves be innovations.
Given the complexities of reconciling perspectives and interests across institutions and sectors, designing innovations in governance to achieve improved social outcomes is challenging (Bardach, 2001; Cels et al, 2012; Forrer et al, 2014). Ansell and Torfing (2014) highlight the importance of setting the right conditions for design processes in collaborative governance contexts. Working in collaborative governance arrangements requires much more attention to questions of composition, role and structure than in traditional government, where these questions are mostly settled and the focus is primarily on policies and instruments. It remains unclear, though, what conditions are conducive to the design of innovative governance arrangements, and how to create them.
Recent design science research has advanced our understanding of the value of design thinking and using the design process for public policy innovations (Bason, 2010; 2017). Existing work tends to focus either on conceptual and theoretical explorations of design processes (Dorst, 2011; Howlett, 2014; Mintrom and Luetjens, 2016) or on case studies (Hillgren et al, 2011; Bryson et al, 2013). It does not formulate and test propositions about the conditions for effective design in today’s collaborative governance contexts. Simultaneously, collaborative governance theorists have explored conditions influencing the success of collaborative processes (Ansell and Gash, 2008; Emerson et al, 2012), but this work does not say whether these conditions are conducive for collaborative innovation engaged in policy design nor how to change these conditions in a dynamic design environment.
Bason (2010: 19) claims that for public sector innovation, ‘random incrementalism still seems to be the rule rather than the exception. New thinking often happens by chance and against the odds, and the potential of a more conscious, strategic and systematic approach to innovation across public organizations and sectors is not realized’. Mintrom and Luetjens (2016: 399) confirm that ‘there are few empirical studies on actual use of design thinking…in the private and public sphere’, let alone in new collaborative governance settings.
This article aims to fill this gap by exploring, through empirical study, what a more general approach to designing environments for experimenting with, learning about and innovating in collaborative governance looks like. We answer the following question: How does one design an environment that creates the conditions that support collaborations in overcoming the most common challenges in their design process? To answer this question, we draw on the literatures on collaborative governance and design science in public policy and gain new insights by designing an environment for experimenting, learning and innovating ourselves: the Organised Crime Field Lab (OCFL).
We launched the OCFL in collaboration with the Public Prosecution Service and the National Police in the Netherlands in 2014. Our objective was twofold: to facilitate practitioners in developing innovative solutions to ‘wicked’ crime problems, and to advance knowledge of the design features that enable or hinder effective collaboration. Between 2014 and 2017, a total of 18 multi-agency crime-fighting collaborations, and a total of 161 individuals, participated in the OCFL, including police officers, public prosecutors, tax inspectors, local and regional government officials and non-state stakeholders (for example, an energy company fraud specialist).
We provided these collaborations and individuals with training and coaching, which evolved over the years in content and form. Topics covered included the analysis of complex problems and stakeholder configurations, strategic public management, cross-sector collaboration and leadership. Our methods were participant-centred and discussion-based, using customised teaching cases. With a five-day intensive residential workshop at its core, the programme grew into a year-long trajectory that started with a kick-off day, followed by pre-workshop preparatory meetings as well as post-workshop reflection and progress review sessions.
As an environment for learning, experimentation and innovation, the OCFL facilitated the design process of the 18 collaborations and generated significant qualitative and quantitative data. In a quasi-experimental action research framework, we observed collaborations and surveyed and interviewed individuals about the challenges they encountered. We used this data both to improve our understanding of what facilitates and impedes collaboration, and to improve the design of the OCFL over the years. As such, we ourselves adopted a design approach, experimenting with and learning about the design features of the collaborative space that supports collaborations in their own design process.
Design processes thus took place at two distinct levels of analysis: first, at the level of the 18 collaborations designing innovative crime-fighting approaches and, second, at the level of the trainers, coaches and researchers in charge of designing and managing the OCFL as an environment for learning, experimentation and innovation. This article focuses on the latter level of analysis.
We begin with a brief literature review on design processes and environments in collaborative governance contexts, then describe our research approach and present the origins, context and evolution of the OCFL design. Based on this, we formulate propositions on the conditions of the design environment that may support collaborative governance design processes. They include a problem-solving space that allows for continuous iteration in practice and provides pre-structuring of the problem-solving process through frameworks and scaffolding; a facilitation process that includes future collaborators in problem definition and the selection of fellow collaborators and provides coaching on teaming practices; and an accountability structure that involves collaborators’ top management and immediate superiors, and that allows for reporting various types of progress, including progress in terms of learning. By putting forward conditions required for design work in novel collaborative governance settings, we make a fresh contribution to both the literatures on collaborative governance and design science in public policy.
Design in a collaborative governance context
The notion of public administration as a multi-disciplinary design science is not new. Simon (1996) referred to the management field broadly as an artificial science – different from the natural sciences because problems and answers differ depending on manmade contexts. Shangraw and Crow (1989: 156) built on this idea: ‘Public administration…draws knowledge from [other behavioural science fields], for the purpose of designing, constructing, and evaluating institutions and mechanisms for the public good.’
Nevertheless, over the past decades, both the concept of ‘design’ in public policy and the notion of ‘design science’ in public administration have been largely absent. Various explanations have been offered. Howlett (2014: 187) argues that in the 1990s and 2000s, design approaches to governance were neglected because ‘design’ was associated with blueprints and top-down government, which seemed incompatible with conceptualisations of governance as a phenomenon of bottom-up, emergent action in networks. Bryson et al (2013: 24) suggest that risk-averse administrative cultures and structural barriers to experimentation have favoured traditional bureaucratic approaches.
In recent years, however, the need to understand and advance the art and science of governance design has resurfaced. In many countries, trends toward decentralisation and privatisation raise new questions about how to coordinate between actors, sectors and levels of government and organise collective decision-making around wicked societal problems. A design approach to collaborative governance, understood as a process of small-scale experimentation, learning by doing, and co-creation of innovative solutions, might fit the dynamism and uncertainty of contemporary social and institutional challenges. Thus, a design thinking approach neither has to be top-down nor risky, but can rather serve as a way to satisfy the increasing demand for governance design that is inclusive of stakeholders, consensus-oriented and deliberative, and mindful of the constraints and complexities inherent in the public sector.
Design processes for collaborative governance
Recent literature has explored the question of what constitutes an effective policy design process in today’s ‘messy’ policy contexts. Mintrom and Luetjens (2016) describe five phases of policy design processes: first, empathetically observing the target group; second, exploring the problem; third, canvassing possible solutions; fourth, developing a prototype solution; and fifth, testing the prototype with the target group. The novelty of this design approach lies in the fact that it is user-centred, non-linear and iterative, and more comfortable with incompleteness, uncertainty and outside-the-box solutions. The goal is not to invent a superior policy design in one take, but rather to develop a Minimum Viable Product (MVP) in collaboration with users. There is no irreversible path from problem definition to solution, but rather a back-and-forth process between the ‘problem space’ and the ‘solution space’.
The type of reasoning employed to draw conclusions from these iterative processes has also received renewed attention. In traditional policy design, inductive and deductive reasoning were the primary logics. Even when assumptions and data were dubious, value was attached to a theory of change grounded in such reasoning. In stark contrast stands the non-linear, iterative process underlying design thinking in today’s policy context, which requires abductive reasoning. Abductive reasoning involves inferring explanations for phenomena from incomplete or uncertain premises (Dorst, 2011). It acknowledges the difficulty of gathering all relevant information at once and rather tests assumptions through rapid prototyping. To reduce randomness and increase the plausibility of inferences, researchers use methods to structure this process of testing assumptions, such as framing, environmental context scanning, collaborative inquiry and stakeholder mapping (Dorst, 2011; Mintrom and Luetjens, 2016).
Theorists are also starting to explore design processes specifically for novel governance arrangements, as opposed to the design of single services or policies. For instance, Hillgren et al (2011) discuss the slow, iterative process of ‘infrastructuring’ new collaborative arrangements, which is significantly different in nature from the traditional fast-paced design approach of single service or policy prototyping. Building on this, Bryson et al (2013) show how the iterative nature of design is critical in efforts to design solutions with and for citizens and external stakeholders. The fact that new governance arrangements include multiple stakeholders and that the issues are multi-dimensional and volatile necessitates multiple ‘rounds’ of consultation, deliberation and design.
Exploring the design environment
While the process of designing public policy and governance has received some attention in the literature as discussed earlier, the notion of an environment for designing has been explored much less. In much of the literature there is an implicit assumption that such an environment simply exists. The process approach emphasises steps or phases, and actors or stakeholders, but does not articulate where the design work gets done, under what conditions, or how to create those conditions (Ansell and Gash, 2008; Sorenson and Torfing, 2011; Ansell and Torfing, 2014).
We define the design environment as the setting, more or less consciously created and maintained, in which actors experiment, learn and innovate to establish institutional arrangements to govern the nomination of complex public problems, the development of multi-agency responses, and the evaluation of societal outcomes. This understanding builds on what Ansell and Torfing (2014: 191) call the ‘interactive arena’, which includes, among other issues, the ‘policy objectives, actor composition, role positions and work form’ adopted to facilitate the design process in collaborative governance contexts. Our conceptualisation of the design environment concretises and extends this idea in two ways. First, it includes what actors actually design and the strategies they employ to overcome challenges. Second, it considers not only the internal interactive arena but also how external actors support collaborators in their design process.
A systematic empirical exploration of what constitutes a design environment that creates the conditions for collaborations to address common design-process challenges has been lacking. There have been numerous conceptual studies exploring effective conditions for collaborative governance to thrive, including Ansell and Gash’s (2008) model of collaborative governance and Emerson et al (2012) integrative framework for collaborative governance. However, none of these studies has, actively explored interventions to affect these conditions for collaborative governance in a design process nor did they explicitly focus on collaborations engaged in design work themselves.
Case studies of particular design environments, like explorations of policy labs by Bailey and Lloyd (2017) and Williamson (2015) do give us a starting point. For instance, Bailey and Lloyd suggest a number of conditions for effectively designing policy labs, including political and institutional embedding, mechanisms to overcome cultural inhibitions that hinder collaboration, and a format that encourages collaboration and progress. Yet, through reflecting on the iterative design process that shaped the OCFL, we seek to formulate a more general understanding of design environment conditions that enable experimentation, learning and innovation in governance.
A research design for design research on collaborative governance
To fill the observed gap in the literature, we used a quasi-experimental action research approach, applying it at the two levels distinguished before. First, through training and coaching, we supported collaborations designing innovative approaches to complex organised-crime issues. Like other action researchers, we participated in a ‘change situation’, working not only to understand but also to improve problem-solving capacity. Simultaneously, we studied the teams’ responses to different iterations of our design environment (the OCFL training and coaching programme we created).
More like engineers or doctors than biologists or physicists, we worked by trial and error, interacting directly with the crime-fighting collaborations under study, introducing and dynamically adjusting conditions in the design environment to test the impact of these changes on their design process. Our approach closely followed the design process described by Mintrom and Luetjens (2016). We ‘empathetically observed’ the collaborative governance challenges of our target group and designed a prototype design environment to help the collaborations overcome these challenges. We tested this prototype with the actual target group and adjusted it based on direct observations and feedback in practice. Because we were pursuing a clear value – effective collaboration to fight organised crime – without a hypothesis about precisely how to produce it, we relied predominantly on abductive reasoning, inferring different potential approaches to produce the sought-after value from incomplete or uncertain premises.
Our action research approach allowed us to experiment with different interventions to support the collaborations’ design process and to move beyond the theoretical explorations of the design process and single case studies that have dominated the literature on collaborative governance and design science. It also allowed us to do so at a stage when more stringent social-science methods, such as randomised controlled trials, seemed premature or even futile, since no clear hypotheses about ideal environments for collaborative governance processes exist and collaborative governance seems too messy to randomise across. Finally, it made it possible to facilitate change while studying collaboration processes and their design environments, and to combine experimentation, learning and innovation – both at the level of the collaborations and at the level of the OCFL.
Case selection
The case of crime-fighting collaborations in the Netherlands is representative of a broader set of collaborative governance efforts seeking to design new solutions for ‘wicked’ problems – problems that are difficult to solve because facts about their origins and character are incomplete and norms about their solutions are contradictory and changing. Collaborations focused on solving wicked problems are themselves a subset of collaborative governance efforts, or ‘structures of public policy decision making and management’ that bring together stakeholders from all sectors ‘in order to carry out a public purpose that could not otherwise be accomplished’ (Emerson et al, 2012: 2).
We selected the case of crime-fighting collaborations for three reasons. First, there are many similarities – and hence possibilities for generalisation or at least wider hypothesis testing – between these collaborations and collaborations tackling wicked problems in other domains. As with similar challenges in the environmental and cyber-security domains, for instance, these wicked problems have surfaced due to the increased interconnectedness and economic intensification the world is facing. Responses by single organisations have proven to be too one-dimensional, sub-scale and inadequate to deal with them. As such, problem-oriented collaborations have formed around solving these issues in more multi-faceted, systemic ways, for instance through energy sector transition strategies or common cyber-security standards across organisations. Such collaborations all face similar growing pains of working with new partners for the first time, attempting to innovate across each organisations’ capabilities to resolve issues that have emerged because of a changing global economic context.
Second, as a practical matter, we had a high degree of access to these crime-fighting collaborations and room to experiment with their design environment, given previous work with the involved stakeholders. Third, and perhaps most importantly, the urgency of the underlying issue of organized crime in the Netherlands led to a high level of political priority and commitment among the stakeholders, which allowed us to work with the collaborations over a relatively long period of time without significant losses in participation or level of activity.
Data collection and analysis
To understand the impact of our design environment on the collaborations’ design process, data collection and analysis followed a two-stage process. First, we had to gather and analyse data on the collaborations’ design processes themselves, particularly the challenges collaborators encountered. Second, to answer our research question, we collected and interpreted data on the extent to which our interventions in the design environment created conditions that helped collaborations overcome those challenges.
We used various techniques to gather data on the collaborations’ design processes and challenges. Observation of the collaborations in practice helped to identify challenges and responses. For every year-long OCFL trajectory, we, the researchers, each spent at least seven working days with the collaborations, equal to roughly two-and-a-half to five full days of direct observation per collaboration (depending on how many participated), or just above 60 days of direct observations in total. We also engaged coaches – experienced practitioners from the collaborating organisations who were not participating in the collaborations themselves – to spend a similar amount of time with each collaboration, which provided us with another source of roughly 90 days of indirect observations in total across all the OCFLs.
We also used interviews and surveys to dig deeper and ask participants directly about the challenges they experienced. We interviewed all of the collaborations once in their group setting during the five-day workshop that was part of the OCFL (total of 18 group interviews) and surveyed the 161 individual collaboration members once during the same weeklong workshop and twice after the workshop week. Almost all collaboration members completed these surveys. Throughout the process, we gathered collaborations’ deliverables and communications to enrich our understanding of the work they were doing and the approaches they were designing. We asked the collaborations to submit a range of documents, from initial problem definitions to implementation plans and progress reports. We were also copied in most email correspondence among collaboration members, revealing detailed insights about the challenges and how they dealt with them.
To track changes in our design environment and their influence on collaborations’ success in overcoming challenges, we logged every change we made in the OCFL and kept records of our deliberations around the reasons for making these changes. These deliberations included debrief sessions immediately after and in between formal sessions in the OCFL (kick-off day, weeklong workshop, reflection sessions, progress review meetings) and evaluation and preparation meetings at the end and beginning of each year-long trajectory. During these sessions, trainers, coaches, researchers and managers from the collaborating organisations shared observations and discussed the effects of changes to the programme. Subsequent iterations of the programme used learnings from these discussions to improve facilitation of the collaborations’ design processes further.
We analysed the resultant data at two successive levels. First, the debrief sessions and meetings at the beginning, middle and end of the year-long OCFL trajectory were themselves moments of analysis during which we not only reflected on how to improve the OCFL as a design environment, but also advanced our understanding of how to design a design environment that supports collaborations in overcoming common challenges in their design process. These resembled the open-to-learning conversations proposed by Mintrom and Luetjens (2016) as an effective design-thinking strategy, questioning various elements of the OCFL as a design environment with a diverse set of individuals around the table.
Second, we engaged in a final sense-making process (Mintrom and Luetjens, 2016). After four OCFL trajectories had transpired, we synthesised our understandings of primary collaborative challenges and effective interventions in the design environment. We organised our observations, ‘physically, placing items that are related next to each other’ (Mintrom and Luetjens, 2016: 398). This allowed us to discover the (often complex) relationships between collaborations’ design processes and the results thereof, and between our interventions and the design environment.
Validity
A key challenge for this approach is its internal validity. Using the tracking approach described earlier, we sought to trace the causal process and effects of certain interventions in the design environment. However, it was often possible to imagine several other explanations for observed changes in the collaborations’ design processes. Ensuring temporal proximity between changes in the design environment and observations of collaborations’ responses made it possible to establish a causal pathway for our observations with a higher degree of reliability.
Herr and Anderson (2014) propose a number of other methods to safeguard internal validity for action-research approaches. To ensure dialogic validity – essentially peer review in action – we conducted reflection sessions not only among ourselves, but also with external practitioners and academics. In addition, extensive documentation helped ensure process validity – recording findings carefully to allow ongoing learning and continuously looping back to refine earlier conclusions.
External validity is another critical challenge for this type of action research. The generalisation of findings from an approach essentially relying on a set of case studies (the collaborations in the OCFL) may be problematic. Given the particularities of the case and its specific context, propositions would not necessarily be valid for all other cases and in other contexts. For example, the OCFL was replicated four times, but always in a specific local crime-fighting context to which the design was adapted. This affected certain OCFL features like the number, composition and size of collaborations participating and the most important collaboration obstacles to deal with, which depended in large part on the history of collaboration in a region. For instance, as explored later in this article, the focus on learning how to work together was much larger in earlier editions of the OCFL as many participants had not worked together before. Such particularities should be kept in mind when trying to generalise from this particular study.
Nevertheless, there are insights to be gained for the design environments of other collaborative efforts dealing with wicked problems in the same or other domains. In our propositions, we heeded particular attention to draw insights across the four OCFLs we conducted, which makes the findings on particularly important features already more universal. In addition, rather than simply transplanting them, these insights should be tried and tested for the particularities of other cases – an inherent feature of the very design process (Howlett, 2014), the subject of our exploration.
The evolution of the Organised Crime Field Lab
Following the publication of a case study we wrote on innovation in the fight against human trafficking in the Netherlands, the chief prosecutor of the Dutch Public Prosecution Service inquired whether the approach might be applied in other areas. As scholars of cross-boundary collaboration and public-sector innovation, eager to study ‘innovation in action’, we agreed to co-create a programme that offered training and coaching to aspiring innovators and an opportunity to learn while doing through action research (Waardenburg et al, 2018).
The programme’s goal was to forge collaborations to fight organised crime through innovative methods. Programme participants would design collaborative approaches to specific, intractable organised crime problems. The idea was to look beyond arrests and prosecutions and identify ways to prevent and deter crime by thinking more strategically and systemically about the problem – for example, by erecting barriers for criminals and putting pressure on individuals and organisations that (consciously or not) facilitate such crime. The general philosophy was to frustrate criminal processes as well as the social, economic and administrative infrastructure that enables them, and put the criminals out of business.
This preventative approach required close collaboration between the Public Prosecution Service, the National Police and other organisations such as local and regional government and tax authorities. It demanded a new mindset and skillset for prosecutors, undertaking responsibilities and activities far beyond their traditional role in collaboration with different actors.
Training and coaching programme (2014)
The first edition of what would become the OCFL took place in 2014. Forty people participated in a five-day workshop, each bringing a personal challenge to reflect on. The programme’s focus was training and coaching to equip public prosecutors, police officers and city officials with the skills to operate strategically and systemically in tackling crime. Conventionally, these professional groups receive training separately. This cross-boundary programme, bringing together multiple professions around a societal issue, was a novelty.
Before the workshop, we asked participants to write down their most pressing collaboration challenge, including a description of the substantive problem, the outcomes they were trying to achieve, the involved partners and their roles, and the roadblocks to effective collaboration. This resulted in a rich overview of challenges on which to build during the workshop. The workshop comprised highly interactive sessions around customised teaching cases relevant to several themes: analysis of wicked problems and stakeholder configurations, strategic thinking and acting, smart cross-boundary collaboration and adaptive leadership. During discussions, we offered analytic frameworks to help examine practical problems and identify action alternatives.
Workshop participants received guidance not only from us as trainers, but also from coaches from their own organisations. This support structure, however, ended after the workshop. Post-workshop surveys showed significant variability in progress made on participants’ challenges. Those who made the most progress reported leaving the programme with a clearer definition of their substantive problem, better articulated goals and indicators of success, and a more focused strategy for accomplishing desired results. Those who made the least progress cited struggles to apply frameworks and define action alternatives in the short timeframe allotted and suffered from the lack of support structure after the workshop to make progress and implement what they had learned.
While the participants and management of the organisations involved reported in evaluation surveys and debrief conversations that the first edition of the OCFL avant-la-lettre was helpful, we were concerned about the ‘real-world’ impact. Would participants be able to apply lessons learned to innovate across organisational boundaries? What if we made application and collaboration more integral to the OCFL?
Towards an experimentation, learning and innovation environment (2015)
Insights into collaborative governance challenges from the first edition of the OCFL helped us fine-tune the curriculum for the second edition. Managing the expectations of participants and stakeholders became an important part of the work of creating the environment for learning, experimentation and innovation. Participants in the first edition tended to have a binary conception of ‘work’ and ‘training’. Echoing the sentiment of others, one participant explained that she ‘thought this was just a training. I didn’t know we actually had to do work as well’. The OCFL was intended to be both things at once, which to most participants was a new – and sometimes confusing – experience.
For the second edition, we therefore made it clear that participants would not merely be taking a ‘course’; they would take part in a year-long trajectory that brought their work into the classroom, organised on-the-job learning, and provided ongoing reflection and progress documentation. Instead of having the collaborating organisations select participants based on a variety of motives, we worked with the organisations to identify priority crime problems in need of innovative approaches and select and invite individuals who could work on them (but were not necessarily doing so already). For this, we partnered with the Task Force Brabant Zeeland, an initiative of the police, the public prosecutor’s office, the tax authority and local governments to stimulate collaborative efforts to fight organised crime in the southwestern part of the Netherlands. We trained and coached participants as members of eight collaborations to redefine their respective crime problems, develop a theory of change, take coordinated actions, and indicate for what kind of results they would like to be held accountable as a collaboration. See Table 1 for three examples of collaborations in the OCFL.
Examples of collaborations in the OCFL (Organised Crime Field Lab)
Crime problem | Collaboration composition | Innovative approaches | |
---|---|---|---|
Eradicating marijuana plantations in local neighbourhoods (OCFL, 2015) | Intimidation of and violence against citizens allured or forced into growing weed in homes; fire hazard from illegally constructed electricity supply | Public prosecutor, police, tax authority, local government, energy system operator | Using electricity network to pinpoint illegal domestic marijuana plantations in neighbourhoods; joint awareness campaigns to raise alertness and targeted enforcement in prone areas |
Putting criminal entrepreneurs out of business (OCFL, 2016) | Entrepreneurs operating in the ‘grey zone’ with a network of (quasi-)illegal business activities including brothels and drugs shops; disregard for the law and tax obligations, intimidation and extortion of citizens | Public prosecutor, police, tax authority, local government | Offering an opportunity for criminal entrepreneurs to ‘confess’ and stop illegal activities. If offer is not taken up, coordinated effort to make business operations unmanageable through stringent tax audits, blanket denial of permit requests, and other similar tools |
Sabotaging Albanian criminal gangs (OCFL, 2017) | Criminal gangs using Amsterdam as logistics hub for drugs trade; violence and intimidation of other gangs as well as facilitators | Public prosecutor, police, tax authority, local government, Regional Information and Expertise Centre | Lifting the veil of anonymity for suspects through smart use of big data and collaboration with Albanian authorities; targeted enforcement actions against priority suspects – for example, checking legal status, tracing money laundering streams |
There were two further structural changes in the training and coaching programme of the 2015 OCFL edition. First, we developed a structured working process over a nine-month period, with designated activities and milestones along the way to support the collaborations including a kick-off meeting several weeks before the workshop and frameworks and models employed to help collaborations proceed strategically and systemically. Second, we helped to create an accountability structure that involved the top-level executives of each participating organisation working together to align their directives and mandates and check in on the collaborations and their progress through two progress review meetings.
Despite these structural changes, designing an environment that facilitates progress in the collaborations’ design process turned out to be a complex challenge. Several participants expressed confusion about why their parent organisations had selected them, sometimes noting that they had never worked on the problem at hand. Interviews also revealed a lingering sense that this was a training on top of their work and expectations that teams would iterate their approach in practice after the workshop and report their progress were not sufficiently clear from the start. Additionally, because top-level executives were far removed from field work, the accountability structure designed contributed to participants not viewing collaborative work as part of their core daily tasks. Finally, our own observations revealed that basic collaboration challenges like trust-building and teaming were among the most significant hurdles the collaborations experienced, an element we did not explicitly integrate in the earliest editions of the OCFL.
Adjusting and improving the design environment (2016–2017)
In 2016, the OCFL moved to the east of the Netherlands with adjustments to the design environment based on the previous edition. Expectations that OCFL work would continue beyond the week-long workshop, during which the six participating collaborations designed their crime-fighting approaches, were made more explicit upfront at the start of the OCFL: trainers instructed collaborations to test their approaches in practice and explained that progress would be monitored at pre-planned intervals. A committee representing all stakeholders nominated problems and selected participants for six collaborations, paying more attention to whether participations were already working on the underlying issues (nonetheless, interviews revealed that not all were).
For this iteration, we also introduced explicit training and coaching on collaboration practices. In addition, we organised a separate training session for middle management to ensure that they were aware of what OCFL work entailed and to help them see it as an integral part of their employees’ daily work. Middle managers also participated in the progress review sessions.
These changes in the trajectory may be one factor that led to fewer observations and mentions (in surveys and interviews) of confusion around expectations, participant selection and accountability. Yet, some deeper-seated issues lingered: participants still felt that the problem and participant selection were externally imposed, which may have been because the collaborations were more or less formed for the OCFL. In addition, we observed that collaborative process issues persisted despite explicit coaching on teaming – most likely because these growing pains may be inevitable for any new collaboration. Finally, despite our efforts, middle management participation was not yet comprehensive and consistent throughout the trajectory.
We took these lessons on-board in the next edition of the OCFL we organised. In 2017 we worked with the Amsterdam and Rotterdam metropolitan regions in the OCFL West NL. Building on insights from the past OCFLs, potential participants selected crime problems in a more bottom-up way, and at least some of the participants selected themselves around these problems. By selecting collaborators in four collaborations that were either emerging or already existent, we could, in theory, shift the focus from overcoming initial challenges like trust building to producing societal outcomes. In practice, however, we observed the usual collaboration challenges, suggesting that facilitating the collaborative process remains critical in any type of collaborative design environment, no matter how mature the collaboration.
In its most mature form, our design environment approximated participants’ real-life accountability environment, including the full involvement of middle management at all stages of the OCFL trajectory. Expectations were explicit from the start not only that collaborations would be required to iterate in practice and report their progress during the OCFL trajectory, but also that they would continue to exist after the final OCFL progress review sessions. Most participants reported in interviews that the OCFL was an integral part of their work and a welcome support in their efforts to find solutions to the selected crime issues.
Between 2014 and 2017, the OCFL evolved from a workshop on smart collaboration into a design environment – optimised over iterations – to facilitate collaborations in the process of designing innovative solutions to wicked organised-crime problems. It began as an externally-imposed training and ended up a vital part of participants’ work, with many potential future participators seeking to enter their own collaborations into the trajectory. Getting results in practice evolved from a conceptual discussion point to the core of the programme. For a summary of adjustments and improvements, see Table 2.
Evolution of the OCFL (Organised Crime Field Lab) as design environment (2014–2018)
Condition in design environment | Smart Collaboration Workshop (2014) | Field Lab South-Netherlands (2015) | Field Lab East-Netherlands (2016) | Field Lab West-Netherlands (2017) |
---|---|---|---|---|
Number of participants/teams | 40 individual participants. | 8 collaborations, comprising 41 individuals in total. | 6 collaborations, comprising 44 individuals in total. | 4 collaborations, comprising 36 individuals in total. |
Focus of training and coaching | Concepts and cases in strategic management in collaborative governance settings. | Concepts and cases plus working in and shaping a real-life problem-oriented collaboration to improve societal outcomes. | Concepts and cases, working on a real-life collaboration plus increased focus on creative thinking and iteration in collaborative design. | Concepts and cases, working on a real-life collaboration in an iterative design process plus increased focus on managing accountability relationships. |
Structure of design process | Input: template for individual challenge; output: personal takeaways. | Input: template for team challenge; output: presentation with team, takeaways and next steps. Two after-programme progress review sessions. | Input: Detailed guide for the 9-month design process with structured templates for deliverables for each step; output: multiple along the stages of the process, including a problem definition, approach presentation and several progress updates. Multiple check-in points and stock-taking sessions and coaching throughout on substance and process. | |
Selection of problems and participants | Participants selected by organisations based on merit and potential. Problems selected by participants. | Problems selected by Task Force Brabant-Zeeland; participants selected top-down by organisations based on merit and potential. | Problems selected top-down by organisations; participants selected based on merit, potential and existing involvement with and responsibility for selected problems. | Problems selected collaboratively and bottom-up by organisations in consultation with participants. Selection of participants primarily based on involvement with and responsibility for selected problems. |
Facilitation of collaboration process | Participants in workshop as individuals – no active collaboration work in training. | Participants part of temporary collaboration around Field Lab – main focus was overcoming initial hurdles of setting up collaborations. | Participants part of longer lasting collaborations around Field Lab – coaching on teaming practices explicitly introduced as core element of programme. | Individuals part of existing collaborative governance arrangements – focus on perfecting teaming practices. |
Integration of OCFL in own organisation and day-to-day work | Top management involved as sponsors; training feels like ‘addition to work’. | Top management actively involved as sounding board for innovations by collaborations; OCFL still feels largely like ‘addition to daily work’. | Top management involved as sponsor and accountability mechanism; middle management involved as sounding board; OCFL feels like more integral part of daily work. | |
Progress reporting mechanism | Not applicable | Two progress review meetings designed and planned ad hoc with selection of high-level representatives. | Multiple check-in and stock-taking meetings designed, planned and formalised with purposeful selection of high-level representatives; | |
Research element | Participant evaluations and feedback and interaction with collaborating organisations informed research agenda. | Research becomes integral part of OCFL; research design and data-collection are focused on documenting progress, understanding obstacles, and identifying enabling conditions for innovation in collaborative governance. Ongoing research feeds back into programme re-design as OCFL uses action research and design thinking to learn about collaborative governance and optimise conditions for collaborative design in the design environment at the same time. |
Propositions for conditions that facilitate the collaborative design process
In our design of the OCFL, we took steps in line with state-of-the-art knowledge on effective policy design processes (Mintrom and Luetjens, 2016). We observed the initial ‘target group’ and the challenges they faced in collaboratively addressing organised crime. We explored possible solutions – from immersive training and coaching programmes to specific interventions such as introducing problem-structuring frameworks and conducting teaming exercises – while prototyping and testing these solutions as part of the OCFL.
Rather than following sequential phases, our design process was highly dynamic. We proceeded in rounds, with our target group expanding and diversifying over time, and the crime problems becoming more rooted in practice. We tested solutions, such as introducing explicit accountability mechanisms, and tailored them to the needs of our target group, for instance by increasingly focusing on how to measure societal outcomes and account for collaborative performance.
As in policy design processes, heavy user involvement, continuous iteration and outside-the-box thinking characterised our design of the design environment. By evaluating activities, results and participant experiences as well as organising feedback and brainstorming sessions with our partners, we developed a deeper understanding of learning needs, aspirations and constraints. Our iterations of the design environment over four years allowed us to experiment with and learn about the conditions that support collaborations in overcoming their collaborative governance challenges and engaging in collaborative innovation design.
Contrary to the implicit assumptions in existing literature, environments for collaborations to design innovative solutions do not simply exist; they must be created and maintained. We thus consciously designed the OCFL as a setting in which collaborations and individual collaborators experimented, learned and innovated. Concretising and expanding existing notions of the setting in which the design process takes place, we explored a more concrete and extended conception of the design environment, specifying what collaborations actually design and how they can be supported in this process through external facilitation. By thinking outside of the mould of existing institutional arrangements, we designed an environment that exists between rather than within formal organisational structures – a kind of parallel context where the design work gets done.
We went beyond the design process and questions of steps or phases, actors or stakeholders, to explore the conditions under which collaborations design innovative solutions, and how to create such conditions as part of the environment in which they design. The resultant design environment helped collaborations face the three most common categories of challenges collaborations face in their design process: substantive problem-solving challenges, collaborative process challenges and multi-relational accountability challenges (Waardenburg et al, forthcoming). We now discuss our findings and formulate propositions about the conditions under which collaborations design innovative solutions that can be tested in further research and inform practice along these three common categories of challenges faced by collaborations engaged in design work.
Conditions supporting substantive problem-solving
Substantive problem-solving challenges comprise the technically and politically difficult work of defining the problem, developing an appropriate solution, and designing performance measures to determine whether the solution has been successful (Agranoff, 2007; Bryson et al, 2015; Head and Alford, 2015). Based on in-group interviews and surveys on collaboration challenges across all OCFL trajectories, we found that defining the problem and finding appropriate approaches was consistently seen as the greatest challenge. This is partly inherent in the paradoxical nature of collaborative problem-solving in the context of solving wicked societal problems, which requires simultaneous analysis and action to move closer to a solution.
Through our iterations of the design environment, we found that sufficient space and time to experiment with solutions and obtain feedback from practice is critical. In the first edition of the OCFL, participants were asked to bring their own substantive challenges, but these challenges were a secondary element, and there were no mechanisms for real-life feedback. In the following years, substantive problem-solving around real-life crime problems in collaborative settings became an increasingly integral part of the trajectory. Feedback from trainers, coaches and superiors and experimentation in practice helped collaborators adjust their designs. In post-workshop evaluation surveys, participants reported that these feedback mechanisms were among the core elements that brought the programme to life. This finding leads us to a first proposition, to be investigated in future research:
A design environment requires designated space and time to experiment with and prototype different approaches in order to obtain feedback from practice.
Another condition of the environment that supported collaborations in their design process was a pre-structured problem-solving process. Within each of the collaborations, participants invariably had different perspectives on the problems at hand, how to solve them and how to measure progress. In collaborative governance settings, there are often multiple (sometimes competing) values to achieve, and these values may change along the way through the collaborative process of defining and redefining a ‘public value proposition’ (Moore, 2013).
In the OCFLs, the crime-fighting collaborations continuously shifted their perceptions of the optimal balance between repressive and preventative approaches and the scope of the problem they wanted to tackle. To facilitate this messy process, we increasingly sought to pre-structure it by providing analytic frameworks and practical scaffolding in the form of questions, suggestions and templates for outputs around, for example, iterations on problem definitions, solutions and implementation plans, and performance reports. A second proposition highlights the importance of frameworks and scaffolding:
When problems and solutions are not clear, and there is a lack of consensus around substance, structuring the substantive problem-solving process through frameworks and scaffolding may help collaborations progress and support their design process.
Conditions supporting the collaborative process
Collaborative process challenges pertain to reconciling the perspectives and interests of partners and building trust (Ansell and Gash, 2008; Provan and Kenis, 2008; Emerson et al, 2012; Bryson et al, 2015). As described by Hillgren et al (2011), this process of ‘infrastructuring’ new collaborative governance arrangements is messy and complex. The involvement of many different (and sometimes new) actors may improve the response to the problem at hand, but sacrifices efficiency and complicates coordination. In the OCFL, we were able to facilitate and observe the collaborative process over time, with participants facing consecutive challenges of initial trust-building, building shared objectives and taking collaborative actions. Unfortunately, a year was still too short to see the full development of the collaborations’ potential. Some collaborations finally secured results after we had left; some continue to work together today. Nevertheless, we were able to glean some insights for the conditions that the design environment should create.
First, over the course of the OCFL trajectories, the importance of the initial collaboration conditions became clear. In contrast to traditional policy design settings, which are often stable and known, the design setting in a collaborative governance context is characterised by dynamic problems and unfamiliar partners. In the first two Field Labs, when individual participants brought their own problems (2014) or were selected rather independently from problems (2015), they were sometimes reluctant to participate in the collaborative design process – likely because they saw this as extra work externally imposed on them. For this reason, by the third Field Lab, we started to embrace a bottom-up approach centred on selecting collaborative governance problems that most participants were at least somewhat involved in solving and allowing for more participant self-selection. This removed a major source of inertia, which is one possible explanation for why collaborations moved to action much more quickly and consistently in the third and fourth OCFLs than in the first two. From this finding follows a third proposition:
Building the initial design environment from the bottom up – including future collaborators in problem selection and definition as well as the selection of ultimate collaboration participants – is critical to ensure buy-in among collaborators and progress on designs.
Second, facilitation of the teaming process was also revealed as an important condition of the design environment. In the first two OCFLs, we focused most of our attention on the substantive problem-solving challenges that collaborations struggled with. However, it became obvious that many of the challenges the collaborations were experiencing were not rooted in an inability to resolve content problems, but rather in an inability to overcome typical teaming hurdles, such as building trust, managing process and aligning perspectives. As a result, in the third and fourth OCFL, we explicitly introduced training on teaming, co-facilitation by coaches and mediation where necessary. This investment upfront helped collaborations to surface and deal with their teaming challenges early on in the design process. This finding is in line with Howlett’s (2017) suggestion that procedural elements of the design environment are just as critical as substantive elements in today’s policy context. This leads us to a fourth proposition:
As part of the design environment, it is critical to provide external facilitation of the collaborative process, including coaching on teaming practices, in addition to facilitating the substantive problem-solving process of collaborations engaged in design.
Conditions supporting accountability relationships
Multi-relational accountability challenges are difficulties collaborations experience in reporting to new channels of accountability, including to other organisations and society at large, that exist in tension with old channels of accountability (Moore and Hartley, 2008; Moynihan et al, 2011; Bryson et al, 2015). In the OCFL, individual collaborators had to account for their progress and performance not only to their own superiors, but also to superiors from the other organisations involved, as well as the political arena.
Accountability challenges were not a central component of the first OCFL edition, but accountability structures were introduced in the second edition to incentivise progress. Yet, the second edition did not make accountability expectations explicit from the start and participants were asked to report to top management without involving their immediate supervisors. This only heightened the sense that the OCFL work was an extra burden. As such, we started to involve middle management in periodic progress review discussions and organised separate training sessions for them in the third edition. This established much clearer incentives for participants to invest in the programme, leading us to a fifth proposition.
As part of the design environment, it is critical to involve the immediate superiors of collaborators in periodic progress review discussions and the broader design process to ensure that the design process is not seen as something external to collaborators’ daily tasks.
The second key condition in relation to accountability structures that we identified is closely related to the first condition for substantive problem-solving processes. Collaborators need wide discretion for testing and prototyping. Prototyping may take longer in collaborative governance settings that require a broad range of actors to coordinate their actions. Moreover, these actors may pursue various values and have different interpretations of progress. As such, results in practice may take longer to materialise, and reporting mechanisms may need to allow for various interpretations of progress.
This was evident when collaborations secured results well after the OCFLs had ended – including some that had looked close to giving up. Their approaches and results often looked markedly different from the approaches and intended results they had initially designed in the weeklong workshop. To accommodate for this, in the second, third and fourth Field Labs, progress review meetings were extended after the OCFL trajectories had ended. In addition, in the third and fourth Field Labs, expectations of a long-run trajectory with frequent pre-planned progress review meetings were made more explicit to the participating collaborations from the start. This finding leads us to a final proposition for future research:
To encourage collaborations to make progress in their design process, progress review meetings are essential, but the evaluation of progress should allow for various interpretations of progress and take into account that collaborative design processes may take longer to unfold than traditional design processes focused on single products or services.
Conclusion
In this article, we set out to answer the following question: How does one design an environment that creates the conditions that support collaborations in overcoming the most common challenges in their design process? To do so, we reviewed the relevant literature on design in public policy and governance, and we presented the experimentation, learning and innovation environment we designed ourselves: the Organised Crime Field Lab (OCFL).
Despite experimentation in practice with policy design in collaborative governance settings, the environment in which policy design processes take place has received relatively little attention in the academic literature. There is an implicit assumption in most academic work on policy design that the environment simply exists and does not need to or cannot be manipulated. Ansell and Torfing (2014), like other governance theorists, do refer to the concept of the ‘interactive arena’ in which collaborations work and operate, but empirical explorations of how to actively shape such a design environment that creates the conditions for collaborations to address common design process challenges have been scarce. Case studies of particular design environments like the explorations of policy labs by Bailey and Lloyd (2017) and Williamson (2015) provide a first stepping stone.
The OCFL allowed us to experiment, learn and innovate with a design environment that is conducive to overcoming common challenges in collaborative design processes. We identified several conditions that flow from the design environment and may support collaborative design processes: first, a problem-solving space that 1) allows for continuous feedback and a lot of iteration in practice, and 2) provides pre-structuring of the problem-solving process through frameworks and scaffolding; second, a facilitative process that 3) includes future collaborators in problem selection and definition as well as the selection of fellow collaborators and 4) addresses the collaborative process, including coaching on teaming practices; finally, an accountability structure that 5) involves not only the collaborators’ top management but also their immediate superiors to ensure the design process is seen as part of their daily work and 6) offers sufficient time, discretion and flexibility to account in different ways for various types of progress, including progress in terms of learning.
We gained a number of more general insights from the design process we undertook. First, setting up a design environment, like policy design itself, is an iterative process. Iteration in practice helped us to recognise the importance of certain conditions of the design environment, such as team creation from the bottom up and facilitation of the team process, which we had initially left out of our design. Second, optimising a design environment requires customisation to the local policy context. For example, we had to adjust for local differences when selecting crime problems and participants and adjust our focus on particular collaboration challenges depending on the history of collaboration in each area. Finally, our work revealed that the facilitation of innovative designs for collaborative governance requires a lot of time and patience, given collaborators’ unfamiliarity with this novel way of working.
This article helps to advance the field by contributing an understanding of how to best influence known conditions for effective collaboration, rather than being passive observers of this phenomenon. As explained, this requires a consciously developed design environment, including coaching, scaffolding and feedback loops. It also views collaborative governance arrangements through a new lens of design science. This allows the field to move past the conceptualisation of governance as a phenomenon of bottom-up, emergent action in networks that can only be observed from afar, to an active concept of collaborative governance as part of which a multiplicity of actors can be supported in their plight to design new policies. The significance of this lies at the very roots of the notion of public administration as a design science: seeking to ‘design, construct, and evaluate institutions and mechanisms for the public good’ (Shangraw and Crow, 1989: 156). Ultimately, this article points at mechanisms to effectively engender problem-oriented collaboration to minimise public value losses from emergent challenges like networked, organised crime.
Our research presents a first step in a systematic empirical exploration of key conditions of the design environment for collaborative governance. Limitations are inherent in the nature of the case study approach we adopted. Remaining questions include the following: was a yearlong cycle long enough to reach conclusive answers and might additional cycles of experimentation lead to a deeper and more granular understanding of conditions for effective collaborative design? Would replication of the research design in other institutional, cultural, geographic and policy contexts lead to different observations and conclusions? Testing the propositions that our research generated in other contexts, and over a longer period of time, through comparative analysis and longitudinal research, would be helpful in moving this field of study forward. Finally, once there is triangulated evidence from different contexts that certain design environment conditions are critical to enable innovative governance design, this could be further tested in more standardised experimental settings with an actual control group.
If it is true that pressing public problems require both innovation and cross-boundary collaboration, we have to face the fact that the academic literature provides little guidance with regards to how this type of work might be supported, guided and governed. The contribution of our research lies in the way it approaches both the practical challenge of designing environments to do the work and the opportunity to learn from the process as it unfolds. As such, it brings together public policy and design science in an attempt to advance the art and science of public problem-solving.
Conflict of interest
The authors declare that there is no conflict of interest.
References
Agranoff, R. (2007) Managing within networks: Adding value to public organizations, Washington, DC: Georgetown University Press.
Ansell, C. and Gash, A. (2008) Collaborative governance in theory and practice, Journal of Public Administration Research and Theory, 18(4): 543–71. doi: 10.1093/jopart/mum032.
Ansell, C. and Torfing, J. (2014) Public innovation through collaboration and design, New York, NY: Routledge.
Bailey, J. and Lloyd, P. (2017) The introduction of design to policymaking: policy lab and the UK government, Annual Review of Policy Design, 5(1): 1–14.
Bardach, E. (2001) Developmental dynamics: interagency collaboration as an emergent phenomenon, Journal of Public Administration Research and Theory, 11(2): 149–64. doi: 10.1093/oxfordjournals.jpart.a003497
Bason, C. (2010) Leading public sector innovation: Co-creating for a better society, Bristol: Policy Press.
Bason, C. (2017) Leading public design: Discovering human-centred governance, Bristol: Policy Press.
Borins, S.F. (2014) The persistence of innovation in government, Washington, DC: Brookings Institution Press with Ash Center for Democratic Governance and Innovation.
Bryson, J.M., Quick, K.S., Slotterback, C.S. and Crosby, B.C. (2013) Designing public participation processes’, Public Administration Review, 73(1): 23–34. doi: 10.1111/j.1540-6210.2012.02678.x.
Bryson, J.M., Crosby, B.C. and Stone, M.M. (2015) Designing and implementing cross-sector collaborations: needed and challenging, Public Administration Review, 75(5): 647–63. doi: 10.1111/puar.12432.
Cels, S., de Jong, J. and Nauta, F. (2012) Agents of change: Strategy and tactics for social innovation, Washington, DC: Brookings Institution Press.
Dorst, K. (2011) The core of ‘design thinking’ and its application, Design Studies, 32(6): 521–32. doi: 10.1016/j.destud.2011.07.006.
Emerson, K., Nabatchi, T. and Balogh, S. (2012) An integrative framework for collaborative governance, Journal of Public Administration Research and Theory, 22(1): 1–29. doi: 10.1093/jopart/mur011.
Forrer, J., Kee, J. J. and Boyer, E. (2014) Governing cross-sector collaboration, San Francisco, CA: John Wiley and Sons.
Head, B.W. and Alford, J. (2015) Wicked problems: implications for public policy and management, Administration and Society, 47(6): 711–39. doi: 10.1177/0095399713481601.
Herr, K. and Anderson, G.L. (2014) The action research dissertation: A guide for students and faculty, Thousand Oaks, California: Sage publications.
Hillgren, P.A., Seravalli, A. and Emilson, A. (2011) Prototyping and infrastructuring in design for social innovation, Codesign-International Journal of Cocreation in Design and the Arts, 7(3–4): 169–83.
Howlett, M. (2014) From the ‘old’ to the ‘new’ policy design: design thinking beyond markets and collaborative governance, Policy Sciences, 47(3): 187–207. doi: 10.1007/s11077-014-9199-0.
Howlett, M. (2017) Policy tools and their role in policy formulation: dealing with procedural and substantive instruments, Handbook of Policy Formulation, Cheltenham: Edward Elgar, pp 96–111.
Mintrom, M. and Luetjens. J, 2016, Design thinking in policymaking processes: opportunities and challenges, Australian Journal of Public Administration, 75(3): 391–402. doi: 10.1111/1467–8500.12211.
Moore, M. and Hartley, J. (2008) Innovations in governance, Public Management Review, 10(1): 3–20. doi: 10.1080/14719030701763161.
Moore, M.H. (2013) Recognizing public value, Cambrige, MA: Harvard University Press.
Moynihan, D.P., Fernandez, S., Kim, S., LeRoux, K.M., Piotrowski, S.J. and Wright, B.E. and Yang, K. F. (2011) Performance regimes amidst governance complexity, Journal of Public Administration Research and Theory, 21: i141–i155. doi: 10.1093/jopart/muq059.
Provan, K.G. and Kenis, P. (2008) Modes of network governance: structure, management, and effectiveness, Journal of Public Administration Research and Theory, 18(2): 229–52. doi: 10.1093/jopart/mum015.
Shangraw, R.F. and Crow, M.M. (1989) Public administration as a design science, Public Administration Review, 49(2): 153–8. doi: 10.2307/977335.
Simon, H.A. (1996) The sciences of the artificial, Cambridge, MA: MIT Press.
Sorensen, E. and Torfing, J. (2011) Enhancing collaborative innovation in the public sector, Administration and Society, 43(8): 842–68. doi: 10.1177/0095399711418768.
Waardenburg, M., Groenleer, M., de Jong, J. and Bolhaar, H. (2018) Evidence-based prevention of organized crime: assessing a new collaborative approach, Public Administration Review, 78(2): 315–17. doi: 10.1111/puar.12889.
Waardenburg, M., Groenleer, M., de Jong, J. and Keijser, B. (forthcoming) Paradoxes of collaborative governance: investigating the real-life dynamics of multi-agency collaborations using a quasi-experimental action-research approach, Public Management Review. doi: 10.1080/14719037.2019.1599056.
Williamson, B. (2015) Governing methods: policy innovation labs, design and data science in the digital governance of education, Journal of Educational Administration and History, 47(3): 251–71. doi: 10.1080/00220620.2015.1038693.