Introduction

We have structured this book so that the chapters move from simpler to more complex ways of using VR in research. This being said, perhaps the majority of projects using VR to date skew toward the more complicated end of this spectrum, with researchers building bespoke 3D environments for their work. This approach gives maximum flexibility to the design of specific worlds and scenarios that can be used in a variety of ways with participants. Developing custom 3D content requires significant expertise, however, and can be seen as a barrier to researchers beginning to employ VR within their own work. Hopefully, the earlier chapters of this book have demonstrated that one does not have to learn how to code or design 3D objects in order to undertake interesting projects using VR. Nonetheless, our concern in this chapter is to examine why and how we might create this kind of bespoke content.

We are not going to get into the details of programming here. There are a number of step-by-step guides available to get researchers started with coding for VR (for example Murray, 2020), as well as countless YouTube tutorials and blogs with handy hints and tips. Instead, we are going to focus here on approaches to developing original material and the kinds of research this enables. As discussed in Chapter 3, there are opportunities to customise existing materials, which can be a relatively straightforward way into testing specific scenarios. This can be as simple as finding a game that allows users to build specific quests or missions. Some of the commercial social VR platforms allow users to create buildings and spaces, decorating them and even importing photographs, video and 3D objects. At a more involved level, one can create ‘mods’ for existing games. Indeed, part of the reason why there are currently over one thousand community-created mods available on the Steam platform for Half-Life: Alyx is because its developers have created an incredibly flexible and sophisticated toolbox to allow fans to create and share their own variations on the core game.

For building VR content from scratch, the Unity and Unreal games engines have become default tools. Unity seems to be more commonly used by academic researchers, in part because it is a little more user-friendly for amateur programmers, whereas Unreal exchanges greater complexity for flexibility and sophistication. Both of these are free to use for non-commercial purposes and both have ready-made code packages and 3D assets available to import, which take much of the heavy lifting out of creating content. Games engines generally rely on 3D content being brought in from other platforms to integrate into a project, such as designing objects and avatars in Blender or importing landscape and terrain features from geographic information systems mapping software (Sermet and Demir, 2019).

For the beginner, with no coding experience, there is a very steep learning curve and many ‘how-to’ guides can assume knowledge that you simply do not possess. If you have no experience with programming, this can be very challenging and, again, the perception that you need to be able to program in order to undertake research using VR, acts as a barrier to more scholars working in this area. As a result, those researchers wanting to create a customised VR experience, but who have neither the time nor skill to learn basic coding, will need to work with an experienced programmer. If you want to create your own 3D assets rather than relying on ready-made materials, you may need to bring in expertise from the digital arts and animation. This might involve paying a commercial developer or collaborating with a specialist researcher. An alternative approach might be to contact one of the coding clubs organised in many universities where students produce games for fun, or recruit undergraduates studying programs in design or architecture who have experience in 3D modelling. This might offer the opportunity to hire student interns for less-demanding projects. Either way, having a basic understanding of how games engines and 3D design work can be invaluable when commissioning third parties to develop content for your research.

The very large number of studies that have created their own VR content means that we can only scratch the surface here rather than attempt a comprehensive review. Instead, the chapter explores the wider reasons why researchers build custom VR experiences, which can be crudely divided into two overlapping categories: testing different scenarios and creating novel environments. We examine each in turn before reflecting on a case study of a small project we undertook in which participants explored highly simplified reproductions that we created of two urban landscapes: the National Mall in Washington DC and an unbuilt Nazi project for the post-war reconstruction of Berlin.

Scenario testing

VR research in the social sciences has been most commonly undertaken by psychologists. As a result, there has been a great deal of work examining the extent to which participant reactions to situations depicted in VR are similar to those expected in real-world scenarios. This has been important to establish, as it means that VR can be used reliably to examine people’s responses to situations that would otherwise be impractical or dangerous to test. An early example of this kind of work was a study by Pan and Slater (2007), which was undertaken prior to the third-wave of VR using a CAVE¹ system rather than an HMD. They compared the responses of socially anxious and more confident heterosexual men to the approach of an attractive woman at a virtual bar using measurements of EEG, electrodermal activity and post-experiment interviews. In short, they found that confident men responded confidently to the simulated interaction with the female avatar, while those with anxiety issues demonstrated the same signs of stress as if approached by a woman in the real world.

A host of studies since have demonstrated that the response of participants to scenarios tested in VR is a reasonable analogue for studies undertaken in the real world. This allows for otherwise impossibly dangerous scenarios to be explored. A recent study by Baker et al (2020), for example, sought to explore fear response and risk-taking behaviours. They used Unreal Engine to build a mountain environment where stepping on the wrong block of ice would lead to participants falling to their (virtual) death. Trackers were attached to participants’ shoes so that the precise movements of their feet as well as head and hands could be reproduced in the game. They were also able to create a track log of players’ movements – capturing hesitations and unsteadiness – effectively making more data available for subsequent analysis than would have been possible without a custom build. As a result, they were able to use an extreme scenario to induce emotional and physiological responses in their participants and then examine how those with different levels of neuroticism responded. Those with higher levels of neuroticism were shown to demonstrate much more risk-averse behaviours, being less willing to commit to standing on a block of ice that could collapse at any moment.

The fact that VR stimulates the same kind of emotional and affectual responses as people experience in real scenarios has been shown to have tremendous implications for medical and training uses. The kind of ‘conquer your fears’ apps that we discussed in Chapter 2, for example, thus do appear to have some kind of scientific validity. Bentz et al (2021) built their own smartphone-based VR app Easy Heights where they attempted to reproduce the kind of conventional but expensive in-vivo exposure therapy that has been demonstrated to be very effective for people with a fear of heights. Participants were exposed to different virtual scenarios and then subjected to a Behavioural Avoidance Test in the real world where they had to climb a lookout tower. The app was built around 360° images taken using drones at different heights, with audio layered in that matched the height at which the image was captured. The app asked participants to indicate their level of discomfort based on the Subjective Units of Distress ranking, not being allowed to progress to the next stage until they were able to give a low rating for the scene they were witnessing. The control group simply viewed ground-level Google Streetview images through the same HMD and without the test of distress. Participants who had used the app performed better in the subsequent test of climbing the real lookout tower compared with the control group.

The Easy Heights app is not especially technically complex and would have been relatively straightforward to design, commission and build, although it does have a fair degree of polish. What the app demonstrates is that even relatively basic VR experiences can be effective when carefully constructed to serve the needs of a wider research design. We see something similar in Salovaara-Hiltunen et al’s (2019) work creating a simulation training app for healthcare workers. The app was built in Unity and designed for the now obsolete Samsung Gear VR smartphone platform. Although it is graphically quite sophisticated, with realistic renderings of patient avatars, medical equipment and so on, a key reason why the app worked well is that it was prototyped with the input of a series of specialist clinical staff. The app was designed to train medical practitioners in the latest European guidance for resuscitating critically ill patients. The main feedback received by the researchers was not about the quality of information that participants absorbed through the simulation but rather more mundane concerns about how to interact with the HMD.

Salovaara-Hiltunen et al’s app was very much a prototype. Medical simulators in particular require considerable prototyping, user testing and refinement prior to deployment, given that this training might make the difference between a patient living and dying. Even in use cases where someone’s life is not at risk, projects creating original VR content need to build in sufficient time and resources to pilot and refine the app with users to ensure it can deliver the needs of the research design. This is not necessarily about highly sophisticated visuals, however. Creating something graphically complex takes considerable time and skill and may not always be crucial to testing the scenario that is being investigated. The sense of immersion generated by wearing an HMD can often significantly offset the lack of realism from abstract or unsophisticated graphics. As Hupont et al (2015) point out, convincing simulation of movement can be more significant than convincing visuals. In their study, they built a forklift truck simulator and put considerable effort into reproducing the way the machine moves, as well as giving their participants a gaming steering wheel and foot pedals to use rather than a conventional controller. These helped to simulate the feel of driving the vehicle and thus enhance the immersion in the training scenario, even though the graphics used were a little underwhelming.

Reproducing environments

Sometimes a more visually sophisticated experience is important, however, and this in part comes down to the user group that the custom VR content is being created for. For prototyping purposes, or experiments in a lab, it is less of a problem that content is a little rough around the edges. Where content is being designed for use outside a research context, however, considerably more work needs to be undertaken to improve the visuals and functionality. Indeed, there is a good argument for using a specialist external developer when producing public-facing materials, rather than researchers cobbling something together with amateur-level design and programming. A really nice example of this more refined content is in a project undertaken by Ryu et al (2018) designed to help children who were being taken in for surgery. The research team worked with a VR games company to produce a sophisticated experience, with a Pixar-like cartoon feel designed to appeal to children. Participants were taken through a representation of being anaesthetised, so that it was less unfamiliar and frightening when subsequently experiencing this for real. Compared with a sample group who were given the hospital’s conventional paediatric pre-surgery orientation exercise, the VR group reported lower pre-op anxiety and demonstrated greater compliance while being anaesthetised. Beyond simply creating a slick experience, another advantage of working with a commercial developer is in the longevity of a project like this. The hospital can commission updates depending on changing procedures and the developers can easily recompile the app for other platforms as the original technology becomes obsolete.

Longevity can be an issue with materials built for one-off projects, as specialist team members leave or the source code goes astray or becomes unusable. The National Holocaust Memorial Museum in Washington DC, for instance, has been sufficiently concerned by this to archive a number of different virtual reconstructions of wartime concentration camps that have been built over the years to ensure that these are not lost – including Ralph Breker’s centimetre-accurate VR model that was used in a war crimes trial (Cieslak, 2016). Fortunately, in recent years it has become more of a standard procedure within research projects to make source code openly available, so that these materials are not lost and other researchers can check the validity of research findings derived from them (Easterbrook, 2014). Many funders now actively require this, though there can be issues around intellectual property rights to negotiate where external developers are commissioned to produce the software – this is something that ideally needs to be carefully thought about at the grant-writing stage.

Within archaeology there is a long-established tradition of creating reconstructions of what sites may have looked like in different periods of their history. Archaeologists in the 1990s were thus unsurprisingly quick to grasp the potential of using digital 3D modelling within their work. It is a small step from creating a 3D model to bringing that model into VR for an audience to explore as if they were present in the historic landscape. Ch’ng et al’s (2020) model of Sanjiankou, an 800-year-old Yuan dynasty site in Ningbo, is a nice example of just how sophisticated these digital reconstructions have become, with highly realistic models made using photogrammetry and other techniques. The model was turned into a VR experience using Unreal Engine and was used to examine whether younger demographic groups could be made more receptive to consuming heritage experiences in VR as a way to encourage them to visit actual museums and historic sites. The researchers concluded that VR could help to create a more constructivist learning environment, which could allow museum visitors to engage more actively with the different aspects of heritage being presented.

The heritage sector is a good example of how carefully one needs to consider the design of VR experiences when designing for public audiences. A useful review of how this immersive material has been employed in heritage contexts has been undertaken as part of the AHRC-EPSRC-funded Scottish National Heritage Partnership project (Pittock, 2018). Here, the researchers examined both the current state of immersive experiences within Scottish museums and heritage sites and the potential for their wider use. Key findings were that immersive experiences needed to move beyond the gimmicky, to present meaningful educational experiences – indeed, to encourage the type of constructivist learning that Ch’ng et al highlighted. Particularly in heritage contexts, audiences wanted strong storylines and a blending of both virtual and physical content. This idea of integrating original VR content into a physical experience is something we will return to in the case study later. In short, however, it is not enough simply to put a pretty 3D model into an HMD and assume that this will add significant value to a heritage experience.

Thus far we have concentrated on talking about VR experiences that are led by visual materials, with audio playing more of a supporting role. As discussed in Chapter 5, VR offers considerable potential to explore the multisensory, which can be built into projects creating original content. A rather lovely example of this is an attempt to reconstruct the acoustics of King James IV’s Chapel Royal at Linlithgow Palace. A 3D model of the now-ruined chapel was built and used as the basis for acoustic modelling. Andrew Kirkman’s Binchois Consort recorded music that would have been heard in the chapel during the period and this was then acoustically manipulated to create a 3D audio reconstruction of how the music would have sounded in that space. HMD users can thus stand in a model of the chapel and hear the music accurately reproduced. The researchers used a collection of off-the-shelf tools to undertake the modelling, meaning that the same technique could be quickly and easily applied to other spaces and recordings (McAlpine et al, 2021).

The ways in which participants can move around a virtual environment is a key consideration when designing original content, not least because of the relationship between movement and cybersickness in VR. By building a basic game, Christensen et al (2018) were able to test how different control mechanisms for movement shaped a multiplayer VR experience. The code could be tweaked to create three different versions of the same game: desktop, seated VR and ambulatory VR. Players seemed most satisfied with the full-VR state, which integrated their body movements into the scene. It should be noted, however, that these kinds of projects assume an able-bodied participant. Indeed, this is true of most commercial VR games. In screen-based gaming, considerable effort has been made to create controllers that can be customised to allow players with disabilities to build a set of control mechanisms around their particular physical constraints – the Xbox Adaptive Controller being an important example of this (Stark and Sarkar, 2018). For VR, there have been some community-led projects, such as WalkinVR, which remaps the buttons on handheld controllers in order to replace body movement when using Steam VR games.

In order to explore how games might be better adapted for wheelchair users, Gerling et al (2020) built a series of game prototypes to test with disabled participants. This included integrating the GAMEWheels tool, which allows game input to be controlled by a wheelchair mounted on rollers. The GAMEWheels tool proved particularly popular because it very accurately reproduced participants’ own bodily movements, which was seen as giving more agency to players than other prototype designs where button presses triggered pre-programmed automated moves. None of the prototype games were going to win any awards – indeed, this is a good example of where producing a highly sophisticated product was not really necessary for the research design. Nonetheless, the prototypes were effective in allowing different modes of navigation for wheelchair users around the virtual environments to be examined. This rigorous user testing simply would not have been possible without the research team being able to build their own software.

Case study: building urban landscapes

We turn now to consider a case study where we created two very basic VR experiences. There was a dual purpose to this. First, we could undertake a small research project that blended virtual and material experiences of urban space and heritage (Osborne and Jones, 2020). Second, it gave Phil an opportunity to learn the basics of working with Unity. This is part of his wider practice of getting to grips with the fundamentals of a new technique in order to have more meaningful conversations and collaborations with experts in that field (an approach to research discussed further in Jones, 2020).

Germania was a Nazi-era plan for the wholesale reconstruction of Berlin intended to be completed shortly after a successful war against the Allied powers. Designed primarily by Albert Speer with direct input from Hitler, the designs envisaged a 3-mile north-south axis lined with neoclassical buildings. At the northern end was a vast public square and parade ground surrounded by giant buildings, including Hitler’s personal palace and the Volkshalle, a 290m-tall domed building intended for public gatherings and rallies. The Reichstag was retained in the plans, which gives a useful sense of the location and scale of the planned design, given that this building still exists today (Scobie, 1990).

Several amateur historians have created 3D models of the proposed scheme, which can be downloaded for use in projects. We selected one of these that effectively reproduced the scale of the main buildings, although it was not a complete reconstruction as it lined the parade route with generic infill structures. This was then imported into Unity running on a Razer Blade Pro gaming laptop (7th gen Core i7, GTX 1060 graphics card). The research design for this project called for the model to be used by participants outdoors and in public spaces, so it was designed in Unity to be exported to the (now obsolete) Oculus Go standalone headset. The Go is a relatively underpowered HMD and there was a process of trial and error to balance the number of assets and detail used in the VR project against the capability of the device. Ultimately, we stripped out additional 3D assets, including models of the contemporaneous neue Reichskanzlei and Tempelhof airport, which made the software run too slowly on the stand-alone HMD.

One of the advantages of working in Unity is that the major VR hardware developers have produced ready-made code packages that can be imported into projects to save on programming time. We used the locomotion assets provided by Oculus, which have a number of preset tools for VR navigation, including the standard ‘point-and-teleport’. Again, there was a process of trial and error here in setting the teleport distance to give a sense of the scale of the simulated environment, while allowing users to move around a very large model in a reasonable time. We also set the height of the game camera to 1.8m so that users had a human-scale view of the environment, and imported a ready-made sky-box that created a sense of a sunny day with a blue sky and light clouds.

We used the same design approach for the second model, of the National Mall in Washington DC. An abstract model of the wider city was imported to Unity, derived from OpenStreetMap data – effectively plain boxes extruded to the correct height for the buildings. Ready-made tree models were imported and used to line the western end of the National Mall area, while the memorial reflecting pond was reproduced using one of Unity’s standard water tools (Figure 6.1). This tool was not able to correctly produce reflections in stereoscopic VR and so these were turned off. (This problem could have been resolved by buying a higher-quality water asset, but this would have been computationally demanding and unlikely to run properly on the low-powered Oculus Go.)

Part of the Washington DC model as it appears in Unity

Source: Phil Jones and Unity

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Part of the Washington DC model as it appears in Unity

Source: Phil Jones and Unity

• Download Figure
• Download figure as PowerPoint slide

Part of the Washington DC model as it appears in Unity

Source: Phil Jones and Unity

• Download Figure
• Download figure as PowerPoint slide

Neither of these VR experiences would win any awards – indeed, they would not even meet the standard of undergraduate work in a cognate discipline. The point here was to explore principles and potential. Oculus allow users to develop and test their own software on its HMDs by setting the headsets to developer mode, rather than having to go through the challenging process of building software to a quality where it could be approved for formal distribution via the Oculus stores. Ten Oculus Go devices were loaded with the two Unity projects containing the city models. We took these to a conference in Washington DC in April 2019 and used them as part of a day of workshop activities around memory and memorialisation. Participants stood at the foot of the Lincoln memorial and put on the HMDs to run the Washington DC model, which we had set to start in the same location. Participants were able to explore the model while listening to extracts from Martin Luther King’s ‘I have a dream …’ speech, which had been given from the steps of the Lincoln Memorial. This gave participants an opportunity to have a blended experience, blurring the boundaries between the virtual and material, somewhat in line with the recommendations of the Scottish National Heritage Partnership discussed earlier.

The second phase of the intervention was to ask participants, still standing in the National Mall, with a view of the domed Capitol building in the distance, to try the Germania model (Figure 6.2). The architectural historian Barbara Miller Lane (1986) noted that the kinds of neoclassical designs favoured by the National Socialists in Germany were not significantly dissimilar to the state architecture of other nations. Indeed, she particularly singled out Paul Cret’s design for the Federal Reserve Board Building, completed in 1937 and located just on the edge of the National Mall, as something that would not have looked out of place in Hitler’s Berlin. Participants were thus able to reflect on the striking similarities between the triumphal axis and neoclassic architecture of the National Mall and that of Germania, giving an uncomfortable juxtaposition, particularly having just listened to part of a speech calling for racial equality in the United States.

Workshop participants exploring the VR environments while standing in the National Mall

Source: Phil Jones

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Workshop participants exploring the VR environments while standing in the National Mall

Source: Phil Jones

• Download Figure
• Download figure as PowerPoint slide

Workshop participants exploring the VR environments while standing in the National Mall

Source: Phil Jones

• Download Figure
• Download figure as PowerPoint slide

This was only a small test project, and the VR simulations were of an exceedingly rudimentary quality. They were, however, appropriate for the limited aims of a research design examining how we might modify our perception of urban heritage. Creating the VR materials in Unity took a few hours for each project, but this followed on from about a week of getting to grips with how Unity works, how models and code assets can be imported and customised and how the final product could then be compiled and exported for use on a different device. We would not, therefore, want to underplay the difficulty of this, though for those with a technical mindset and enough time, it can be quite a fun exercise. Both Unity and Unreal can be used to do considerably more interesting things than simply letting users navigate a pre-existing 3D model, but this means going beyond simply bolting together existing pieces of code and beginning to learn the nuances of different programming languages. For most researchers, this will mean collaborating with a specialist. Similarly, while much can be done with inexpensive or free assets, for more complex projects consideration may need to be given to bringing in a designer to create realistic-looking 3D models.

Conclusion

Much of the existing research that uses VR is based around constructing custom pieces of software. While we have argued throughout this book that there is much potential for research using existing VR materials, there are situations where the only way to meet the needs of a research design is to create original content. In this chapter, we have considered two main reasons why this might be the case: where one wants to place participants into a specific scenario, and when asking them to explore a particular environment. Of course, in practice, these two can and do overlap in interesting ways. What we have not been able to do here is present a comprehensive survey of all the different types of projects that have been undertaken where researchers have produced their own VR experiences. This is now a very large field of work. Conferences dedicated to VR have proliferated in recent years – some sessions were even held in VR during the COVID-19 pandemic – with findings from hundreds of exciting projects being presented. Some of this material is highly technical, with a great deal of engineering detail that is entirely opaque to the outsider. Nonetheless, much of the work in this area shows just how innovative and dynamic the emerging field of VR research has become, with applications across a whole range of different disciplines.

As with any research project, it is important to start by thinking about what you want to achieve with work using VR. It might be that the research design can function more cheaply and easily by reusing or customising an existing piece of software. If original content is required, it does not necessarily need to be particularly graphically refined. A surprising number of pre-built 3D objects are available for free or at low cost, from plants and flowers through to vehicles and machinery. These can give a relatively sophisticated look and feel to a VR project without the cost of bringing in a specialist designer. For projects that test user responses to a particular situation, a graphically advanced environment may be surplus to requirements. In other applications, a more visually convincing design may well be needed. This is particularly the case where VR experiences are designed to be public facing rather than for simply testing an idea in a lab setting. Indeed, where a public audience is sought, much more thought needs to be given to how the VR content fits into a wider set of user experiences; blending the virtual and material, for example, can help to drive more active engagement.

Whatever the intended purpose of the project, building in enough time for prototyping is crucial. Working with a pilot group not only smooths out bugs, it can also highlight design issues that form a barrier to proper engagement by participants. Beyond mere user testing, co-design can be an incredibly useful approach, as we saw with the School days app discussed in Chapter 4. User groups are often much better able to identify their own needs and interests than a research team, which can lead to more effective VR experiences. Building rapid prototyping into the time frame (and budget!) of a project can take research in unexpected and productive directions. Co-design can also move us beyond the power-centric model of research subjects having no role in the design and outcome of projects.

We would encourage researchers to have a play with programming tools such as Unity, but not to feel that becoming a coder is an essential barrier to overcome. Similarly, getting a sense of what can be done with 3D-modelling tools such as Blender can be interesting, but will not turn researchers into skilled designers overnight. Trying these tools might spark a genuine enthusiasm for design and programming among some scholars, but, perhaps more prosaically, it opens up avenues for discussion with collaborators and a better sense of what might be possible in a research project. Thus, while we are keen to emphasise that one does not have to a technical expert to undertake research with VR, engaging with some of the possibilities offered by creating custom content can open the door to an incredibly wide range of potential projects.