Abstract
Climate change and population growth will increase vulnerability to natural and human-made disasters or pandemics. Longitudinal research studies may be adversely impacted by a lack of access to study resources, inability to travel around the urban environment, reluctance of sample members to attend appointments, sample members moving residence and potentially also the destruction of research facilities. One of the key advantages of longitudinal research is the ability to assess associations between exposures and outcomes by limiting the influence of sample selection bias. However, ensuring the validity and reliability of findings in longitudinal research requires the recruitment and retention of respondents who are willing and able to be repeatedly assessed over an extended period of time. This study examined recruitment and retention strategies of 11 longitudinal cohort studies operating during the Christchurch, New Zealand earthquake sequence which began in September 2010, including staff perceptions of the major impediments to study operations during/after the earthquakes and respondents’ barriers to participation. Successful strategies to assist recruitment and retention after a natural disaster are discussed. With the current COVID-19 pandemic, longitudinal studies are potentially encountering some of the issues highlighted in this paper including: closure of facilities, restricted movement of research staff and sample members, and reluctance of sample members to attend appointments. It is possible that suggestions in this paper may be implemented so that longitudinal studies can protect the operation of their research programmes.
Key messages
Recruitment and retention of longitudinal study participants is challenging following a natural disaster.
The long-lasting, global effects of the COVID-19 pandemic will increase this problem.
Longitudinal study researchers should develop protocols to support retention before a disaster occurs.
Researchers need to be pragmatic and flexible in the design and implementation of their studies.
Introduction
As climates change and populations increase, natural and human-made disasters and disease will increase in frequency and impacts. Longitudinal research studies represent a powerful tool by which researchers, service providers and policy makers can quantify the public health impacts and community needs of these events. Longitudinal studies are distinguished from other research designs by their repeated observations of the same participants over time. Participants are recruited based on measurable or identifiable characteristics such as their area of residence or year of birth to form cohorts with similar initial exposures who are observed at regular intervals (Coggon et al, 2009; Caruana et al, 2015). There are a number of advantages of longitudinal designs for disaster research. These include the ability to: limit recall bias through prospective assessment; assess causality, recovery and resilience by observing before and after events using cohort members as their own controls; and avoid biases in recruitment attributable to disaster exposure.
While the maximum benefits of longitudinal research are realised where multiple observations prior to the disaster and following the disaster are available, the inherently unpredictable nature of disasters mean that opportunities for longitudinal research into impacts and recovery from disasters are rare (Norris et al, 2006; Neria et al, 2009). Indeed, longitudinal study designs in any settings are expensive and time-consuming, requiring careful planning and skilled staff (Neria et al, 2009; Caruana et al, 2015). To ensure the validity and reliability of findings in longitudinal research, the use of appropriate methods is essential (Wright et al, 1995; Ribisl et al, 1996; Mason, 1999; Gul and Ali, 2010; Satherley et al, 2015). One way to reduce threats to study-validity is to effectively and efficiently recruit and retain cohort members, as attrition can produce biased results (Ribisl et al, 1996; Mason, 1999; Nohr et al, 2006; Norris et al, 2006; Gul and Ali, 2010; Forcey et al, 2014; D.B. Friedman et al, 2015; L.M. Friedman et al, 2015; Fry et al, 2017; Davison et al, 2017; Bartlett et al, 2018; Howcutt et al, 2018). However, while several longitudinal studies have reported impacts of disasters on their participant cohorts (La Greca et al, 1996; Warheit et al, 1996; Norris et al, 2006; Proctor et al, 2007; McGonagle et al, 2008; Lankenau et al, 2010; Satherley et al, 2015; Byrd-Bredbenner et al, 2017), few have addressed specific recruitment, retention or attrition issues associated with these impacts and their potential effect on results.
Methods of retention
A number of papers have been published which report recruitment, tracking and retention strategies of participants in longitudinal studies (Ribisl et al, 1996; Lee et al, 2000; Scott, 2004; Scott et al, 2006; Davison et al, 2017). For recruitment, these methods have included, for example: the selection of appropriate recruitment venues/methods, emphasis of the benefits of the research to the participants and the community, and highlighting a commitment to participant confidentiality. For retention, methods have included for example: collection of comprehensive contact information for each respondent (such as secondary contacts of family members and social media contact details) to simplify tracking and tracing; allowing time for the study team to develop rapport with the participant; and writing regular newsletters regarding study progress to encourage notification of address changes.
While these strategies are effective, their practicality and utility in the face of significant disruption is unknown. This is because relatively few studies report recruitment and retention techniques of cohort members displaced and traumatised by natural disaster. In addition, there is limited literature that addresses the challenges that longitudinal studies encounter regarding tracking displaced cohort members, loss of study facilities and infrastructure, and impacts on research staff after a natural disaster (Ribisl et al, 1996; Matthieu and Ivanoff, 2006). Natural disasters can have a serious impact on population mobility due to the loss of housing, employment, educational institutions and infrastructure (La Greca et al, 1996; Norris et al, 2006; Lankenau et al, 2010; Gray and Mueller, 2012; Hallegatte, 2016). By extension, natural disasters may have serious consequences for participant recruitment and retention (Norris et al, 2006). A number of longitudinal studies have published papers following a natural disaster affecting their cohort members (La Greca et al, 1996; Warheit et al, 1996; Norris et al, 2006; Proctor et al, 2007; McGonagle et al, 2008; Lankenau et al, 2010; Satherley et al, 2015; Byrd-Bredbenner et al, 2017), however only two studies (McGonagle et al, 2008; Lankenau et al, 2010) address specific recruitment, retention or attrition issues following disruption to their studies by Hurricane Katrina in 2005.
Lankenau et al (2010) reported on the longitudinal recruitment and retention of young substance users across three US cities: New York, Los Angeles and New Orleans. In the New Orleans study location, the hurricane halted enrolment progress for two months. In addition, Hurricane Katrina also impacted the 34th wave of the Panel Study of Income Dynamics (PSID) a genealogical household panel survey of a nationally representative sample of US families (McGonagle et al, 2008). McGonagle et al (2008) describe the steps taken to locate the families residing in areas impacted by Hurricane Katrina, to support retention in the 2007 wave. Initially, postcards or newsletters were sent to elicit address verification or update. If there was no response, researchers would attempt to locate the family by telephone contact with alternative contact person provided by participants, or by gathering publicly listed details on websites. If this was unsuccessful, visits to home or other addresses (such as neighbours) were conducted. Both studies reported that these efforts were successful in retaining cohort members.
Current study
Considering a likely increase in natural disasters due to climate change (Hallegatte, 2016), and in light of the importance of longitudinal studies in providing insight into the impacts of disasters or infectious diseases such as the current worldwide COVID-19 pandemic, it is important that studies develop protocols to support recruitment and retention prior to a disaster in their area. Specifically, studies need to consider how to keep their longitudinal cohort operating in adverse circumstances while maximising recruitment and minimising sample attrition
One such natural disaster was the Canterbury earthquake sequence. In September 2010, the Canterbury region of New Zealand experienced the first of a number of earthquakes. This was a 7.1 magnitude earthquake originating 40 kilometres west of the city of Christchurch. Christchurch is New Zealand’s second largest city and had nearly 390,000 residents at the time. Due to the epicentre of the earthquake being in a rural location and the earthquake taking place while most people were in bed, the resulting injuries and damage were minimal. Unfortunately, this earthquake triggered a large number of aftershocks, with the most devastating occurring five months later, on 22 February 2011. This was magnitude 6.3, and killed 185 people while injuring several thousand (Ardagh et al, 2012; McSaveney, 2017). Given the catastrophic nature of this and the subsequent aftershock series (McSaveney, 2017), this paper will refer to the event as the ‘February earthquake’ in keeping with previous literature (Smith et al, 2017; Bell et al, 2018).
A state of emergency was declared the following day and the central business district (CBD) including many residential homes were cordoned off (Ardagh et al, 2012; Ministry for Culture and Heritage, 2016). Of particular note, was the extensive damage to the iconic Christ Church Cathedral which is situated within the CBD (Christ Church Cathedral Reinstatement Project, 2021). The cordon remained, at least in part, until June 2013 (Ministry for Culture and Heritage, 2016). Overall, the region experienced thousands of aftershocks, with over 4,000 of magnitude 3 or greater occurring within the first two years after the September 2010 earthquake (Spittlehouse et al, 2014). Significant aftershocks were still occurring even into 2016 (Geonet, 2016). It has been estimated that nearly 170,000 homes were damaged and the New Zealand Government bought and demolished over 7,000 homes that were located in areas deemed unsuitable for residential housing. The damage and displacement initiated considerable disruption and population movement (Spittlehouse et al, 2014; Mitchell, 2015; Potter et al, 2015).
A number of longitudinal cohort studies were operating in the region prior to the earthquake sequence and a number of research projects were subsequently initiated to examine the impact of the disaster on the Canterbury population. The studies that were operating prior to the earthquake sequence, or shortly after the earthquake sequence began included:
Adult Speciality Services Earthquake Trauma Treatment study (ASSETT)
Earthquake Resilient Controls (ERC)
Canterbury Health, Ageing and Lifecourse study (CHALICE)
Christchurch Health and Development Study (CHDS)
Health and Wellbeing Study (HWS)
Methadone in Pregnancy Study (MIPS)
Micronutrient Study (MS)
Canterbury Preterm Study (CPS)
New Zealand Health, Work and Retirement study (NZHWR)
Relationship Quality Study (RQS)
New Zealand, 1986 Very Low Birthweight (VLBW) study
These studies represent a broad range of longitudinal study designs who recruited samples prior to the earthquake sequence including: birth cohorts (CHDS, MIPS, CPS, VLBW), national representative samples (NZHWR), community samples (CHALICE, HWS), and longitudinal study designs of samples recruited after the earthquake sequence began including: patient-referred cohorts (ASSETT, ERC), randomised controlled trials (MS) and community samples (RQS). More detailed characteristics of the study designs, population, number of post-earthquake assessments, proportion of sample members who experienced the earthquakes, and retention rates are shown in Table 1 and Online Resource 1.
Characteristics and retention rates of included studies
| Study | Design | Population | Number of assessment waves conducted post-quakes (years of assessment) |
Experienced quakes or resident in Canterbury during quake sequence % (n) |
Retention rate from baseline to latest collection % (n) |
|---|---|---|---|---|---|
| ASSETT | Intervention study of a referred patient cohort | People suffering psychological distress following an earthquake | 3 (2011–15) |
100 (184) | 25.5 (47/184) |
| ERC | Recruited control group for treatment/patient groups | Parallel control group to ASSETT | 2 (2012–13) |
100 (101) | 59.4 (60/101) |
| CHALICE | Multidisciplinary prospective random cohort | Canterbury District Health Board area residents aged 49–51 years | 4 (Wave 1: 2010–13; Wave 2–4: 2011–16) |
99 (399)1 | 62.8 (252/401)2 |
| CHDS | Prospective longitudinal birth cohort | Babies born in the Christchurch urban region over a four-month period during 1977 | 2 (2012–17) |
51.4 (495) age 35; 45.5 (411) age 40 Estimated from completed earthquake interviews |
78.7 (962/1223) Wave 23: 2012; 74.1 (904/1220) Wave 24: 2017 Surviving cohort members |
| HWS | Longitudinal community sample | Christchurch and Selwyn District residents | 3 (2010–12) |
100 (354) | 40.7 (144/354) |
| MIPS | Prospective longitudinal regionally representative cohort | A group of Methadone-exposed children and comparison group of children born in the same time-period 2003–08 | 2 (September 2010–December 2013); (March 2013–September 2018) |
4.5 year 92.2 (178); 9 year 84.8 (156) Estimated from address data: |
4.5 year: 89 (89) Methadone-exposed; 94 (99) Comparison group 9 year: 85 (85) Methadone-exposed; 90 (99) Comparison group |
| MS | Randomised controlled trial | Adults who were experiencing acute stress following the earthquake | 2 (2011–12) |
100 (116) | 73% |
| CPS | Prospective longitudinal regional birth cohort | Consecutively born very preterm infants admitted to Christchurch Women’s Hospital Level 3 Neonatal Intensive Care Unit July 1998–January 2001 | 1 (October 2010–May 2013) |
84.5 (180) Estimated from address data |
Very preterm group: 97.2%; Comparison group: 96.5% |
| NZHWR | National longitudinal study of a representative sample | New Zealand adults aged 55 to 70 years | 2 (2010, 2012) |
90% Estimated |
86.2 (423/491) |
| RQS | Longitudinal community survey | Heterosexual couples | 4 (2011–13) |
100 (99 couples) | 75.8 (75/99) |
| New Zealand 1986 VLBW | Data audit; prospective longitudinal national birth cohort with recruited control group | Very low birth weight infants, born in 1986, who survived to discharge | 1 (2013–16) |
10% Estimated from address data |
77% of survivors (250/323) |
Notes:
Estimated as earthquake sequence began after the assessment of the first five participants.
Three participants died during follow-up.
As few publications exist on the practicality and utility of recommended recruitment and retention techniques following a natural disaster, these studies provide an opportunity to examine recruitment, participation rates and retention strategies of sample members living in the Canterbury region during the earthquake sequence 2010 to 2016. The aim of this study is to provide insights and strategies that can be implemented by longitudinal studies following a natural disaster. Specifically, the aim of the study is to:
alert research managers and those designing new longitudinal studies to be aware of potential issues of running a study in the aftermath of a disaster; and
highlight some of the potentially important strategies that may be the most impactful for recruiting and retaining cohort members during this time.
Methods
Participants and procedure
Studies were identified through personal contact, referral and literature searches. Eligibility criteria were that the study was longitudinal and contained sample members who had experienced at least some of the Canterbury earthquake sequence. Principal investigators of 12 eligible longitudinal studies were contacted via email and invited to participate in the study.
The questionnaire was emailed to study principal investigators / research staff for completion. All information was gathered with the consent of study principal investigators. Only the 11 studies that returned the completed questionnaire were included in the analysis.
Study location, onset year, sample sizes, sex and age of the samples, Institutional Review Board approval information and a brief description of the studies are reported in Online Resource 1.1
Measure
The questionnaire was developed and refined by two staff with expertise in the recruitment and retention of cohort members in longitudinal studies (GM and JB). The questionnaire followed a priori themes established through staff experience of the earthquake sequence and a review of recruitment and retention strategies literature.
The measure consisted of a short answer questionnaire, divided into two parts:
Part 1 included questions about the study design, inclusion criteria, recruitment strategies, sample number at baseline, the number of assessment waves undertaken after the beginning of the earthquake sequence, assessment methods, retention rates and the characteristics of the sample (gender, age and proportion of participants resident in Canterbury at the beginning of the earthquake sequence).
Part 2 included questions around the contact methods that worked best at getting assessments completed. These questions were: the best methods to track cohort members who had changed address and were difficult to find; perceptions of the biggest hurdles facing research staff after the earthquakes; and research staff perceptions of the challenges experienced by cohort members after the earthquakes that adversely affected sample recruitment and/or retention.
Information from the questionnaires was collated and tabulated to summarise each study’s characteristics, recruitment strategies and retention rates.
Results
Research staff from 11 longitudinal studies agreed to participate and supplied their completed questionnaires. Of the 11 studies, eight studies were running prior to the earthquake sequence and three began following the February earthquake. Studies were based at the University of Otago, Christchurch (n = 5), the University of Canterbury (n = 5), and Massey University (n = 1). Further information on the included studies can be found in Table 1 and Online Resource 1.
While all studies were longitudinal, they were heterogeneous, covering diverse populations from very low birthweight infants to the elderly. Study designs ranged from birth cohorts through randomised controlled trials to intervention studies of referred patients. The longest-running cohort began in 1977, while the most recent studies began after the February earthquake. Retention rates also varied from 25% to over 90%. Most studies did not report having a formal contact protocol. Recruitment strategies also varied with some studies using the publicly available electoral rolls to source participants, while others used advertising or approach by research staff in local hospitals. Sample members were not able to be traced through national government databases and incentives to participate were limited to a small koha (expression of gratitude in the form of a donation/gift) which were approved by the relevant ethics committee. Data was collected through a variety of mediums including paper-based questionnaires, Skype interviews, online assessments and MRI scans. Table 2 summarises strategies used by research staff to increase sample recruitment and retention, and the barriers they had to overcome in the post-earthquake environment. Despite the heterogeneity of the studies involved, the experience of the earthquake sequence resulted in specific challenges. These challenges are elaborated below from the research staff perspective.
Summary of recruitment and retention strategies used by the longitudinal studies
| Strategy | ASSETT | ERC | CHALICE | CPS | CHDS | HWS | MIPS | MS | NZHWR | RQS | VLBW |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Word of mouth | ✓ | ✓ | ✓ | ||||||||
| Social media | ✓ | ✓ | ✓ | ✓ | ✓ | ||||||
| Advertisements | ✓ | ✓ | ✓ | ✓ | |||||||
| Electoral roll1 | ✓ | ✓ | ✓ | ✓ | |||||||
| Newsletter | ✓ | ✓ | |||||||||
| Notices on community bulletin boards | ✓ | ||||||||||
| Telephone/SMS | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Post | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | |||
| ✓ | ✓ | ✓ | ✓ | ✓ | |||||||
| Home visitation | ✓ | ✓ | ✓ | ✓ | ✓ | ||||||
| Online contact/questionnaires | ✓ | ✓ | ✓ | ✓ | |||||||
| Contact protocol | ✓ | ✓ | ✓ | ✓ | |||||||
| Flexible assessment locations/methods | ✓ | ✓ | ✓ | ✓ | ✓ | ||||||
| Provided transport/parking | ✓ | ✓ | ✓ | ✓ | |||||||
| Contact-tracing | ✓ | ✓ | ✓ | ✓ | ✓ | ||||||
| Reminders | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| Number of strategies | 7 | 7 | 10 | 9 | 6 | 2 | 10 | 6 | 5 | 5 | 9 |
Note:
1 Electoral roll was used to select participants, but also to try to find updated contact details
Research staff perceptions of the best methods to contact sample members
Telephone/SMS
Most studies reported that their first attempt to contact a sample member was by telephone/SMS. Some MIPS sample members would not answer mobile phone calls from unknown numbers; they often did not have funds on their accounts to call research staff back. To overcome this problem, study staff would text the participant stating the call was from MIPS, why staff were trying to contact them and that they would call again in five minutes. Staff that did this often found that their mobile phone calls were answered and appointments were scheduled.
Earthquake disruption or the unavailability of telephone numbers were the main reasons that study staff would use another contact method. For example, MS only used telephone contact at the beginning of their study as, following the disruption of the earthquakes, staff changed to online and email contact through the study website. Telephone numbers were often not available if sample members were recruited from the electoral rolls. CHALICE and MIPS study staff reported that if a telephone number could not be found or the sample member could not be contacted by telephone, a staff member would conduct a home visit.
Email was not a preferred form of initial contact. Only MS reported using email as a preferred contact after the earthquakes. In some cases, email addresses were not collected during the identification of sample members, and were only obtained at the first assessment. However, once rapport had been established, email was useful for research staff (MIPS; CPS) to send follow-up reminders of scheduled appointments.
Post
Two studies (NZHWR; HWS) reported that they relied on postal methods in the first instance and only used telephone contact if the first postal contact was returned to sender. Despite the earthquake disruption, CHALICE study continued to rely on posted invitation letters as the electoral rolls information gave postal addresses but not telephone numbers. This strategy was effective, as people tended to continue checking their mailboxes even if the property had been abandoned. In cases where the person was no longer at the address provided at baseline, there was always an undelivered address stamped on to the envelope so that the questionnaire could be returned to the study.
Social media
Social media is now a common way to contact sample members for either recruitment or follow-up. Results showed that approximately half of the studies surveyed used some form of social media to contact sample members. For example, the VLBW study found that Facebook Messenger was an effective way to contact younger sample members, while online recruitment and follow-up was initiated by MS after the earthquakes.
Follow-up after appointment scheduled
Many studies considered that contact after an appointment was scheduled was very important to secure the completion of assessments. Five studies (ASSETT; CHALICE; MIPS; CPS; VLBW) sent out letters (in some cases including map and photo of the research building) either by post or email after contact was made. As many sample members were not familiar with the best way to get to their appointments due to road closures, this strategy was particularly helpful. In these studies, at least one SMS reminder was sent or phone call was made prior to the appointment and if there was no response, subsequent phone calls were made to follow up. The phone call made by CHALICE staff also served to remind sample members that they needed to fast prior to their appointment.
Home visits
Five studies (CHALICE; MIPS; CPS; CHDS; VLBW) reported that they conducted home visits if contact by telephone or post had been unsuccessful. The CHALICE staff reported that home visits were especially effective on Saturday or Sunday mornings. If houses were abandoned due to earthquake damage, research staff would leave a contact slip in their mailbox as many people were still checking their mailboxes.
Tracking and tracing strategies
Contact-tracing
At enrolment and follow-up, five studies (CHALICE; CHDS; MIPS; CPS; VLBW) collected comprehensive information on additional contacts – people the sample members would be happy for research staff to contact if the sample member’s information had become out of date. These alternative contact people were an invaluable resource for the study staff when they were having difficulty tracking sample members. Research staff (MIPS; CHDS) found that it was important to retain the mobile phone numbers of sample members, as occasionally the phones were given to a family member or acquaintance.
As the MIPS sample members were young children, research staff found that often the best secondary contacts were the grandparents. Grandparents tended to have stable home addresses and have their telephone numbers listed in the publicly available telephone directories. In contrast, parents of MIPS children moved frequently and changed their mobile phone numbers often.
Children in CPS were school-aged during the assessments at the time of the earthquake sequence. Research staff had a record of the school the child attended from the previous assessment wave and on one or two occasions were able to leave a letter for the caregiver which the school passed on.
Public records
In cases where contact by letter or telephone had not been successful, VLBW and CHALICE staff checked the newly published electoral rolls each year for potential participants’ latest addresses. Unfortunately, this process tended to be time consuming and not particularly effective.
Perceptions of the biggest hurdles facing study staff after the earthquakes
Building closures
Studies that were based in the Canterbury region encountered numerous impediments to their research. One of the main issues was that, due to ongoing aftershocks (Spittlehouse et al, 2014; Geonet, 2016), all Canterbury-based research facilities and buildings were frequently closed, lasting between one day and several months. For example, the building that housed CHALICE was closed permanently and demolished. However, prior to being demolished the building had been reoccupied.
Staff had to adopt a very flexible approach to work space; including at the kitchen table or in cars. The building closures led to many other unforeseen problems including lack of access to patient records, treatment manuals and contact information. To ensure patient privacy and security of information, clinicians working for ASSETT, ERC and MS had to obtain resources such as treatment rooms, office equipment and lockable filing cabinets from other District Health Board services. As previously noted, studies that required physical interaction with cohort members to gain the data needed from each assessment, were most impacted by building closures. The CHALICE, ASSETT, ERC and MS studies required physical measurements to be recorded, face-to-face counselling or specialised nutritional supplements supplied to their cohort members.
Transport around city for staff and clients
Research staff reported that due to the destruction and disruption they felt disoriented when navigating around the city. Many roads were closed or damaged and public transport services were constantly changing, making it difficult for people to attend appointments. Due to a lack of parking (the main hospital parking building in the CBD was closed and then demolished), patients often missed their appointments. To enable attendance, research staff (CPS; MIPS; ASSETT, ERC; VLBW) provided transport to and from appointments, if requested. The CHALICE study provided free and guaranteed parking spaces. This was an important strategy to help research staff retain the sample members in the study. The MIPS and CHALICE reported that this was budgeted for prior to the earthquakes.
Cleaning up of office spaces
Once staff were allowed back into their buildings, the offices needed cleaning and clearing. Most research staff reported that filing cabinets which had not been secured to the wall had fallen over, books and materials stored on bookshelves had fallen to the floor, and ceiling panels fell down into the office space. The clean-up took valuable time away from study operations and ongoing problems with working in offices that felt unsafe was stressful for staff.
Damage to equipment
Some recording equipment used by the CPS and MIPS was damaged and needed to be replaced. Unfortunately, this delayed assessments. This issue had a flow-on effect as assessments could not be completed without it.
Information technology problems
Following building closures, many staff could not access necessary electronic information. When remote computer access was enabled by both the University of Otago and the University of Canterbury IT departments, it was noted as being particularly helpful.
Supporting traumatised staff
Clinicians found that some auxiliary staff such as receptionists and administrators working for ASSETT were vicariously traumatised when dealing with severely affected patients or their information. In this instance it was important for clinicians to remember that their support staff may not have had any training on coping with traumatic patient information. In addition, some ASSETT staff knew victims of the earthquake who had died in collapsed buildings. Some auxiliary staff were offered supportive supervision that is normally only offered to clinical staff.
Further, many Christchurch-based staff were also dealing with their own earthquake-related housing issues, from having to change where they were living to having to leave their homes permanently. Some staff who worked in multistorey buildings found the continuing aftershocks distressing.
Staff retention
A number of staff (CPS) resigned to take up positions outside of Christchurch. The lost work force either needed to be replaced, or remaining staff had to take on additional duties.
Staff perceptions of the biggest hurdles for sample member’s post-earthquakes
Fear of attending appointments
Studies that were based in the CBD had difficulty getting their patients to attend appointments. This was because the patients feared going into the CBD or being inside multistorey buildings due to the building collapses on 22 February 2011 that killed 185 people. Anecdotally, ASSETT clinicians noted that it was the fear of the location of the appointments not the content of the discussion at the appointment that was causing the reluctance to attend. To increase attendance, some appointments were scheduled at a single-storey building away from the CBD. Further, many of the VLBW sample members lived outside of Christchurch and they also did not want to come to the city for assessments as they believed it was too dangerous due to the perceived risk of being in the CBD during an aftershock.
Sample members with physical health problems
Some ASSETT, ERC and VLBW participants had physical health problems (prior to and as a consequence of the earthquakes) that limited their accessibility to study facilities. Research staff needed to take these needs into account when planning assessments.
Financial problems
Overall, some assessments that required physical attendance at appointments (ASSETT; ERC; VLBW; CHALICE) were problematic for participants. After the earthquakes, some businesses closed or were left severely financially disadvantaged, which made their employees worried about taking time off work for assessment/treatment. Due to the lack of housing supply, rent increases made finding appropriate and affordable accommodation very difficult for some families. These additional stresses made some sample members unwilling to continue their participation.
Damaged homes
Some sample members had to abandon their homes immediately following the February earthquake. Those whose homes were still able to be occupied also experienced ongoing problems with electricity, water, sewerage and access to the internet. Houses which were located in the Residential Red Zone had their homes bought by the government and demolished (Land Information New Zealand, 2015; Potter et al, 2015; Greater Christchurch Group, 2019). Many sample members had to reside in temporary housing while their houses were repaired, or they relocated outside of Christchurch. If homes were too badly damaged, CHDS staff would arrange for interviews to take place in other locations such as cafés that were convenient for the sample members. Most research staff reported that they did not collect data on damage to the sample members’ homes or number of relocations.
Overall, the retention rates of the studies, particularly those that had collected data prior to and after the beginning of the earthquake sequence, were admirable. Online Resource 2, Figure 1 shows the summarised retention rates for the cohorts for each assessment wave. Only one study of a community sample (HWS) reported retention rates below 70%. Most studies’ retention rates remained stable; the sharp drop in retention rates for the VLBW cohort may be explained by the cohort having very little contact with the research team over their lives, as only two assessments had been conducted at age 7–8 years and age 26–30 years.
Discussion
This study gathered data from 11 longitudinal studies who had sample members living in the Canterbury region during the earthquake sequence which began in 2010. The Canterbury earthquakes caused great disruption (Ardagh et al, 2012; Potter et al, 2015; McSaveney, 2017), not only for the sample members, but also for the research staff living in the area. The aim of the study was to alert research managers as to the importance of planning for unexpected events such as natural disasters and also to report recruitment and retention strategies researchers used during this time. This research is unique; no previous study has reported recruitment and retention strategies among a group of longitudinal cohorts that have sample members who experienced a natural disaster.
Although all studies were longitudinal, they were heterogeneous in nature; using diverse designs, populations and assessment methods. This was reflected in the range of recruitment and retention strategies study staff used. A number of recruitment and retention strategies were reported by study staff who believed these tactics helped them to keep their studies operating.
Appropriately, recruitment methods used reflected the population that was being targeted. For example, electoral rolls recruitment for the older people in CHALICE and NZHWR; community and social network recruitment advertising for younger people in MS and VLBW; and word of mouth for recruitment of earthquake traumatised people in ASSETT. Most studies used a combination of telephone, SMS or post to contact participants. If these contact methods did not work, some studies had the ability to conduct home visits to track and trace sample members.
Retention rates varied from just over 25% to over 90%. Again, this was probably a reflection of the population from which the sample members were drawn, indicating a combination of barriers to participation and investment by sample members in study outcomes. The lowest retention rates were associated with studies of traumatised patients; while the highest were associated with studies of children.
Key recruitment and retention issues and potential solutions
In a systematic review and meta-analysis Teague et al (2018) found that reducing barriers to participation was the most effective method to retain sample members. This study also found that to retain sample members, research staff also had to reduce barriers to participation. In most cases studies were flexible and offered different ways in which to interact with and assess sample members. For example, ASSETT had to find alternative treatment spaces outdoors for traumatised patients who refused to enter CBD multistorey office buildings; CHDS cohort members whose homes were badly damaged consented to being interviewed at workplaces and cafés; MS had to change their assessments to online methods. To further reduce participation barriers, four studies provided transportation to and from assessments or guaranteed free parking spaces, if required.
Counter-intuitively Teague et al (2018) found that studies that sent out reminder notices lost significantly more cohort members, although the mechanism for this was unclear. In contrast, the current study found that follow-up after scheduling appointments with letters, SMS or telephone call was important. However, it should be noted that these follow-ups included helpful information about the upcoming assessments including maps showing the location of the assessment building. This was supported with the provision of transportation.
Teague et al (2018) also found that using community-building or contact-tracing did not increase retention rates. The current study did not specifically ask if studies used community-building strategies (such as newsletters, branding and merchandise). Only two studies (MIPS; CPS) reported sending out newsletters from which it was believed that sample members would update their address details. This may be an effective strategy for these studies because the sample members were children; parents may be more invested in enabling their children to participate. No study mentioned having logos, merchandise or other community-building strategies as being effective for retaining their samples.
Contact-tracing was used successfully by research staff. Contact-tracing is the use of alternative contacts, given by the sample member, such as family or friends. It can again be seen that positive results from contact-tracing were those studies in which the sample members were children. In one case, MIPS reported that contacting the grandparents of the children was a very effective strategy.
Only four studies (ASSETT; ERC; CHALICE; NZHWR) had a formal contact protocol. A contact protocol lists the number and type of contacts sample members are to receive from the study staff. Once the research staff have exhausted the methods of contact on the protocol, the sample member can be deemed as lost to contact. Two studies that did not have a formal contact protocol (MIPS; CPS) did record all contacts made with sample members; including the number of telephone, SMS, emails and letters. Strategies for tracing and contacting sample members that were difficult to retain were discussed at regular team meetings.
This research also identified a number of issues that should be considered during the design of new studies. If thought is given to the potential for a disaster occurring during the lifetime of the longitudinal study, then research staff will have guidelines and procedures to use during the events. This could include:
Staff retention: staff may leave their roles and relocate away from the disaster area. Research managers need to ensure that no one staff member is indispensable to the project.
Assessment methods: in the event that buildings cannot be accessed, it is important that a pragmatic approach is used and assessment methods are able to be changed. This could include moving from a paper-based questionnaire to online or emailed assessments.
Checks on the welfare of sample members: after significant aftershocks, research staff should, where possible, conduct welfare checks on their sample members. This is particularly important for vulnerable sample members because social issues and barriers to participation can be identified, connections with relevant services made and contact details can be updated.
Transportation: after the earthquakes, public transport was halted and many roads were impassable. Road works, detours and parking problems meant that sample members could not attend, were late or missed appointments. Therefore, consideration needs to be made of how sample members can travel to and from appointments. Budgets need to allow for research staff to provide transport or free guaranteed parking spaces.
Research institutions: institutions housing research studies need to prioritise the reopening of facilities to minimise disruption. It is important for the institution to find alternative assessment locations in the meantime that are safe for staff and participants. Regular updates from computer/IT services regarding remote access of records is needed to minimise delays getting important information to research staff. It is also important to ensure that staff have provision to take time to move house, arrange repairs to their homes or access counselling.
Strengths and limitations
This study is unique and possibly the first of its kind. Few studies examine recruitment and retention after a natural disaster, and none were identified that examine a group of longitudinal studies whose participants experienced the same natural disaster. Eleven of the 12 studies that met the eligibility criteria were recruited into the study. The major strength of this study was to expose the difficulties that research staff and sample members encounter after a natural disaster. This study has also identified some notable weaknesses in prior study planning; no discernible guidelines existed for continuing a study in which essential resources were suddenly unavailable.
However, it is important to address limitations of this research. Overall, it was impossible to compare the usefulness of the recruitment and retention strategies across the group of studies due to the diverse populations and methods used, with the exception of research staff reporting which techniques were useful. Further, research staff were not specifically asked about what strategies did not work for recruiting or retaining their sample members. Some additional limitations included that: it was also unclear whether loss to follow-up was due specifically to earthquake issues; very few studies recorded the number of contacts that were made to each participant; and only one study recorded housing problems and residential movements of participants due to the earthquake damage.
Concluding remarks
In conclusion, recruitment and retention of study participants in longitudinal cohorts was challenging following the Canterbury earthquake sequence that began in 2010. However, in general, the retention rates of sample members was admirable under the circumstances. Overall, this study highlights post-disaster issues that may not have been considered previously by researchers when designing their longitudinal studies. As it is likely that there will be an increase in natural disasters due to climate change (Hallegatte, 2016) it is important for studies to develop protocols to support retention by maximising recruitment and minimising attrition prior to a disaster occurring. Currently, the COVID-19 pandemic has most likely highlighted this issue for many researchers and academics (Atkeson, 2020; Lau et al, 2020; World Health Organization, 2020). Social distancing and lockdowns have restricted movement and interactions of both research staff and their participants in a similar fashion to the earthquake sequence. However, the COVID-19 pandemic differs from most disasters because the effects are global not local and they are likely to be longer in duration than a one-off disaster or a series of earthquakes. This may further impact recruitment and retention as the social and economic consequences of COVID-19 take effect. For example, loss of employment may result in participants moving area or country, making their contact details obsolete. Researchers will have to use additional contacts (if these have been collected) to track participants, potentially threatening retention rates if they cannot be found. Recruitment may become more difficult for similar reasons and the general stress created by the pandemic may result in less willingness to participate in research. Data collection will also be more challenging or not possible at all, if face-to-face interaction is required (for example, collecting blood samples). Researchers involved in ongoing and new longitudinal studies will have to be pragmatic and flexible in both the design and implementation of their research if they are to overcome the additional problems of natural disasters and pandemics.
Note
Online Appendix is available here: https://doi.org/10.6084/m9.figshare.14410700
Funding
The preparation of this manuscript was supported by a tertiary institution.
Acknowledgements
The authors would like to acknowledge the assistance of support staff for their help with manuscript preparation. The authors would also like to acknowledge the commitment of the many sample members who have given their time participating in the studies.
Ethics approval
Ethical approval information is contained in Online Resource 1.
Conflict of interest
The authors declare that there is no conflict of interest.
References
Ardagh, M.W., Richardson, S.K., Robinson, V., Than, M., Gee, P., Henderson, S., Khodaverdi, L., McKie, J., Robertson, G., Schroeder, P.P. et al. (2012) The initial Health-system response to the earthquake in Christchurch, New Zealand, in February, 2011, The Lancet, 379(9831): 2109–15. doi: 10.1016/S0140-6736(12)60313-4
Atkeson, A. (2020) What Will Be the Economic Impact of COVID-19 in the US? Rough Estimates of Disease Scenarios, Working Paper 26867, Cambridge, MA: National Bureau of Economic Research.
Bartlett, R., Milne, R. and Croucher, R. (2018) Strategies to improve recruitment of people with dementia to research studies, Dementia, 18(7/8): 2494–504. doi: 10.1177/1471301217748503
Bell, C.J., Frampton, C.M., Colhoun, H.C., Douglas, K.M., McIntosh, V.W., Carter, F.A., Jordan, J., Carter, J.D., Smith, R.A., Marie, L.M. et al. (2018) Earthquake brain: impairment of spatial memory following long-term earthquake-related stress, Australian & New Zealand Journal of Psychiatry, 53(1): 37–47.
Byrd-Bredbenner, C., Delaney, C., Martin-Biggers, J., Koenings, M. and Quick, V. (2017) The marketing plan and outcome indicators for recruiting and retaining parents in the HomeStyles randomized controlled trial, Trials, 18(1): art. 540, https://doi.org/10.1186/s13063-017-2262-3. doi: 10.1186/s13063-017-2262-3
Caruana, E.J., Roman, M., Hernández-Sánchez, J. and Solli, P. (2015) Longitudinal studies, Journal of Thoracic Disease, 7(11): E537–E540, https://doi.org/10.3978/j.issn.2072-1439.2015.10.63.
Christ Church Cathedral Reinstatement Project (2021) The project to reinstate Christ Church Cathedral, https://reinstate.org.nz/.
Coggon, D., Barker, D. and Rose, G. (2009) Epidemiology for the Uninitiated, London: Wiley.
Davison, K.K., Charles, J.N., Khandpur, N. and Nelson, T.J. (2017) Fathers’ perceived reasons for their underrepresentation in child health research and strategies to increase their involvement, Maternal Child Health Journal, 21(2): 267–74. doi: 10.1007/s10995-016-2157-z
Forcey, D.S., Walker, S.M., Vodstrcil, L.A., Fairley, C.K., Bilardi, J.E., Law, M., Hocking, J.S., Fethers, K.A., Petersen, S., Bellhouse, C. et al. (2014) Factors associated with participation and attrition in a longitudinal study of bacterial vaginosis in Australian women who have sex with women, PLoS ONE, 9(11): e113452, https://doi.org/10.1371/journal.pone.0113452.
Friedman, D.B., Foster, C., Bergeron, C.D., Tanner, A. and Kim, S.H. (2015) A qualitative study of recruitment barriers, motivators, and community-based strategies for increasing clinical trials participation among rural and urban populations, American Journal of Health Promotion, 29(5): 332–8. doi: 10.4278/ajhp.130514-QUAL-247
Friedman, L.M., Furberg, C.D., DeMets, D.L., Reboussin, D.M. and Granger, C.B. (2015) Recruitment of study participants, Fundamentals of Clinical Trials, 5th edn, Cham: Springer, pp 215–32.
Fry, A., Littlejohns, T.J., Sudlow, C., Doherty, N., Adamska, L., Sprosen, T., Collins, R. and Allen, N.E. (2017) Comparison of sociodemographic and health-related characteristics of UK Biobank participants with those of the general population, American Journal of Epidemiology, 186(9): 1026–34. doi: 10.1093/aje/kwx246
Geonet (2016) Magnitude 5.7, Sun Feb 14 2016 1:13 PM, https://www.geonet.org.nz/earthquake/2016p118944.
Gray, C.L. and Mueller, V. (2012) Natural disasters and population mobility in Bangladesh, Proceedings of the National Academy of Sciences, 109(16): 6000–5. doi: 10.1073/pnas.1115944109
Greater Christchurch Group (2019) Canterbury Earthquake Recovery Learning and Legacy Programme: The EQ Recovery Learning Website, https://www.eqrecoverylearning.org/about/earthquake-recovery-in-canterbury/.
Gul, R.B. and Ali, P.A. (2010) Clinical trials: the challenge of recruitment and retention of participants, Journal of Clinical Nursing, 19(1/2): 227–33. doi: 10.1111/j.1365-2702.2009.03041.x
Hallegatte, S. (2016) Natural Disasters and Climate Change: An Economic Perspective, Cham: Springer.
Howcutt, S.J., Barnett, A.L., Barbosa‐Boucas, S. and Smith, L.A. (2018) Research recruitment: a marketing framework to improve sample representativeness in health research, Journal of Advanced Nursing, 74(4): 968–75. doi: 10.1111/jan.13490
La Greca, A.M., Silverman, W.K., Vernberg, E.M. and Prinstein, M.J. (1996) Symptoms of posttraumatic stress in children after Hurricane Andrew: a prospective study, Journal of Consulting and Clinical Psychology, 64(4): 712–23. doi: 10.1037/0022-006X.64.4.712
Land Information New Zealand (2015) Christchurch Residential Red Zone, https://www.linz.govt.nz/crown-property/types-crown-property/christchurch-residential-red-zone.
Lankenau, S.E., Sanders, B., Hathazi, D. and Jackson Bloom, J. (2010) Recruiting and retaining mobile young injection drug users in a longitudinal study, Substance Use and Misuse, 45(5): 684–99. doi: 10.3109/10826081003594914
Lau, H., Khosrawipour, V., Kocbach, P., Mikolajczyk, A., Schubert, J., Bania, J. and Khosrawipour, T. (2020) The positive impact of lockdown in Wuhan on containing the COVID-19 outbreak in China, Journal of Travel Medicine, 27(3): art taaa037, https://doi.org/10.1093/jtm/taaa037. doi: 10.1093/jtm/taaa037
Lee, C., Dobson, A., Brown, W. and Adamson, L. (2000) Tracking participants: lessons from the Women’s Health Australia Project, Australian and New Zealand Journal of Public Health, 24(3): 334–6. doi: 10.1111/j.1467-842X.2000.tb01580.x
Mason, M.J. (1999) A review of procedural and statistical methods for handling attrition and missing data in clinical research, Measurement and Evaluation in Counseling and Development, 32(2): 111–18. doi: 10.1080/07481756.1999.12068976
Matthieu, M. and Ivanoff, A. (2006) Treatment of human-caused trauma: attrition in the adult outcomes research, Journal of Interpersonal Violence, 21(12): 1654–64. doi: 10.1177/0886260506294243
McGonagle, K., Leissou, E., Beaule, A. and Stafford, F. (2008) Tracking PSID Families in the Gulf States of Louisiana, Mississippi, and Alabama, PSID (Panel Study of Income Dynamics) Technical Series Paper 08-01.
McSaveney, E. (2017) Historic Earthquakes: The 2011 Christchurch Earthquake, Te Ara: The Encyclopedia of New Zealand, http://www.TeAra.govt.nz/en/historic-earthquakes/page-13.
Ministry for Culture and Heritage (2016) February 2011 Christchurch Earthquake, https://nzhistory.govt.nz/culture/february-2011-christchurch-earthquake.
Mitchell, M. (2015) Relocation After Disaster: Engaging with Insured Residential Property Owners in Greater Christchurch’s Land-Damaged ‘Residential Red Zone’, Washington, DC: Brookings Institution.
Neria, Y., Galea, S. and Norris, F.H. (2009) Disaster mental health research: current state, gaps in knowledge, and future directions, in Y. Neria, S. Galea and F.H. Norris (eds) Mental Health and Disasters, Cambridge: Cambridge University Press, pp 594–610.
Nohr, E.A., Frydenberg, M., Henriksen, T.B. and Olsen, J. (2006) Does low participation in cohort studies induce bias?, Epidemiology, 17(4): 413–18. doi: 10.1097/01.ede.0000220549.14177.60
Norris, F.H., Galea. S., Friedman, M.J. and Watson, P.J. (eds) (2006) Methods for Disaster Mental Health Research, New York: Guilford Press.
Potter, S.H., Becker, J.S., Johnston, D.M. and Rossiter, K.P. (2015) An overview of the impacts of the 2010–2011 Canterbury earthquakes, International Journal of Disaster Risk Reduction, 14(1): 6–14. doi: 10.1016/j.ijdrr.2015.01.014
Proctor, L.J., Fauchier, A., Oliver, P.H., Ramos, M.C., Rios, M.A. and Margolin, G. (2007) Family context and young children’s responses to earthquake, Journal of Child Psychology and Psychiatry, 48(9): 941–9. doi: 10.1111/j.1469-7610.2007.01771.x
Ribisl, K.M., Walton, M.A., Mowbray, C.T., Luke, D.A., Davidson, W.S. II and Bootsmiller, B.J. (1996) Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: review and recommendations, Evaluation Program Planning, 19(1): 1–25. doi: 10.1016/0149-7189(95)00037-2
Satherley, N., Milojev, P., Greaves, L.M., Huang, Y., Osborne, D., Bulbulia, J. and Sibley, C.G. (2015) Demographic and psychological predictors of panel attrition: Evidence from the New Zealand attitudes and values study, PLoS ONE, 10(3): art e0121950. doi: 10.1371/journal.pone.0121950
Scott, C.K. (2004) A replicable model for achieving over 90% follow-up rates in longitudinal studies of substance abusers, Drug and Alcohol Dependence, 74(1): 21–36. doi: 10.1016/j.drugalcdep.2003.11.007
Scott, C.K., Sonis, J., Creamer, M. and Dennis, M.L. (2006) Maximizing follow‐up in longitudinal studies of traumatized populations, Journal of Traumatic Stress. 19(6): 757–69. doi: 10.1002/jts.20186
Smith, R., McIntosh, V.V., Carter, J.D., Colhoun, H., Jordan, J., Carter, F.A. and Bell, C.J. (2017) In some strange way, trouble is good for people: posttraumatic growth following the Canterbury earthquake sequence, Australasian Journal of Disaster and Trauma Studies, 21(1): 31–42.
Spittlehouse, J.K., Joyce, P.R., Vierck, E., Schluter, P.J. and Pearson, J.F. (2014) Ongoing adverse mental health impact of the earthquake sequence in Christchurch, New Zealand, Australian & New Zealand Journal of Psychiatry, 48(8): 756–63.
Teague, S., Youssef, G.J., Macdonald, J.A., Sciberras, E., Shatte, A., Fuller-Tyszkiewicz, M., Greenwood, C., McIntosh, J., Olsson, C.A. and Hutchinson, D. (2018) Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis, BMC Medical Research Methodology, 18(1): art 151, https://doi.org/10.1186/s12874-018-0586-7. doi: 10.1186/s12874-018-0586-7
Warheit, G.J., Zimmerman, R.S., Khoury, E.L., Vega, W.A. and Gil, A.G. (1996) Disaster related stresses, depressive signs and symptoms, and suicidal ideation among a multi‐racial/ethnic sample of adolescents: a longitudinal analysis, Journal of Child Psychology and Psychiatry, 37(4): 435–44. doi: 10.1111/j.1469-7610.1996.tb01424.x
World Health Organization (2020) Coronavirus disease (COVID-19) Situation Report 136, https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200604-covid-19-sitrep-136.pdf?sfvrsn=fd36550b_2.
Wright, J.D., Allen, T.L. and Devine, J.A. (1995) Tracking non-traditional populations in longitudinal studies, Evaluation and Program Planning, 18(3): 267–77. doi: 10.1016/S0149-7189(95)00020-8
