Data quality and response distributions in a mixed-mode survey

View author details View Less
  • 1 University of Michigan, , USA
Full Access
Get eTOC alerts
Rights and permissions Cite this article

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition.

Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode.

We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.

Abstract

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition.

Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode.

We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.

Key messages

  • Data quality is slightly lower for web versus telephone protocols, but differences are small.

  • Rates of asset ownership are lower for web versus telephone protocol.

  • There is little evidence of differential reporting by assigned mode for most other measures examined.

  • Mode differences may be due to interviewer probing more than respondent satisficing or social desirability.

Introduction

The challenges facing survey data collection are well documented (see Couper, 2017; Williams and Brick, 2018). Non-response rates are rising. Increasing efforts to stem the tide of declining responses rates have led to rapid rises in the costs of data collection, especially those involving interviewers. The ongoing (as at the time of writing) COVID-19 pandemic has brought alternative modes in sharp focus, with many surveys being forced to abandon (at least temporarily) interviewer-administered data collection. These factors are leading researchers to explore alternatives. Sequential mixed-mode data collection, in which sample members are initially invited to complete the survey via a cheaper method (such as the web), and those who don’t initially respond are then followed up with more expensive methods (telephone or face-to-face interviews) are becoming increasingly popular. However, the literature on the effects of such changes on the quality of survey responses, and indeed on response distributions and measures of change in longitudinal studies, is still very limited. This paper adds to the small but growing literature.

Longitudinal surveys are not immune from the negative effects of declining response rates (Cheshire et al, 2011; Lynn, 2017). Because of ongoing interaction with panel members, mixed-mode options are particularly attractive for longitudinal surveys, given that contact information and the ability and willingness to complete the survey online could be ascertained in a previous wave. Furthermore, information from prior waves could be used to target panel members for particular modes or sequences of modes (see Lynn, 2013; 2017; Freedman et al, 2018). For these reasons, a number of large panel and cohort studies (including some cohort studies and Understanding Society in the UK, the Panel Study of Income Dynamics [PSID] and Health and Retirement Study [HRS] in the US and the Swiss Household Panel Study [SHP]) are transitioning to mixed-mode designs or evaluating the efficacy of doing so (see, for example, Bianchi et al, 2017).

A key concern with such mode transitions is the effect of these changes on key estimates and analyses. This is especially true of longitudinal surveys where mode switches may affect not only levels of estimates of key outcomes (both point estimates and cross-sectional models), but also changes over time, threatening a core premise of longitudinal measurement: continuity of measurement to facilitate evaluation of change. Mode transitions are typically done carefully and involve an evaluation of the effects of the mode changes on estimates of key outcomes.

A variety of different approaches have been used to study mode effects (see Jäckle et al, 2010), including testing for data ‘completeness’ (for example, Jäckle et al, 2015), differences in marginal effects, effects on overall psychometric properties, comparisons of relationships between variables and trajectories by mode, or formal equivalence tests (for example, Klausch et al, 2013; Cernat, 2015a; 2015b; Cernat et al, 2016; Mariano and Elliott, 2017). We take advantage of a simple sequential mixed-mode experimental design that was implemented in the 2018 wave of the HRS and use an intent-to-treat (ITT) analytic approach. Our goal is not to explore how and why mode effects may occur, but rather to evaluate the effect of a mode switch on key survey estimates (that is, the first type of mode study identified by Jäckle et al, 2010). We investigate differences in response quality (for example, item missing, incomplete responses, focal responses) and response distributions by assigned mode (web versus telephone) for a variety of different measures that may be subject to mode effects, including assets, expectations, physical and psychological health, and family and household composition. Given that the HRS is unlikely to switch to an exclusively self-administered mode (that is, online only), but rather to continue to deploy a sequential mixed-mode approach, the ITT approach allows us to compare estimates obtained from a mixed-mode (web and telephone) approach with those obtained from an interviewer-administered-only approach (specifically, computer-assisted telephone interviewing or CATI). That is, we are comparing two systems of data collection (de Leeuw, 2005), rather than attempting to isolate a ‘pure’ mode effect. Our focus is on whether mixing modes would affect substantive conclusions (see Jäckle et al, 2010).

Background

A number of features of survey modes have been identified as potentially affecting measurement (see, for example, Tourangeau et al, 2000, Chapter 5; Jäckle et al, 2010; Couper, 2011). Our goal is not to attempt an exhaustive review of all of the mechanisms that may produce mode effects, but rather to focus on a select few that are relevant to the data we have available.

One key element is that modes differ in the presence of an interviewer. Interviewers are associated with higher levels of socially desirable responding, but interviewers also reduce missing data and other errors in surveys. Both of these elements (socially desirable responding and item-missing data) are relevant to our study.

One of the consistently found differences between interviewer-administered and self-administered surveys relates to social desirability bias, or the tendency to present oneself in a favourable light. A number of studies have found higher reports of socially undesirable behaviours, attributes or attitudes in self-administered surveys and lower reports of socially desirable ones. For reviews, see Groves et al (2009), Tourangeau et al (2000) and Heerwegh and Loosveldt (2011). For a selection of empirical examples involving internet surveys, see Goodman et al (2020), Heerwegh (2009), Kreuter et al (2008), Milton et al (2017), and Zhang et al (2017).

In the absence of an interviewer to provide motivation and encouragement, respondents may be inclined to invest less effort in answering the survey questions, resulting in suboptimal responding or satisficing (Krosnick, 1991). This can take many forms, including higher rates of item-missing data, more rounding or heaping, motivated misreporting (such as answering ‘no’ to avoid follow-up questions), acquiescence, and straightlining or non-differentiation (providing the same response to a series of related questions).

Several studies report evidence of higher item-missing data in web surveys relative to interviewer administration (see, for example, de Leeuw, 2005; 2018; Heerwegh and Loosveldt, 2008; Jäckle et al, 2015; Bruine de Bruin et al, 2016; McGonagle et al, 2017; Goodman et al, 2020). Interviewers are trained to probe if a respondent offers an initial response of ‘don’t know’ (DK) or ‘prefer not to answer’ (NA). Offering explicit DK or NA options tends to increase item-missing data. While additional probing is possible in web surveys (see, for example, Al Baghal and Lynn, 2015; de Leeuw et al, 2016) this is typically reserved for a subset of key questions. In the absence of such probes, item-missing data rates are expected to be higher in web surveys.

There is some evidence of more straightlining or non-differentiation in web surveys (see Fricker et al, 2005; Bowyer and Rogowski, 2017; but for a contrary finding, see Chang and Krosnick, 2010). With regard to rounding or heaping, we know of few studies that have explored mode effects on questions that are susceptible to such effects. In one exception, Lui and Wang (2015) reported higher rates of rounding on a number of feeling thermometer questions on the web than in a face-to-face survey, contrary to their (and our) expectation. In a review of questions potentially susceptible to estimation (as opposed to providing exact answers), d’Ardenne et al (2017: 5) hypothesised that: ‘In self-completion modes participants may be less motivated to get documentation to improve the accuracy of their answer’. Similarly, Hope et al (2014: 36) found evidence that ‘interviewers helped respondents carry out complicated tasks’, leading to better performance than in the web mode. A competing hypothesis is that there are fewer time pressures in a web survey than a telephone survey, giving respondents more time to think about their answers or to consult records (see Couper et al, 2013), potentially leading to answers with greater precision or less rounding.

A related concern, consistent with the satisficing hypothesis, is that reduced effort may be expended by web respondents to review and change information presented to them. Dependent interviewing, in which information is fed forward from a previous wave and presented to the respondent for confirmation or correction (see Jäckle et al, 2021), may result in greater acquiescence on the web, resulting in fewer changes or corrections. Jäckle et al (2014) reported higher levels of concern about the confidentiality of such fed-forward data among web respondents, but few effects on the quality of resultant data.

The updating of household rosters, which involves feeding forward information about household members from prior waves for confirmation or correction, raises similar questions regarding reduced effort or acquiescence. As Burton and Jäckle (2020: 9) note, an early concern for Understanding Society was that the web mode might fail to pick up new entrants to a household or people who have left the household. As they note,

With a complex task such as this, it was possible that a respondent completing it online would just confirm that all the previous residents were still resident, and we may miss new entrants and leavers in the household, resulting in poorer quality data and errors in a key measure of interest (dynamics of household composition).

However, analysis of the household grids in the sequential mixed-mode experiments in the Understanding Society Innovation Panel has indicated that there are no significant differences in the numbers of leavers or new entrants by mode (Burton and Jäckle, 2020).

Given that this is a secondary analysis, we did not design our study to test specific hypotheses. Rather, we offer the following expectations with regard to differences between web and telephone modes in the HRS experiment, based on the relatively sparse literature:

  1. While we expect low rates of missing data overall, we expect higher rates of missing data for the web-assigned sample than for the control (telephone) sample.

  2. Consistent with the social desirability findings, we expect differences in reported health measures. Specifically, we expect more reports of chronic conditions and activity limitations, higher rates of depression and lower life satisfaction on the web.

  3. We expect less precision in the reporting of asset amounts and greater use of focal values (that is, 0%, 50% and 100%) in response to expectation questions on the web.

  4. We expect greater acceptance of the preloaded rosters in the web mode. That is, we expect those in the web-assigned sample to make fewer changes (additions, deletions or corrections) to the rosters presented for review.

We next turn to a description of the data.

Data

The HRS is an ongoing panel study of people over age 50 in the US. The study conducts biennial (core) interviews with age-eligible individuals and their spouses of any age. The interviews collect detailed information on a wide range of topics, including physical health, disability, cognition and psychosocial well-being; employment, housing, income and wealth; health insurance and use of health services; family composition and support exchanges; and expectations. In addition to the core interviews, HRS also conducts a number of supplemental studies, mainly in the form of mail and web surveys, in the years between interview waves.

Up through 2016, the core interviews were conducted using a mix of face-to-face (FTF) and telephone modes. In the early waves, FTF was mainly reserved for baseline interviews. Starting in 2006, the study added physical and biological measurements requiring an FTF visit, for a random and rotating half-sample each wave. Thus, half of the sample is interviewed FTF and the other half by telephone each wave and that assignment then flips in the next wave. As a result, everyone is interviewed FTF every other wave and by telephone in the intervening waves. For telephone mode, the full interview is administered by field interviewers using a CATI interface.

Introduction of a sequential mixed-mode design

In the 2018 wave, HRS began the transition to a sequential mixed-mode design. The full core interview (about 90 minutes in length) was programmed to be self-administered on the web. A substantial amount of developmental work went into the design, programming and pretesting of the web instrument, with the goal of maximising the comparability of data obtained from the web and interviewer-administered surveys.

Prior to data collection for the 2018 wave, HRS identified 3,750 sample members who were scheduled to receive a telephone interview in 2018 and who had reported having internet access in their most recent interview.1 A random subsample of about 2,250 of these participants was invited to complete their interview online.2 The web sample received an advance letter with the survey URL, followed a day or two later by an email with a link to access the survey. Periodic reminders were sent by email and mail during the first several weeks to those who had not yet completed the survey, and non-respondents (and partial respondents) were contacted to schedule a telephone interview after 6–12 weeks of no web activity. A control sample of 1,500 participants with web access received the usual telephone interview, with no option to complete by web. This design provides a strong framework for making comparisons between the samples that were assigned to the sequential web+telephone versus telephone-only protocols (hereafter referred to as ‘web-assigned’ and ‘control’ groups).

Final response rates were the same for both groups: 81% for the web-assigned sample (64% completed by web and 17% by telephone; that is, 21% of respondents in the web-assigned sample completed the survey by telephone) and 81% for the control sample. Our base analysis sample comprises 2,834 respondents: 1,741 in the web-assigned group and 1,093 in the control group.3 For some outcomes we use different subsets of this sample, such as self-respondents or financial respondents, as noted in the tables and discussion later in this article. Data used in the analysis is from the 2018 Early Release Version 1.0 file.

Measures

For our analysis of mode differences in response quality and response distributions, we focused on several domains key to HRS: health (physical and psychological), financial status, expectations and family composition. To evaluate response quality we adopted a broad approach by examining all of the items in these domains that were asked of all or most respondents. (Items that were asked of select subgroups or embedded in skip patterns were generally excluded.) To evaluate response distributions we selected measures within the domains that are used widely in research and that may be sensitive to mode differences. We do not examine cognition because several of the cognitive measures differ between the interviewer-administered and web modes and because these have been the focus of other work (Ofstedal et al, 2021). A list of the data quality and response distribution measures that we examined is provided in Figure 1. These measures are described in detail below and can be found in the online questionnaires.4

Figure 1:
Figure 1:

Measures of data quality and response distributions, by survey domain

Citation: Longitudinal and Life Course Studies 2022; 10.1332/175795921X16494126913909

Response quality

We examined several indicators of response quality, including item-missing rates, completeness of asset values, focal responses for expectation measures, and the rate of corrections and/or updates in rosters.

For item-missing rates, we generally included all items within a given domain that were asked of most or all respondents. The rate is simply the number of questions to which the respondent gave a don’t know or refuse answer divided by the total number of questions.

For assets, we examined 11 questions on asset ownership that were asked of all financial respondents.5 These included primary residence; second home; other real estate, business or farm; stocks or mutual funds; bonds; treasury bills or certificates of deposit (CDs); checking or savings accounts; vehicles; individual retirement accounts or Keogh accounts; other assets; and debt. For each of these assets, respondents were first asked if they (or their partner) owned it and, if so, how much it was worth. Respondents who do not provide an exact dollar amount are asked a series of follow-up questions (referred to as unfolding brackets) in an attempt to obtain a range within which the value falls. It yields the following possible responses: exact value, complete bracket (the value falls between a known lower and upper bound), incomplete bracket (either the lower or upper bound is known, but not both), no information. We constructed two item-missing rates: one for the ownership questions and a second for the value questions, conditional on ownership. The item-missing rate for the value questions is a count of the number of owned assets for which the respondent provided no information on the value, divided by the number of owned assets. This subsample has 1,946 respondents (financial respondents in the experiment who were asked the asset questions; 1,182 web-assigned and 764 control).

Respondents were also asked a series of questions to elicit the likelihood of various events in the form of a probability response, referred to as expectations. For each item, respondents were asked how likely they think the event is to happen by giving a number ranging from 0 (absolutely no chance that the event will happen) to 100 (absolutely sure that the event will happen). There are 29 expectations questions in the survey, but only five of the questions are asked of all or most respondents. They include the probability that: (1) the respondent will live to a target age (determined by their age); (2) the respondent’s out-of-pocket medical expenses will exceed $1,500 in the following year; (3) the respondent will leave an inheritance of $10,000 or more; (4); the stock market will go up in the following year; (5) the Medicare programme will become less generous over the following ten years. The item-missing rate for expectations is based on these five questions. The subsample for this analysis has 2,791 respondents in 2018 (1,709 assigned to the web; 1,082 assigned to the control group).

In addition to the item-missing rate, for assets and expectations we also examined the completeness and/or quality of responses for these items. For assets, we examine the proportion of all owned assets for which the respondent provided an exact value, complete bracket, incomplete bracket and no information. These proportions sum to 1. For the expectation items, we examined several other indicators of response quality: the proportion of non-focal responses (numbers other than 0%, 50% or 100%) among all valid responses; the proportion of 50% among all valid responses; and the proportion of 0% or 100% among all valid responses. Responses of 0%, 50% and 100% are referred to in the literature as focal responses and thought to signify potential lack of understanding of the question or lack of effort in responding to the question (Huynh and Jung, 2005).

We constructed separate item-missing rates for the physical health, disability and psychological health measures. For physical health, the rate was composed of 38 items in the health section, including diagnosis of ten chronic conditions (hypertension, diabetes, cancer, lung disease, heart disease, stroke, arthritis, emotional or psychiatric problem, depression, Alzheimer’s), problems with incontinence, sleep disorders, sensory impairments, physical activity, pain and the use of pain medications. We constructed a separate item-missing rate for the disability measures, which included eight physical functioning (PF) items (walking several blocks, jogging one mile, walking one block, stooping/crouching/kneeling, reaching over shoulder height, pulling or pushing a large object, lifting ten pounds, picking up a dime from a table) and six Instrumental Activities of Daily Living (IADL) (using maps, preparing meals, shopping for groceries, making phone calls, taking medication, managing money) for a total of 14 items. The indicator for psychological health comprised eight items that were adapted from the Centers for Epidemiologic Studies Depression Scale (CES-D) (Radloff, 1977; Steffick, 2000), the stem question from the Short-Form Composite International Diagnostic Interview (CIDI-SF) (Kessler et al, 1998), and a life satisfaction question for a total of ten items. The physical health and disability indicators are based on 2,834 respondents (1,741 in the web-assigned and 1,093 in the control group). The psychological health indicators were not asked of proxy respondents, so the analysis is based on 2,820 respondents (1,735 web-assigned and 1,085 control).

We also examined a measure of discordance between previous wave and current wave reports of chronic conditions. For chronic conditions that were reported in a previous wave, respondents are reminded of their prior report and they may either confirm or dispute it. The level of disputes is very low but potentially sensitive to mode. To evaluate this we create a count of previously reported chronic conditions that were disputed by the respondent in the 2018 survey (disputes). This count is based on 2,757 respondents (1,678 web-assigned and 1,079 control) who had non-missing data on the chronic condition measures in both the current and previous waves.

Our final indicator of data quality comes from the rosters. There are two key rosters in the HRS of relevance. One is the roster of household members, where the interviewer or respondent is presented with information on household members collected in a previous wave, and asked to review the information and update if necessary. This could include additions, deletions and/or corrections. A second roster focuses on non-resident children and their spouses. Given the age of the HRS sample, the child roster is likely to have significantly more changes, including deaths, changes in marital status and the like.

Of the 2,834 respondents in the base sample, a total of 2,455 respondents (1,512 web-assigned and 943 control) nested within 2,011 households were presented with the rosters and comprise our sample for the roster analyses.6

For this data quality indicator, we simply sum the number of changes that are made in the household and child rosters. As already noted, our expectation is that respondents completing the survey online would be inclined to accept the information presented without careful review, resulting in fewer changes to the rosters than in the interviewer-administered mode, where this information is read out loud to the respondent for confirmation or correction.

Response distributions

We also examined response distributions for the health, asset and expectation items. These are based on respondents who provided non-missing responses to the items.

For assets we examine reports of ownership for each of the 11 assets individually, as well as a count of the number of assets owned. Likewise, we examine the reported value for each of the assets individually, as well as the total value of all assets combined. The value measures are based on reports of exact amounts only. For expectations, we also examine each of the five expectation items individually. In addition, we created a standardised optimism score from these five items that include the negative events with a negative sign (medical expenses increasing, Medicare to become less generous).

For physical health we included counts of the total number of chronic conditions (0–10) and the number of newly reported (incident) chronic conditions (0–10), and a measure of self-rated health (1 = excellent, 5 = poor). Disability measures include the number of physical functioning limitations (0–8) and number of IADL limitations (0–6). For psychological health, we included the number of CES-D depressive symptoms (0–8) and life satisfaction (1 = completely satisfied to 5 = not at all satisfied).

Analytic methods

To investigate average mode effects we carried out an ITT effect analysis. We compared average outcomes of respondents assigned to the web to the average outcomes of respondents assigned to the control group. There are three reasons to focus on ITT effects. First, they can be identified in a credible and transparent way due to random assignment. Second, they are the effects that are practically relevant for surveys that allow for switching to the telephone mode in a mixed-mode protocol. Third, in our case, the ITT effects are probably close to the average treatment effects anyway, because compliance to assignment was high: 79% of the respondents assigned to web mode completed their interview in the web mode, and none of the respondents in the comparison group completed the questionnaire online. However, the ITT approach does not allow us to isolate the effects of mode of completion; only those of mode assignment.

For each outcome variable j, we estimate the average ITT effects by a simple regression
M1

where yji is outcome j for respondent i; web-assignedi is one if the respondent is assigned to web mode and zero otherwise; αj is average of outcome j in the comparison group; βj is the average ITT effect for outcome j, estimated as the average difference between the two groups.

In the course of our analysis we consider the outcomes we outlined in the Data section, focusing first on response quality and second on response distributions. We present the outcome variables separately for the four domains we investigate: assets, expectations, health, and household and child rosters.

Our main analysis focuses on average ITT effect estimates using outcome variables from the 2018 survey. In an auxiliary analysis we repeat all regressions with dependent variables constructed analogously from the 2016 survey. These regressions provide estimates of pre-treatment differences and serve as checks for random assignment. If assignment was random, there should be no pre-treatment differences between the two groups and, thus, the regression estimates should be all close to zero.

Given the selection criteria into the web-assigned and control samples and because we are not making inferences to a population, the analyses presented here are not weighted. We did not adjust significance levels for multiple comparisons. Rather, we attempted to use summary measures wherever possible, and adopt a conservative approach in interpreting significant effects.

Results

Response quality

Assets

We show results for 11 asset and debt items in the main text. Ownership of these 11 items was asked of all financial respondents in the survey. The Appendix contains the corresponding results for all 16 items, including those that were asked only of a subset of the respondents, conditional on ownership of a different asset (for example, mortgages, conditional on home ownership).

The quality of information on assets is affected by item non-response for asset ownership and asset value; the latter can have various results depending on the bracket responses. Table 1 shows the results for (1) the proportion of don’t know or refused ownership responses, (2) the proportion of asset value responses that are exact number values (that is, the response was a valid number), (3) the proportion of responses in closed brackets, (4) the proportion of responses with open-ended brackets, and (5) the proportion of responses with no information on asset value. Higher proportions of (1), (3), (4) and (5) indicate lower quality; a higher proportion of (2) indicates higher quality.

Table 1:

Quality of asset responses

(1)(2)(3)(4)(5)
Mode assignment in 2018OwnershipProportion of value responses that are
Proportion don’t know or refusedExact numberComplete bracketIncomplete bracketNo value
Web0.017** (0.004)−0.035** (0.012)0.010 (0.008)0.002 (0.003)0.023** (0.009)
Constant0.012** (0.002)0.867** (0.009)0.068** (0.006)0.013** (0.002)0.053** (0.006)
Observations1,9461,9051,9051,9051,905
R-squared0.0080.0040.0010.0000.003
Control group mean0.0120.8670.0680.0130.053
Control group sd0.0650.2450.1620.0690.167

Notes:

Average intent-to-treat effect estimates (2018); linear regression results; variables constructed from 11 asset and debt items.

Standard errors in parentheses.

** p < .01, * p < .05

The results indicate statistically significant and moderately strong mode effects. Being assigned to the web mode increased the proportion of missing data (don’t know responses or refusals) to the asset and debt ownership questions; it decreased the proportion of exact number responses to the asset value questions; it increased the proportion of responses giving no value at all to the asset value questions. These three effect estimates are statistically significant. The magnitude of the effects is substantial in relative terms, but because baseline response quality is high, the absolute magnitudes are not large. The proportion of missing responses is increased from 1.2% to 2.9% (an effect size of 0.26 standard deviation); the proportion of exact number responses is reduced from 87% to 83% (0.14 standard deviation); the proportion of no value responses is increased from 5.3% to 7.6% (0.14 standard deviation). The results are very similar when all 16 asset and debt items are considered instead of the 11 included here (see Appendix Table A1).

To check random assignment we replicated the same ITT analysis with the 2016 asset data. If assignment was random the pre-treatment differences between the two groups should be approximately zero, and so the coefficient on the ‘assigned to web’ variable should be approximately zero in all regressions. Appendix Table A2 shows the results. All coefficient estimates are very close to zero, and none of them is statistically significant.

Expectations

Recall that the expectations questions are formulated in terms of subjective probabilities: for various events, respondents are invited to answer what they think the percentage chance of the event is. We show results on five items asked of all respondents. We examine ITT effects on four measures of response quality: (1) the proportion of don’t know or refused responses; (2) the proportion of non-focal responses (numbers other than 0%, 50% or 100%) among all valid responses; (3) the proportion of 50% among all valid responses; (4) the proportion of 0% or 100% among all valid responses. Higher proportions of (1), (3) and (4) indicate lower quality; a higher proportion of (2) indicates higher quality. Table 2 shows the ITT effect estimates from four linear regressions.

Table 2:

Quality of subjective probability responses

(1)(2)(3)(4)
Mode assignment in 2018Proportion of don’t know or refused responsesProportion of value responses that are
non-focal50%0% or 100%
Web0.034** (0.006)−0.088** (0.011)0.045** (0.008)0.043** (0.010)
Constant0.029** (0.003)0.557** (0.009)0.212** (0.006)0.231** (0.007)
Observations2,7912,7372,7372,737
R-squared0.0100.0240.0110.008
Control group mean0.0290.5570.2120.231
Control group sd0.1100.2750.1930.236

Notes:

Average intent-to-treat effect estimates (2018); linear regression results; variables constructed from 5 items.

Non-focal responses are all valid responses except 0%, 50%, and 100%. Standard errors clustered by household in parentheses.

** p < .01, * p < .05

Similar to the mode effects on the quality of asset responses, the results for expectations indicate statistically significant ITT effects. The estimated effect sizes are larger here. Assignment to the web increases the proportion of missing (don’t know or refused) responses from 2.9% to 6.3% (an effect size of 0.31 standard deviation); it decreases the proportion of non-focal responses from 56% to 47% (effect size 0.32 standard deviation); it increases the proportion of 50% responses from 21% to 26% (0.23 standard deviation); it increases the proportion of 0% or 100% responses from 23% to 27% (0.18 standard deviation). These magnitudes are substantial. They indicate a strong negative mode effect on the quality of subjective probability responses to expectations questions. The results are very similar if we consider all subjective probability questions besides the five examined here (see Appendix Table A3).

Table A4 in the Appendix shows the regression estimates of the pre-treatment differences, using the same variables constructed from the 2016 responses. The coefficient estimates for web assignment are all very close to zero, and none of them is statistically significant.

Health

To investigate mode assignment effects on health measures we included all questions from the health, disability and psychosocial domains that were asked of all respondents. Table 3 presents ITT results for the proportion of don’t know or refused responses from each of these question sets.

Table 3:

Quality of physical and subjective health responses

Proportion of don’t know or refused responses
Mode assignment in 2018(1)(2)(3)(4)
HealthDisabilityPsychological health# disputed conditions
Web.0018** (.0005).0054** (.0021).0056** (.0012).0876** (.0140)
Constant.0019** (.0004).0016 (.0016).0018* (.0009).0519** (.0109)
Observations2,8342,8342,8202,757
R-squared.0051.0024.0075.0136
Control group mean.0019.0016.0018.0519
Control group sd.0111.0295.0176.2372

Notes:

Average intent-to-treat effect estimates (2018); linear regression results.

Standard errors in parentheses.

** p < .01 * p < .05

The proportion of missing responses for each of these health domains is significantly higher for those assigned to the web mode compared to those in the control group. For the physical health items, respondents assigned to the web had a proportion twice that of the control group (1.9% versus 3.7%); for the disability and psychological health items, the differences are nearly fourfold (1.6% versus 7.0% for disability and 1.8% versus 7.4% for psychological health). Web-assigned respondents also had a significantly higher number of disputed conditions on average than the control group (0.14 versus 0.05), although the number for both groups is extremely small. This suggest some evidence of social desirability effects of interviewer administration if respondents in the control (telephone) group are more reluctant to dispute a condition than those in the web-assigned group.

Table A5 in the Appendix shows the regression results for the pre-treatment differences, using the same variables constructed from the 2016 responses. As was the case for assets and expectations, the coefficient estimates for web assignment are all very close to zero, and none of them is statistically significant.

Family rosters

Our final data quality indicator relates to the household and child rosters. As noted earlier, respondents with no other household members or with no living children would have nothing to change (although additions are feasible). We conducted the analyses both including and excluding these cases. The substantive conclusions do not change, so we present the results for the full set of respondents who were presented with the roster below.

Table 4 presents a summary of changes (additions, deletions and corrections) made to the two rosters, separately and combined. We present both an indicator for any change, and the mean number of changes. We see slightly more changes made among respondents assigned to the web condition than those assigned to the control condition. That is, contrary to expectation, we see no evidence of less diligence applied by web-assigned respondents than among those in the control group. Additionally (not shown in Table 4) we also examined specific types of changes (for example, additions versus corrections) and a variety of specific roster fields (for example, name, sex, birth year, relationship, residence), and see no patterns suggesting differential quality of reporting by mode assignment.

Table 4:

Roster changes (among all respondents presented with rosters)

Web-assigned groupControl groupSign. test
Child roster changes
 % 1 or more50.7%47.2%Chi2 = 2.8, p = .094
 Mean1.3961.294Poisson Wald Chi2 = 4.49, p = .034
Household member roster changes
 % 1 or more17.9%16.9%Chi2 = 0.4, p = .53
 Mean0.3110.265Poisson Wald Chi2 = 4.13, p = .034
Total roster changes
 % with any change54.0%50.4%Chi2 = 3.13, p = .077
 Mean1.7071.559Poisson Wald Chi2 = 7.72, p = .0055
(n)(1,512)(943)

Response distributions

We now turn to results of our analysis of response distributions for the asset, expectations and health measures. Note that these analyses are based on respondents who had non-missing data on the specific measure being examined, or who reported an exact value in the case of the asset value measures. Differences in the level of missing data that were documented in the previous section confound the comparisons of response distributions to some extent. Nevertheless, given that the magnitude of the difference in missing data was fairly small for most of the measures, the results of this analysis can provide useful information about the direction and magnitude of mode effects on response distributions.

Assets

Table A6 in the Appendix shows the ITT effects on the ownership of each of the 11 asset and debt items that were asked of all financial respondents. In addition to the likelihood of having each of the 11 items separately, it estimates the effect on the count of asset ownership among these 11 items. Table A7 in the Appendix shows the estimates on the value of each asset as well as their total value. The asset value variables here are based on exact number responses only. The estimated effects on ownership are all negative (positive for debt), but they are small. The item-level estimates are statistically insignificant, with the exceptions of checking or savings accounts and second homes. The estimated effect on the count of assets is small but statistically significant. Conditional on ownership, there are no detectable mode effects on the value of assets. The estimated effects on asset values are all statistically insignificant, and their sign varies across the items.

From these results we can conclude that there is a small mode effect on the reported ownership of two assets, checking/savings accounts and second homes, with web-assigned respondents showing slightly lower likelihoods of ownership. We found no mode effects on average value reports.

Expectations

To investigate mode effects on average reported expectations, we estimated ITT effects on the responses to each of the five subjective probability questions that were asked of all respondents, as well as the standardised optimism score. Table 5 shows the results from simple linear regressions.

Table 5:

Subjective probability responses

(1)(2)(3)(4)(5)(6)
Mode assignment in 2018Responses to specific questions
living to target agemedical expensesleaving inheritancestock market to go upMedicare to become less generousOptimism score
Web−3.2** (1.21)9.4** (1.48)−5.9** (1.45)−2.3* (1.00)−0.8 (1.16)−0.177** (0.044)
Constant59.4** (0.93)45.7** (1.13)76.8** (1.07)52.3** (0.78)56.8** (0.88)0.000 (0.033)
Observations2,6562,6742,6912,5032,6662,737
R-squared0.0030.0160.0070.0020.0000.007
Control group mean59.445.776.852.356.80.0
Control group sd28.335.632.423.928.01.0

Notes:

Average intent-to-treat effect estimates (2018); linear regression results; 5 items and a composite score.

Standard errors clustered by household in parentheses.

** p < .01, * p < .05

The results indicate a moderately strong and statistically significant mode effect. The effect on the composite optimism score is −0.18 standard deviations. The effect on four of the five items is statistically significant, each one is larger than one standard deviation, and all have the same sign (negative for positive events, positive for the negative event of increasing medical expenses). The composite optimism score also showed a significant negative coefficient for web-assigned cases. Taken together, these results suggest that the web mode is associated with less optimistic subjective probability responses.

Health

To investigate ITT effects on health measures we use several items that are commonly used by researchers. These include counts of health conditions, functional limitations and depressive symptoms, as well as subjective ratings of health and life satisfaction. Higher values on all of these measures are indicative of poorer health. Regression results for these measures are presented in Table 6.

Table 6:

Physical and psychological health responses

Mode assignment in 2018(1)(2)(3)(4)(5)(6)(7)
Total # conditions# incident conditionsSelf-rated health# PF limitations# IADL limitations# CES-D symptomsLife satisfaction
Web−.0665 (.0637).0077 (.0187)−.0068 (.0369)−.1501* (.0654).0273 (.0226).1172 (.0696).0626* (.0302)
Constant2.4259** (.0497).1961** (.0146)2.6484** (.0289)1.6501** (.0511).1554** (.0179)1.0521** (.0541)2.0970** (.0237)
Observations2,7732,7052,8332,7842,7942,7162,817
R-squared.0000−.0003.0000.0015.0002.0007.0012
Control group mean2.4259.19612.64841.6501.15541.05212.0970
Control group sd1.5785.4604.95501.6697.52191.6519.7948

Notes:

Average intent-to-treat effect estimates (2018); linear regression results; 5 counts and 2 individual items.

Standard errors in parentheses.

** p < .01 * p < .05

In general, there is little evidence of mode effects on response distributions to the health measures. For four of the seven measures (# total and # incident conditions, self-rated health, number of IADL limitations) there are no significant differences between the web and control groups. Although the number of depressive symptoms is slightly higher for the web versus control group (1.17 versus 1.05 symptoms), this difference is only marginally significant (p = .094). There is a significant difference between groups in the number of physical functioning limitations, with those assigned to the web reporting a smaller number of limitations than those in the control group. Substantively, however, this difference is very small (1.50 limitations for web versus 1.65 for control). Finally, those assigned to the web have significantly poorer ratings of life satisfaction (2.16 for web and 2.10 for control), although this difference is also substantively very small.

Table A8 in the Appendix presents results for the pre-treatment differences, using health measures from 2016. The coefficients for web assignment are small and non-significant for all of the measures, lending further support to the randomness of the web assignment.

Discussion

We conducted an ITT analysis comparing cases assigned to web completion (where 79% of respondents completed the survey online) with those assigned to telephone (where all were interviewed by phone). In terms of data quality, we find that missing data rates (don’t know and refused responses) were significantly higher among web-assigned respondents for both asset and expectation questions, and for health measures. Assignment to web mode is also associated with less precision in reports of asset values (that is, lower proportion of exact value and higher proportion of no value responses) and increases in focal responses (0%, 50%, 100%) to expectation questions. This supports our expectations about higher missing rates and less precise responses, suggesting the possibility of greater levels of satisficing (that is, less optimal reporting) among online respondents.

However, we find slightly more respondents disputing their preloaded health conditions in the web-assigned group, but the rates are very low in both modes. We also find no evidence of differential quality in verifying and updating roster information, although if anything slightly more respondents in the group assigned to the web made changes to the roster entries. Thus we find little evidence to support our expectations about fewer changes initiated by web respondents. This matches the reports of Burton and Jäckle (2020).

It should be noted that, while many of the data quality indicators showed statistically significant differences between the web-assigned and control groups, most of the differences are quite modest from a practical standpoint. We examined a total of 15 proportion-based indicators (proportion with missing data, focal values and so on), 13 of which showed significant differences. Of these, the differences were less than 5 percentage points for ten of the indicators and between 5 and 10 percentage points for the remaining three indicators. The largest differences we observed were for the proportion of non-focal values for the expectations questions (8.8 percentage points lower for web-assigned versus control) and the proportion missing for the disability and psychosocial indicators (5.4 and 5.6 percentage points higher for web-assigned, respectively). Likewise, we examined four count-based indicators, all of which were statistically significant, but with mean differences of 0.15 or less.

Our findings of similar or slightly higher rates of dispute and roster updating in the web-assigned group suggest that web-assigned respondents may not exert lower effort. Yet we have also found higher missing rates and less precision in the web-assigned group. Taken together, these results suggests that it may not be less effort expended by web respondents, but perhaps the ability to probe and encourage more complete responses among telephone respondents that is producing these results. If this is the case, targeted online encouragement (follow-up probes for missing answers or focal responses to key items) could reduce the observed differences between assigned modes.

In terms of response distributions, we find slightly lower rates of asset ownership reported by those assigned to the web mode for only 2 of the 11 assets examined (checking/savings account and second home), as well as for the count of assets owned. Conditional on ownership, we find no detectable mode effects on the value of assets. In terms of expectations, we find more negative (pessimistic) expectations for those assigned to the web mode, though this may be due in part to the differential use of focal values across the two modes. For the health items we examined, we find little evidence to support our expectation that those assigned to the web mode would report poorer health. The effects for depressive symptoms and life satisfaction are in the expected direction, but are not statistically significant in the case of depression and substantively very small in the case of life satisfaction. Thus, with the exception of expectations, we do not find compelling evidence of strong social desirability effects of interviewer administration. This finding is consistent with that from the within-person analyses reported by Cernat et al (2016).

Our study suffers from several limitations. First, this is not a pure mode experiment. We are comparing a sequential mix-mode approach (with the majority completing the survey online and the balance completing a telephone interview) with a telephone-only mode. Our ITT analysis is appropriate given the inferences we want to make, but this design does not allow us to isolate the effect of mode of administration. It is possible that the inclusion of telephone cases in the web-assigned group (making up about 21% of cases) could dampen the detection of mode differences. As a check, we repeated several of the analyses using actual mode of interview (rather than assigned mode), and our overall conclusions remain unchanged.

Second, a number of key groups were excluded from the mixed-mode experiment. The study was restricted to those who were self-respondents in the previous wave (that is, proxies were excluded) and to those who reported having internet access in the previous wave. Including both of these groups would probably lead to a larger proportion of telephone interviews in the sequential mixed-mode condition. Thus, we must be careful in making generalisations to the full HRS population, or even more so to the entire US adult population.

Third, this study is embedded in a longitudinal study. HRS respondents are already familiar with the study, having responded in previous waves of data collection. They are generally well-motivated and well-incentivised. This may mitigate the impact of mode on any responses differences.

Fourth, this paper is based on a secondary analysis of existing data. The items we examined were not designed explicitly to test the specific hypotheses about mode effects. Rather this could be viewed as a real-world comparison of an existing survey under realistic design conditions.

Finally, we focused on cross-sectional comparisons of the two mode groups. In part the complex design of the HRS with a FTF interview every other wave makes it more difficult to explore cumulative effects of mode changes on longitudinal estimates of change. We did not examine mode differences between telephone and FTF interviews, or between web and FTF modes.

Despite these limitations, the controlled experimental nature of the design allows us to examine the effect of assigned mode on responses to several types of key items asked in the HRS.

Overall, the effects we find comparing a sequential mixed-mode approach with a telephone-only mode are generally modest. This is consistent with findings from other large studies (most notably, Understanding Society in the UK) transitioning from a single mode of data collection to a mixed-mode design. Given this, finding ways to remediate the differences in item-missing data and focal values should be a focus of future research. Targeted probes to encourage complete responses and use of non-focal values have already been tested elsewhere (see, for example, Al Baghal and Lynn, 2015), and are a promising avenue to pursue.

HRS plans to continue this sequential mixed-mode design for the half-sample that is not receiving an in-person interview and who have internet access in each wave going forward. In addition, HRS now offers web as a last resort follow-up strategy to participants in the other half of the sample who are resistant to an in-person interview.

Generally, we view our results as somewhat reassuring that measurement differences in switching to a mixed-mode design for a survey like the HRS are unlikely to produce serious distortions in estimates of key outcomes. Nonetheless, we urge designers and producers of longitudinal survey data to continue to evaluate the effects of mixed-mode approaches on a variety of different quality indicators and substantive variables.

Notes

1

To simplify protocols, the sample was restricted to English-speaking participants who were self-respondents (that is, proxy interviews were excluded) in their most recent interview and who had completed at least two interviews prior to 2018.

2

The assignment was made at the household level so that both participants in coupled households were assigned the same mode.

3

Our analysis sample of 2,834 excludes a small subsample of respondents (80 respondents in total; 54 in the web-assigned group, 26 in the control group) who were assigned to an overlapping experiment that may have affected response rates differently among web-assigned and control respondents.

5

For coupled respondents, HRS designates one respondent to be the financial respondent (the one who reports being most knowledgeable about the household finances) and the other respondent as the family respondent.

6

In some instances both respondents in a coupled household are presented with the rosters. Specifically, in coupled households the person who is interviewed first is always presented with the household and child grids. If the first respondent is the family respondent, the second respondent (the financial respondent) would not see the rosters; however, if the first respondent is the financial respondent, the second respondent (family respondent) is also presented with the rosters.

Funding

The HRS is sponsored by the National Institute of Aging (grant NIA U01AG009740) and the Social Security Administration and is conducted by the University of Michigan.

Data availability

The data used in the analyses in this article are available free of charge and are distributed by the Health and Retirement Study (https://hrsonline.isr.umich.edu/). The authors take responsibility for the integrity of the data and the accuracy of the analysis.

Acknowledgements

Dr Ofstedal is former Co-Associate Director and Dr Couper is a Collaborator on the HRS project. Dr Kézdi, recently deceased, was a Co-Investigator on the HRS. Drs Ofstedal and Couper were involved in the design and implementation of the 2018 HRS web experiment, the data from which is used in this paper. We wish to thank Abdelaziz Adawe for assistance with data management and analysis of the family rosters.

We dedicate this article to the memory of Gábor, our dear colleague and co-author, who fully participated in the analysis and all phases of the writing of this article up through the initial submission, but who died before it could be published.

Conflict of interest

The authors declare that there is no conflict of interest.

References

  • Al Baghal, T. and Lynn, P. (2015) Using motivational statements in web-instrument design to reduce item-missing rates in a mixed-mode context, Public Opinion Quarterly, 79(2): 56879. doi: 10.1093/poq/nfv023

    • Search Google Scholar
    • Export Citation
  • Bianchi, A., Biffignandi, S. and Lynn, P. (2017) Web-face-to-face mixed mode design in a longitudinal survey: effects on participation rates, sample composition, and costs, Journal of Official Statistics, 33(2): 385408. doi: 10.1515/jos-2017-0019

    • Search Google Scholar
    • Export Citation
  • Bowyer, B.T. and Rogowski, J.C. (2017) Mode matters: evaluating response comparability in a mixed-mode survey, Political Science Research and Methods, 5(2): 295313. doi: 10.1017/psrm.2015.28

    • Search Google Scholar
    • Export Citation
  • Bruine de Bruin, W., van der Klaauw, W., van Rooij, M., Teppa, F. and de Vos, K. (2016) Measuring Expectations of Inflation: Effects of Survey Mode, Wording, and Opportunities to Revise, DNB Working Paper No 506, Amsterdam: De Nederlandsche Bank.

    • Search Google Scholar
    • Export Citation
  • Burton, J. and Jäckle, A. (2020) Mode Effects, Understanding Society Working Paper No 2020-05, Colchester: University of Essex, ISER.

  • Cernat, A. (2015a) The impact of mixing modes on reliability in longitudinal studies, Sociological Methods & Research, 44(3): 42757. doi: 10.1177/0049124114553802

    • Search Google Scholar
    • Export Citation
  • Cernat, A. (2015b) Impact of mixed modes on measurement errors and estimates of change in panel data, Survey Research Methods, 9(2): 8399.

    • Search Google Scholar
    • Export Citation
  • Cernat, A., Couper, M.P. and Ofstedal, M.B. (2016) Estimation of mode effects in the Health and Retirement Study using measurement models, Journal of Survey Statistics and Methodology, 4(4): 50124. doi: 10.1093/jssam/smw021

    • Search Google Scholar
    • Export Citation
  • Chang, L. and Krosnick, J.A. (2010) Comparing oral interviewing with self-administered computerized questionnaires: an experiment, Public Opinion Quarterly, 74(1): 15467. doi: 10.1093/poq/nfp090

    • Search Google Scholar
    • Export Citation
  • Cheshire, H., Ofstedal, M.B., Scholes, S. and Schröder, M. (2011) A comparison of response rates in the English Longitudinal Study of Ageing and the Health and Retirement Study, Longitudinal and Life Course Studies, 2(2): 12744.

    • Search Google Scholar
    • Export Citation
  • Couper, M.P. (2011) The future of modes of data collection, Public Opinion Quarterly, 75(5): 889908. doi: 10.1093/poq/nfr046

  • Couper, M.P. (2017) New developments in survey data collection, Annual Review of Sociology, 43: 12145. doi: 10.1146/annurev-soc-060116-053613

    • Search Google Scholar
    • Export Citation
  • Couper, M.P., Ofstedal, M.B. and Lee, S. (2013) Encouraging record use for financial asset questions in a web survey, Journal of Survey Statistics and Methodology, 1(2): 17182. doi: 10.1093/jssam/smt007

    • Search Google Scholar
    • Export Citation
  • d’Ardenne, J., Collins, D., Gray, M., Jessop, C. and Pilley, S. (2017) Assessing the Risk of Mode Effects: Review of Proposed Survey Questions for Waves 7–10 of Understanding Society, Understanding Society Working Paper No. 2017-04, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation
  • De Leeuw, E.D. (2005) To mix or not to mix data collection modes in surveys, Journal of Official Statistics, 21(2): 23355.

  • De Leeuw, E.D. (2018) Mixed mode: past, present, and future, Survey Research Methods, 12(2): 7589.

  • De Leeuw, E.D., Hox, J.J. and Boevé, A. (2016) Handling Do-not-know answers: exploring new approaches in online and mixed-mode surveys, Social Science Computer Review, 34(1): 11632. doi: 10.1177/0894439315573744

    • Search Google Scholar
    • Export Citation
  • Freedman, V.A., McGonagle, K. and Couper, M.P. (2018) Use of a targeted sequential mixed mode protocol in a nationally representative panel study, Journal of Survey Statistics and Methodology, 6(1): 98121. doi: 10.1093/jssam/smx012

    • Search Google Scholar
    • Export Citation
  • Fricker, S., Galesic, M., Tourangeau, R. and Yan, T. (2005) An experimental comparison of web and telephone surveys, Public Opinion Quarterly, 69(3): 37092. doi: 10.1093/poq/nfi027

    • Search Google Scholar
    • Export Citation
  • Goodman, A., Brown, M. Silverwood, R.J., Sakshaug, J.W., Calderwood, L., Williams, J. and Ploubidis, G.B. (2020) The Impact of Using the Web in a Mixed Mode Follow-up of a Longitudinal Birth Cohort Study: Evidence from the National Child Development Study, CLS Working Paper 2020/9, London: UCL Centre for Longitudinal Studies.

    • Search Google Scholar
    • Export Citation
  • Groves, R.M., Fowler, F.J. Jr., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2009) Survey Methodology, 2nd edn, Hoboken, NJ: Wiley.

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. (2009) Mode differences between face-to-face and web surveys: an experimental investigation of data quality and social desirability effects, International Journal of Public Opinion Research, 21(1): 11121. doi: 10.1093/ijpor/edn054

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. and Loosveldt, G. (2008) Face-to-face versus web surveying in a high-internet-coverage population: differences in response quality, Public Opinion Quarterly, 72(5): 83646. doi: 10.1093/poq/nfn045

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. and Loosveldt, G. (2011) Assessing mode effects in a national crime victimization survey using structural equation models: social desirability bias and acquiescence, Journal of Official Statistics, 27(1): 4963.

    • Search Google Scholar
    • Export Citation
  • Hope, S., Campanelli, P., Nicolaas, G., Lynn, P. and Jäckle, A. (2014) The Role of the Interviewer in Producing Mode Effects: Results from a Mixed Modes Experiment Comparing Face-to-Face, Telephone and Web Administration, Understanding Society Working Paper No 2014-20, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation
  • Huynh, K.P. and Jung, J. (2005) Subjective health expectations, Journal of Policy Modeling, 37(4): 693711. doi: 10.1016/j.jpolmod.2015.04.001

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Roberts, C. and Lynn, P. (2010) Assessing the effect of data collection mode on measurement, International Statistical Review, 78(1): 320.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Eckman, S. and Nichols, A. (2014) How do web respondents react to dependent interviewing? Confidentiality concerns and reported change compared to face-to-face administration, paper presented at the 4th International Panel Survey Methods Workshop, Ann Arbor, MI.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Lynn, P. and Burton, J. (2015) Going online with a Face-to-Face household panel: effects of a mixed mode design on item and unit Non-response, Survey Research Methods, 9(1): 5570.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Al Baghal, T., Eckman, S. and Sala, E. (2021) How to pop the question? Interviewer and respondent behaviours when measuring change with proactive dependent interviewing, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, Chichester: Wiley, pp 36898.

    • Search Google Scholar
    • Export Citation
  • Kessler, R.C., Andrews, G., Mroczek, D., Ustun, B. and Wittchen, H.U. (1998) The world health organization Composite International Diagnostic Interview Short-Form (CIDI-SF), International Journal of Methods in Psychiatric Research, 7(4): 17185. doi: 10.1002/mpr.47

    • Search Google Scholar
    • Export Citation
  • Klausch, T., Hox, J.J. and Schouten, B. (2013) Measurement effects of survey mode on the equivalence of attitudinal rating scale questions, Sociological Methods & Research, 42(3): 22763. doi: 10.1177/0049124113500480

    • Search Google Scholar
    • Export Citation
  • Kreuter, F., Presser, S. and Tourangeau, R. (2008) Social desirability bias in CATI, IVR, and web surveys the effects of mode and question sensitivity, Public Opinion Quarterly, 72(5): 84765. doi: 10.1093/poq/nfn063

    • Search Google Scholar
    • Export Citation
  • Krosnick, J.A. (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied Cognitive Psychology, 5(3): 21336. doi: 10.1002/acp.2350050305

    • Search Google Scholar
    • Export Citation
  • Liu, M. and Wang, Y. (2015) Data collection mode effect on feeling thermometer questions: a comparison of face-to-face and web surveys, Computers in Human Behavior, 48: 21218. doi: 10.1016/j.chb.2015.01.057

    • Search Google Scholar
    • Export Citation
  • Lynn, P. (2013) Alternative sequential mixed-mode designs: effects on attrition rates, attrition bias, and costs, Journal of Survey Statistics and Methodology, 1(2): 183205. doi: 10.1093/jssam/smt015

    • Search Google Scholar
    • Export Citation
  • Lynn, P. (2017) From standardised to targeted survey procedures for tackling nonresponse and attrition, Survey Research Methods, 11(1): 93103.

    • Search Google Scholar
    • Export Citation
  • Mariano, L.T. and Elliott, M.N. (2017) An item response theory approach to estimating survey mode effects: analysis of data from a randomized mode experiment, Journal of Survey Statistics and Methodology, 5(2): 23353. doi: 10.1093/jssam/smw033

    • Search Google Scholar
    • Export Citation
  • McGonagle, K.A., Freedman, V.A. and Griffin, J. (2017) Web Development in the PSID: Translation & Testing of a Web Version of the 2015 PSID Telephone Instrument, PSID Technical Series Paper No 17-02, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation
  • Milton, A.C., Ellis, L.A., Davenport, T.A., Burns, J.M. and Hickie, I.B. (2017) Comparison of Self-reported telephone interviewing and Web-based survey responses: findings from the second Australian Young and Well National Survey, JMIR Mental Health, 4(3): art e37, https://mental.jmir.org/2017/3/e37/. doi: 10.2196/mental.8222

    • Search Google Scholar
    • Export Citation
  • Ofstedal, M.B., McClain, C.A. and Couper, M.P. (2021) Measuring cognition in a multi-mode context, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, New York: Wiley, pp 25071.

    • Search Google Scholar
    • Export Citation
  • Radloff, L.S. (1977) The CES-D scale: a self-report depression scale for research in the general population, Applied Psychological Measurement, 1(3): 385401. doi: 10.1177/014662167700100306

    • Search Google Scholar
    • Export Citation
  • Steffick, D.E. (2000) Documentation of affective functioning measures in the health and retirement study, HRS Documentation Report DR-005, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation
  • Tourangeau, R., Rips, L.J. and Rasinski, K. (2000) The Psychology of Survey Response, Cambridge: Cambridge University Press.

  • Williams, D. and Brick, J.M. (2018) Trends in U.S. Face-to-Face household survey nonresponse and level of effort, Journal of Survey Statistics and Methodology, 6(2): 186211. doi: 10.1093/jssam/smx019

    • Search Google Scholar
    • Export Citation
  • Zhang, X.C., Kuchinke, L., Woud, M.L., Velten, J. and Margraf, J. (2017) Survey method matters: online/offline questionnaires and face-to-face or telephone interviews differ, Computers in Human Behavior, 71: 17280. doi: 10.1016/j.chb.2017.02.006

    • Search Google Scholar
    • Export Citation

Appendix

Table A1:

Quality of asset responses: average intent-to-treat effect estimates, 2018 (linear regression results; all 16 asset and debt items)

(1)(2)(3)(4)(5)
OwnershipValue
Mode assignment in 2018Don’t know or refusedExact numberComplete bracketIncomplete bracketNo value
Web0.016** (0.004)−0.038** (0.012)0.009 (0.008)0.003 (0.004)0.026** (0.009)
Constant0.012** (0.002)0.859** (0.009)0.069** (0.006)0.016** (0.003)0.056** (0.006)
Observations1,9461,9061,9061,9061,906
R-squared0.0070.0040.0010.0000.004
control group mean0.0120.8590.0690.0160.056
control group sd0.0630.2490.1580.0720.171

Notes:

Standard errors in parentheses.

** p < .01, * p < .05

Table A2:

Quality of asset responses: pre-treatment effect estimates, 2016 (linear regression results)

(1)(2)(3)(4)(5)
OwnershipValue
Mode assignment in 2018Don’t know or refusedExact numberComplete bracketIncomplete bracketNo value
Web0.000 (0.003)0.003 (0.012)−0.005 (0.008)0.001 (0.002)0.002 (0.007)
Constant0.011** (0.002)0.866** (0.009)0.077** (0.006)0.007** (0.002)0.044** (0.006)
Observations1,8661,8481,8481,8481,848
R-squared0.0000.0000.0000.0000.000
control group mean0.0110.8660.0770.0070.044
control group sd0.0660.2490.1660.0490.150

Notes:

Standard errors in parentheses.

** p < .01, * p < .05

Table A3:

Quality of subjective probability responses using all items: average intent-to-treat effect estimates, 2018 (linear regression results)

(1)(2)(3)(4)
Mode assignment in 2018Proportion of don’t know or refused responsesProportion of value responses that are
non-focal50%0% or 100%
Web0.040** (0.006)−0.079** (0.009)0.029** (0.005)0.050** (0.008)
Constant0.022** (0.003)0.568** (0.007)0.167** (0.004)0.265** (0.006)
Observations2,7912,7382,7382,738
R-squared0.0150.0270.0110.014
control group mean0.0220.5680.1670.265
control group sd0.0920.2280.1220.201

Notes:

Non-focal responses are all valid responses except 0%, 50%, and 100%.

Standard errors clustered by household in parentheses.

** p < .01, * p < .05

Table A4:

Quality of subjective probability responses: pre-treatment effect estimates, 2016 (linear regression results)

(1)(2)(3)(4)
Mode assignment in 2018Proportion of don’t know or refused responsesProportion of value responses that are
non-focal50%0% or 100%
Web0.003 (0.004)0.009 (0.011)0.000 (0.008)–0.009 (0.010)
Constant0.022** (0.003)0.534** (0.009)0.213** (0.006)0.253** (0.008)
Observations2,7572,7472,7472,747
R-squared0.0000.0000.0000.000
control group mean0.0220.5340.2130.253
control group sd0.0870.2850.1970.243

Notes:

Non-focal responses are all valid responses except 0%, 50%, and 100%.

Standard errors clustered by household in parentheses.

** p < .01, * p < .05

Table A5:

Quality of physical and psychological health measures: pre-treatment effect estimates, 2016 (linear regression results)

Proportion of don’t know or refused responses
Mode assignment in 2018(1)(2)(3)(4)
HealthDisabilityPsychological health# disputed conditions
Web.0004 (.0005).0011 (.0008).0002 (.0008).0015 (.0096)
Constant0.0838** (.0004).0005 (.0007).0013* (.0006).0548** (.0075)
Observations2,7712,7712,7602,762
R-squared.0003.0006.0000.0000
Control group mean.0838.0005.0013.0548
Control group sd.0118.0062.0129.2358

Notes:

Standard errors in parentheses.

** p < .01 * p < .05

Table A6:

Asset ownership: average intent-to-treat effect estimates, 2018 (linear probability models)

(1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)
Mode assignment in 2018Real estateBusinessStocksBondsChecking accountCDsVehiclesOther assetsDebtHomeSecond homeOwnership count
Web−0.012 (0.016)−0.004 (0.013)−0.030 (0.021)−0.016 (0.011)−0.083** (0.018)0.004 (0.014)−0.022 (0.017)−0.027 (0.019)0.029 (0.022)−0.014 (0.020)−0.035* (0.018)−0.272** (0.075)
Constant0.150** (0.013)0.084** (0.010)0.282** (0.017)0.066** (0.009)0.853** (0.013)0.101** (0.011)0.858** (0.013)0.199** (0.015)0.349** (0.017)0.786** (0.015)0.183** (0.014)3.863** (0.057)
Observations1,9221,9301,8721,8811,8841,8531,9171,8731,9021,9091,9301,946
R-squared0.0000.0000.0010.0010.0100.0000.0010.0010.0010.0000.0020.007

Notes:

Standard errors in parentheses.

** p < .01 * p < .05

Table A7:

Asset values (thousand USD): average intent-to-treat effect estimates, 2018 (exact number responses only)

(1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)
Mode assignment in 2018Real estateBusinessStocksBondsChecking accountCDsVehiclesOther assetsDebtHomeSecond homeExact values total
Web46.3 (98.7)−140.8 (169.3)−41.2 (111.8)287.0 (295.0)−27.0 (29.4)12.8 (23.6)−0.9 (1.6)−55.3 (44.4)4.7 (2.9)−1.5 (16.7)37.4 (31.3)−83.1 (50.2)
Constant321.0** (64.3)456.7** (143.4)462.6** (78.7)183.9** (41.3)74.5* (29.1)75.8** (15.4)21.6** (1.2)142.3** (41.3)13.7** (2.0)314.9** (12.8)172.7** (19.0)531.8** (41.0)
Observations218111352671,2581561,3922546361,3862591,946
R-squared0.00.00.00.00.00.00.00.00.00.00.00.0

Notes:

Standard errors in parentheses.

** p < .01 * p < .05

Table A8:

Physical and psychological health responses: pre-treatment effect estimates, 2016 (linear regression results)

Mode assignment in 2018(1)(2)(3)(4)(5)(6)(7)
Total # conditions# incident conditionsSelf-rated health# PF limitations# IADL limitations# CES-D symptomsLife satisfaction
Web.0054 (.0632).0145 (.0207)−.0386 (.0369)−.0230 (.0651).0044 (.0193)−.0524 (.0665).0007 (.0307)
Constant2.2835** (.0494).2400** (.0162)2.6226** (.0288)1.5939** (.0508).1427** (.0151)1.0690** (.0520)2.0640** (.0239)
Observations2,7322,6792,7712,7542,7652,7512,763
R-squared.0000.0002.0004.0000.0000.0002.0000
Control group mean2.2835.24002.62261.5939.14271.06902.0640
Control group sd1.5726.5015.93191.6821.45591.7103.7891

Notes:

Standard errors in parentheses.

** p < .01 * p < .05

  • View in gallery

    Measures of data quality and response distributions, by survey domain

  • Al Baghal, T. and Lynn, P. (2015) Using motivational statements in web-instrument design to reduce item-missing rates in a mixed-mode context, Public Opinion Quarterly, 79(2): 56879. doi: 10.1093/poq/nfv023

    • Search Google Scholar
    • Export Citation
  • Bianchi, A., Biffignandi, S. and Lynn, P. (2017) Web-face-to-face mixed mode design in a longitudinal survey: effects on participation rates, sample composition, and costs, Journal of Official Statistics, 33(2): 385408. doi: 10.1515/jos-2017-0019

    • Search Google Scholar
    • Export Citation
  • Bowyer, B.T. and Rogowski, J.C. (2017) Mode matters: evaluating response comparability in a mixed-mode survey, Political Science Research and Methods, 5(2): 295313. doi: 10.1017/psrm.2015.28

    • Search Google Scholar
    • Export Citation
  • Bruine de Bruin, W., van der Klaauw, W., van Rooij, M., Teppa, F. and de Vos, K. (2016) Measuring Expectations of Inflation: Effects of Survey Mode, Wording, and Opportunities to Revise, DNB Working Paper No 506, Amsterdam: De Nederlandsche Bank.

    • Search Google Scholar
    • Export Citation
  • Burton, J. and Jäckle, A. (2020) Mode Effects, Understanding Society Working Paper No 2020-05, Colchester: University of Essex, ISER.

  • Cernat, A. (2015a) The impact of mixing modes on reliability in longitudinal studies, Sociological Methods & Research, 44(3): 42757. doi: 10.1177/0049124114553802

    • Search Google Scholar
    • Export Citation
  • Cernat, A. (2015b) Impact of mixed modes on measurement errors and estimates of change in panel data, Survey Research Methods, 9(2): 8399.

    • Search Google Scholar
    • Export Citation
  • Cernat, A., Couper, M.P. and Ofstedal, M.B. (2016) Estimation of mode effects in the Health and Retirement Study using measurement models, Journal of Survey Statistics and Methodology, 4(4): 50124. doi: 10.1093/jssam/smw021

    • Search Google Scholar
    • Export Citation
  • Chang, L. and Krosnick, J.A. (2010) Comparing oral interviewing with self-administered computerized questionnaires: an experiment, Public Opinion Quarterly, 74(1): 15467. doi: 10.1093/poq/nfp090

    • Search Google Scholar
    • Export Citation
  • Cheshire, H., Ofstedal, M.B., Scholes, S. and Schröder, M. (2011) A comparison of response rates in the English Longitudinal Study of Ageing and the Health and Retirement Study, Longitudinal and Life Course Studies, 2(2): 12744.

    • Search Google Scholar
    • Export Citation
  • Couper, M.P. (2011) The future of modes of data collection, Public Opinion Quarterly, 75(5): 889908. doi: 10.1093/poq/nfr046

  • Couper, M.P. (2017) New developments in survey data collection, Annual Review of Sociology, 43: 12145. doi: 10.1146/annurev-soc-060116-053613

    • Search Google Scholar
    • Export Citation
  • Couper, M.P., Ofstedal, M.B. and Lee, S. (2013) Encouraging record use for financial asset questions in a web survey, Journal of Survey Statistics and Methodology, 1(2): 17182. doi: 10.1093/jssam/smt007

    • Search Google Scholar
    • Export Citation
  • d’Ardenne, J., Collins, D., Gray, M., Jessop, C. and Pilley, S. (2017) Assessing the Risk of Mode Effects: Review of Proposed Survey Questions for Waves 7–10 of Understanding Society, Understanding Society Working Paper No. 2017-04, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation
  • De Leeuw, E.D. (2005) To mix or not to mix data collection modes in surveys, Journal of Official Statistics, 21(2): 23355.

  • De Leeuw, E.D. (2018) Mixed mode: past, present, and future, Survey Research Methods, 12(2): 7589.

  • De Leeuw, E.D., Hox, J.J. and Boevé, A. (2016) Handling Do-not-know answers: exploring new approaches in online and mixed-mode surveys, Social Science Computer Review, 34(1): 11632. doi: 10.1177/0894439315573744

    • Search Google Scholar
    • Export Citation
  • Freedman, V.A., McGonagle, K. and Couper, M.P. (2018) Use of a targeted sequential mixed mode protocol in a nationally representative panel study, Journal of Survey Statistics and Methodology, 6(1): 98121. doi: 10.1093/jssam/smx012

    • Search Google Scholar
    • Export Citation
  • Fricker, S., Galesic, M., Tourangeau, R. and Yan, T. (2005) An experimental comparison of web and telephone surveys, Public Opinion Quarterly, 69(3): 37092. doi: 10.1093/poq/nfi027

    • Search Google Scholar
    • Export Citation
  • Goodman, A., Brown, M. Silverwood, R.J., Sakshaug, J.W., Calderwood, L., Williams, J. and Ploubidis, G.B. (2020) The Impact of Using the Web in a Mixed Mode Follow-up of a Longitudinal Birth Cohort Study: Evidence from the National Child Development Study, CLS Working Paper 2020/9, London: UCL Centre for Longitudinal Studies.

    • Search Google Scholar
    • Export Citation
  • Groves, R.M., Fowler, F.J. Jr., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2009) Survey Methodology, 2nd edn, Hoboken, NJ: Wiley.

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. (2009) Mode differences between face-to-face and web surveys: an experimental investigation of data quality and social desirability effects, International Journal of Public Opinion Research, 21(1): 11121. doi: 10.1093/ijpor/edn054

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. and Loosveldt, G. (2008) Face-to-face versus web surveying in a high-internet-coverage population: differences in response quality, Public Opinion Quarterly, 72(5): 83646. doi: 10.1093/poq/nfn045

    • Search Google Scholar
    • Export Citation
  • Heerwegh, D. and Loosveldt, G. (2011) Assessing mode effects in a national crime victimization survey using structural equation models: social desirability bias and acquiescence, Journal of Official Statistics, 27(1): 4963.

    • Search Google Scholar
    • Export Citation
  • Hope, S., Campanelli, P., Nicolaas, G., Lynn, P. and Jäckle, A. (2014) The Role of the Interviewer in Producing Mode Effects: Results from a Mixed Modes Experiment Comparing Face-to-Face, Telephone and Web Administration, Understanding Society Working Paper No 2014-20, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation
  • Huynh, K.P. and Jung, J. (2005) Subjective health expectations, Journal of Policy Modeling, 37(4): 693711. doi: 10.1016/j.jpolmod.2015.04.001

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Roberts, C. and Lynn, P. (2010) Assessing the effect of data collection mode on measurement, International Statistical Review, 78(1): 320.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Eckman, S. and Nichols, A. (2014) How do web respondents react to dependent interviewing? Confidentiality concerns and reported change compared to face-to-face administration, paper presented at the 4th International Panel Survey Methods Workshop, Ann Arbor, MI.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Lynn, P. and Burton, J. (2015) Going online with a Face-to-Face household panel: effects of a mixed mode design on item and unit Non-response, Survey Research Methods, 9(1): 5570.

    • Search Google Scholar
    • Export Citation
  • Jäckle, A., Al Baghal, T., Eckman, S. and Sala, E. (2021) How to pop the question? Interviewer and respondent behaviours when measuring change with proactive dependent interviewing, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, Chichester: Wiley, pp 36898.

    • Search Google Scholar
    • Export Citation
  • Kessler, R.C., Andrews, G., Mroczek, D., Ustun, B. and Wittchen, H.U. (1998) The world health organization Composite International Diagnostic Interview Short-Form (CIDI-SF), International Journal of Methods in Psychiatric Research, 7(4): 17185. doi: 10.1002/mpr.47

    • Search Google Scholar
    • Export Citation
  • Klausch, T., Hox, J.J. and Schouten, B. (2013) Measurement effects of survey mode on the equivalence of attitudinal rating scale questions, Sociological Methods & Research, 42(3): 22763. doi: 10.1177/0049124113500480

    • Search Google Scholar
    • Export Citation
  • Kreuter, F., Presser, S. and Tourangeau, R. (2008) Social desirability bias in CATI, IVR, and web surveys the effects of mode and question sensitivity, Public Opinion Quarterly, 72(5): 84765. doi: 10.1093/poq/nfn063

    • Search Google Scholar
    • Export Citation
  • Krosnick, J.A. (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied Cognitive Psychology, 5(3): 21336. doi: 10.1002/acp.2350050305

    • Search Google Scholar
    • Export Citation
  • Liu, M. and Wang, Y. (2015) Data collection mode effect on feeling thermometer questions: a comparison of face-to-face and web surveys, Computers in Human Behavior, 48: 21218. doi: 10.1016/j.chb.2015.01.057

    • Search Google Scholar
    • Export Citation
  • Lynn, P. (2013) Alternative sequential mixed-mode designs: effects on attrition rates, attrition bias, and costs, Journal of Survey Statistics and Methodology, 1(2): 183205. doi: 10.1093/jssam/smt015

    • Search Google Scholar
    • Export Citation
  • Lynn, P. (2017) From standardised to targeted survey procedures for tackling nonresponse and attrition, Survey Research Methods, 11(1): 93103.

    • Search Google Scholar
    • Export Citation
  • Mariano, L.T. and Elliott, M.N. (2017) An item response theory approach to estimating survey mode effects: analysis of data from a randomized mode experiment, Journal of Survey Statistics and Methodology, 5(2): 23353. doi: 10.1093/jssam/smw033

    • Search Google Scholar
    • Export Citation
  • McGonagle, K.A., Freedman, V.A. and Griffin, J. (2017) Web Development in the PSID: Translation & Testing of a Web Version of the 2015 PSID Telephone Instrument, PSID Technical Series Paper No 17-02, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation
  • Milton, A.C., Ellis, L.A., Davenport, T.A., Burns, J.M. and Hickie, I.B. (2017) Comparison of Self-reported telephone interviewing and Web-based survey responses: findings from the second Australian Young and Well National Survey, JMIR Mental Health, 4(3): art e37, https://mental.jmir.org/2017/3/e37/. doi: 10.2196/mental.8222

    • Search Google Scholar
    • Export Citation
  • Ofstedal, M.B., McClain, C.A. and Couper, M.P. (2021) Measuring cognition in a multi-mode context, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, New York: Wiley, pp 25071.

    • Search Google Scholar
    • Export Citation
  • Radloff, L.S. (1977) The CES-D scale: a self-report depression scale for research in the general population, Applied Psychological Measurement, 1(3): 385401. doi: 10.1177/014662167700100306

    • Search Google Scholar
    • Export Citation
  • Steffick, D.E. (2000) Documentation of affective functioning measures in the health and retirement study, HRS Documentation Report DR-005, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation
  • Tourangeau, R., Rips, L.J. and Rasinski, K. (2000) The Psychology of Survey Response, Cambridge: Cambridge University Press.

  • Williams, D. and Brick, J.M. (2018) Trends in U.S. Face-to-Face household survey nonresponse and level of effort, Journal of Survey Statistics and Methodology, 6(2): 186211. doi: 10.1093/jssam/smx019

    • Search Google Scholar
    • Export Citation
  • Zhang, X.C., Kuchinke, L., Woud, M.L., Velten, J. and Margraf, J. (2017) Survey method matters: online/offline questionnaires and face-to-face or telephone interviews differ, Computers in Human Behavior, 71: 17280. doi: 10.1016/j.chb.2017.02.006

    • Search Google Scholar
    • Export Citation

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 2 2 2
Full Text Views 68 68 68
PDF Downloads 32 32 32

Altmetrics

Dimensions