Cover Longitudinal and Life Course Studies
Longitudinal and Life Course Studies
An international journal

Data quality and response distributions in a mixed-mode survey

Authors: Mary Beth Ofstedal1, Gábor Kézdi1, and Mick P. Couper1
View author details View Less
  • 1 University of Michigan, , USA
Article Category:
Research Article
Copyright:
© Authors 2022
Online Publication Date:
01 May 2022
Pages:
1–26
Publisher:
Bristol University Press
Volume/Issue:
Early View
DOI:
https://doi.org/10.1332/175795921X16494126913909
Restricted access
Get eTOC alerts
Rights and permissions Cite this article
Add to cart (GBP  £25.00)

Longitudinal surveys traditionally conducted by interviewers are facing increasing pressures to explore alternatives such as sequential mixed-mode designs, which start with a cheaper self-administered mode (online) then follow up using more expensive methods such as telephone or face-to-face interviewing. Using a designed experiment conducted as part of the 2018 wave of the Health and Retirement Study (HRS) in the US, we compare a sequential mixed-mode design (web then telephone) with the standard telephone-only protocol. Using an intent-to-treat analysis, we focus on response quality and response distributions for several domains key to HRS: physical and psychological health, financial status, expectations and family composition.

Respondents assigned to the sequential mixed-mode (web) had slightly higher missing data rates and more focal responses than those assigned to telephone-only. However, we find no evidence of differential quality in verifying and updating roster information. We find slightly lower rates of asset ownership reported by those assigned to the web mode. Conditional on ownership, we find no detectable mode effects on the value of assets. We find more negative (pessimistic) expectations for those assigned to the web mode. We find little evidence of poorer health reported by those assigned to the web mode.

We find that effects of mode assignment on measurement are present, but for most indicators the effects are small. Finding ways to remediate the differences in item-missing data and focal values should help reduce mode effects in mixed-mode surveys or those transitioning from interviewer- to self-administration.

  • Al Baghal, T. and Lynn, P. (2015) Using motivational statements in web-instrument design to reduce item-missing rates in a mixed-mode context, Public Opinion Quarterly, 79(2): 568–79. doi: 10.1093/poq/nfv023

    • Search Google Scholar
    • Export Citation

  • Bianchi, A., Biffignandi, S. and Lynn, P. (2017) Web-face-to-face mixed mode design in a longitudinal survey: effects on participation rates, sample composition, and costs, Journal of Official Statistics, 33(2): 385–408. doi: 10.1515/jos-2017-0019

    • Search Google Scholar
    • Export Citation

  • Bowyer, B.T. and Rogowski, J.C. (2017) Mode matters: evaluating response comparability in a mixed-mode survey, Political Science Research and Methods, 5(2): 295–313. doi: 10.1017/psrm.2015.28

    • Search Google Scholar
    • Export Citation

  • Bruine de Bruin, W., van der Klaauw, W., van Rooij, M., Teppa, F. and de Vos, K. (2016) Measuring Expectations of Inflation: Effects of Survey Mode, Wording, and Opportunities to Revise, DNB Working Paper No 506, Amsterdam: De Nederlandsche Bank.

    • Search Google Scholar
    • Export Citation

  • Burton, J. and Jäckle, A. (2020) Mode Effects, Understanding Society Working Paper No 2020-05, Colchester: University of Essex, ISER.

  • Cernat, A. (2015a) The impact of mixing modes on reliability in longitudinal studies, Sociological Methods & Research, 44(3): 427–57. doi: 10.1177/0049124114553802

    • Search Google Scholar
    • Export Citation

  • Cernat, A. (2015b) Impact of mixed modes on measurement errors and estimates of change in panel data, Survey Research Methods, 9(2): 83–99.

    • Search Google Scholar
    • Export Citation

  • Cernat, A., Couper, M.P. and Ofstedal, M.B. (2016) Estimation of mode effects in the Health and Retirement Study using measurement models, Journal of Survey Statistics and Methodology, 4(4): 501–24. doi: 10.1093/jssam/smw021

    • Search Google Scholar
    • Export Citation

  • Chang, L. and Krosnick, J.A. (2010) Comparing oral interviewing with self-administered computerized questionnaires: an experiment, Public Opinion Quarterly, 74(1): 154–67. doi: 10.1093/poq/nfp090

    • Search Google Scholar
    • Export Citation

  • Cheshire, H., Ofstedal, M.B., Scholes, S. and Schröder, M. (2011) A comparison of response rates in the English Longitudinal Study of Ageing and the Health and Retirement Study, Longitudinal and Life Course Studies, 2(2): 127–44.

    • Search Google Scholar
    • Export Citation

  • Couper, M.P. (2011) The future of modes of data collection, Public Opinion Quarterly, 75(5): 889–908. doi: 10.1093/poq/nfr046

  • Couper, M.P. (2017) New developments in survey data collection, Annual Review of Sociology, 43: 121–45. doi: 10.1146/annurev-soc-060116-053613

    • Search Google Scholar
    • Export Citation

  • Couper, M.P., Ofstedal, M.B. and Lee, S. (2013) Encouraging record use for financial asset questions in a web survey, Journal of Survey Statistics and Methodology, 1(2): 171–82. doi: 10.1093/jssam/smt007

    • Search Google Scholar
    • Export Citation

  • d’Ardenne, J., Collins, D., Gray, M., Jessop, C. and Pilley, S. (2017) Assessing the Risk of Mode Effects: Review of Proposed Survey Questions for Waves 7–10 of Understanding Society, Understanding Society Working Paper No. 2017-04, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation

  • De Leeuw, E.D. (2005) To mix or not to mix data collection modes in surveys, Journal of Official Statistics, 21(2): 233–55.

  • De Leeuw, E.D. (2018) Mixed mode: past, present, and future, Survey Research Methods, 12(2): 75–89.

  • De Leeuw, E.D., Hox, J.J. and Boevé, A. (2016) Handling Do-not-know answers: exploring new approaches in online and mixed-mode surveys, Social Science Computer Review, 34(1): 116–32. doi: 10.1177/0894439315573744

    • Search Google Scholar
    • Export Citation

  • Freedman, V.A., McGonagle, K. and Couper, M.P. (2018) Use of a targeted sequential mixed mode protocol in a nationally representative panel study, Journal of Survey Statistics and Methodology, 6(1): 98–121. doi: 10.1093/jssam/smx012

    • Search Google Scholar
    • Export Citation

  • Fricker, S., Galesic, M., Tourangeau, R. and Yan, T. (2005) An experimental comparison of web and telephone surveys, Public Opinion Quarterly, 69(3): 370–92. doi: 10.1093/poq/nfi027

    • Search Google Scholar
    • Export Citation

  • Goodman, A., Brown, M. Silverwood, R.J., Sakshaug, J.W., Calderwood, L., Williams, J. and Ploubidis, G.B. (2020) The Impact of Using the Web in a Mixed Mode Follow-up of a Longitudinal Birth Cohort Study: Evidence from the National Child Development Study, CLS Working Paper 2020/9, London: UCL Centre for Longitudinal Studies.

    • Search Google Scholar
    • Export Citation

  • Groves, R.M., Fowler, F.J. Jr., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2009) Survey Methodology, 2nd edn, Hoboken, NJ: Wiley.

    • Search Google Scholar
    • Export Citation

  • Heerwegh, D. (2009) Mode differences between face-to-face and web surveys: an experimental investigation of data quality and social desirability effects, International Journal of Public Opinion Research, 21(1): 111–21. doi: 10.1093/ijpor/edn054

    • Search Google Scholar
    • Export Citation

  • Heerwegh, D. and Loosveldt, G. (2008) Face-to-face versus web surveying in a high-internet-coverage population: differences in response quality, Public Opinion Quarterly, 72(5): 836–46. doi: 10.1093/poq/nfn045

    • Search Google Scholar
    • Export Citation

  • Heerwegh, D. and Loosveldt, G. (2011) Assessing mode effects in a national crime victimization survey using structural equation models: social desirability bias and acquiescence, Journal of Official Statistics, 27(1): 49–63.

    • Search Google Scholar
    • Export Citation

  • Hope, S., Campanelli, P., Nicolaas, G., Lynn, P. and Jäckle, A. (2014) The Role of the Interviewer in Producing Mode Effects: Results from a Mixed Modes Experiment Comparing Face-to-Face, Telephone and Web Administration, Understanding Society Working Paper No 2014-20, Colchester: University of Essex, ISER.

    • Search Google Scholar
    • Export Citation

  • Huynh, K.P. and Jung, J. (2005) Subjective health expectations, Journal of Policy Modeling, 37(4): 693–711. doi: 10.1016/j.jpolmod.2015.04.001

    • Search Google Scholar
    • Export Citation

  • Jäckle, A., Roberts, C. and Lynn, P. (2010) Assessing the effect of data collection mode on measurement, International Statistical Review, 78(1): 3–20.

    • Search Google Scholar
    • Export Citation

  • Jäckle, A., Eckman, S. and Nichols, A. (2014) How do web respondents react to dependent interviewing? Confidentiality concerns and reported change compared to face-to-face administration, paper presented at the 4th International Panel Survey Methods Workshop, Ann Arbor, MI.

    • Search Google Scholar
    • Export Citation

  • Jäckle, A., Lynn, P. and Burton, J. (2015) Going online with a Face-to-Face household panel: effects of a mixed mode design on item and unit Non-response, Survey Research Methods, 9(1): 55–70.

    • Search Google Scholar
    • Export Citation

  • Jäckle, A., Al Baghal, T., Eckman, S. and Sala, E. (2021) How to pop the question? Interviewer and respondent behaviours when measuring change with proactive dependent interviewing, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, Chichester: Wiley, pp 368–98.

    • Search Google Scholar
    • Export Citation

  • Kessler, R.C., Andrews, G., Mroczek, D., Ustun, B. and Wittchen, H.U. (1998) The world health organization Composite International Diagnostic Interview Short-Form (CIDI-SF), International Journal of Methods in Psychiatric Research, 7(4): 171–85. doi: 10.1002/mpr.47

    • Search Google Scholar
    • Export Citation

  • Klausch, T., Hox, J.J. and Schouten, B. (2013) Measurement effects of survey mode on the equivalence of attitudinal rating scale questions, Sociological Methods & Research, 42(3): 227–63. doi: 10.1177/0049124113500480

    • Search Google Scholar
    • Export Citation

  • Kreuter, F., Presser, S. and Tourangeau, R. (2008) Social desirability bias in CATI, IVR, and web surveys the effects of mode and question sensitivity, Public Opinion Quarterly, 72(5): 847–65. doi: 10.1093/poq/nfn063

    • Search Google Scholar
    • Export Citation

  • Krosnick, J.A. (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied Cognitive Psychology, 5(3): 213–36. doi: 10.1002/acp.2350050305

    • Search Google Scholar
    • Export Citation

  • Liu, M. and Wang, Y. (2015) Data collection mode effect on feeling thermometer questions: a comparison of face-to-face and web surveys, Computers in Human Behavior, 48: 212–18. doi: 10.1016/j.chb.2015.01.057

    • Search Google Scholar
    • Export Citation

  • Lynn, P. (2013) Alternative sequential mixed-mode designs: effects on attrition rates, attrition bias, and costs, Journal of Survey Statistics and Methodology, 1(2): 183–205. doi: 10.1093/jssam/smt015

    • Search Google Scholar
    • Export Citation

  • Lynn, P. (2017) From standardised to targeted survey procedures for tackling nonresponse and attrition, Survey Research Methods, 11(1): 93–103.

    • Search Google Scholar
    • Export Citation

  • Mariano, L.T. and Elliott, M.N. (2017) An item response theory approach to estimating survey mode effects: analysis of data from a randomized mode experiment, Journal of Survey Statistics and Methodology, 5(2): 233–53. doi: 10.1093/jssam/smw033

    • Search Google Scholar
    • Export Citation

  • McGonagle, K.A., Freedman, V.A. and Griffin, J. (2017) Web Development in the PSID: Translation & Testing of a Web Version of the 2015 PSID Telephone Instrument, PSID Technical Series Paper No 17-02, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation

  • Milton, A.C., Ellis, L.A., Davenport, T.A., Burns, J.M. and Hickie, I.B. (2017) Comparison of Self-reported telephone interviewing and Web-based survey responses: findings from the second Australian Young and Well National Survey, JMIR Mental Health, 4(3): art e37, https://mental.jmir.org/2017/3/e37/. doi: 10.2196/mental.8222

    • Search Google Scholar
    • Export Citation

  • Ofstedal, M.B., McClain, C.A. and Couper, M.P. (2021) Measuring cognition in a multi-mode context, in P. Lynn (ed) Advances in Longitudinal Survey Methodology, New York: Wiley, pp 250–71.

    • Search Google Scholar
    • Export Citation

  • Radloff, L.S. (1977) The CES-D scale: a self-report depression scale for research in the general population, Applied Psychological Measurement, 1(3): 385–401. doi: 10.1177/014662167700100306

    • Search Google Scholar
    • Export Citation

  • Steffick, D.E. (2000) Documentation of affective functioning measures in the health and retirement study, HRS Documentation Report DR-005, Ann Arbor, MI: University of Michigan, Institute for Social Research.

    • Search Google Scholar
    • Export Citation

  • Tourangeau, R., Rips, L.J. and Rasinski, K. (2000) The Psychology of Survey Response, Cambridge: Cambridge University Press.

  • Williams, D. and Brick, J.M. (2018) Trends in U.S. Face-to-Face household survey nonresponse and level of effort, Journal of Survey Statistics and Methodology, 6(2): 186–211. doi: 10.1093/jssam/smx019

    • Search Google Scholar
    • Export Citation

  • Zhang, X.C., Kuchinke, L., Woud, M.L., Velten, J. and Margraf, J. (2017) Survey method matters: online/offline questionnaires and face-to-face or telephone interviews differ, Computers in Human Behavior, 71: 172–80. doi: 10.1016/j.chb.2017.02.006

    • Search Google Scholar
    • Export Citation

Content Metrics

May 2022 onwards Past Year Past 30 Days
Abstract Views 30 30 28
Full Text Views 98 98 7
PDF Downloads 64 64 9

Altmetrics

Dimensions