Computer assisted telephone interviewing (CATI) for health surveys in public health surveillance:

Methodological issues and challenges ahead

Vol. 25 No. 2, 2004

Bernard CK Choi

Abstract

This article describes methodological issues, challenges, and a vision for using computer assisted telephone interviewing (CATI) in a comprehensive public health surveillance system in the 21st century. Methodological issues include funding of surveys, survey frequency, sample size considerations, response rates, and types of bias to be considered in questionnaire design. Challenges include the recognition of the merits and limitations of CATI, and the potential for greater use in surveillance of public health issues in health regions requiring rapid and regular data. The vision of a CATI survey-based, rapid, flexible, cost-effective public health surveillance system is described. It is concluded that further discussion and views on improvements with regard to CATI methodological and practical issues will help build a better CATI survey-based public health surveillance system for the future.

Key words: CATI; computer assisted telephone interviewing; methodology; public health surveillance; questionnaire bias; survey

Introduction

In the development of methods for comprehensive public health surveillance in the 21st century,^1-3 health surveys using computer assisted telephone interviewing (CATI) play an important role.⁴ CATI is a technique based on a combined use of long-standing methods, such as interview and surveys, and modern technologies, such as the telephone and the computer.⁵ It is becoming the data collection method of choice in an increasing number of health surveys.^6-8 However, CATI health surveys raise methodological issues.^9-11 The question is how to move forward with CATI health surveys in public health surveillance.

This article reviews a number of methodological issues and proposes a number of strategies to tackle the challenges ahead. It describes the usefulness of the CATI method for surveys as a component of public health surveillance system. Two important concepts are proposed to stimulate further discussion and debate on a vision of a CATI survey-based surveillance: the importance of local CATI surveys in public health surveillance and the ability to generate rapid data.

Methodological issues

1. Sources of funding

An important question concerning health surveys is: Who pays for the survey?

Frequency distributions of the funding sponsors for the 67 Canadian health surveys carried out between 1950 and 1997, derived from the results of Kendall et al.,¹² indicate that the major sources of funding were the federal government (75% of the surveys), followed by the provincial government (54%) and research institutes, foundations or societies (15%) (Table 1).

Many federal government departments besides Health Canada also funded health surveys (see the “Other” category in Table 1). Local government has traditionally not played a role in funding health surveys (0%). In terms of data collection, the provincial government played a more important role (42% of the surveys) than the federal government (30%), universities (21%) and survey contractors (6%). The management role of surveys was shared by the provincial government (42% of the surveys), the federal government (34%) and the universities (24%).

It is important to identify stakeholders and funding sources for CATI surveys. Local governments are a potential funding source that traditionally has not been tapped. Methods may be developed to better coordinate the funding efforts of all levels of government and stakeholders.

2. Frequency of surveys

Another question concerning health surveys is: How frequently should a survey be conducted?

The frequency of health surveys in Canada from 1950 to 1997 ranged from four times a year (2% of the surveys) to only once (the majority, or 61%) (Table 2). Most of the surveys collected data only once (50% of the 30 national health surveys and 70% of the 37 provincial surveys). In general, national health surveys tended to be more frequent (14% were done every year) than provincial health surveys (16% were done every five years).

Extreme survey frequency leads to the concept of continuous survey. For example, the annual sample can be spread evenly over 12 months to keep the survey going continuously. In other words, 100 interviews per month can be carried out instead of 1,200 interviews per year in a survey region. Indeed, in Cycle 1.1 of the 2000 Canadian Community Health Survey (CCHS) data were collected on a monthly basis.13 There are many advantages of a continuous survey, such as better data quality, timeliness of data, ability to detect seasonal trends, and continuous employment of staff and interviewers, which means consistency of the data. In addition, there is no increased cost; in fact, continuous surveys may lead to a decreased expenditures through elimination of the cost of advertising positions, hiring staff, and training interviewers, etc., every time the survey is restarted.

The concept of "repeated sampling of the same population" was first described by Cochran in 1963.14 With a dynamic population, a large survey at infrequent intervals is of limited use; a series of small samples at shorter intervals is more informative, and may be published regularly. With a population in which time changes are slow, on the other hand, an annual average taken over 12 monthly samples or four quarterly samples may be adequate. With just a slight change in the sample methodology, the survey becomes so much more versatile and informative.

3. Sample size

The next question is: What sample size is adequate?

In Canada, national health surveys have bigger sample sizes than provincial health surveys, e.g., 30% with samples of 10,000 to 19,999, versus 33% with samples of 2,000 to 2,999 (Table 3). However, given the fact that there are 10 provinces in Canada, the provincial health surveys may provide a larger coverage of the population than national health surveys.

Samples sizes are strongly affected by available funding. In Australia, the 1989 South Australia Health Omnibus Survey conducted 3,000 interviews/year¹⁵ and the 1997-1998 New South Wales Health Survey conducted 17,000 (1,000/year per Area Health Service).¹⁶ In the US, the 1998 US Behavioral Risk Factor Surveillance System (BRFSS) involved 120,000 participants per year (100 to 400/month per state).¹⁷ In Canada, the 1994 National Population Health Survey (NPHS) longitudinal panel had a sample size of 17,276¹⁸ and the 2000 CCHS had 130,000 respondents (2,000 to 42,000/year per province).¹³

It is easy to calculate sample size if one hypothesis is tested. However, it is difficult to perform sample size calculations for a CATI survey because there are usually many items of interest in the survey. Survey experience indicates that for most purposes a minimum sample size of 100 interviews per month per survey region is probably adequate.

4. Response rate

What response rate is adequate?

Most health surveys in Canada from 1950 to 1997 achieved a high response rate of 70% to 100% (Table 4). National health surveys in general had a higher response rate than provincial surveys. It has been suggested that the target response rate should be 80% or more.19 In Canada, the 1994 NPHS had a response rate of 96%,20 and in 1996 a response rate of 94%.21 The 2000 CCHS had a response rate of 85%.²²

There are ways to promote high response rates, such as by using trained interviewers and voices that encourage compliance (e.g., female), paying attention to speech pace and rhythm, and selecting the appropriate design and length of the questionnaire. There is usually a lower response to interviewers with "foreign" accents and young male interviewers. Questionnaires must be pretested for wording, order and misinterpretation of questions.

5. Questionnaire design

Choi and Pak provided a catalogue of 109 epidemiologic biases in biostatistics: literature review four, study design 31, study execution three, data collection 46, analysis 15, interpretation seven, and publication three.²³ Most of the biases (46/109 or 42%) occur in the data collection stage.

There are a number of questionnaire biases that specifically affect telephone interviews. For example, complex and lengthy questions should be avoided in a telephone interview. When the precise wording of a question changes from survey to survey, or the measurement scale for a quantity changes over surveys, the results may not be comparable. Some questions may be framed in such a manner that respondents are misled to a wrong choice (framing bias). The mindset of the respondents can affect their perception of questions, and therefore their answers. Questionnaires that are too long can induce fatigue in respondents and result in uniform and inaccurate answers (response fatigue bias). In a lengthy interview, e.g., a telephone interview that lasts for more than an hour, interviewees are unable to concentrate and give correct answers, and tend to say "yes" so as to quickly get through the interview (yes-saying bias).

Characteristics of CATI

In order to further advance the technique of CATI for public health surveillance, the merits and limitations must be fully appreciated (Table 5).

1. Merits of CATI

Interview

The interview is an efficient way to obtain a wide variety of information (compared with record searching, dosimeter reading, laboratory testing, etc.). It is the only way to obtain opinion and information on attitudes and, in many cases, it can collect information not available from other sources. An interview can target specific subgroups in the population (e.g., using surnames to identify ethnic groups).²⁴

Telephone

Use of the telephone is a cost-effective method that can provide access to large samples (compared with personal interview) and instant results (compared with mailed surveys). Random digit dialing (RDD) can select reasonably random samples, by reaching unlisted numbers and recent connections. Telephone interviews can facilitate responses to sensitive questions (e.g., about sexual behaviours) as compared with personal interviews and are less likely to lead to socially desirable answers.

Computer assisted

CATI can link with a Survey Management System (SMS) to log interviewer activity, schedule repeat calls, select interviewees randomly, remove numbers from the call queue, produce operational reports, and perform automatic dialing. CATI can help interviewers to do their job more efficiently by eliminating tedious paperwork and by providing on-screen instructions. For example, the screen may display previously provided names of family members so that the interviewer can refer to them by name in follow-up questions. It can facilitate data collection by randomizing the order of questions and response options, by programming edits and consistency checks into the questionnaire and by providing automatic question skips for complex questions. The CATI technique may be especially useful if the question structure is complicated or there are many possible responses. This was the case in a survey that required information about asthma medication in terms of 486 possible combinations of drug, dose and delivery systems.²⁵ Furthermore, checks can be built in and an immediate warning given if a reply lies outside an acceptable range or is inconsistent with previous replies.

CATI permits direct data entry in an electronic format (reducing processing time and costs) and therefore quick data turnaround. Coding procedures can be programmed into the computer. A number of alternative coding methods are available, ranging from a simple pick list (shown on the screen) to more sophisticated coding tools that guide the selection of an appropriate code according to pre-specified rules and which require only a few letters to be typed. This not only reduces the costs of office coding, but also allows for better data quality.

2. Limitations of CATI

Interview

Self-report in any survey is a problem. First, respondents may not understand the questions. Second, they may not be able to identify their status. For example, up to 50% of people with diabetes do not know they have diabetes.²⁶ Third, they may not want to report their status (e.g., sexual behaviours). Data collected from interview still need to be occasionally validated by record searching.

Telephone

A telephone survey is not applicable if the percentage of households owning a telephone is low, because it can only reach those who can afford a telephone.²⁷ People who normally stay at home (e.g., older or retired people, or housewives) are the ones who tend to be interviewed. RDD may reach disconnected, non-household, fax or other invalid numbers. In general, respondents' concentration and patience are shorter over the telephone than in personal interview. Therefore, long blocks of text should be avoided, concepts should be relatively simple, rating scales straightforward, response alternatives short (maximum of five or six), and the interview normally should be limited to 20 minutes. Interviewers cannot show cards, life history calendars, or other tools that they need to show the respondents during the interview. A telephone survey loses personal touch and rapport compared with a personal interview. Finally, people nowadays are tired of telephone interviews, as there are already too many consumer and market surveys.

Computer assisted

CATI requires a large cost for initial set-up (hardware, software and personnel). It requires software specifically developed for the questionnaire, and more time and effort to develop and test the questionnaire. Automatic question skips can cause problems if not fully tested. More effort is needed to train interviewers to use computers. There is a possibility of data entry errors, e.g., typing slips when entering the answers. It has been found that 2.0% of recorded responses were erroneous, as compared with 1.1% when a paper-and-pencil technique was used.²⁸

Discussion

To practise public health, we need information that is timely, appropriate, affordable, and accurate. There is great potential for CATI survey technology to address surveillance needs within a public health context. In this regard, the vision and framework of a rapid, flexible, cost-effective, and CATI survey-based public health surveillance system needs to be further developed and tested.²⁹ The feasibility of using on-going local CATI sample surveys to supplement national health surveys is worth further exploring.

Traditionally, health surveys are funded by the federal and, to a lesser extent, the provincial governments. It is important to recognize that there are shared and separate responsibilities of federal, provincial, regional and local agencies and organizations. Recent experience in Canada in the development of a Rapid Risk Factor Surveillance System (RRFSS) indicates that regional and local governments are perhaps interested in providing funding for a nation-wide local CATI-survey based surveillance system that can address regional and local needs.³⁰ Local CATI surveys have enormous potential within public health and health surveillance. Further exposition of their roles and their advantages would better serve that potential. In the US, work is underway to further develop a local and state level CATI survey-based surveillance system, the State and Local Area Integrated Telephone Survey, to supplement national CATI surveys.³¹

It is also necessary to place the use of local CATI surveys such as RRFSS in the context of the expensive and ambitious National Health Survey program which includes the CCHS being jointly funded by Statistics Canada, Health Canada, and the Canadian Institute for Health Information. For example, what is the added value of rapid health region data, given that official national data are now also available at the health region level but after a certain time lag? Are specialized CATI survey surveillance units within individual health regions necessarily the answer to obtaining critical information?

There is a question of CATI survey capacity at the local and regional level, even when local CATI surveys are justified. Given the complexity of setting up CATI surveys, is it practicable for regional health units to conduct their own CATI surveys? Or is it more cost-effective to employ specialized contractors to conduct local CATI surveys through a national or provincial coordination scheme?

At the local health region level, many health units do not have analytical or data processing resources. An example is the recent CCHS that collected data at the health region level. The CATI survey has great amounts of data that might be relevant to health regions, but many of them cannot access and analyze the data independently because of a lack of staff and resources, including hardware and software. However, these types of problems can be worked out through communications with the parties that have an interest in the data. Automated standard data analysis software, which produces a standard set of statistics, tables and graphs, may be helpful to the health regions in this regard.^1,29

One must stress the importance of having an analytical plan for the data that are collected in the local CATI surveys. There should be a clear understanding of what descriptive statistics should be compiled quickly and then a plan should be in place for more detailed analysis.¹ If the data are timely, there is a great sense of immediacy if important changes in population health are detected. Standard statistics and standard analysis protocols help to maintain the saliency and therefore the policy value of the information.

In modern society, the type of surveillance information that might be critical to the infectious disease patterns of a population is more likely to come from networks of clinicians rather than through surveys of the general client population, as in the case, for example, of West Nile virus, E. coli contamination of the Walkerton water supply and severe acute respiratory syndrome. Of course, this does not entirely diminish the importance of more general survey surveillance information. But does this mean that the local and national CATI surveys should be geared more towards chronic diseases and their risk factors?

Many national health surveys are conducted every year or every two years. In many situations, monthly data are shown to be desirable. For example, the CCHS, which started in 2000, is administered on a monthly basis.¹³ The BRFSS has been a continuous surveillance system in the US for some years.¹⁷ In Australia, NSW Health has converted its annual risk factor survey to a continuous collection system.¹⁶ Despite the ongoing debate about whether data collected more frequently than annually (e.g., monthly) are needed for chronic diseases, monthly and seasonal trends in chronic disease risk factors, e.g., physical inactivity, or low intake of fruits and vegetables, are indeed observed, according to initial analysis of the 2000 CCHS and 1999 RRFSS pilot results. Monthly data can allow better and statistically more powerful detection and evaluation of the success of public health prevention and control programs and campaigns. Further data from the monthly CCHS and RRFSS should be able to clarifiy the true benefit of monthly data compared with the traditional annual or biennial data.

Sample size is a statistical question. There are formal procedures to calculate what sample size is needed to allow the reliable detection of differences from administration to administration (or between sample subgroups within a single administration).³² However, it is very difficult to statistically estimate a sample size for CATI surveys, because there are many items of interest in the questionnaire. CATI survey sample sizes are better estimated empirically. For example, on the basis of the experience of the 1997-1998 New South Wales Health Survey,¹⁶ the 1998 US BRFSS¹⁷ and the 2000 Canadian CCHS¹³ it seems that for most purposes a sample size of 100 interviews per month per survey region is adequate.

The complex nature of survey design and the various issues associated with surveys, such as methods to increase response rate, questionnaire design, and data collection biases, are difficult to address in a short paper like this. Moser's textbook on Survey Methods in Social Investigation is over 350 pages long.³³ Interested readers are referred to a number of recent textbooks on survey methods.^34-36 The important point is that one has to be careful when conducting survey operations, especially CATI.

The use of CATI has the advantage of built-in data quality feedback. However, there is a problem of data entry errors. This implies the need to design data quality checks at the entry level. The complexity of setting up CATI surveys, including extensive programming requirements and testing, needs to be carefully tackled.

Telephone surveys have a number of difficulties. Non-operating telephone banks or telephone numbers that are assigned to businesses are one problem that leads to difficulties in obtaining samples. The issue of cellular phones is emerging as a potential problem in CATI survey design because there are cases in which every member of a family has his or her own cellular phone and there may not be a conventional telephone in the household. The existence of multiple telephone numbers relating to the same household may influence the likelihood that a household would be selected to participate in the survey. Cellular phone numbers are excluded in current survey sample frames. That implies a possibility of bias in telephone samples.

CATI health surveys also face increasing difficulties in competing with telemarketing surveys for interview time and cooperation from the general public. With the increasing number of daily CATI surveys, including health and telemarketing, the response rate to any specific CATI survey will drop. Are we nearing the end of the useful life of the telephone, and therefore up CATI survey technology?

Besides CATI surveys, there are other vehicles for obtaining health information, such as routine records, registry data, and research databases. It may be time to explore, discover, or invent new data sources to supplement CATI data with regard to health information for public health and health surveillance.

CATI technology has only been in use for several decades. It has a number of advantages to be utilized, and a number of disadvantages to be minimized. Resolving the CATI methodological issues and building a CATI survey-based public health surveillance system will help us in charting a path forward in the 21st century.

Acknowledgements

The author gratefully thanks the members of the 1999 (RRFSS) Steering Committee, and, in particular, Dr. Philippa Holowaty and Dr. Eric Holowaty, for providing relevant data from the 1999 Durham CATI health survey pilot project and technical assistance. Dr. David Mowat contributed significantly to the vision of a rapid, flexible, cost-effective, survey-based public health surveillance system.

This paper is based on an invited keynote speech at the "CATI (Computer Assisted Telephone Interviewing) Health Surveys in Australia - the Challenges Ahead" Forum, November 28, 2001, Sydney, Australia.

References

Author References

Bernard CK Choi, Centre for Chronic Disease Prevention and Control, Population and Public Health Branch, Health Canada, Ottawa, Ontario, Canada and Department of Public Health Sciences, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada and Department of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada Correspondence: Bernard CK Choi, Evidence and Information for Chronic Disease Policy Division, Centre for Chronic Disease Prevention and Control, Health Canada, PL 6701A,120 Colonnade Road, Ottawa, Ontario, Canada K1A 1B4; Fax: (613) 941-2633; E-mail: Bernard_Choi@hc-sc.gc.ca