Sources of Error in the BRFSS

Noncoverage Error: Noncoverage error occurs because not all members of the general population are capable of being included in the sample. Population groups typically excluded from most general population surveys include persons living in nonresidential settings, such as hospitals, nursing homes, prisons, military bases, and college dormitories. Compared with the size of the adult population of the state as a whole, the number of persons within the above-mentioned groups is generally small. Because the BRFSS uses telephone surveys, households without telephones are not included, making this a larger source of noncoverage error. For some populations (e.g. American Indians, rural blacks in some southern states), telephone noncoverage is much higher than for most populations. Persons without telephones tend to have lower household incomes, and low income is associated with certain health risk behaviors.

Sampling Error: Sampling error occurs because estimates are based on only a sample of the population rather than on the entire population. Strictly adhering to the BRFSS calling rules and randomly selecting a household member can avoid some sampling error.

Nonresponse Error: Nonresponse error Ė the inability to obtain data for all questionnaire items from a person in the sample population- is a common problem in surveillance work. There are two levels of nonresponse: unit nonresponse and item nonresponse. Unit nonresponse occurs when an eligible sampling unit (i.e. household, person) does not respond or a respondent refuses to participate in the survey. Item nonresponse occurs when useful data are not obtained for all questionnaire items. Because nonresponse bias is inversely related to response rate, surveys with higher response rates will generally have lower nonresponse bias.

Measurement Error: The quality of measurements in a telephone survey can be affected by the question wording, question order, response-code precision, length of interview, recall error, interviewer technique, coding errors and simple data entry error.

Quality Assurance and the BRFSS

The goal of BRFSS quality assurance activities is to ensure that the BRFSS data are collected with scientific rigor and consistency in order to provide the most accurate data possible.

Interviewer Monitoring: BRFSS supervisors routinely monitor interviews in progress. The BRFSS protocol specifies that systematic, unobtrusive electronic monitoring will be a routine and integral part of monthly survey procedures for all interviewers.

Assessing Quality Assurance Indicators: Selected statistics are compiled and reviewed at the end of each monthís data collection so that any corrective action can be taken before the next monthís data collection activities begin. Objectives have been established for the BRFSS for a number of quality assurance indicators. Some of these measures are interviewer statistics, question-response frequencies and response rates.

Data Editing: Each month the data are checked for coding errors and the data files are edited if necessary. The Computer Assisted Telephone Interviewing software system greatly reduces coding errors by automating question skips and doing internal range checks.

Weighting the BRFSS Data

The purposes of weighting the BRFSS data are to compensate for unequal probabilities of selection, to adjust for non-response and telephone non-coverage, to ensure that results are consistent with population data and to make population estimates.

BRFSS data are directly weighted for the probability of selection of a telephone number, the number of adults in a household, and the number of phones in a household. The weights for number of adults in a household and number of phones are needed because we want to make statistically valid inferences about individuals but we are sampling telephone numbers. Because only one person per household is interviewed, respondents in larger households have a smaller probability of selection than respondents in smaller households. For example, once the telephone number is selected, a person in a one-adult household has a 100% chance of being selected whereas a person in a two-adult household has only a 50% chance. A respondent in a one-adult household thus would get a weight of 1 for the number of adults factor whereas a respondent in a two-adult household would get a weight of 2. A similar logic applies to the number of phones: the more phones in the household, the greater the probability of selection of an individual and thus the smaller the weight.

When Utah used a Mitofsky-Waksberg sample design, an additional adjustment was made for sampled clusters accepted for further sampling for which the number of completes was other than the target number of three. In a Waksberg design as used in the BRFSS, exactly three completes should be obtained from each primary sampling unit. When this is the case, each complete gets a weight of 1. When the number of completes is other than three, the weights or each complete in a primary sampling unit are adjusted so that they sum to 3. For example, when the number of completes in a primary sampling unit is 2, then each complete gets a weight of 1.5 on this factor, which is called the cluster size adjustment.

With disproportionate stratified sampling (DSS), an adjustment is made based on whether the sampled telephone is from a bank of phones that is presumed to contain many households (a high density stratum) and telephone numbers from a bank that is presumed to contain few households (a low density stratum).

Since 1995, Utah has used a design in which prefixes are assigned to strata representing local health districts, where the local health districts with smaller populations are sampled at a higher rate than the local health districts with larger populations. Therefore, an adjustment is made to the weight that accounts for differences in the basic probability of selection among these strata. It is the inverse of the ratio of the estimated sampling fraction of each stratum to the stratum with the largest estimated sampling fraction. The sampling fraction is the number of completed interviews in the stratum divided by the number of adults ages 18 years and older in the stratum.

Two other weighting factors are determined by the outcome of the data collection process. These two adjustments are attempts to estimate missing values. They assume that missing respondents are, on the average, just like the respondents in the sample. Although this assumption is, strictly speaking, unlikely to be true, the theory is that the adjustment gets the data closer to the true value. The more important adjustment is to make the sample proportions of selected demographic characteristics (gender and age in Utah) equal the estimated proportions in the Utah population. A second adjustment is to make the sum of the weights equal the population of the state. This process is known as post-stratification.

Six categories of age are usually used for post-stratification: 18-24, 25-34, 35-44, 45-54, 55-64, 65+. Race is categorized as White/Non-White, when it is used (Utah does not use race). For example, suppose that the weighted sample size after weighting for the other factors among males aged 35-44 is 113.73 and that there are 350,000 males aged 35-44 in the population. Then, each 35-44 year old male would receive a post-stratification weight of 350,000/113.73 = 3077.46. (Designing post-stratification weights so that the sum of the weights for each combination of demographic characteristics equals the population size automatically ensures that the weighted proportion in the sample matches the proportion in the population so the two adjustments don't have to be made separately. As noted, however, the more important adjustment is getting the proportions to match.)

All the weighting factors are multiplied together to get the final weight for each respondent.

 

The BRFSS Weighting Formula

FINALWT = GEOWT*DENWT * 1 OVER NPH * NAD *CSA* POSTSTRAT

The computational formula above is intended to reflect all the possible factors that could be taken into account in weighting a stateís data. Where a factor does not apply its value is set to one.

FINALWT is the final weight assigned to each respondent.

GEOWT accounts for differences in the basic probability of selection among strata.

DENWT accounts for differences in the basic probability of selection between telephone numbers from a stratum that is presumed to contain many households (a high density stratum) and telephone numbers from a stratum that is presumed to contain few households (a low density stratum).

1/NPH is the inverse of the number of residential telephone numbers in the respondentís household.

NAD is the number of adults in the respondentís household.

CSA is the ratio of the expected cluster size to the actual cluster size.

POSTSTRAT is the number of people in an age-by-sex category in the population of a region or a state divided by the sum of the products of the preceding weights for the respondents in that same age-by-sex category. It adjusts for non-coverage and non-response and, since 1995, also adjusts for different probabilities of selection by region, where applicable.

Analytic Procedures Used With BRFSS Data in IBIS-Q

The BRFSS data are analyzed using SAS Proprietary Software Release 9.1 (Copyright © 2002-2003 by SAS Institute Inc., Cary, NC, USA). The SURVEYMEANS procedure produces estimates of survey population means and totals from sample survey data. The procedure also produces variance estimates, confidence limits, and other descriptive statistics. When computing these estimates, the procedure takes into account the sample design used to select the survey sample. The sample design can be a complex survey sample design with stratification, clustering, and unequal weighting. PROC SURVEYMEANS uses the Taylor expansion method to estimate sampling errors of estimators based on complex sample designs. This method obtains a linear approximation for the estimator and then uses the variance estimate for this approximation to estimate the variance of the estimate itself. When there are clusters, or primary sampling units (PSUs), in the sample design, the procedure estimates variance from the variation among PSUs. When the design is stratified, the procedure pools stratum variance estimates to compute the overall variance estimate.

References:

Userís Guide: Behavioral Risk Factor Surveillance System, U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. Available on line at www.cdc.gov/brfss

SAS Proprietary Software Release 9.1 (Copyright © 2002-2003 by SAS Institute Inc., Cary, NC, USA) SAS System Help.

--------------------------------------------------------------------------------------------------------------

Further information about Utahís BRFSS may be obtained by contacting Jennifer Wrathall, Utah BRFSS Coordinator, in the Office of Public Health Assessment, Utah Department of Health, P.O.Box 142101, Salt Lake City, UT 84114-2101. Phone: (801) 538-9259. Email: jwratha@utah.gov.

Page updated on 2/11/2013