Abraham, Manja D., Hendrien L. Kaal, & Peter D.A. Cohen (2002), Licit and illicit drug use in the Netherlands 2001. Amsterdam: CEDRO/Mets en Schilt. Pp. 99-113.
© Copyright 2002 CEDRO Centrum voor Drugsonderzoek.

[Press release] [Index] [Previous Chapter] [Next Chapter]

Licit and illicit drug use in the Netherlands 2001

Chapter 4: Mode comparisons

Manja D. Abraham, Hendrien L. Kaal, & Peter D.A. Cohen

Tables

4.1 Introduction

This chapter describes the comparison of CAPI and MM in terms of the representativeness of the CAPI and MM samples, drug use prevalence rates measured in each mode and socio-economic variables. In the NPO 2001 survey both CAPI and MM were used alongside. Approximately 8,000 persons were approached for CAPI and approximately 32,500 persons were addressed following the MM protocol.

The 2001 CAPI sample was approached in the same way as in 1997. Persons received a letter from the University of Amsterdam that asked for co-operation with the survey and announced the visit of an interviewer at their home address. The interviewer questioned the respondent, guided by a laptop computer. If the selected person was not found at home, the interviewer revisited the address at least three times. As in 1997, CAPI respondents did not receive an incentive for participation.

Persons in the MM sample received a letter from the University of Amsterdam to participate in the survey. They could participate in their preferred mode by:

  • Paper questionnaire;
  • Questionnaire on disk; or
  • Downloading, completing and uploading the questionnaire on the Internet.

The incentive for participation was ƒ 25,- (€ 11,34). Non-response was followed up within three weeks. People who did not react were approached to be interviewed by phone or, if their phone number was not listed, were send a reminder by mail and received a second copy of the questionnaire and disk. People who had explicitly refused to participate were not approached again. See chapter 1 for more detail of the protocols.

The aim of the comparison of CAPI and MM described in this chapter is to examine whether it is justified to combine two data sets that are collected by two different approaches (i.e. the CAPI and the MM sample) and whether the two samples are equally representative for the research population. With these issues in mind, it is discussed whether the results can be compared to those of the 1997 survey.

4.2 Representativeness of the CAPI and MM surveys

The CAPI response rate (48.7 per cent) is almost identical to the MM response rate (46.8 per cent) (see table 4.1). The gross sample is categorised in: successful interviews, non-response, frame errors and non-used addresses. For CAPI it is clear to which category each visited person belongs, but this is less clear for MM. 'No contact' in MM could be a 'frame error' as well as a 'refusal'. Those cases have been classified as non-response. As a result the response reported for MM is likely to underestimate the actual response rate, which means that MM is slightly more effective than the reported 46.8 per cent of response.

Non-response is classified in: refusal, no contact achieved, contact achieved (soft refusal), language or cultural problems, and non-valid cases. The different approaches of CAPI and MM result in different non-response categories. Not surprisingly, the number of (solid) refusals is much higher with the CAPI than with the MM approach, whereas the number of occurrences of 'no contact achieved' in the CAPI approach is a quarter of that with the MM approach. This is a direct consequence of the presence of a face-to-face interviewer in the CAPI approach. CAPI compares favourably with MM when looking at the proportion of non-valid cases (see also chapter 1).

Table 4.2 shows the distribution of the population, sample and response according to age, gender and marital status in the CAPI and MM parts of the survey. There is a difference between CAPI and MM in response from the various age groups: juveniles are more likely to respond with MM and senior citizens respond better with CAPI. As a result response amongst the unmarried is a little lower in the CAPI sample and higher in the MM sample. Men seem more inclined to respond when they are approached with CAPI than with MM. The response rate of women is independent of the method used.

To provide national estimates for each sample, it is necessary to adjust the data for differences in the selection probabilities because of the sample design. To be able to compare CAPI estimates with MM estimates, weights are calculated separately for the CAPI sample as well as the MM sample. Weights were produced to balance the oversampling of two groups of people: 1) those aged 12 to 20; and 2) those living in Amsterdam and Rotterdam, and to correct for selective non-response. This weighting procedure is based on post-stratification relating to the level of address density, age, gender and marital status, according to the population registry (GBA). See paragraph 1.6 for more description.

4.3 Survey outcomes for CAPI and MM

Socio-economic and lifestyle characteristics

To verify whether the response groups of the CAPI and MM surveys are similarly composed, some socio-economic and lifestyle characteristics are compared. Table 4.3 shows the educational levels and income levels for both groups. CAPI and MM show roughly the same pattern for the level of education, with variations up to 2.5 per cent points, but no clear pattern can be seen. Regarding income, it is found that the group of persons with an income lower than ƒ 750,- (€ 340,-) is remarkably larger in MM than in CAPI. The rest of the income classes show a similar distribution for CAPI and MM. It should be stressed that the variable 'income' is notorious for being left unanswered. Of the CAPI respondents 37 per cent did not give their income while 22 of the MM respondents. The average number of evenings a week spent at home is equal for both CAPI and MM groups.

Drug use prevalence rates

Tables 4.4 and 4.5 show lifetime and last month drug use prevalence rates for the CAPI and MM sample. The prevalence rates are given for Amsterdam as well as for the Netherlands as a whole. The NPO 2001 sample was designed to enable methodological comparisons between CAPI and MM within Amsterdam and on a national level. Amsterdam is looked at separately because it served as playground for the pilot study; many adjustments in the MM fieldwork were made during the pilots. Also, the CAPI fieldwork in Amsterdam experienced a lot of problems. It was hard to find sufficient interviewers and the response willingness for CAPI in Amsterdam was low.

Small but consistent differences can be seen between the outcomes of the two survey methods. Generally speaking CAPI results in lower drug use prevalence estimates than MM.

In the Netherlands this difference can be seen for lifetime use of almost all licit and illicit drugs, with the exception of alcohol (equivalent rates in MM and CAPI), sedatives, and codeine (both drugs show lower rates in MM than in CAPI). A similar pattern is shown in Amsterdam: CAPI rates are lower than MM rates. Other than what was found for the national estimates, in Amsterdam this finding also seems to hold for the rates of tobacco, sedative and codeine use, although differences found are not significant.

Last month prevalence rates show more resemblance between the CAPI and MM group. In comparison with CAPI, MM still shows slightly higher prevalence rates, but these differences are statistically significant for a few drugs only. The exception to this rule is again the prevalence of tobacco; the prevalence rate in MM is lower than in CAPI. Amsterdam last month prevalence rates are in line with the national last month rates. Last month tobacco prevalence is lower for MM than for CAPI. However, last month prevalence of alcohol amongst the MM group is remarkably higher than amongst the CAPI group.

In short, a clear pattern can be found when comparing CAPI and MM prevalence rates. With some exceptions, CAPI rates are lower than those for MM. Differences are small but the consistency could direct towards mode effects such as socially desirable answers. According to the - mainly American - literature on this issue, the presence of an interviewer may lead to socially desirable answers when asking about a respondent's drug use. This is typically the case for CAPI and not for MM. If a person is approached following the MM protocol, the respondent has the opportunity to opt for an interview method without interviewer. Social desirability in the CAPI sample would explain the fact that almost all drug prevalence rates are lower with CAPI than with MM. The exceptions are tobacco, sedatives and codeine, but these substances are legally obtainable and therefore might be less prone to socially desirable answers. Therefore the explanation of social desirability is not accepted as the one and only explanation of the difference between CAPI and MM.

4.4 Predicting mode by logistic regression analysis

The previous sections provided data suggesting that there is some relationship between drug use figures and method used. This relationship was checked with a logistic regression. It was tried to predict the 'dependent' dichotomy variable 'mode' (CAPI vs. MM) by adding 'independent' variables age, gender, marital status, stratum, education and lifetime drug use of alcohol, cannabis, ecstasy, cocaine, amphetamines and heroin. A model with low explanatory power would suggest that differences between the outcomes of CAPI and MM are not systematic enough to consider CAPI and MM as distinctive groups.

The analysis was performed on unweighted data. Not all MM cases were included in the analysis; a random sample of 2,995 cases was drawn from the 13,876 MM cases so that the groups CAPI and MM were equally sized. NOOT [In practice the analysis was performed on slightly viewer cases because of item missing] In the first step of the logistic regression, the variables age, gender, marital status, stratum and education were entered in the prediction model. Drug use variables were entered in the second step. By doing so it could be determined if drug use variables contributed to the 'prediction' of mode after having 'corrected' for the background variables, or if their presence did not make a significant difference.

The results of the logistic regression analysis are summarised in table 4.6. The variables in the equation are: age, gender, education, stratum, alcohol lifetime use and cocaine lifetime use. The 'B' values express the level to which the accompanying variable contributes to the model. Not all drug variables contribute to the model; only alcohol and cocaine use do. After step one the model rightly predicted mode in 57.6 per cent of the cases, only marginally more than one would predict on the basis of chance. This suggests that only slightly different groups of respondents were reached with CAPI than with MM in terms of background characteristics. The results of step two showed that, if these background variables are kept equal, the groups differ only slightly, namely for the drug use variables alcohol and cocaine lifetime use. The total model rightly predicts 57.4 per cent of the cases. The explanatory power of the model is also expressed by the R2. The low values of R2 (0.039 and 0.052) indicate that the model is not a good predictor of mode- implying that the outcomes of CAPI and MM are not hugely different. These findings do not stop us from combining the two response groups approached with different survey methods.

4.5 Conclusion/discussion

This chapter described the mode effects resulting from using two different survey methods. Potential mode effects include: sensitivity to social desirability (influencing prevalence rates), response selectivity (affecting the composition of the group of respondents), and data errors like item missing (as is described more extensively in chapter 1).

Literature is equivocal about mode effects. According to Aquilino and LoSciuto (1999), the mode used affects response percentages and response set bias, most notably socially desired responding. These findings are to a certain extent confirmed in the NPO 2001 survey: response rates hardly vary between CAPI and MM modes but CAPI respondents seem slightly lower educated while MM respondents have a lower income. This possibly indicates that different types of non-respondents are created by CAPI compared to MM; alternatively, it can be that the same type of person gives different answers when questioned with CAPI or MM.

The differences in drug prevalence rates found between CAPI and MM, although statistically significant, are small. The outcomes of the logistic regression also suggest that the CAPI and MM groups are not very different. This finding reinforces De Leeuw (1992), who states that different modes will result in comparable outcomes as long as procedures are performed in a corresponding way.

On the basis of small differences found and the outcomes of the logistic regression it can be concluded that both samples can be combined. In addition, it can be argued that MM already exists of a combination of modes: since MM included CAPI results in the pilot phase of the survey it seems harmless to add the separate CAPI study to the MM findings.

After comparing the representativity of the data of the MM samples it was concluded that the total sample could be regarded as representative for the research population. After comparing estimates between modes, it was conclude that the total data set probably delivers a reliable estimates of drug use, near enough to the 'true value' of drug use in the Netherlands.

Of more concern is the fact that some of the CAPI results are also lower than those of the 1997 survey. It is important to realize that, in general, the lifetime drug use prevalence rates increase because this is a cumulative measure. Lifetime prevalence rates only decrease when a large part of the drug use population deceases. In the past, some slightly decreasing drug use estimates were found between 1987 and 1990 as well, only to see estimates rise again in the measurements of 1994. It is hard to draw conclusions regarding the possible causes of these decreases; one possible explanation for the difference between 1997 and 2001 is that the lower response rate in 2001 caused a stronger bias.

MM was introduced as it had been experienced that CAPI was no longer feasible. The above provides us with another reason to favour MM over CAPI: different modes were shown to result in different mode related response biases and answer behaviours. The different effect of various MM modes might to some extend equal each other out, and thus lead to estimates closer to the 'true values' because fewer 'types' of respondents are excluded from the final data set.