Sampling units and units of analysis
Studies of drug use practices sometimes use an incorrect unit of analysis. The samples drawn in a study of prescribing practices in a group of health facilities can be thought of in a variety of different ways which include:
• areas or locations - a sample may include different regions or districts; or within a single region, it might include urban, peri-urban and rural areas;
• health facilities - a sample may be drawn from a number of health facilities of the same type, or from different types of facilities such as hospital outpatient departments, polyclinics and health centers;
• health providers - sometimes it will be possible to know the identity and background of the individual providers (doctors, nurses, paramedical workers, pharmacists) who treated patients in the sample, and to examine provider-specific differences in treatment patterns;
• prescribing encounters - encounters are collected from several health facilities in the sample, and studied as a whole.
Why are these distinctions important? One difficult aspect of designing a sample is deciding how many areas and health facilities to include, and how many encounters to collect for each prescriber at each facility. In many drug use studies the primary sampling unit is the health facility. This means that facilities are the first units randomly selected from a larger group. However, results can not then be reported with the prescribing encounter as the unit of analysis, with results as simple percentages or averages across all encounters sampled from the various facilities, as this would ignore the fact that sampling took place at two levels (health facility and patient encounter) and that real differences may exist at facility level. The methods of sampling and data analysis recommended in this manual attempt to deal with this problem.
For theoretical and practical reasons the health facility is the key unit for drug use studies. Many factors related to drug supply and utilization patterns vary at the facility level, that is, between facilities. For that reason it would be better to include as many different facilities as possible. However, this is generally more costly than it is to add additional encounters within facilities, because of extra transportation and lodging costs. Therefore, the primary goal in the design is to have a sample large enough to provide reliable answers to the major study questions, yet a sample which includes the smallest possible number of areas and facilities.
Guiding principles for sample size
On what grounds should decisions about the number of areas, facilities and encounters to include in the sample be based? The main guidelines that have been used as the basis for recommendations in this manual are listed below.
• Individual health providers tend to exhibit consistent practices over time, so that a sample drawn at one point in time will provide basically the same results as a sample that covers a longer period.
• Within facilities, differences between the type of prescriber (doctors, paramedical workers) are best ignored in both sampling and data analysis, unless a study or supervision of individual prescribers is an explicit objective.
• The goal of a drug use study should be to estimate percentage indicators that summarize values for the sample as a whole with a 95% confidence interval of plus or minus 7.5%.
• A study of individual facilities should measure facility-specific percentage indicators with a 95% confidence interval of plus or minus 10%.
• Above a certain number of encounters, adding additional encounters to a sample within a facility adds very little new information. However, increasing the number of facilities in the sample is a much better way to obtain more accurate and reliable estimates of overall prescribing practices.
• The study should be planned in such a way that data collection in one facility can be completed in a single day by a team of two investigators.
Recommendations on sample size
Surveys describing current treatment practices
There should be at least 600 encounters included in a cross-sectional survey, with a greater number if possible. If 20 health facilities are included, as recommended, this means about 30 encounters per facility. If fewer facilities are included, a larger number of cases should be selected in each, so that the minimum of 600 encounters is reached. Wherever possible, retrospective data collection over the past year should be used for prescribing indicators. Where records do not exist or key components are missing, use a prospective data collection, being aware of the possible problems of this method. Patient care and facility indicators are always collected prospectively.
Comparisons between individual facilities or prescribers
When it is important to compare individual facilities or prescribers, the size of samples drawn within each facility or per prescriber must be higher than 30 in order to get more reliable within-facility estimates of prescribing patterns. At least 100 cases per health facility or per prescriber would be recommended. If possible, retrospective data should be used. If groups of facilities are to be compared, at least 10 facilities should be included in each group.
Periodic monitoring and supervision
Indicators for individual facilities or prescribers can be used for monitoring purposes, e.g. when they are measured regularly. However, indicator data can also be collected with the specific objective to identify those facilities or prescribers that are grossly different from a set standard with respect to one or more indicators. In that case the number of cases collected at any one time can be much lower, with the size of the sample defined by the degree of accuracy needed. The basic principles of this method of “Lot Quality Assurance Sampling” are discussed in Annex 4. Generally, prospective data would be used for such monitoring, but if of good quality, retrospective data could be used.
Studies to assess the impact of an intervention
Consecutive studies can be used to measure changes in practice that result from an intervention. However, they must be designed in such a way that the response to the intervention can be distinguished from changes that would have occurred anyway. One critical issue in designing such studies is to compare changes in the intervention group with change (or the lack of it) in an appropriate control group. Without such a comparison it is impossible to know whether or not it was the intervention that caused any observed change.
It is important to establish the data collection process for both the intervention and control groups in exactly the same way. If baseline data for the intervention group are collected prior to the intervention, baseline data for the control group should be collected then as well. Alternatively, if retrospective records in the intervention and control facilities are good, both pre- and post-intervention data might be collected in a single data collection exercise at the end of the follow-up period. In this way it might be possible to guard against the possibility that changes in the study groups result from changes in the data collection system, or from the knowledge that their practices were being observed.
Although it is not important that they be identical, the sizes of the intervention and control groups should be roughly similar, both in the number of facilities sampled and the number of encounters per facility. To provide reasonable accuracy when drawing conclusions from observed differences between the intervention and comparison groups, there should usually be at least 10 facilities in each group, with 20 for more reliable comparisons. The easiest way to collect prescribing data is by retrospective data collection after the intervention is completed and the possible effect has occurred. If prospective data collection is used, exactly the same procedures should be undertaken in the intervention and control groups to control for bias that might result from the observation process.