- Medicine Information and Evidence for Policy > Medicines Policy
- Medicine Information and Evidence for Policy > Monitoring and Evaluation
(1999; 250 pages) [French]
Organizing the data collection
Step 1: Choose monitoring unit members
A team at central level, most probably in the ministry of health or national drug authority, should be responsible for the monitoring system, i.e. for data collection, analysis and reporting. This team should include at least three senior professionals: (i) a pharmacist or a medical doctor conversant with the national pharmaceutical policy, (ii) an economist or a manager with strong economic expertise, and (iii) a statistician. Data collectors should be recruited for the field work, when needed. This team can undertake other activities and not work full time on NDP monitoring; however, it is important that its existence be institutionalized and its functions well defined to ensure sustainability.
Step 2: Decide on the final set of indicators
Deciding on the final set of indicators needed for monitoring the NDP should take place at the beginning of the monitoring process. It is an extremely important step as the selection of indicators will have major implications for the results. This selection should be done once the objectives, strategies and targets of the NDP have been defined (policy formulation and planning). The team should then read the manual carefully and review all the indicators using the model lists of indicators (Chapter III) and the description of the indicators (Chapter IV). It should:
• review the purpose of each indicator (what is it supposed to measure? And why is it important to measure this in the national context?);
• assess whether the indicator is relevant in terms of the NDP’s objectives and strategies and whether it needs to be modified or adapted;
• review whether the necessary data can be obtained, and if so where?; and
• decide whether additional indicators are needed; if so, which ones? how should the data be obtained?
The team may also decide that some indicators in this manual are not appropriate for national circumstances or policy and delete them. When developing supplementary indicators to cover additional policy objectives beyond those specified in this manual, the team should identify carefully the objectives and strategies in order to select indicators that will provide useful information about the effectiveness of policy implementation. It can also decide to apply certain indicators included in the manual to other sectors, for instance the illegal market. Finally, the monitoring unit should also at this stage collect information on any other work to develop health indicators in the country - as examples of what has been done - to assist in deciding on the final list and to avoid duplication of effort.
Step 3: Identify data sources
The manual was designed in such a way that most of the data needed for the four categories of indicators should come from the national health information system and/or from the national drug management information system and can be collected at central level12. The monitoring unit should always try to collect existing reliable data rather than new data. The process of collecting new data on a regular basis can have high start-up and operational costs compared with improvements in existing data collection systems.
12 All background information, structural indicators and 75% of the process indicators can be collected at central level; however, 9 of the 10 outcome indicators will probably require surveys.
Sources of data vary from one country to another. In general, the following are useful sources:
• the ministry of health is the main source of data. The chief pharmacist, the essential drugs programme manager, the head of the central medical stores, the drug regulatory authority, the head of the planning department, etc., will be able to provide information on the main aspects of the pharmaceutical sector and most data for the background information and structural indicators;
• drug producers, wholesalers and pharmaceutical associations can provide data on importation, local production, distribution and supply of drugs;
• health facilities and pharmacies, whether public or private, will be a good source of specific information on essential drugs’ availability, prices, quality, etc.;
• other health programmes may collect data on the drug sector, on population, on morbidity and on health status which may be appropriate for use in the indicator calculations;
• nongovernmental organizations, and international and bilateral organizations may also have data on specific projects dealing with drugs at the community level;
• other sectors such as the ministry of finance, customs authorities and the central bank, may have information on total government expenditures, total health expenditures, drug expenditures, drug imports, etc.;
• international organizations such as the International Monetary Fund, the United Nations Development Programme, the World Bank, the United Nations Children’s Fund and WHO may have data on demographic and economic trends, health status and health systems, and financial data related to drugs.
Detailed information on the main sources of data for each indicator is provided in Chapter V.
Step 4: Determine methods for data collection
Three methods are used in this manual to collect the data:
• record and document review;
Identifying data which should be collected through interviews and record reviews, and data which will need a survey, is the fourth step in the process. This is specific to each country and depends on the available sources. Once the final list of indicators is established, the monitoring team should make a list of all the data that are needed, including data for the numerators and the denominators of the process and outcome indicators, and identify potential sources for the various data and ways to collect them.
Most of the data can be obtained by reviewing records and documents, such as ministry of health reports, evaluations and reports of international and bilateral agencies, inspection reports, stock inventories, sales figures, prescriptions and patient logbooks. All these sources, when reliable, should be used as much as possible, as they are an efficient way of collecting information.
Interviews are also an important source of information when the people to be interviewed have been selected carefully. Interviews can be conducted with varying degrees of flexibility, from structured to semi- or unstructured interviews. For collecting indicators data it is recommended that structured interviews should be used. In certain cases, a more open and informal interview style may allow a wider range of information to be explored. Aspects not adequately covered by the indicators may emerge during the course of a conversation. In this case, interviewers should continue to ask questions until they fully understand the situation. Cross-checking the findings obtained from other sources can also be done by this type of interview. Non-structured interviews may also determine whether the respondents are answering truthfully or not (e.g. in the case of the number of samples tested, the head of the laboratory may be inclined to provide figures that are higher than the actual ones). For this kind of approach, much depends on the interviewer's skills. Interviewers should have a good understanding of the subject and be able to ask probing questions. The interviews are therefore mainly useful for clarifying and better understanding points which seem strange or contradictory in the information provided through other methods. Alternatively, a list of data to be collected at each source can be developed and data can be sent regularly to the monitoring unit from a number of recognized sources.
However, in most countries, data for some indicators will not be routinely recorded and will need to be collected through specific surveys. The selection of samples, the size of the samples, and the sites where data are to be collected, should be decided at this stage as these are important for planning the process and calculating the budget. These issues are discussed in more detail below (see pages 44 to 58).
Step 5: Plan the process and calculate the time needed
The time needed for collecting, processing and analysing all the information at central and peripheral levels should be carefully calculated. Experience shows that four to six weeks are enough, with the exception of the first year when some preliminary tasks will be necessary. These include organization of the team, planning of the work at central level, budgeting, recruitment and training of staff, and development of data collection forms.
It is also important to determine how often the chosen indicators will be measured since monitoring is essentially an ongoing process. Guiding considerations include the ease with which the information can be obtained and the resources available. In some countries, most of the information will flow from the drug management information system. When it has been set up, the process and analysis can take place every year without great difficulty. In others, data will need to be collected through specific studies and should probably take place every two to three years, until the monitoring system is in place.
Step 6: Estimate resources and budget
A detailed budget should be prepared, based on the total number of staff and the time necessary to collect, process and analyse the data, at central and peripheral levels. The budget should include costs of field work (personnel, transport, supplies) and of the monitoring unit (personnel, supplies, meetings, etc.). Lack of resources is usually a major limiting factor. The budget can be reduced (i) by identifying other health-related institutions or other departments within the ministry of health willing to temporarily allocate personnel to the unit, (ii) by training personnel from the survey sites to collect the data, (iii) by using students and community volunteers, where appropriate, and (iv) by planning for strict control of expenditure. It is important that the ministry of health or the authority responsible for monitoring the NDP secures a budget implementing the monitoring system, including the necessary surveys. Compared with the large drug expenditures in all countries and the potential savings which can be achieved with a good monitoring system, a small budget for such a system is a good investment. This budget allocation should be strongly advocated by drug sector personnel to decision-makers in the health sector and at government levels.