(2001; 96 pages) [French] [Spanish]
2.4 Monitoring and evaluation
Why are monitoring and evaluation important?
Monitoring and evaluating the impact of a national drug policy are challenging. Apart from a lack of time, human resources and budget, there is often a basic lack of understanding of the value of monitoring in the first place, and even a certain resistance to objectively or critically reviewing the effects of activities formulated in the master plan.
Monitoring is a form of continuous review which gives a picture of the implementation of planned activities and indicates whether targets are being met. It can be carried out using a combination of various methods, including supervisory visits and both routine and sentinel reporting.
Evaluation is a way of analysing progress towards meeting agreed objectives and goals. It should build on, and use, monitoring systems. At the start of a programme it is used to provide a clear needs assessment. A mid-term evaluation can provide valuable information about whether the programme is working, and if not, why not. Final evaluation allows a complete review of programme achievements from which lessons can be drawn for the future.
A system for monitoring and evaluation is a constructive management tool that enables a continuous assessment of progress, and helps to make the necessary management decisions. It also provides transparency and accountability, and creates a standard by which comparisons can be made between countries and areas and over time. All of this may produce the necessary evidence that progress is being made (or not), in order to support the policy in discussions with interested parties and policy-makers. Some practical aspects are presented in Box 1.
Indicators for monitoring national drug policies
To determine whether adequate progress is being achieved it is helpful to set realistic targets or performance standards. Indicators can be selected and used to measure changes, make comparisons and assess whether the targets are being met. If indicators are used they should be clear, useful, measurable, reliable and valid.
WHO and MSH have done a great deal of operational research to develop and refine indicators for monitoring drug policies.9,10 Currently there are four categories of drug policy indicators: background information, structural indicators, process indicators and outcome indicators. It is possible to use selected subsets of these indicators to meet the needs of countries.
WHO and MSH have also agreed on a subset of core indicators for routine use and sentinel reporting. These indicators are highly standardized so that trends can be identified. A detailed manual on their use is available.11 Data collection is relatively easy, so that monitoring can be done on a regular basis. Core indicators cover the following aspects:
• access to essential drugs, and other indicators on drug financing schemes, and public supply management; these provide information on access to essential drugs;
• functions and efficiency of the drug regulatory authority, the quality control laboratory and how drugs are handled to maintain good quality; these provide information about drug quality;
• drug prescribing and dispensing, use of the list of essential drugs and clinical guidelines; these provide information about drug use patterns.
Routine reporting or sentinel reporting?
A routine reporting system, as part of a drug management information system, can provide much of the information needed to monitor the drug policy’s implementation. However, in reality much of the routine information is not collected systematically and whatever is collected is rarely used. With a sentinel reporting system a selected sample of health facilities is regularly surveyed. Some countries, such as Zimbabwe, have successfully used a sentinel reporting system to collect, every two years, standard information about the status, strengths and weaknesses of the national pharmaceutical system. This has proved to be a very useful management tool.12
Who can use the results?
The indicator-based method of assessing the pharmaceutical situation can be useful for various parties involved in the pharmaceutical field. Policy-makers, implementers and managers can get a clear picture of the problems in the country so that they can reassess their strategies and priorities. Results can be used as a guide to setting priorities and to strengthening those strategies that can have the best impact.
Results can also be used by the government and health ministry to synchronize policies. For example, a low level of availability and affordability of essential drugs could indicate that the directions on health and drug financing should be reviewed. Economic policies may be too focused on joining the global economy without looking into the implications for drug pricing, or affordability and availability of relevant drugs. The presence of substandard drug products in the market could indicate the need to assess the implications of the government’s emphasis on a free market that allows more products and business entities than the drug regulatory authority (DRA) can efficiently regulate.
International agencies and donors can use the results to focus on priority areas that need support and to determine the viability of investing resources in areas where best impact can be achieved. Professional groups and NGOs can also be guided on areas where advocacy campaigns and information can be focused.
Multi-country comparative evaluations
Multi-country studies based on standard methodologies can assist national policymakers in learning about innovative approaches that may be applicable in their own country. They also encourage international exchanges and collaboration on drug policy issues. One example is the 12-country study on national drug policy monitoring.13 Its results indicate that in most countries the structure and the system are in place, but that it is much easier to create structures than to make them work. One example is the DRA: although it has the mandate to register drugs and inspect manufacturer and retail outlets, enforcement of regulations is often limited. In many countries public drug financing is weak.
The core indicators mentioned above are collected by WHO for international comparisons and are kept in the Organization’s database and used for WHO’s report on the world drug situation.14
Practical aspects of monitoring national drug policies
• Identify the right questions: focus on questions with answers that are relevant for management decisions.
• Limit data collection to data that are relevant and are likely to be used. If too many data are collected the process will become expensive, and data analysis will become too complicated and probably less accurate.
• Establish a reliable data collection system; remember that the data will be reliable only if they are also of relevance to the people who collect them. Wherever possible, build on and strengthen existing systems; data collection should as much as possible be built into the routine functioning of the system. This requires staff to be trained and resources to be allocated. Rapid feed-back of results is important.
• Apart from being used by district or provincial health managers, the aggregated results should flow back to the central policy and management level, and be used for management decisions at the central level. If the data are used to prepare a monitoring report, the report should be shared with all those who contributed to it, including those who collected the data.
Periodic evaluations of the national drug policy
The national drug policy should be periodically evaluated, for example every four years. Independent consultants or professionals from other countries or from WHO may be invited to complement a national evaluation team. Such evaluations should form an integral part of the pharmaceutical master plan, with the necessary resources allocated from the start.