Indicators for Monitoring National Drug Policies
(1999; 250 pages) [French] View the PDF document
Table of Contents
View the documentACKNOWLEDGEMENTS
Open this folder and view contentsCHAPTER I: Introduction
Open this folder and view contentsCHAPTER II: Development of the manual
Open this folder and view contentsCHAPTER III: Model lists of indicators
Close this folderCHAPTER IV: Methodology for indicator calculation
View the documentOrganizing the data collection
View the documentCollecting data
View the documentAnalysis and reporting
View the documentConducting surveys
View the documentCalculating the value of a basket of drugs
Open this folder and view contentsCHAPTER V: Detailed presentation of indicators
Open this folder and view contentsANNEX 1: Data collection forms
View the documentANNEX 2: Glossary
View the documentANNEX 3: Table of random numbers
View the documentBACK COVER
 

Analysis and reporting

Step 13: Calculate indicators

Once the data needed for the calculation of each indicator have been collected through interviews, record reviews or surveys, the next step is to process them. The data processing should be carefully designed and resources for processing should be identified before starting data collection. Correct design of data collection forms will facilitate processing. The main part of the processing will be done at central level by the monitoring unit, which will receive all the forms completed during interviews, record reviews and surveys. All the calculations can be done manually, but a computerized spreadsheet can also be used to enter the data, consolidate the results and prepare reports. A computerized programme (Excel) is available on request in the Department of Essential Drugs and Medicines Policy to facilitate the data processing and calculation.

For the background information, the monitoring unit may report the figures collected on a summary form, such as the one provided in Annex 1 (Summary Form 1: National drug policy background information). For the structural indicators, the monitoring unit may also record the Yes/No answers on a summary form such as the one provided in Annex 1 (Summary Form 2: National drug policy structural indicators).

For the process and outcome indicators, more attention needs to be paid to the data processing, as data for the same indicator (numerator and denominator) are sometimes collected in different places (central/peripheral), with different methods (record review/interview/survey), and are reported on different data collection forms. For most of the indicators, processing the data involves two phases: the first one consists of "ticking" and "counting" on each data collection form (e.g. "ticking" whether a certain drug is on the essential drugs list (EDL), counting the total number of drugs prescribed, etc.). It can be done either by the monitoring unit, or by the data collectors under the supervision of the monitoring unit.

The second phase consists of summarizing the results of all the individual data forms (one per facility surveyed). A form can be created for each indicator or can cover a number of indicators as in Example 1. This phase is carried out by the monitoring unit, which, at the beginning of the exercise, should be careful to separate data from different types of facilities (e.g. private/public sector) and give each facility a code.

Example 1

To calculate the indicators PR9, OT7, OT8, etc. it is first necessary to summarize information collected in all the health centres surveyed and recorded on separate data collection forms.

HC code

Number of prescriptions

Number of drugs

Number of drugs from EDL

Number of prescriptions with injections

HC-1

30

85

75

8

HC-2

30

69

44

12

HC-3

29

68

68

5

Total:

89

222

187

25


For these three health centres:
PR9 (187 : 222) x 100% = 84% of drugs prescribed are from the EDL
OT7 (222 : 89) x 100% = 2.49 drugs per prescription
OT8 (25 : 89) x 100% = 28% of prescriptions include at least 1 injection

All the process indicators and most of the outcome indicators are answered by a percentage. Once obtained, this percentage can be recorded on a summary form such as the ones provided in Annex 1 (Summary Form 1 and Summary Form 2).

The method of calculating the indicator will depend on the type of information needed for the numerator and the denominator. It is important to understand that the purpose of the indicators is to identify trends; therefore, it is not necessary to present the results with two decimals, since this will give a false impression of their accuracy. For most process and outcome indicators, a maximum of one decimal is recommended.

Example 2

To calculate the percentage of drug outlets inspected in one year (PR1) for example, it will be necessary to count the number of drug outlets inspected, to divide this number by the total number of drug outlets in the country, and then to multiply the figure obtained by 100. The result will be:

For this indicator, the data are relatively easy to retrieve as the numerator and the denominator normally come from the same source at central level.

Example 3

Other indicators are more time-consuming to obtain. For instance, to calculate the percentage of drugs from the national essential drugs list (EDL) sold (PR10), it will be necessary to count all the drugs from the EDL recorded on Data Collection Form 1 (see page 167), to divide this number by the total number of drugs sold, and then to multiply the figure obtained by 100. The result will be:

For this indicator some work will be needed, most probably from the monitoring unit, to classify the drugs into the two categories: EDL drugs and non-EDL drugs. Data Collection Form 1 has been designed to assist in this task.

Example 4

In other cases, information for the numerator and the denominator come from different sources. For instance, to calculate the value difference for a drug basket between retail value and CIF/ex-factory value (PR30), it will be necessary to use data from a survey in a sample of drug outlets for retail value (see Data Collection Form 1, page 167) and data from central level for CIF/ex-factory value.

Example 5

Finally, when calculating the indicators for the first time, some denominators will not be readily available, e.g. average expenditure per prescription out of average expenditure per prescription in the past three years (PR31). In this case an estimate can be made with the assistance of responsible personnel in the area concerned and from a review of studies, if any. If this is not possible, the indicator will be calculated only from the second year.

Further detailed information on the calculation of each indicator is given in Chapter V.

Step 14: Perform quality control checks

Usually the data have already been checked in the field to ensure that all the information has been properly collected and recorded. However, the information should be checked again during data processing for completeness and internal consistency. If some data collection forms have not been filled in completely, data will be missing for some of the indicators. It may still be possible in most cases to collect the missing information. However, consistency of data collection may be more important than completeness of information, as the information needs to contribute to monitoring activities, to identifying problems and to proposing changes in management and policy. If data seem inconsistent, it may be possible to check with the data collector or to return to the respondent and ask for clarification.

If it is not possible to correct information that is clearly inconsistent, it can be decided to exclude this particular part of the data from processing and analysis. However, this decision should be considered carefully, since if too many data are excluded it may affect the validity of the results.

When using a computer, quality control checks must also include a verification of how the data have been coded and subsequently entered.

Step 15: Analyse and interpret the results

After all data have been processed and all indicators calculated, a mass of information will be available; this information should be analysed. It is recommended that a systematic approach is followed which includes the following two steps:

• Take just the summary sheets (Annex 1) with the reply/value obtained for the indicators and, where appropriate, the value of the numerators and denominators.

• Look at subgroups of indicators, and "evaluate" each subgroup (see Example 6).


Example 6: Legislation and regulation

In Example 6, the structural indicators related to legislation and regulation show an improvement in 1995; in 1996 the process indicators have improved considerably. This shows that the performance in the field of legislation and regulation has improved; it does not show why. In order to understand why, it is necessary to review all the indicators for key component 1. A possible reason could be that inspection activity has increased; PR 1, PR 3, PR 4 and probably PR 6 and PR 7 will show this. On the other hand, increased adherence to the rules by the facilities inspected could also have caused this improvement; in this case, indicators PR 2 and PR 6 should have decreased.

The indicators can thus give an indication on "why there is a change", but in order to know exactly what has happened, a more detailed investigation is often needed.

If, in the example above, an increase in inspection activities has been the main cause of the improvement, there are various possible underlying reasons for this increase. To list a few: inspectors may have become more active, they may have started to address some areas that were neglected previously, they could have been given more resources to carry out inspections, or the number of inspectors may have increased.

Identifying the questions will help to get information on the reasons for improvements, deterioration or stagnation in implementation of the NDP; this in turn will help to focus attention, resources and efforts, and to identify the inputs that have had a real impact.

Step 16: Report results

As a summary of the whole process, a report should be prepared by the monitoring unit in order to communicate the information obtained, the results and conclusions to the department which has requested the information (usually the department in charge of national drug policy at the ministry of health). This report should be the basis for decisions on drug policy and strategies. It can be a persuasive and influential tool if it is well prepared, and if results are interpreted and discussed in a comprehensive manner, taking into account the objectives and the strategies of the drug policy.

In addition to this official report, there are many other ways to make the results widely known (presentations, articles, etc.). One important rule to follow in order to get people interested in the subjects discussed is to limit the scope to issues that will be of interest to the audience: the part they are responsible for, can influence or are (directly) concerned by. Therefore, presentations, reports, etc. should be tailored to specific audiences and results should be provided in a clear and visually attractive way.

Reporting and presenting the results of a monitoring exercise is not difficult - but it does require some effort.

Three mistakes are to be avoided, when you bring your work to the attention of others:

• do not present results that are not relevant: people will feel that they are of no concern to them;

• do not present results in a chaotic or "complicated" way: people will be lost;

• do not present results in a boring way: people will fall asleep; make graphics, limit the information given, present results not indicators, etc.

Step 17: Use the results for action

No monitoring or evaluation exercise can be considered useful unless it leads to action. Results of monitoring NDP implementation should lead to action on at least three levels: the central level, the "specialist" audiences and/or intermediate levels, and the health facility level.

At each of these levels, there are three types of action: recognize what is well done and encourage its continuation, correct what is not well done, and change what does not work (if necessary after further investigation into the reasons why it does not work).

Central level

At central level (MOH policy-makers) the results from the indicators can be used to define (new or additional) strategies, to update them, to reallocate resources (human as well as other) or to adjust plans and targets. Recognition of persons/units/departments which are performing well is important in order to motivate them, as is identification of bottlenecks and problem areas. Problems may require additional research to understand better the underlying reasons for them before corrective action can be taken.

"Action" at central level can mean either action by the central level (this will usually be needed when structures are not in place) or stimulating other levels in the health care system to take action or to improve the quality of their action.

The "specialist" audiences and/or intermediate levels

This refers to all those levels and structures that are involved in implementation of (part of) the NDP: drug inspection, central medical stores, private pharmacies, manufacturers etc.

They can take action and start to improve their work after being stimulated to do so by the central level. However, they can also take action (within the limits of their means and responsibilities) for improvement themselves.

Wide dissemination of the results is therefore important: it can help create awareness at different levels that performance is sub-optimal, initiate discussions on how to improve, and eventually lead to action.

The health facilities

Health facilities are not different from the previous group: they are responsible for implementing part of the NDP (and certainly not the least important part!). And they can do their part in improving the performance of some indicators (on "rational" prescription, e.g. PR 9 and OT 9), whilst other indicators may only partly depend on them (e.g. PR 29 concerning the average stockout duration at remote health facilities. Those stockouts can occur for various reasons - the central procurement unit may not perform very well, but bad stock management at the facilities may also be the cause).

For individual health facilities to improve their performance, they need to be given feedback on how they are performing and how/where they can (should) improve. The more specific this feedback is, the more helpful it will be to the health facility concerned, and the more likely it is that action will be taken. Furthermore, feedback should be given in a constructive way (that is, focusing on how to improve), rather than "telling them what they do wrong".

Step 18: Evaluate the whole process

The monitoring unit should review and evaluate the full process with the staff involved. The best approach for this evaluation is to analyse systematically during a seminar all the procedures used in each phase, including sampling techniques, methods for data collection, training of data collectors, organization of surveys, etc. All obstacles and difficulties encountered during the process should be reviewed and ways to address them in future work should be identified. This activity can provide a good opportunity for further discussion on the ways to set up a more regular monitoring system.

to previous section
to next section
 
 
The WHO Essential Medicines and Health Products Information Portal was designed and is maintained by Human Info NGO. Last updated: October 7, 2014