(1997; 175 pages)
1.3 Relevance of the methods
When defining the research objectives and questions, the coordinators of the project selected the WHO set of indicators for monitoring NDP and the political mapping as the two best methods for obtaining the answers to the questions raised. It was therefore important to review during the workshop the relevance of the research: Were the country teams able to assess NDP performance with the indicators and the policy process with the political mapping?
Relevance of the NDP indicators
The participants reviewed the capacity of the indicators:
• to assess the level of implementation of the NDP,
• to evaluate the outcomes of the NDP,
• to reveal new aspects of the policy, unknown facts and realities.
A questionnaire was prepared by the facilitators and used by the country teams to support this process (see Annex 4). The main preliminary conclusions of the workshop can be summarized as follows:
(a) Assessing the structures and processes of the NDP
All the country teams found the two categories of indicators (structural and process) useful to assess the implementation of the various components of the NDP (see page 4) and identify areas that need improvement. Structural indicators were easy to collect. Most countries collected between 92% to 100% of them. Countries had more problems to calculate process indicators; however, with the exception of countries which had not finalized the study at the moment of the workshop (Mali, Zambia) and countries which decided not to collect all the indicators (Thailand), countries were able to calculate between 55% (the Philippines) and 94% of the process indicators (Bulgaria, India and Zimbabwe). For Zimbabwe "the quantifiable indicators gave an assessment of the degree and trend of the implementation. Further as Zimbabwe had most data available for 1993-1995 it strengthened the evaluation of the implementation". Bulgaria was also able to assess the performance of NDP, the same applies for Guinea, India, the Philippines and Viet Nam. Colombia could assess the implementation of the NDP except in the area of financing of drugs as there was a shift from traditional funding to public facilities with fiscal revenues to insurance. For the time being, there was not enough available data during the period of transition to properly assess this component. Sri Lanka felt also that the indicators work better for some areas than for others. Viet Nam stressed the fact that the indicators point out the lack of some important activities and strategies which need to be developed such as tenders and drug policies, and problems but not the reasons for these problems; it is also often difficult to know what is the ideal value of an indicator. Finally, Colombia partly supported by Viet Nam and Zimbabwe, felt that although the main aspects of the NDP were covered, it was necessary to have specific indicators on the implementation of the NDP in the context of the health reform. They should measure:
• The performance of the health authorities (at national, regional and local level) in a decentralized context.
• The performance of the health resources administrators (health plans, insurance) to use efficiently the per capita expenditure allocated in the system.
• The structure and process in the management and technological development of pharmaceutical services.
• The performance of the academic sector in function of the professionalization and qualification of human resources needed in the pharmaceutical sector; and provision of funding and technical assistance for the modernization of public sector procurement procedures, distribution and logistics (drug selection, programming, marketing, reception, storage, distribution and dispensing).
• The structure and process of the quality of the pharmaceutical care services: user's satisfaction, professional competence, security and continuity of care.
(b) Evaluating the outcomes of the NDP
All the country teams calculated most of the outcome indicators included in the WHO manual; Guinea, India and Viet Nam calculated the 10 outcome indicators; the Philippines and Sri Lanka could not calculate one indicator, different in each case; Thailand was able to collect only a few indicators, however as said earlier the Thai team did not perform the indicators study in full. Most countries felt that the 10 indicators provided a good measure of what the situation was in relation to the four main objectives of a policy, namely availability and accessibility to essential drugs and quality and rational use of drugs. For Zimbabwe, "the indicators give a good overview and it is easy to include additional specific national indicators". For India "considering the inputs used, the results are quite rewarding". For Chad and Mali the indicators provide "good baseline data which will allow to look at progress". A few countries felt that some additional outcome indicators - quality of pharmaceutical care and coverage of the social security system in relation to the total population in Colombia, self-medication in the Philippines - would have been useful to evaluate better the four objectives: Guinea and Viet Nam which added a few indicators on patient care and RUD felt that this "helped to get a more precise idea of the outcome of the policy".
Most countries would have liked more outcome indicators for additional objectives of their policies: traditional medicine was the most frequent missing objective (Chad, Guinea, Mali, the Philippines, Thailand and Viet Nam) followed by self-reliance and domestic industry (Bulgaria, India, the Philippines and Thailand) and by illegal market (Chad, Guinea, Mali, Thailand and Viet Nam). Guinea, the Philippines, Viet Nam and Zimbabwe would welcome some additional indicators for rational use of drugs at patient and community levels including indicators on standard treatment guidelines.
(c) Discovering new aspects of NDP
Through the assessment of the performance of the NDP using the different categories of indicators all the country teams found things they and their policy-makers did not know or expect. For instance, Bulgaria, Chad, Mali, Viet Nam and Zimbabwe discovered that the area of drug pricing was not receiving enough attention. Colombia "had a general idea about what was going on but the usage of the indicators allow them to have a systematic approach to specific components of the policy and have a much clearer picture of the present stage in the implementation of the NDP"; the same applies for the Philippines, Sri Lanka, Viet Nam and Zimbabwe. Guinea discovered that use of drugs was better than expected and quality worse, that the cost of a prescription has not changed during the last three years at least in the capital, that the average cost of a prescription up country was 50% lower than in Conakry and that some wholesalers supply also the illegal market. In India it was noted that the increase in the price of the basket of drugs was linked to the New Drug Price Control Order and that the quality of the drugs in the basket was comparatively poorer than the quality of drugs in general; this seems to reflect a shift in the practice of the industry, the essential drugs under generic name being left for production to small manufacturers, not all of them enforcing GMP. In Mali, it showed that prices decreased in the public sector mainly due to a new and improved procurement sector. In Sri Lanka the study demonstrated the lack of information and data on key areas and specific issues. There was a general belief "that something done upstream would have some influence downstream: pharmacies are registered and were therefore assumed to be not breaking the law too much. Surveys showed that this was not true and the rules were being broken much more than it was assumed. The Government has produced a Sri Lanka Hospital Formulary and made available sufficient copies for distribution to the prescribers in the State Health Sector. However, problems in the distribution system at the various levels meant that only a small minority of the prescribers had the formulary available with them". Zimbabwe learnt that the drug policy needs to be realistic and more efforts should be devoted to priority setting, that there should be a continuous development of the indicators and that the data collection system should be simple and sustainable. In Viet Nam, it showed also the lack of information and data, the problems of enforcing regulations and the difficulties to adapt to market economy.
Relevance of the political mapping
Three main aspects of the political mapping were discussed by the workshop participants: the nature and quality of the political mapping analyses that were performed, the findings and interpretations of the analyses, and an assessment of this method's ability to achieve the broader objectives of the research project. The participants identified the following ways in which the political mapping method contributed to an analysis of policy-making processes for national drug policies:
Systematic political picture: The method provided a systematic assessment of the political environment in which a national drug policy is formulated and implemented. At a minimum, the analysis provided a description of the political dimensions of a policy decision, and a method to organize the multiple dimensions of a complex decision. One country team reported that they had not previously considered the political dimensions involved in implementation of a national drug policy.
Assistance in strategy design: For those countries actively considering policy changes and therefore engaged in a prospective political mapping analysis (Colombia, the Philippines, Sri Lanka, Thailand and Viet Nam), the analysis provided practical assistance in the design of political strategies. The systematic political picture and the strategy suggestions (included in the software) were reported to be particularly helpful in designing strategies by some countries.
Challenge to assumptions: One team reported that the political mapping method was useful in making explicit the team's assumptions about how a new policy would be adopted and in forcing the team to explain and justify those assumptions. This process of questioning helped to enhance the coherence and feasibility of the policy. The analysis thus allowed the team to reflect on some of the unstated assumptions about how a new national drug policy would be accepted.
Reality testing: One team found the interviews to be particularly useful in checking the team's perceptions about other stakeholders. This advantage has been reported in other instances of political mapping analysis, and is one of the strengths of this method. The analysis helps to present a policy decision from multiple perspectives, and helps the country team view a problem from the perspectives of other stakeholders.
Team building: The process of performing the political mapping analysis helped create a sense of common language and sense of mission for some country teams. The analysis encourages the teams to rethink their strategies, taking into account the interactions among policies, players, and positions. Finally, the process was thought to strengthen the team's capacity to advocate for a national drug policy.
Relevance of the methods
NDP indicators: The participants agreed that the various categories of indicators achieve their goals, i.e. the outcome indicators provide a good picture of the national situation in relation to the four objectives of the NDP, the structural and process indicators allow the assessment of the level of implementation of the main components of the NDP; the indicators reveal new things on the policy and illustrate the importance of using certain types of data such as the financial data; finally, they are helpful tools for focusing actions on deficiencies, and for finding solutions based on data and analysis. The results of the exercise could also be used to achieve political consensus on the national drug policy. Most countries recommended that such collection of data should be undertaken regularly and budgeted in the MOH budget. There should however be a balance between the advantages of having good indicators but which are difficult to calculate and the amount of effort and cost needed to collect related data. In addition, the workshop recognized that it is difficult to focus on monitoring of NDP when the society as such is without "a culture of monitoring". The findings and lessons from the analysis are discussed below (in sections 2.1 and 2.2).
Political mapping: Workshop participants agreed that political mapping was a new and effective tool that could assist in the analysis of policy-making processes and in the design of strategies for both the formulation and implementation of national drug processes. While the method has a number of inherent limitations and application difficulties (discussed above in section 1.2.2), workshop participants believed that they had learned important insights from performing the analysis in their country. Overall, the participants concluded that the method of political mapping helps policy-makers to systematically analyse the support and opposition for a proposed policy; consult with the major stakeholders on their views, analyse opportunities and obstacles to change; design a set of creative and effective strategies for change; and assess and track the processes of implementation. The substantive findings and lessons from the analysis are discussed below (in section 2.3).